OpenShift is a container platform based on docker and kubernetes.
This isn’t a deep dive into OpenShift architecture design. But a few basics:
It uses a master/node architecture (pictured below).Figure 1. Graphic credit: https://docs.openshift.org/latest/architecture/index.html
One of the worker nodes, the infrastructure node, runs an haproxy container that handles traffic into and out of OpenShift. If you create a 2-node cluster, one will be the master and the other the node that runs all of your applications as well as the haproxy container. If you decide to create a cluster larger than 2 clusters, then you can read more about infrastructure nodes in the OpenShift documentation.
This walkthrough will be setting up a 3 node cluster, with 1 master and 2 nodes that server the applications.
3 x CentOS servers
These servers can be on any platform that can run Linux. I installed mine as kvm virtual machines on my Fedora 25 laptop. But you can do this on a couple of workstations or any hypervisor that can run CentOS 7.
As for installing CentOS, I am not going to get into that. There are tons of howtos on that topic. But there are a few configuration details.
In my 3 node OpenShift cluster, the master server runs the web interface, API server, and etcd key store. The nodes run the actual containerized applications. There are a few configuration points that are important for your cluster to work correctly.
I’m a little lazy. I don’t like to have to set up a DNS server unless I absolutely have to. I have tested this setup with the nip.io domain. This is a nifty little domain that maps anything.<IP_ADDRESS>.nip.io to the IP address in the domain. For example, app18.104.22.168.2.nip.io will return an A record of 10.1.1.2. some.server.192.168.122.104.nip.io returns a DNS lookup of 192.168.122.104.
$ nslookup bar.192.168.150.98.nip.io Server: 172.20.0.1 Address: 172.20.0.1#53 Non-authoritative answer: Name: bar.192.168.150.98.nip.io Address: 192.168.150.98 $ nslookup app22.214.171.124.14.nip.io Server: 172.20.0.1 Address: 172.20.0.1#53 Non-authoritative answer: Name: app126.96.36.199.14.nip.io Address: 192.168.122.14
IP addresses and hostnames
The IP addresses will become important. You will want them to be the same consistently. For that reason I am setting static IP addresses for each of my 3 servers. You can accomplish the same thing with a permanent DHCP lease if that is available in your environment.
In my setup, and am configuring static IP addresses when I provision the servers.
For hostnames, I am using the same nip.io DNS service that I described above.
You virtual machines should have 2 disks. When you go through the installation, install the operating system onto the bootable disk. The second disk will be used by the container engine for storage space. Each disk should be 15-20 GB. For the storage disk, the more you have available the more container space you can have.
Software prerequisites for all servers
Install the OpenShift Origin repository
curl -L https://tdawson.fedorapeople.org/centos/CentOS-OpenShift-Origin.repo > /etc/yum.repos.d/CentOS-OpenShift-Origin.repo
Install the yum-utils package
yum -y install yum-utils
enable the OpenShift repository
yum-config-manager --enable centos-openshift-origin
Installing needed packages on the OpenShift master
Install a few packages that will be helpful later on.
yum -y install origin origin-clients atomic-openshift-utils vim-enhanced httpd-tools
Installing packages on the OpenShift nodes
Install a few packages that will be helpful later on.
yum -y install origin atomic-openshift-utils docker vim-enhanced
Once this is complete, you are ready to configure your container engine’s storage on your OpenShift nodes.
Configuring storage on the OpenShift nodes
There is an application that is packaged with docker called docker-storage-setup. It configures the desired storage backend for docker to use for space when it creates containers for OpenShift. We will be using thinpool LVM volumes. You can find additional details about the storage setup in the OpenShift documentation
Creating the docker-storage-setup configuration file
cat <<EOF > /etc/sysconfig/docker-storage-setup DEVS=/dev/vdb VG=docker-vg EOF
Running the docker-storage-setup application
# docker-storage-setup Checking that no-one is using this disk right now ... OK Disk /dev/vdb: 41610 cylinders, 16 heads, 63 sectors/track ... Rounding up size to full physical extent 24.00 MiB Logical volume "docker-pool" created. Logical volume docker-vg/docker-pool changed.
The disk that is specified in /etc/sysconfig/docker-storage-setup is the second disk that you created for your virtual machine.
Enabling and starting docker on your OpenShift nodes
Now we get into some of the fun stuff. We are going to start up the docker service on our OpenShift nodes. This is the service that will actually start and stop containers inside the OpenShift workflows.
Enabling docker with systemctl
systemctl enable docker.service
Starting the docker service with systemctl
systemctl start docker.service
Preparing your master server to run the OpenShift installer
the OpenShift installation program is an Ansible playbook. Before you launch the installer you need to create an ssh key on your master server and get it installed on all of your OpenShift nodes. This is because Ansible uses ssh to connect to the other servers and perform all of the configuration steps to get OpenShift up and running.
Creating an ssh key on your OpenShift master
ssh-keygen -f ~/.ssh/id_rsa -t rsa -N ''
This command will automatically create an ssh key with no password on the master server. * Copying over the ssh public key to each server
for i in 192.168.122.100 192.168.122.101 192.168.122.102;do ssh-copy-id root@$i;done
Notice that I included the IP address for the master server as well. Ansible will actually ssh into the master from itself to configure the master. That means you could actually run the OpenShift installer from outside the cluster. For this demo, however, it’s just easier to document this way. * Downgrading ansible
One glitch I have run into is that you need to downgrade the version of Ansible that is installed by default on the master server. This may change over time, but as of the time I’m writing this it is still required. * Downgrading ansible on the master server
yum -y downgrade ansible
Once this is complete you are ready to run the interactive installer for OpenShift.
There are multiple installation methods for openshift. You can install it in containers or with tradtional RPM packages. You can do an interactive install, or pass the installer a pre-configured configuration file for an unattended installation.
For this first attempt, I am going to use traditional packages for installation and an interactive installation process. The installer is a text utliity that is launched from the command line. We will walk through each screen.
To start the installer, on the master server, type atomic-openshift-installer install.
OpenShift interactive installer process
The first screen
Welcome to the OpenShift Enterprise 3 installation. Please confirm that following prerequisites have been met: * All systems where OpenShift will be installed are running Red Hat Enterprise Linux 7. * All systems are properly subscribed to the required OpenShift Enterprise 3 repositories. * All systems have run docker-storage-setup (part of the Red Hat docker RPM). * All systems have working DNS that resolves not only from the perspective of the installer, but also from within the cluster. When the process completes you will have a default configuration for masters and nodes. For ongoing environment maintenance it's recommended that the official Ansible playbooks be used. For more information on installation prerequisites please see: https://docs.openshift.com/enterprise/latest/admin_guide/install/prerequisites.html Are you ready to continue? [y/N]: y
We have satisfied all of these requirements. so go ahead and type y and press Enter. * specifying the installation user
This installation process involves connecting to remote hosts via ssh. Any account may be used. However, if a non-root account is used, then it must have passwordless sudo access. User for ssh access [root]:
We have set up ssh keys for our root users on our cluster. Go ahead and accept the default here and press Enter. * Selecting the OpenShift version to install
Which variant would you like to install? (1) OpenShift Origin (2) OpenShift Container Platform (3) Registry Choose a variant from above: :
There are multiple variations of OpenShift available. OpenShift Conainer Platform is the supported release of OpenShift by Red Hat. Registry is a limited version of OpenShift that acts as a standalone container image registry. We are going to install OpenShift Origin, so you can take the default value here as well and hit Enter. * Specifying your OpenShift cluster nodes
*** Host Configuration *** You must now specify the hosts that will compose your OpenShift cluster. Please enter an IP address or hostname to connect to for each system in the cluster. You will then be prompted to identify what role you want this system to serve in the cluster. OpenShift masters serve the API and web console and coordinate the jobs to run across the environment. Optionally, you can specify multiple master systems for a high-availability (HA) deployment. If you choose an HA deployment, then you are prompted to identify a *separate* system to act as the load balancer for your cluster once you define all masters and nodes. Any masters configured as part of this installation process are also configured as nodes. This enables the master to proxy to pods from the API. By default, this node is unschedulable, but this can be changed after installation with the 'oadm manage-node' command. OpenShift nodes provide the runtime environments for containers. They host the required services to be managed by the master. http://docs.openshift.com/enterprise/latest/architecture/infrastructure_components/kubernetes_infrastructure.html#master http://docs.openshift.com/enterprise/latest/architecture/infrastructure_components/kubernetes_infrastructure.html#node Enter hostname or IP address: 192.168.122.100
Now we are getting to the meat of the process. We will start out with our master server. In my configuration, 192.168.122.100 is my master server. Enter the IP address or hostname for your master server and hit Enter. * Declaring the master server
Will this host be an OpenShift master? [y/N]: y
We tell OpenShift that this is our master server, and hit Enter. * Specifying the installation method
Will this host be RPM or Container based (rpm/container)? [rpm]:
We want to use the rpm installation method for this cluster. You can accept the default value and hit Enter. * Adding another node
*** Installation Summary *** Hosts: - 192.168.122.100 - OpenShift master - OpenShift node - Etcd Total OpenShift masters: 1 Total OpenShift nodes: 1 NOTE: Add a total of 3 or more masters to perform an HA installation. Do you want to add additional hosts? [y/N]: y
This screen gives us a summary of the cluster so far, and asks us if we want to add another host. You do, so enter y and press Enter. * Specifying our second node and its configuration
Enter hostname or IP address: 192.168.122.101 Will this host be an OpenShift master? [y/N]: N Will this host be RPM or Container based (rpm/container)? [rpm]:
Similarto our first node, we specifiy a hostname or IP address, whether or not this is a master in the node, and the installation method to use. The only differences here are the IP address and telling the installer that this is NOT a master server. * Repeating this for our third server
*** Installation Summary *** Hosts: - 192.168.122.100 - OpenShift master - OpenShift node (Unscheduled) - Etcd - 192.168.122.101 - OpenShift node (Dedicated) Total OpenShift masters: 1 Total OpenShift nodes: 2 NOTE: Add a total of 3 or more masters to perform an HA installation. Do you want to add additional hosts? [y/N]: y Enter hostname or IP address: 192.168.122.102 Will this host be an OpenShift master? [y/N]: N Will this host be RPM or Container based (rpm/container)? [rpm]:
Non-HA cluster confirmation
*** Installation Summary *** Hosts: - 192.168.122.100 - OpenShift master - OpenShift node (Unscheduled) - Etcd - 192.168.122.101 - OpenShift node (Dedicated) - 192.168.122.102 - OpenShift node (Dedicated) Total OpenShift masters: 1 Total OpenShift nodes: 3 NOTE: Add a total of 3 or more masters to perform an HA installation. Do you want to add additional hosts? [y/N]: N You have chosen to install a single master cluster (non-HA). In a single master cluster, the cluster host name (Ansible variable openshift_master_cluster_public_hostname) is set by default to the host name of the single master. In a multiple master (HA) cluster, the FQDN of a host must be provided that will be configured as a proxy. This could be either an existing load balancer configured to balance all masters on port 8443 or a new host that would have HAProxy installed on it. (Optional) If you want to override the cluster host name now to something other than the default (the host name of the single master), or if you think you might add masters later to become an HA cluster and want to future proof your cluster host name choice, please provide a FQDN. Otherwise, press ENTER to continue and accept the default. Enter hostname or IP address [None]:
Since we only have a single master, this is not considered an HA OpenShift cluster. This is fine for our demo lab, but for production you want to specify at least 3 masters. If there are enough master servers, OpenShift will automatically configure itself in an HA configuration.
Since we are not creating an HA cluster, we can just accept the default here and move on. * Registry storage
Setting up high-availability masters requires a storage host. Please provide a host that will be configured as a Registry Storage. Note: Containerized storage hosts are not currently supported. Enter hostname or IP address [192.168.122.100]:Setting up high-availability masters requires a storage host. Please provide a host that will be configured as a Registry Storage. Note: Containerized storage hosts are not currently supported. Enter hostname or IP address [192.168.122.100]:
This is another HA-specific configuration parameter. Just accept the default value and press Enter. * Specifying an application domain
You might want to override the default subdomain used for exposed routes. If you don't know what this is, use the default value. New default subdomain (ENTER for none) : apps.192.168.122.101.nip.io
This is one of the hardest parts of OpenShift to comprehend. The easiest way to publicly expose applications in Openshift is to create a wildcard DNS record that points to your OpenShift infrastructure node. By default, the infrastructure nodes is the first non-master node that you specify during your installation process. In my setup, that is 192.168.122.101.
I can take advantage of the nip.io domain here as well. Using apps.192.168.122.101.nip.io, all of the applications I create in OpenShift will have routes in that domain.
After I specify this, I can move forward. * Gathering information and getting going
*** Installation Summary *** Hosts: - 192.168.122.100 - OpenShift master - OpenShift node (Unscheduled) - Etcd - Storage - 192.168.122.101 - OpenShift node (Dedicated) - 192.168.122.102 - OpenShift node (Dedicated) Total OpenShift masters: 1 Total OpenShift nodes: 3 NOTE: Add a total of 3 or more masters to perform an HA installation. Gathering information from hosts... All specified hosts in specified environment are installed. Do you want to (re)install the environment? Note: This will potentially erase any custom changes. [y/N]: y
At this stage the OpenShift installer makes sure it can connect to all of the hosts, and give you one final warning before we get going. If you’re ready, reply y and press Enter. * Confirming hostnames and IP addresses from the host query.
The following is a list of the facts gathered from the provided hosts. The hostname for a system inside the cluster is often different from the hostname that is resolveable from command-line or web clients, therefore these settings cannot be validated automatically. For some cloud providers, the installer is able to gather metadata exposed in the instance, so reasonable defaults will be provided. Please confirm that they are correct before moving forward. 192.168.122.100,192.168.122.100,192.168.122.100,ocp-188.8.131.52.100.nip.io,ocp-184.108.40.206.100.nip.io 192.168.122.101,192.168.122.101,192.168.122.101,ocp-220.127.116.11.101.nip.io,ocp-18.104.22.168.101.nip.io 192.168.122.102,192.168.122.102,192.168.122.102,ocp-22.214.171.124.102.nip.io,ocp-126.96.36.199.102.nip.io Format: connect_to,IP,public IP,hostname,public hostname Notes: * The installation host is the hostname from the installer's perspective. * The IP of the host should be the internal IP of the instance. * The public IP should be the externally accessible IP associated with the instance * The hostname should resolve to the internal IP from the instances themselves. * The public hostname should resolve to the external IP from hosts outside of the cloud. Do the above facts look correct? [y/N]: y
This is a confirmation screen to confirm the IP addresses and hostnames that were queried by the installer. If this all looks good to you, go ahead and respond with a y and hit Enter. * Writing config files and last chance saloon
Wrote atomic-openshift-installer config: /root/.config/openshift/installer.cfg.yml Wrote Ansible inventory: /root/.config/openshift/hosts Ready to run installation process. If changes are needed please edit the config file above and re-run. Are you ready to continue? [y/N]: y
Just in case something goes sideways, the OpenShift installer writes out the configuration that you have specified into a file. This is it! It’s time to install OpenShift. So respond with a final y and press Enter.
The OpenShift installation
Now Ansible goes out to all of the hosts and does its thing. The output will look similar to:
Play 1/29 (Create initial host groups for localhost) .. Play 2/29 (Create initial host groups for all hosts) . Play 3/29 (Populate config host groups)
This will take some time, depending on your internet connection speed and other factors.
If everything was successful, you should see output indicating that your installation was successful.
Once its done, you should be able to browse to the hostname of your master server on port 8443 with https at this point. It should look similar to this screenshot. Before the page loads, you will probably get a warning about the site being insecure because the SSL certificate hasn’t been properly signed. Don’t worry about this. OpenShift created its own SSl certificates as part of the installation process.
Now it’s time to flex the muscles of your shiny new OpenShift container platform! I will cover that in subsequent posts.