OpenShift in about an hour

Posted on Fri 28 April 2017 in howto • 12 min read

OpenShift is a container platform based on docker and kubernetes.

It is created by Red Hat. The upstream project for OpenShift is called OpenShift Origin. In this howto, we will walk through a full multi-node test-grade installation of OpenShift origin on CentOS. For a 2-node cluster, it takes about an hour and involves about 13 steps. So let’s get started!

OpenShift Architecture

This isn’t a deep dive into OpenShift architecture design. But a few basics:

  • It uses a master/node architecture (pictured below).

  • One of the worker nodes, the infrastructure node, runs an haproxy container that handles traffic into and out of OpenShift. If you create a 2-node cluster, one will be the master and the other the node that runs all of your applications as well as the haproxy container. If you decide to create a cluster larger than 2 clusters, then you can read more about infrastructure nodes in the OpenShift documentation.

This walkthrough will be setting up a 3 node cluster, with 1 master and 2 nodes that server the applications.

Infrastructure Prerequisites

3 x CentOS servers

These servers can be on any platform that can run Linux. I installed mine as kvm virtual machines on my Fedora 25 laptop. But you can do this on a couple of workstations or any hypervisor that can run CentOS 7.

As for installing CentOS, I am not going to get into that. There are tons of howtos on that topic. But there are a few configuration details.

In my 3 node OpenShift cluster, the master server runs the web interface, API server, and etcd key store. The nodes run the actual containerized applications. There are a few configuration points that are important for your cluster to work correctly.

DNS resolution

I’m a little lazy. I don’t like to have to set up a DNS server unless I absolutely have to. I have tested this setup with the nip.io domain. This is a nifty little domain that maps anything.<IP_ADDRESS>.nip.io to the IP address in the domain. For example, app1.10.1.1.2.nip.io will return an A record of 10.1.1.2. some.server.192.168.122.104.nip.io returns a DNS lookup of 192.168.122.104.

$ nslookup bar.192.168.150.98.nip.io
Server:         172.20.0.1
Address:        172.20.0.1#53

Non-authoritative answer:
Name:   bar.192.168.150.98.nip.io
Address: 192.168.150.98

$ nslookup app1.192.168.122.14.nip.io
Server:         172.20.0.1
Address:        172.20.0.1#53

Non-authoritative answer:
Name:   app1.192.168.122.14.nip.io
Address: 192.168.122.14

Server configuration

IP addresses and hostnames

The IP addresses will become important. You will want them to be the same consistently. For that reason I am setting static IP addresses for each of my 3 servers. You can accomplish the same thing with a permanent DHCP lease if that is available in your environment.

In my setup, and am configuring static IP addresses when I provision the servers.

For hostnames, I am using the same nip.io DNS service that I described above.

Role hostname IP Address

OpenShift master

ocp-1.192.168.122.100.nip.io

192.168.122.100

OpenShift node

ocp-2.192.168.122.101.nip.io

192.168.122.101

OpenShift node

ocp-3.192.168.122.102.nip.io

192.168.122.102

2 disks

You virtual machines should have 2 disks. When you go through the installation, install the operating system onto the bootable disk. The second disk will be used by the container engine for storage space. Each disk should be 15-20 GB. For the storage disk, the more you have available the more container space you can have.

Software prerequisites for all servers

  • Install the OpenShift Origin repository

    curl -L https://tdawson.fedorapeople.org/centos/CentOS-OpenShift-Origin.repo > /etc/yum.repos.d/CentOS-OpenShift-Origin.repo
  • Install the yum-utils package

    yum -y install yum-utils
  • enable the OpenShift repository

    yum-config-manager --enable centos-openshift-origin

Installing needed packages on the OpenShift master

  • Install a few packages that will be helpful later on.

    yum -y install origin origin-clients atomic-openshift-utils vim-enhanced httpd-tools

Installing packages on the OpenShift nodes

  • Install a few packages that will be helpful later on.

    yum -y install origin atomic-openshift-utils docker vim-enhanced

Once this is complete, you are ready to configure your container engine’s storage on your OpenShift nodes.

Configuring storage on the OpenShift nodes

There is an application that is packaged with docker called docker-storage-setup. It configures the desired storage backend for docker to use for space when it creates containers for OpenShift. We will be using thinpool LVM volumes. You can find additional details about the storage setup in the OpenShift documentation

  • Creating the docker-storage-setup configuration file

    cat <<EOF > /etc/sysconfig/docker-storage-setup
    DEVS=/dev/vdb
    VG=docker-vg
    EOF
  • Running the docker-storage-setup application

    # docker-storage-setup
    Checking that no-one is using this disk right now ...
    OK
    
    Disk /dev/vdb: 41610 cylinders, 16 heads, 63 sectors/track
    ...
      Rounding up size to full physical extent 24.00 MiB
      Logical volume "docker-pool" created.
      Logical volume docker-vg/docker-pool changed.

The disk that is specified in /etc/sysconfig/docker-storage-setup is the second disk that you created for your virtual machine.

Enabling and starting docker on your OpenShift nodes

Now we get into some of the fun stuff. We are going to start up the docker service on our OpenShift nodes. This is the service that will actually start and stop containers inside the OpenShift workflows.

  • Enabling docker with systemctl

    systemctl enable docker.service
  • Starting the docker service with systemctl

    systemctl start docker.service

Preparing your master server to run the OpenShift installer

the OpenShift installation program is an Ansible playbook. Before you launch the installer you need to create an ssh key on your master server and get it installed on all of your OpenShift nodes. This is because Ansible uses ssh to connect to the other servers and perform all of the configuration steps to get OpenShift up and running.

  • Creating an ssh key on your OpenShift master

    ssh-keygen -f ~/.ssh/id_rsa -t rsa -N ''

    This command will automatically create an ssh key with no password on the master server. * Copying over the ssh public key to each server

    for i in 192.168.122.100 192.168.122.101 192.168.122.102;do ssh-copy-id root@$i;done

    Notice that I included the IP address for the master server as well. Ansible will actually ssh into the master from itself to configure the master. That means you could actually run the OpenShift installer from outside the cluster. For this demo, however, it’s just easier to document this way. * Downgrading ansible

    One glitch I have run into is that you need to downgrade the version of Ansible that is installed by default on the master server. This may change over time, but as of the time I’m writing this it is still required. * Downgrading ansible on the master server

    yum -y downgrade ansible

Once this is complete you are ready to run the interactive installer for OpenShift.

Installing OpenShift

There are multiple installation methods for openshift. You can install it in containers or with tradtional RPM packages. You can do an interactive install, or pass the installer a pre-configured configuration file for an unattended installation.

For this first attempt, I am going to use traditional packages for installation and an interactive installation process. The installer is a text utliity that is launched from the command line. We will walk through each screen.

To start the installer, on the master server, type atomic-openshift-installer install.

OpenShift interactive installer process

  • The first screen

    Welcome to the OpenShift Enterprise 3 installation.
    
    Please confirm that following prerequisites have been met:
    
    * All systems where OpenShift will be installed are running Red Hat Enterprise
      Linux 7.
    * All systems are properly subscribed to the required OpenShift Enterprise 3
      repositories.
    * All systems have run docker-storage-setup (part of the Red Hat docker RPM).
    * All systems have working DNS that resolves not only from the perspective of
      the installer, but also from within the cluster.
    
    When the process completes you will have a default configuration for masters
    and nodes.  For ongoing environment maintenance it's recommended that the
    official Ansible playbooks be used.
    
    For more information on installation prerequisites please see:
    https://docs.openshift.com/enterprise/latest/admin_guide/install/prerequisites.html
    
    Are you ready to continue? [y/N]: y

    We have satisfied all of these requirements. so go ahead and type y and press Enter. * specifying the installation user

    This installation process involves connecting to remote hosts via ssh. Any
    account may be used. However, if a non-root account is used, then it must have
    passwordless sudo access.
    
    User for ssh access [root]:

    We have set up ssh keys for our root users on our cluster. Go ahead and accept the default here and press Enter. * Selecting the OpenShift version to install

    Which variant would you like to install?
    
    
    (1) OpenShift Origin
    (2) OpenShift Container Platform
    (3) Registry
    
    Choose a variant from above:  [1]:

    There are multiple variations of OpenShift available. OpenShift Conainer Platform is the supported release of OpenShift by Red Hat. Registry is a limited version of OpenShift that acts as a standalone container image registry. We are going to install OpenShift Origin, so you can take the default value here as well and hit Enter. * Specifying your OpenShift cluster nodes

    *** Host Configuration ***
    
    You must now specify the hosts that will compose your OpenShift cluster.
    
    Please enter an IP address or hostname to connect to for each system in the
    cluster. You will then be prompted to identify what role you want this system to
    serve in the cluster.
    
    OpenShift masters serve the API and web console and coordinate the jobs to run
    across the environment. Optionally, you can specify multiple master systems for
    a high-availability (HA) deployment. If you choose an HA deployment, then you
    are prompted to identify a *separate* system to act as the load balancer for
    your cluster once you define all masters and nodes.
    
    Any masters configured as part of this installation process are also
    configured as nodes. This enables the master to proxy to pods
    from the API. By default, this node is unschedulable, but this can be changed
    after installation with the 'oadm manage-node' command.
    
    OpenShift nodes provide the runtime environments for containers. They host the
    required services to be managed by the master.
    
    http://docs.openshift.com/enterprise/latest/architecture/infrastructure_components/kubernetes_infrastructure.html#master
    http://docs.openshift.com/enterprise/latest/architecture/infrastructure_components/kubernetes_infrastructure.html#node
    
    Enter hostname or IP address: 192.168.122.100

    Now we are getting to the meat of the process. We will start out with our master server. In my configuration, 192.168.122.100 is my master server. Enter the IP address or hostname for your master server and hit Enter. * Declaring the master server

    Will this host be an OpenShift master? [y/N]: y

    We tell OpenShift that this is our master server, and hit Enter. * Specifying the installation method

    Will this host be RPM or Container based (rpm/container)? [rpm]:

    We want to use the rpm installation method for this cluster. You can accept the default value and hit Enter. * Adding another node

    *** Installation Summary ***
    
    Hosts:
    - 192.168.122.100
      - OpenShift master
      - OpenShift node
      - Etcd
    
    Total OpenShift masters: 1
    Total OpenShift nodes: 1
    
    NOTE: Add a total of 3 or more masters to perform an HA installation.
    
    Do you want to add additional hosts? [y/N]: y

    This screen gives us a summary of the cluster so far, and asks us if we want to add another host. You do, so enter y and press Enter. * Specifying our second node and its configuration

    Enter hostname or IP address: 192.168.122.101
    Will this host be an OpenShift master? [y/N]: N
    Will this host be RPM or Container based (rpm/container)? [rpm]:

    Similarto our first node, we specifiy a hostname or IP address, whether or not this is a master in the node, and the installation method to use. The only differences here are the IP address and telling the installer that this is NOT a master server. * Repeating this for our third server

    *** Installation Summary ***
    
    Hosts:
    - 192.168.122.100
      - OpenShift master
      - OpenShift node (Unscheduled)
      - Etcd
    - 192.168.122.101
      - OpenShift node (Dedicated)
    
    Total OpenShift masters: 1
    Total OpenShift nodes: 2
    
    NOTE: Add a total of 3 or more masters to perform an HA installation.
    
    Do you want to add additional hosts? [y/N]: y
    Enter hostname or IP address: 192.168.122.102
    Will this host be an OpenShift master? [y/N]: N
    Will this host be RPM or Container based (rpm/container)? [rpm]:
  • Non-HA cluster confirmation

    *** Installation Summary ***
    
    Hosts:
    - 192.168.122.100
      - OpenShift master
      - OpenShift node (Unscheduled)
      - Etcd
    - 192.168.122.101
      - OpenShift node (Dedicated)
    - 192.168.122.102
      - OpenShift node (Dedicated)
    
    Total OpenShift masters: 1
    Total OpenShift nodes: 3
    
    NOTE: Add a total of 3 or more masters to perform an HA installation.
    
    Do you want to add additional hosts? [y/N]: N
    
    You have chosen to install a single master cluster (non-HA).
    
    In a single master cluster, the cluster host name (Ansible variable openshift_master_cluster_public_hostname) is set by default to the host name of the single master. In a multiple master (HA) cluster, the FQDN of a host must be provided that will be configured as a proxy. This could be either an existing load balancer configured to balance all masters on
    port 8443 or a new host that would have HAProxy installed on it.
    
    (Optional)
    If you want to override the cluster host name now to something other than the default (the host name of the single master), or if you think you might add masters later to become an HA cluster and want to future proof your cluster host name choice, please provide a FQDN. Otherwise, press ENTER to continue and accept the default.
    
    Enter hostname or IP address [None]:

    Since we only have a single master, this is not considered an HA OpenShift cluster. This is fine for our demo lab, but for production you want to specify at least 3 masters. If there are enough master servers, OpenShift will automatically configure itself in an HA configuration.

    Since we are not creating an HA cluster, we can just accept the default here and move on. * Registry storage

    Setting up high-availability masters requires a storage host. Please provide a
    host that will be configured as a Registry Storage.
    
    Note: Containerized storage hosts are not currently supported.
    
    Enter hostname or IP address [192.168.122.100]:Setting up high-availability masters requires a storage host. Please provide a
    host that will be configured as a Registry Storage.
    
    Note: Containerized storage hosts are not currently supported.
    
    Enter hostname or IP address [192.168.122.100]:

    This is another HA-specific configuration parameter. Just accept the default value and press Enter. * Specifying an application domain

    You might want to override the default subdomain used for exposed routes. If you don't know what this is, use the default value.
    
    New default subdomain (ENTER for none) []: apps.192.168.122.101.nip.io

    This is one of the hardest parts of OpenShift to comprehend. The easiest way to publicly expose applications in Openshift is to create a wildcard DNS record that points to your OpenShift infrastructure node. By default, the infrastructure nodes is the first non-master node that you specify during your installation process. In my setup, that is 192.168.122.101.

    I can take advantage of the nip.io domain here as well. Using apps.192.168.122.101.nip.io, all of the applications I create in OpenShift will have routes in that domain.

    After I specify this, I can move forward. * Gathering information and getting going

    *** Installation Summary ***
    
    Hosts:
    - 192.168.122.100
      - OpenShift master
      - OpenShift node (Unscheduled)
      - Etcd
      - Storage
    - 192.168.122.101
      - OpenShift node (Dedicated)
    - 192.168.122.102
      - OpenShift node (Dedicated)
    
    Total OpenShift masters: 1
    Total OpenShift nodes: 3
    
    NOTE: Add a total of 3 or more masters to perform an HA installation.
    
    Gathering information from hosts...
    
    All specified hosts in specified environment are installed.
    
    Do you want to (re)install the environment?
    
    Note: This will potentially erase any custom changes. [y/N]: y

    At this stage the OpenShift installer makes sure it can connect to all of the hosts, and give you one final warning before we get going. If you’re ready, reply y and press Enter. * Confirming hostnames and IP addresses from the host query.

    The following is a list of the facts gathered from the provided hosts. The
    hostname for a system inside the cluster is often different from the hostname
    that is resolveable from command-line or web clients, therefore these settings
    cannot be validated automatically.
    
    For some cloud providers, the installer is able to gather metadata exposed in
    the instance, so reasonable defaults will be provided.
    
    Please confirm that they are correct before moving forward.
    
    
    192.168.122.100,192.168.122.100,192.168.122.100,ocp-1.192.168.122.100.nip.io,ocp-1.192.168.122.100.nip.io
    192.168.122.101,192.168.122.101,192.168.122.101,ocp-2.192.168.122.101.nip.io,ocp-2.192.168.122.101.nip.io
    192.168.122.102,192.168.122.102,192.168.122.102,ocp-3.192.168.122.102.nip.io,ocp-3.192.168.122.102.nip.io
    
    Format:
    
    connect_to,IP,public IP,hostname,public hostname
    
    Notes:
     * The installation host is the hostname from the installer's perspective.
     * The IP of the host should be the internal IP of the instance.
     * The public IP should be the externally accessible IP associated with the instance
     * The hostname should resolve to the internal IP from the instances
       themselves.
     * The public hostname should resolve to the external IP from hosts outside of
       the cloud.
    
    Do the above facts look correct? [y/N]: y

    This is a confirmation screen to confirm the IP addresses and hostnames that were queried by the installer. If this all looks good to you, go ahead and respond with a y and hit Enter. * Writing config files and last chance saloon

    Wrote atomic-openshift-installer config: /root/.config/openshift/installer.cfg.yml
    Wrote Ansible inventory: /root/.config/openshift/hosts
    
    Ready to run installation process.
    
    If changes are needed please edit the config file above and re-run.
    
    Are you ready to continue? [y/N]: y

    Just in case something goes sideways, the OpenShift installer writes out the configuration that you have specified into a file. This is it! It’s time to install OpenShift. So respond with a final y and press Enter.

The OpenShift installation

Now Ansible goes out to all of the hosts and does its thing. The output will look similar to:

Play 1/29 (Create initial host groups for localhost)
..
Play 2/29 (Create initial host groups for all hosts)
.
Play 3/29 (Populate config host groups)

This will take some time, depending on your internet connection speed and other factors.

Installation complete!

If everything was successful, you should see output indicating that your installation was successful.

Once its done, you should be able to browse to the hostname of your master server on port 8443 with https at this point. It should look similar to this screenshot. Before the page loads, you will probably get a warning about the site being insecure because the SSL certificate hasn’t been properly signed. Don’t worry about this. OpenShift created its own SSl certificates as part of the installation process.

images/ocp_origin_login.png
Figure 2. OpenShift Origin login dialog after your successful installation

Next steps

Now it’s time to flex the muscles of your shiny new OpenShift container platform! I will cover that in subsequent posts.