Docker, Kubernetes, and OpenShift¶

Install Minishift¶

Minishift requires a hypervisor and since we already use VirtualBox for Vagrant, you should install VirtualBox from http://virtualbox.org .

Download the Minishift tarball from https://docs.openshift.org/latest/minishift/getting-started/installing.html and put the minishift binary in /usr/local/bin or somewhere in your $PATH. This assumes Mac or Linux. These instructions were last tested on version v1.14.0+1ec5877 of Minishift.

At this point, you might want to consider going through the Minishift quickstart to get oriented: https://docs.openshift.org/latest/minishift/getting-started/quickstart.html

Start Minishift¶

minishift start --vm-driver=virtualbox --memory=8GB

Make the OpenShift Client Binary (oc) Executable¶

eval $(minishift oc-env)

Log in to Minishift from the Command Line¶

Note that if you just installed and started Minishift, you are probably logged in already. This oc login step is included in case you aren’t logged in anymore.

oc login --username developer --password=whatever

Use “developer” as the username and a couple characters as the password.

Create a Minishift Project¶

Calling the project “project1” is fairly arbitrary. We’ll probably want to revisit this name in the future. A project is necessary in order to create an OpenShift app.

oc new-project project1

Note that oc projects will return a list of projects.

Create a Dataverse App within the Minishift Project¶

The following command operates on the conf/openshift/openshift.json file that resides in the main Dataverse git repo. It will download images from Docker Hub and use them to spin up Dataverse within Minishift/OpenShift. Later we will cover how to make changes to the images on Docker Hub.

oc new-app conf/openshift/openshift.json

Log into Minishift and Visit Dataverse in your Browser¶

After running the oc new-app command above, deployment of Dataverse within Minishift/OpenShift will begin. You should log into the OpenShift web interface to check on the status of the deployment. If you just created the Minishift VM with the minishift start command above, the oc new-app step is expected to take a while because the images need to be downloaded from Docker Hub. Also, the installation of Dataverse takes a while.

Typing minishift console should open the OpenShift web interface in your browser. The IP address might not be “192.168.99.100” but it’s used below as an example.

• https://192.168.99.100:8443 (or URL from minishift console)
• username: developer
• password: <any password>

In the OpenShift web interface you should see a link that looks something like http://dataverse-project1.192.168.99.100.nip.io but the IP address will vary and will match the output of minishift ip. Eventually, after deployment is complete, the Dataverse web interface will appear at this URL and you will be able to log in with the username “dataverseAdmin” and the password “admin”.

Another way to verify that Dataverse has been succesfully deployed is to make sure that the Dataverse “info” API endpoint returns a version (note that minishift ip is used because the IP address will vary):

curl http://dataverse-project1.`minishift ip`.nip.io/api/info/version

From the perspective of OpenShift and the openshift.json config file, the HTTP link to Dataverse in called a route. See also documentation for oc expose.

Troubleshooting¶

Here are some tips on troubleshooting your deployment of Dataverse to Minishift.

Cleaning up¶

If you simply wanted to try out Dataverse on Minishift and want to clean up, you can run oc delete project project1 to delete the project or minishift stop and minishift delete to delete the entire Minishift VM and all the Docker containers inside it.

Making Changes¶

Scaling Dataverse by Increasing Replicas in a StatefulSet¶

Glassfish, Solr and PostgreSQL Pods are in a “StatefulSet” which is a concept from OpenShift and Kubernetes that you can read about at https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

As of this writing, the openshift.json file has a single “replica” for each of these two stateful sets. It’s possible to increase the number of replicas from 1 to 3, for example, with this command:

oc scale statefulset/dataverse-glassfish --replicas=3

The command above should result in two additional Glassfish pods being spun up. The name of the pods is significant and there is special logic in the “zeroth” pod (“dataverse-glassfish-0” and “dataverse-postgresql-0”). For example, only “dataverse-glassfish-0” makes itself the dedicated timer server as explained in Dataverse Application Timers section of the Admin Guide. “dataverse-glassfish-1” and other higher number pods will not be configured as a timer server.

Once you have multiple Glassfish servers you may notice bugs that will require additional configuration to fix. One such bug has to do with Dataverse logos which are stored at /usr/local/glassfish4/glassfish/domains/domain1/docroot/logos on each of the Glassfish servers. This means that the logo will look fine when you just uploaded it because you’re on the server with the logo on the local file system but when you visit that dataverse in the future and you’re on a differernt Glassfish server, you will see a broken image. (You can find some discussion of this logo bug at https://github.com/IQSS/dataverse-aws/issues/10 and http://irclog.iq.harvard.edu/dataverse/2016-10-21 .) This is all “advanced” installation territory (see the Advanced Installation section of the Installation Guide) and OpenShift might be a good environment in which to work on some of these bugs.

Multiple PostgreSQL servers are possible within the OpenShift environment as well and have been set up with some amount of replication. “dataverse-postgresql-0” is the master and non-zero pods are the slaves. We have just scratched the surface of this configuration but replication from master to slave seems to we working. Future work could include failover and making Dataverse smarter about utilizing multiple PostgreSQL servers for reads. Right now we assume Dataverse is only being used with a single PostgreSQL server and that it’s the master.

Solr supports index distribution and replication for scaling. For OpenShift use, we choose replication. It’s possible to scale up Solr using the method method similar to Glassfish, as mentioned aboved In OpenShift, the first Solr pod, dataverse-solr-0, will be the master node, and the rest will be slave nodes

Configuring Persistent Volumes and Solr master node recovery¶

Solr requires backing up the search index to persistent storage. For our proof of concept, we configure a hostPath, which allows Solr containers to access the hosts’ file system, for our Solr containers backups. To read more about OpenShift/Kubernetes’ persistent volumes, please visit: https://kubernetes.io/docs/concepts/storage/persistent-volumes

To allow containers to use a host’s storage, we need to allow access to that directory first. In this example, we expose /tmp/share to the containers:

# mkdir /tmp/share
# chcon -R -t svirt_sandbox_file_t
# chgrp root -R /tmp/share
# oc login -u system:admin
# oc edit scc restricted     # Update allowHostDirVolumePlugin to true and runAsUser type to RunAsAny

To add a persistent volume and persistent volume claim, in conf/docker/openshift/openshift.json, add the following to objects in openshift.json. Here, we are using hostPath for development purposes. Since OpenShift supports many types of cluster storages, if the administrator wishes to use any cluster storage like EBS, Google Cloud Storage, etc, they would have to use a different type of Persistent Storage:

{
  "kind" : "PersistentVolume",
  "apiVersion" : "v1",
  "metadata":{
    "name" : "solr-index-backup",
    "labels":{
      "name" : "solr-index-backup",
      "type" : "local"
    }
  },
  "spec":{
    "capacity":{
      "storage" : "8Gi"
    },
    "accessModes":[
      "ReadWriteMany", "ReadWriteOnce",  "ReadOnlyMany"
    ],
    "hostPath": {
      "path" : "/tmp/share"
    }
  }
},
{
  "kind" : "PersistentVolumeClaim",
  "apiVersion": "v1",
  "metadata": {
    "name": "solr-claim"
  },
  "spec": {
    "accessModes": [
      "ReadWriteMany", "ReadWriteOnce",  "ReadOnlyMany"
    ],
    "resources": {
      "requests": {
        "storage": "3Gi"
      }
    },
    "selector":{
      "matchLabels":{
        "name" : "solr-index-backup",
        "type" : "local"
        }
      }
    }
  }

To make solr container mount the hostPath, add the following part under .spec.spec (for Solr StatefulSet):

{
  "kind": "StatefulSet",
  "apiVersion": "apps/v1beta1",
      "metadata": {
    "name": "dataverse-solr",
      ....

  "spec": {
    "serviceName" : "dataverse-solr-service",
            .....

      "spec": {
        "volumes": [
          {
            "name": "solr-index-backup",
            "persistentVolumeClaim": {
              "claimName": "solr-claim"
            }
          }
        ],

                    "containers": [
          ....

            "volumeMounts":[
              {
                "mountPath" : "/var/share",
                "name" : "solr-index-backup"
              }

Solr is now ready for backup and recovery. In order to backup:

oc rsh dataverse-solr-0
curl 'http://localhost:8983/solr/collection1/replication?command=backup&location=/var/share'

In solr entrypoint.sh, it’s configured so that if dataverse-solr-0 failed, it will get the latest version of the index in the backup and restore. All backups are stored in /tmp/share in the host, or /home/share in solr containers.

Running Containers to Run as Root in Minishift¶

It is not recommended to run containers as root in Minishift because for security reasons OpenShift doesn’t support running containers as root. However, it’s good to know how to allow containers to run as root in case you need to work on a Docker image to make it run as non-root.

For more information on improving Docker images to run as non-root, see “Support Arbitrary User IDs” at https://docs.openshift.org/latest/creating_images/guidelines.html#openshift-origin-specific-guidelines

Let’s say you have a container that you suspect works fine when it runs as root. You want to see it working as-is before you start hacking on the Dockerfile and entrypoint file. You can configure Minishift to allow containers to run as root with this command:

oc adm policy add-scc-to-user anyuid -z default --as system:admin

Once you are done testing you can revert Minishift back to not allowing containers to run as root with this command:

oc adm policy remove-scc-from-user anyuid -z default --as system:admin

Minishift Resources¶

The following resources might be helpful.

• https://blog.openshift.com/part-1-from-app-to-openshift-runtimes-and-templates/
• https://blog.openshift.com/part-2-creating-a-template-a-technical-walkthrough/
• https://docs.openshift.com/enterprise/3.0/architecture/core_concepts/templates.html

Installing Docker¶

On Linux, you can probably get Docker from your package manager.

On Mac, download the .dmg from https://www.docker.com and install it. As of this writing is it known as Docker Community Edition for Mac.

On Windows, we have heard reports of success using Docker on a Linux VM running in VirtualBox or similar. There’s something called “Docker Community Edition for Windows” but we haven’t tried it. See also the Windows Development section.

As explained above, we use Docker images in two different contexts:

• Testing using an “all in one” Docker image (ephemeral, unpublished)
• Future production use on Minishift/OpenShift/Kubernetes (published to Docker Hub)

All In One Docker Images for Testing¶

The “all in one” Docker files are in conf/docker-aio and you should follow the readme in that directory for more information on how to use them.

Future production use on Minishift/OpenShift/Kubernetes¶

FIXME: rewrite this section to talk about only pushing stable images to Docker Hub.

When working with Docker in the context of Minishift, follow the instructions above and make sure you get the Dataverse Docker images running in Minishift before you start messing with them.

As of this writing, the Dataverse Docker images we publish under https://hub.docker.com/u/iqss/ are highly experimental. They were originally tagged with branch names like kick-the-tires and as of this writing the latest tag should be considered highly experimental and not for production use. See https://github.com/IQSS/dataverse/issues/4040 for the latest status and please reach out if you’d like to help!

Change to the docker directory:

cd conf/docker

Edit one of the files:

vim dataverse-glassfish/Dockerfile

At this point you want to build the image and run it. We are assuming you want to run it in your Minishift environment. We will be building your image and pushing it to Docker Hub.

Log in to Docker Hub with an account that has access to push to the iqss organization:

docker login

(If you don’t have access to push to the iqss organization, you can push elsewhere and adjust your openshift.json file accordingly.)

Build and push the images to Docker Hub:

./build.sh

Note that you will see output such as digest: sha256:213b6380e6ee92607db5d02c9e88d7591d81f4b6d713224d47003d5807b93d4b that should later be reflected in Minishift to indicate that you are using the latest image you just pushed to Docker Hub.

You can get a list of all repos under the iqss organization with this:

curl https://hub.docker.com/v2/repositories/iqss/

To see a specific repo:

curl https://hub.docker.com/v2/repositories/iqss/dataverse-glassfish/