Innovation in IT, there’s got to be a better way
Last year I installed Kerberized Kafka with Ranger for authorisation and Solr for auditing by a manual installation of handpicked versions of components it needs.
Working through the documentation took me a few months. And yes, I also did installations of Kerberized Kafka via open source Ambari which were not to my full satisfaction.
It took quite some time to get it all sorted out.
Before you have something running at a customer site there is also the product selection and license negotiations that are involved.
Though it is possible to innovate with such an approach it takes the speed out of innovation, not to mention scaling up or down an installation.
It was not the first time that I worked through the documentation on how to install a product. New products come out frequently and the rate accelerates each year.
Before you know it, another quarter has passed and what if your customer does not want to adopt the product that you have prepared for?
Then early 2018 IBM offered me to go on a training for IBM Cloud Private (with the focus on Kubernetes) where I learned how to install containerized IBM middleware.
Installing containerized middleware can now be done in minutes. And with a catalogue of about 50 containerized products in IBM Cloud Private at this moment and more coming, it made sense to me that this is the route to go.
If you have IBM Cloud Private you can have a new middleware platform in an afternoon even with your own customized container images if you want that. Good job, IBM!
And so, I started my Kubernetes journey somewhere during my preparations for the IBM Cloud Private boot camp early this year.
I bought Marco Luksa’s ‘ Kubernetes in Action’ which is a very good buy.
And therefore, I needed a Kubernetes environment.
What do you need?
For those who are short pressed and understand that time is money: 64 GB of RAM, 16 cores and 1 TB SSD will do fine for starters. But please check the hardware pre-requisites.
I started off by installing minikube as well as a regular Kubernetes cluster with 1 master node and 3 worker nodes in virtualbox. It did work although such an installation is very basic. For example, when you want a dashboard you need to install it. I had set it up with bridged nic’s on my home network and all was fine and dandy with very limited resource requirements.
During the ICP boot camp we installed a standard ICP installation consisting of 1 master node, 1 proxy node, 3 worker nodes and an NFS server in an afternoon.
After the training I performed an installation of IBM Cloud Private Community via the vagrant approach on my Lenovo W5210 with 16 GB of RAM in virtualbox which did not work out for me.
Next, I performed a single node install which succeeded. For this I used a virtualbox guest with a NAT nic. It turned out that I could only access the console inside the image. Port forwarding to the console did not work. Also, the 10 GB VM was severely short of memory.
After re-doing the install with a bridged nic I had something up and running but it left little room on my 16 GB laptop with 8 cores. You might be lucky if your laptop has 32 GB.
I then managed to acquire 1 old computer with a 4 core I3 CPU on which I installed a single node ICP 2.1.0.2 cluster directly on bare metal to get the most out of the 16 GB of RAM it had. The machine thus ran the master, the proxy and a single worker node. I had to exclude installing the management and vulnerability advisor node from the installation because of the lack of compute resources available.
I did manage to install a Jenkins pipeline on it to deploy the blue compute shop. It did work but the amount of surplus memory was not a lot.
At the time that ICP 2.1.0.3 came out I managed to get a new computer with an 8 core I7 core. I decided to setup a VM containing the master and the proxy on the I3 and 2 VM’s containing worker nodes on the I7. I used RHEL ‘s KVM instead of Windows 10 VirtualBox running Ubuntu and I must say that that was a very positive experience in terms of start-up times. I can recommend it.
The installation took almost 3.5 hours and in retrospect I believe that this was because the I3 machine has an old-fashioned HDD. After all the installation on the IBM sky-tap environment took about 20 minutes or so, if I re-call it correctly.
I performed the “pod auto scaling“ exercise from chapter 15 of Marko’s book and I used my 8 core I7 laptop to send calls to the I3 running the master and the proxy and then I discovered that sometimes the auto-scaling did not work. The machine containing the master (controller) and the proxy (CNI) was overloaded during the test as evidenced by a Linux utility called atop. When I throttled the rate down to 250 calls per second, the master had sufficient compute power to scale the deployment up.
Now, you understand why IBM has chosen a different topology for real life situations. IBM has service offerings to get you on the right track with adopting ICP from the start, which makes sense in a professional engagement.
I looked at my laptop and asked myself, where do a get a new computer to replace the I3? Oh, … well, … I bought a Lenovo from the X series (light to carry) and moved the KVM running the master/proxy to my laptop and did the test again where I used the Lenovo X as load driver.
The throughput increased, but although the horizontal pod auto-scaler did allow for scaling up to 6 replicas (with a target CPU utilization of 5%), it did not scale up the number of pods higher than 4. None of the I7 nodes were the constraint, … well, you have guessed it, now the X has become the bottleneck.
The current setup looks as follows:

In September 2018 ICP 3.1.0 comes out. I have a cunning plan, …