Day three and we have treated the wounded and regrouped after our initial encounter with the enemy.
There are yaks everywhere, get your razors ready, boys!
How to deal with configuration? Kubernetes likes app config in environment variables, not config files. This is easy in our node apps using convict, pretty easy in our ruby apps and ranging from relatively easy to bloody hard in our java apps. But how to get config into the replication controllers? We opted for using configmaps (a kubernetes object) to store the config, reference the variables from the rc files and maintain it in git controlled files. So when we want to change to app config, update the config files and run a script which updates the configmap and reloads all the pods for the app. Incidentally, the way we do that, is to delete them, and let kubernetes recreate them. Don’t do this if you run one cluster ;-) We should make the apps read the config automatically, but since none our apps do that, we needed a solution that works now.
This also means we can have separate config for different environments while the file definition of the RC remains the same. We started out with only the configuration variables external to the RC file, but soon realized we needed to externalize the version of the image, the replica number and the limits too.
This has worked really well so far.
The deploy script which either creates everything on the first deploy (from yaml files) or performs a rolling upgrade from one version to another. The deploy script also need to handle the configmap and substituting the values from external files.
And then the script for just updating the config of an app, without deploying anything.
(Yeah, it could do with some refactoring)
The question that popped up was: when do we know the cluster is running out of resources, and preferably before the deploy fails with events saying you’re shit out of memory or cpu? Container metrics to the rescue.
We have a existing metric system backed in graphite which has worked well for us. We have used the host name as the metric path to give us the oppurtunity to isolate metrics per host. When kubernetes manages the containers, the host on which it runs becomes ever changing. to solve this, we landed on using heapster for gathering container metrics, and then storing them in influxdb. The influx query language gives easy ways to abstract away over changing host names. I am seeing use of influxdb for app metrics in the future too, for the same reason.
Next time on the kubernetes wars: the curious case of the slow node apps.