On day 2 of KubeCon + Cloud Native Con, Alwyn and Freark went to the talks of Benjy Portnoy, Maciej Szulik, Suresh Visvanathan and others. First up, we’ll share Freark’s notes on the presentations that he visited.


In a presentation titled “Applying Least Privileges through Kubernetes Admission Controllers,” Benjy Portnoy of Aqua Security explains how you can use admission controllers to hook in security features. For instance, before a container is actually deployed, an admission controller could verify if the used image is on a whitelist of allowed images and stop deployment going further if it is not whitelisted. This ensures tighter controls on container deployment and could be used to prevent malicious containers from even starting up.


Both Maciej Szulik of Red Hat and Alena Prokharchyk of Rancher Labs gave separate talks about building Kubernetes controllers. Both pretty much had a full room, so there seems to be huge interest from the community into leveraging custom resource definitions and controllers to customize and extend the functionality of Kubernetes. Maciej gave a good set of ground rules to follow when building controllers and Alena gave a live demo of a custom controller which function it was to spin up new Kubernetes clusters from a Kubernetes controller.


Also interesting to mention that in the Q&A of both controller talks there were questions about using Python instead of Go to build these controllers, but at this moment best practices and library support for Python and other languages are significantly lagging behind Go. Sounds like a golden opportunity for ambitious Pythonistas to be able to fill in that space in the near future! ūüôā


With “101 Ways to ‘Break and Recover’ a Kubernetes Cluster” Suresh Visvanathan and Nandhakumar Venkatachalam from Oath shared some stories and experiences from the trenches of maintaining a large Kubernetes cluster. Calamities ranging from accidentally deleting a Namespace object to an Etcd upgrade misconfiguration were encountered and overcome. For each incident, conquered lessons were learned for the future and these lessons were shared with the audience.


Alwyn also went to the presentation about 101 ways to break and recover a Kubernetes Cluster. And although he noticed that it weren’t¬†exactly 101 ways to break and recover, he did find some good examples of things that can go wrong. For example, with a¬†multitenant cluster with namespace isolation and lots of namespaces.¬†An operator deleted a directory containing one ‚Äúnamespace‚ÄĚ file. This had a cascading effect on the cluster, as everything within that namespace got wiped.¬†They implemented a custom AdmissionController that guards the deletion of namespaces so you explicitly have to force the deletion if you really want to do that, to prevent it from happening again. Another example occurred after upgrading etcd. It somehow started pointing to an empty directory. Kubelet tried synchronizing with the data stored in etcd, which was nothing, so it started evicting every pod in the cluster.¬†After that, Kubelet was configured to not synchronize with etcd if it points to an empty dir.

I visited just one presentation on day 2, since I’m just getting to know Kubernetes. John Morello from Twistlock talked about how containers make security and compliance instantly easier. I learned why applications should run in containers and some benefits that developing your applications in containers gives you compared to the traditional way.¬†