Pains of kubernetes

It's not all roses.

This is a small writeup of my experience with Kubernetes, and some of the drawbacks of the project. It is not fair or balanced, but it was cathartic to write about it.

Bootstrapping

When we started it was all custom ansible scripts to set up machines and bootstrap a cluster, since then this has gotten so much more smooth with kubeadm & kubespray, built in certificate management (we had to roll our own for that in the beginning) and more.

Firewalling

Kubernetes seriously doesn't play well with firewalling, and that in turn makes multi-cloud access both fragile, painful and sometimes will cause split-brain in a cluster, desyncronization between the network layers and force a complete reboot of the cluster to come back to production values.

It's such a painful thing that I'd recommend not doing it for most. It's not worth it. Just set up more kubernetes clusters instead.

Authentication & Credentials

Authentication and management is a pain in the arse, and kubernetes design around this leaves a lot lacking. Both from being disabled by default, making anything that turns on authentication a "custom" solution, and by having no method of enforcing 2fa and time-based re-authentication natively.

A Special mention here is the fact that installing Helm & tiller used to be the same as giving random people root on all your servers. The social behaviour of not defaulting to functional RBAC or authentication has set back k8s security quite a lot.

Upgrades

Upgrade paths are a pain in the rear and never makes anyone happy. This is a where the core of kubernetes has gotten so many improvements lately. But it still hurts.

There is no way to enable automatic upgrades of core components, and most aren't even packaged, so from an adminstrative standpoint, it feels like going back to manually hand-patch each piece of software & dependency in your chain without the traditional Linux repository system where you can just schedule an maintenance window and apply upgrades automatically.

This is further exasberated by the fluency of the ecosystem and the inability of kubernetes to point at as single default setup as being "reommended" with functional upgrade paths. CNI and ingress comes to mind.

Secret management

Secret drift between various namespaces, secrets are a kind of configuration but not, and there's a lack of schema, unless you're vigilant each secret will end up with a slightly different name, and bootstrapping private access to things like registries sometimes ends up with a lot of copy-pasted credentials between places, simply because of lacking management and schema for secrets.

Templating failures

The kustomize project seemed to be ready to solve this, but political project management made it worse, the fact that there's an amazing declarative way to express how you want applications and infrastructure to connect, and then no way to natively template this without external template engines, helm, tiller, etc. etc. is not a pleasant experience.

There is hope that this will improve, but still painful today.

Final words

Overall, kubernetes has been a boon. It made service description declarative, handles a lot of very very complex issues for us, and is an very robust product for all it's failures. I have many annoyances with it, but that's because I haven't made a list of all the good things it does for us.

Dealing with all the good things and all the gains of kubernetes would make this document many many times longer, and would throw shade on a lot of other systems that came before it.

Surprisingly, I'm happy with kubernetes. It solves a definite need, and while it brings somtimes undue complexity and headaches, the trade off has been well worth it.