Stability in Kubernetes Nodepool: The Challenge with Cronjobs

The cronjobs are only created on nodes that have the label "cronjob: allowed". Otherwise, nothing runs there. Why? Because we have given all of our applications NodeSelectors and thus force them to run on their assigned node pools. One node pool for the cronjobs, one for the customers, and one for the applications, all of which need things like IAM.

This helped initially and eased our conscience. But it quickly became apparent that this didn't really help. The nodes in the node pool for the cronjobs still had the same problems and were unstable. The only thing that helped was to clean up these nodes, delete them, and let the node pool automatically generate new nodes.

To avoid manual work, we had to think of something else. It was obvious. We have to do away with these cronjobs!

Optimization 2: No more cronjobs!

Not having any more cronjobs would be nice. Unfortunately, we haven't found a solution that would have abolished the cronjobs. I would have liked to tell you that a cronjob "x" is no longer needed. But we still need it. We have found a solution for ourselves, so that the cronjobs no longer need to be managed by the cluster, and thus there is no longer any instability. We have a central application that knows all workspaces. We have had this central management from the beginning. Now we let all workspaces notify this management that a cronjob is to be executed. Each workspace has an API, which then starts the mentioned cronjob in the shell. This puts more load on the workspace, but usually nothing happens, because there is nothing to do. We have achieved the same effect as with separate cronjob deployments. This way, we avoid too much load on our cluster by constantly starting up cronjobs.

Even more potential for optimization

Unfortunately, this solution also has a catch. It only works as long as we have fewer than 300 customers: A request to a workspace to start a cronjob takes about 200ms. Thus, we can start up to 300 cronjobs within a minute. After that, the action of starting all cronjobs takes longer than a minute, and we have a lag, and a cronjob no longer runs every minute, but somewhat less often.

But that can also be controlled... you can run the requests to the workspaces in parallel processes. But we'll write about that another time.

Are we alone with unstable nodes due to cronjobs?

How do you handle cronjobs when many applications want to start a cronjob at the same time?