There is a lot of articles about kubernetes is hard and you should use managed cloud provider services.
Usually the argumentation is written by an army of marketing people to providers of managed services, or companies making platforms like VmWare, Nutanix, RedHat etc. unfortunately some of or a lot of the tech leaders tend to follow those statements. But for what reason?
The tool stack that is available from the cncf community can, actually help you provide, your organization with your own “managed” kubernetes service. You do not even have to be tied down by the slow progression of a vendor roadmap.
I have been tech lead/architect in a project to do exactly that, setup bare metal k8s on-prem.
Pick from the CNCF shop Link to heading
The cncf landscape is huge and growing with rapid speed. There are lots to choose from in that nice “shop”.
One of the main drivers for building this on-prem solution was to make on-prem resources easy to manage and control, we settled on the k8s api, as it is proven at scale and is extendable via CRD’s to make it fit our needs.
Some of our technology choices was backed by that reason. We have consolidated around the k8s api.
Metal3 Link to heading
The project started some years back, just before KubeCon San Diego 2019 where I was in a session where Metal3 was introduced. I got really intrigued by the fact that not only did you get a way to do k8s LCM on bare metal, but you also got a very cloud like way to manage bare metal servers, with known user-data and meta-data.
It was actually a really simple thing for me to pick this project as there where not really no other project1 that simplified the stack and turned bare metal handling into a CRD and therefor provided us with the nice interface and handling like all other k8s objects.
kubernetes-sigs Link to heading
The kubernetes-sigs provide a lot of value in different areas in the k8s space, we used some of the projects to make the best of the community and the years of gathered best praxis and knowledge.
Cluster Api Link to heading
The k8s community has been kind enough to build a api to manage clusters the same way you do with k8s deployments etc. via Cluster Api.
The concepts of Cluster Api is described in the Cluster Api book.
When combining Cluster Api with Cluster APi Provider Metal3 from the Metal3 project you have a really nice way to manage your bare metal k8s clusters.
image-builder Link to heading
The last piece of the puzzle is having OS images that can be installed during the provisioning. Once again the kubernetes-sigs has the answer, the image-builder project provides ready to use images that are intended to be used with Cluster Api and kubeadm bootstrap provider.
Deploy bare metal k8s Link to heading
I know how much you love yaml, so deploying a bare metal k8s cluster is as easy as running. kubectl apply -f cluster.yaml in the management cluster and the workings of Cluster Api and Cluster Api Provider Metal3 will provide you with the k8s cluster you defined.
Next Link to heading
In the following posts I will get into more details about getting a production environment running with these components and a few more.
No bare metal Link to heading
If you do not have access to metal do not worry, the metal3 has a dev setup that you can try on your own.
-
It was years later that Thinkerbell was introduced. ↩︎