This repository contains a batteries included, production ready, opinionated Kubernetes cluster setup. It only requires editing two simple configuration files to deploy a fully functional cluster using NixOps.

Together, NixOS and Kubernetes is a powerful combination. NixOS enables programmatic configuration of the base Kubernetes system services at the OS level. Just enabling the basic service is still a long way from having a production ready, multi node Kubernetes cluster up and running. Setting up a cluster at the OS level gives you a functional system shell, lacking most of the services needed for actually deploying, accessing and monitoring services.

This project will do to things: It will configure the cluster at the OS level, setting up services, networking, certificates, access tokens etc. It will then bootstrap the running Kubernetes instance, fixing permissions and configuring the following services:

Helm for deploying services
Nginx-ingress for external access
cert-manager with issuers (Let's encrypt, cluster-ca and self-sign)
nfs-client-provisioner for automatic volume management
kubernetes-dashboard for managing the cluster and services
metrics-server for monitoring
Prometheus, node exporters and grafana for monitoring

Prerequisites

Install n basic nodes (node-1, ..., node-n) running nixos.

git submodule init
git submodule update

Installation

cd clusters
cp -r template cluster-1
cd cluster-1
../../bin/initca.sh # generates the cluster wide CA certificate
for i in node-1 node-2 node-3; do
  scp $i:/etc/nixos/hardware-confifuration.nix $i.nix
done
vi default.nix # add nodes and ip:s, etc.
../../bin/deploy.sh cluster-1

TODO

Self-provisioning NixOs iso image
Terraform setup for provisioning a cluster