This repository contains a batteries included, production ready, opinionated Kubernetes cluster setup. It only requires editing two simple configuration files to deploy a fully functional cluster using NixOps.

Together, NixOS and Kubernetes is a powerful combination. NixOS enables programmatic configuration of the base Kubernetes system services at the OS level. Just enabling the basic service is still a long way from having a production ready, multi node Kubernetes cluster up and running. Setting up a cluster at the OS level gives you a functional system shell, lacking most of the services needed for actually deploying, accessing and monitoring services.

This project will do to things: It will configure the cluster at the OS level, setting up services, networking, certificates, access tokens etc. It will then bootstrap the running Kubernetes instance, fixing permissions and configuring the following services:

Helm for deploying services
Nginx-ingress for external access
cert-manager with issuers (Let's encrypt, cluster-ca and self-sign)
nfs-client-provisioner for automatic volume management
kubernetes-dashboard for managing the cluster and services
metrics-server for monitoring
Prometheus, node exporters and grafana for monitoring

Prerequisites

Install n basic nodes (node-1, ..., node-n) running nixos.

git submodule init
git submodule update

Installation

cd clusters
cp -r template cluster-1
cd cluster-1
../../bin/initca.sh # generates the cluster wide CA certificate
for i in node-1 node-2 node-3; do
  scp $i:/etc/nixos/hardware-confifuration.nix $i.nix
done
vi default.nix # add nodes and ip:s, etc.
../../bin/deploy.sh cluster-1

etcd clustering guide

Using fish:

Install standard nixos k8s
Run the init-admin-kubeconfig.sh script in the k8s-charts repo.
Snapshot the etcd database: etcdctl snapshot save (date --iso-8601).etcd

Add the external interface to the etcd peers:

etcdctl member list
etcdctl member update [id] --peer-urls=https://[extenal ip]:2380
etcdctl member list

In the nixos cluster config, enable clustering for all nodes:

  etcdNodes = {
    kN-0 = "https://[ip1]:2380";
    kN-1 = "https://[ip2]:2380";
    kN-2 = "https://[ip3]:2380";
  };

  features.k8s.etcdCluster = {
    enable = true;
    existing = false; # true for master node!
    nodes = {}; # vitally important!
  };

Add the next node kN-1 to the cluster (on the master node):
```
etcdctl member add kN-1 --peer-urls=https://[ip2]:2380
```

ssh into kN-1 and run the etcd-join-cluster script:

sudo etcd-join-cluster kN-0=https://[ip0]:2380

Add the next node kN-2 to the cluster on the master node kN-0:
```
etcdctl member add kN-2 --peer-urls=https://[ip2]:2380
```

ssh into kN-2 and run the etcd-join-cluster script:

sudo etcd-join-cluster kN-0=https://[ip0]:2380,kN-1=https://[ip1]:2380

Zap (ctrl-c) the running etcd:s on kN-1 and kN-2.

Uncomment etcdNodes and set existing to true for all, and redeploy cluster

  features.k8s.etcdCluster = {
    enable = true;
    existing = true;
    nodes = etcdNodes;
  };

Check logs, restart etcd on nodes if necessary, etc.