NixOps Kubernetes clusters

This repository contains a batteries included, production ready, opinionated Kubernetes cluster setup. It only requires editing two simple configuration files to deploy a fully functional cluster using NixOps.

Together, NixOS and Kubernetes is a powerful combination. NixOS enables programmatic configuration of the base Kubernetes system services at the OS level. Just enabling the basic service is still a long way from having a production ready, multi node Kubernetes cluster up and running. Setting up a cluster at the OS level gives you a functional system shell, lacking most of the services needed for actually deploying, accessing and monitoring services.

This project will do to things: It will configure the cluster at the OS level, setting up services, networking, certificates, access tokens etc. It will then bootstrap the running Kubernetes instance, fixing permissions and configuring the following services:

• Helm for deploying services
• Nginx-ingress for external access
• cert-manager with issuers (Let's encrypt, cluster-ca and self-sign)
• nfs-client-provisioner for automatic volume management
• kubernetes-dashboard for managing the cluster and services
• metrics-server for monitoring
• Prometheus, node exporters and grafana for monitoring

Prerequisites

1. Install n basic nodes (node-1, ..., node-n) running nixos.

git submodule init
git submodule update

Installation

cd clusters
cp -r template cluster-1
cd cluster-1
../../bin/initca.sh # generates the cluster wide CA certificate
for i in node-1 node-2 node-3; do
  scp $i:/etc/nixos/hardware-confifuration.nix $i.nix
done
vi default.nix # add nodes and ip:s, etc.
../../bin/deploy.sh cluster-1

etcd clustering guide

Using fish:

1. Install standard nixos k8s
2. Run the init-admin-kubeconfig.sh script in the k8s-charts repo.
3. Snapshot the etcd database: etcdctl snapshot save (date --iso-8601).etcd
4. Add the external interface to the etcd peers:

   etcdctl member list
   etcdctl member update [id] --peer-urls=https://[extenal ip]:2380
   etcdctl member list
   

5. In the nixos cluster config, enable clustering for all nodes:

     etcdNodes = {
       kN-0 = "https://[ip1]:2380";
       kN-1 = "https://[ip2]:2380";
       kN-2 = "https://[ip3]:2380";
     };
   
     features.k8s.etcdCluster = {
       enable = true;
       existing = false; # true for master node!
       nodes = {}; # vitally important!
     };
   

6. Add the next node kN-1 to the cluster (on the master node):

   etcdctl member add kN-1 --peer-urls=https://[ip2]:2380
   

7. ssh into kN-1 and run the etcd-join-cluster script:

   sudo etcd-join-cluster kN-0=https://[ip0]:2380
   

8. Add the next node kN-2 to the cluster on the master node kN-0:

   etcdctl member add kN-2 --peer-urls=https://[ip2]:2380
   

9. ssh into kN-2 and run the etcd-join-cluster script:

   sudo etcd-join-cluster kN-0=https://[ip0]:2380,kN-1=https://[ip1]:2380
   

10. Zap (ctrl-c) the running etcd:s on kN-1 and kN-2.
11. Uncomment etcdNodes and set existing to true for all, and redeploy cluster

      features.k8s.etcdCluster = {
        enable = true;
        existing = true;
        nodes = etcdNodes;
      };
    

12. Check logs, restart etcd on nodes if necessary, etc.