# NixOps Kubernetes clusters

This repository contains a batteries included, production ready, opinionated
Kubernetes cluster setup. It only requires editing two simple configuration
files to deploy a fully functional cluster using NixOps.

Together, NixOS and Kubernetes is a powerful combination. NixOS enables
programmatic configuration of the base Kubernetes system services at the OS
level. Just enabling the basic service is still a long way from having a
production ready, multi node Kubernetes cluster up and running. Setting up a
cluster at the OS level gives you a functional system shell, lacking most of the
services needed for actually deploying, accessing and monitoring services.

This project will do to things: It will configure the cluster at the OS level,
setting up services, networking, certificates, access tokens etc. It will then
bootstrap the running Kubernetes instance, fixing permissions and configuring
the following services:

* Helm for deploying services
* Nginx-ingress for external access
* cert-manager with issuers (Let's encrypt, cluster-ca and self-sign)
* nfs-client-provisioner for automatic volume management
* kubernetes-dashboard for managing the cluster and services
* metrics-server for monitoring
* Prometheus, node exporters and grafana for monitoring

## Prerequisites

1. Install n basic nodes (node-1, ..., node-n) running nixos.

```sh
git submodule init
git submodule update
```

## Installation

```sh
cd clusters
cp -r template cluster-1
cd cluster-1
../../bin/initca.sh # generates the cluster wide CA certificate
for i in node-1 node-2 node-3; do
  scp $i:/etc/nixos/hardware-confifuration.nix $i.nix
done
vi default.nix # add nodes and ip:s, etc.
../../bin/deploy.sh cluster-1
```

## etcd clustering guide

Using fish:

1. Install standard nixos k8s
2. Run the `init-admin-kubeconfig.sh` script in the `k8s-charts` repo.
3. Snapshot the etcd database: `etcdctl snapshot save (date --iso-8601).etcd`
4. Add the external interface to the etcd peers:
    ```
    etcdctl member list
    etcdctl member update [id] --peer-urls=https://[extenal ip]:2380
    etcdctl member list
    ```
4. In the nixos cluster config, enable clustering for all nodes:
    ```
      etcdNodes = {
        kN-0 = "https://[ip1]:2380";
        kN-1 = "https://[ip2]:2380";
        kN-2 = "https://[ip3]:2380";
      };

      features.k8s.etcdCluster = {
        enable = true;
        existing = false; # true for master node!
        nodes = {}; # vitally important!
      };
    ```
5. Add the next node `kN-1` to the cluster (on the master node):
    ```
    etcdctl member add kN-1 --peer-urls=https://[ip2]:2380
    ```
6. ssh into `kN-1` and run the `etcd-join-cluster` script:
   ```
   sudo etcd-join-cluster kN-0=https://[ip0]:2380
   ```
7. Add the next node `kN-2` to the cluster on the master node `kN-0`:
    ```
    etcdctl member add kN-2 --peer-urls=https://[ip2]:2380
    ```
8. ssh into `kN-2` and run the `etcd-join-cluster` script:
   ```
   sudo etcd-join-cluster kN-0=https://[ip0]:2380,kN-1=https://[ip1]:2380
   ```
9. Zap (ctrl-c) the running etcd:s on `kN-1` and `kN-2`.
10. Uncomment etcdNodes and set existing to true for all, and redeploy cluster
    ```
      features.k8s.etcdCluster = {
        enable = true;
        existing = true;
        nodes = etcdNodes;
      };
    ```
11. Check logs, restart etcd on nodes if necessary, etc.