104 lines
3.3 KiB
Markdown
104 lines
3.3 KiB
Markdown
# NixOps Kubernetes clusters
|
|
|
|
This repository contains a batteries included, production ready, opinionated
|
|
Kubernetes cluster setup. It only requires editing two simple configuration
|
|
files to deploy a fully functional cluster using NixOps.
|
|
|
|
Together, NixOS and Kubernetes is a powerful combination. NixOS enables
|
|
programmatic configuration of the base Kubernetes system services at the OS
|
|
level. Just enabling the basic service is still a long way from having a
|
|
production ready, multi node Kubernetes cluster up and running. Setting up a
|
|
cluster at the OS level gives you a functional system shell, lacking most of the
|
|
services needed for actually deploying, accessing and monitoring services.
|
|
|
|
This project will do to things: It will configure the cluster at the OS level,
|
|
setting up services, networking, certificates, access tokens etc. It will then
|
|
bootstrap the running Kubernetes instance, fixing permissions and configuring
|
|
the following services:
|
|
|
|
* Helm for deploying services
|
|
* Nginx-ingress for external access
|
|
* cert-manager with issuers (Let's encrypt, cluster-ca and self-sign)
|
|
* nfs-client-provisioner for automatic volume management
|
|
* kubernetes-dashboard for managing the cluster and services
|
|
* metrics-server for monitoring
|
|
* Prometheus, node exporters and grafana for monitoring
|
|
|
|
## Prerequisites
|
|
|
|
1. Install n basic nodes (node-1, ..., node-n) running nixos.
|
|
|
|
```sh
|
|
git submodule init
|
|
git submodule update
|
|
```
|
|
|
|
## Installation
|
|
|
|
```sh
|
|
cd clusters
|
|
cp -r template cluster-1
|
|
cd cluster-1
|
|
../../bin/initca.sh # generates the cluster wide CA certificate
|
|
for i in node-1 node-2 node-3; do
|
|
scp $i:/etc/nixos/hardware-confifuration.nix $i.nix
|
|
done
|
|
vi default.nix # add nodes and ip:s, etc.
|
|
../../bin/deploy.sh cluster-1
|
|
```
|
|
|
|
## etcd clustering guide
|
|
|
|
Using fish:
|
|
|
|
1. Install standard nixos k8s
|
|
2. Run the `init-admin-kubeconfig.sh` script in the `k8s-charts` repo.
|
|
3. Snapshot the etcd database: `etcdctl snapshot save (date --iso-8601).etcd`
|
|
4. Add the external interface to the etcd peers:
|
|
```
|
|
etcdctl member list
|
|
etcdctl member update [id] --peer-urls=https://[extenal ip]:2380
|
|
etcdctl member list
|
|
```
|
|
4. In the nixos cluster config, enable clustering for all nodes:
|
|
```
|
|
etcdNodes = {
|
|
kN-0 = "https://[ip1]:2380";
|
|
kN-1 = "https://[ip2]:2380";
|
|
kN-2 = "https://[ip3]:2380";
|
|
};
|
|
|
|
features.k8s.etcdCluster = {
|
|
enable = true;
|
|
existing = false; # true for master node!
|
|
nodes = {}; # vitally important!
|
|
};
|
|
```
|
|
5. Add the next node `kN-1` to the cluster (on the master node):
|
|
```
|
|
etcdctl member add kN-1 --peer-urls=https://[ip2]:2380
|
|
```
|
|
6. ssh into `kN-1` and run the `etcd-join-cluster` script:
|
|
```
|
|
sudo etcd-join-cluster kN-0=https://[ip0]:2380
|
|
```
|
|
7. Add the next node `kN-2` to the cluster on the master node `kN-0`:
|
|
```
|
|
etcdctl member add kN-2 --peer-urls=https://[ip2]:2380
|
|
```
|
|
8. ssh into `kN-2` and run the `etcd-join-cluster` script:
|
|
```
|
|
sudo etcd-join-cluster kN-0=https://[ip0]:2380,kN-1=https://[ip1]:2380
|
|
```
|
|
9. Zap (ctrl-c) the running etcd:s on `kN-1` and `kN-2`.
|
|
10. Uncomment etcdNodes and set existing to true for all, and redeploy cluster
|
|
```
|
|
features.k8s.etcdCluster = {
|
|
enable = true;
|
|
existing = true;
|
|
nodes = etcdNodes;
|
|
};
|
|
```
|
|
11. Check logs, restart etcd on nodes if necessary, etc.
|
|
|