Update readme for etcd clusters

README.md

@@ -33,7 +33,6 @@ git submodule init
 git submodule update
 ```
 
-
 ## Installation
 
 ```sh
@@ -48,7 +47,53 @@ vi default.nix # add nodes and ip:s, etc.
 ../../bin/deploy.sh cluster-1
 ```
 
-## TODO
+## etcd clustering guide
+
+Using fish:
+
+1. Install standard nixos k8s
+2. Run the `init-admin-kubeconfig.sh` script in the `k8s-charts` repo.
+3. Snapshot the etcd database: `etcdctl snapshot save (date --iso-8601).etcd`
+4. Add the external interface to the etcd peers:
+    ```
+    etcdctl member list
+    etcdctl member update [id] --peer-urls=https://[extenal ip]:2380
+    etcdctl member list
+    ```
+4. In the nixos cluster config, enable clustering for all nodes:
+    ```
+      etcdNodes = {};
+
+      features.k8s.etcdCluster = {
+        enable = true;
+        existing = false; # true for master node!
+        nodes = etcdNodes;
+      };
+    ```
+5. Add the next node `kN-1` to the cluster (on the master node):
+    ```
+    etcdctl member add kN-1 --peer-urls=https://[ip2]:2380
+    ```
+6. ssh into `kN-1` and run the `etcd-join-cluster` script.
+7. Add the next node `kN-2` to the cluster (on the master node):
+    ```
+    etcdctl member add kN-2 --peer-urls=https://[ip2]:2380
+    ```
+8. ssh into `kN-2` and run the `etcd-join-cluster` script.
+9. Zap (ctrl-c) the running etcd:s on kN-1 and 2.
+10. Uncomment etcdNodes and set existing to true for all, and redeploy cluster
+    ```
+      etcdNodes = {
+        kN-0 = "https://[ip1]:2380";
+        kN-1 = "https://[ip2]:2380";
+        kN-2 = "https://[ip3]:2380";
+      };
+
+      features.k8s.etcdCluster = {
+        enable = true;
+        existing = true;
+        nodes = etcdNodes;
+      };
+    ```
+11. Check logs, restart etcd on nodes if necessary, etc.
 
-* Self-provisioning NixOs iso image
-* Terraform setup for provisioning a cluster