Prepare a cluster for Nereus¶
Make a cluster that offers what Nereus needs, by hand, for Install with Helm; nereus init --create-cluster does the same for you.
Before you start¶
What any cluster must offer, and how k3d and AKS meet it. AKS is there to compare and is not tested: no installation has run on it yet.
| Need | k3d | AKS (not tested) |
|---|---|---|
| Tools | Docker 20.10.5+, k3d 5.0.0+, Helm 4.0.0+ | az, Helm 4.0.0+ |
| Host | fs.inotify.max_user_instances=1024 |
the nodes are Azure's |
| Entrance: Gateway API kinds, a way to the Gateway | no Traefik; 127.0.0.1:8080 to the load balancer's port 80 |
a LoadBalancer Service with a public IP, DNS and TLS in front |
| Images | the local registry, or ghcr.io/getnereus/nereus |
ghcr.io, or a copy in an Azure Container Registry |
| Registry trust: every node trusts the local registry's CA | the CA's certificate mounted on every node | not needed: the nodes pull from a container registry such as ACR or ghcr.io, with credentials |
| API server: anonymous reads of the issuer's discovery only | an AuthenticationConfiguration on the server |
--enable-oidc-issuer; the anonymous read is still to check |
| Storage: a default StorageClass | local-path |
managed-csi |
| Network: NetworkPolicy enforced | k3s enforces it | --network-policy azure, calico or cilium |
Steps¶
The steps make a k3d cluster; the registry CA of steps 2 and 3 is for its local registry. AKS follows after step 5.
1. Raise the host's inotify limit¶
With Ubuntu's default of 128, a second k3d cluster never starts. Raise it once per machine.
echo "fs.inotify.max_user_instances=1024" \
| sudo tee /etc/sysctl.d/99-k3d-inotify.conf
sudo sysctl --system
2. Make the registry CA¶
A Registry answers only over TLS, with a certificate from one CA every node trusts. Its name constraints allow only localhost and names under svc.cluster.local, so trusting it trusts nothing else. Keep its files in a folder of your own, nereus/.
mkdir --parents nereus/registry
(umask 077 && openssl ecparam -name prime256v1 -genkey -noout -out nereus/ca.key)
openssl req -new -x509 -sha256 -days 3650 \
-key nereus/ca.key \
-out nereus/registry/ca.crt \
-subj "/CN=Nereus registry CA" \
-addext "basicConstraints=critical,CA:true,pathlen:0" \
-addext "keyUsage=critical,keyCertSign,cRLSign" \
-addext "nameConstraints=critical,permitted;DNS:localhost,permitted;DNS:svc.cluster.local,excluded;IP:0.0.0.0/0.0.0.0,excluded;IP:::/::"
chmod 0755 nereus nereus/registry
chmod 0644 nereus/registry/ca.crt
3. Trust it for every registry¶
nereus/registries.yaml adds the CA to the roots of every registry the nodes pull from.
configs:
"*":
tls:
ca_file: /etc/nereus/registry/ca.crt
4. Open the issuer's discovery¶
zot reads the cluster issuer's discovery and keys with no token, to check a build's push token. k3s turns anonymous requests off; nereus/authentication-config.yaml lets them in on those two paths only.
apiVersion: apiserver.config.k8s.io/v1
kind: AuthenticationConfiguration
anonymous:
enabled: true
conditions:
- path: /.well-known/openid-configuration
- path: /openid/v1/jwks
5. Create the cluster¶
No Traefik, since the profile brings Istio's Gateway; the Gateway on port 8080 of the loopback; the CA on every node and the configuration on the server. Docker mounts absolute paths.
k3d cluster create nereus \
--k3s-arg "--disable=traefik@server:*" \
--port "127.0.0.1:8080:80@loadbalancer" \
--registry-config nereus/registries.yaml \
--volume "${PWD}/nereus/registry:/etc/nereus/registry@all" \
--volume "${PWD}/nereus/authentication-config.yaml:/etc/rancher/nereus/authentication-config.yaml@server:*" \
--k3s-arg "--kube-apiserver-arg=authentication-config=/etc/rancher/nereus/authentication-config.yaml@server:*" \
--wait
Keep nereus/: the nodes mount it, the API server reads its file on every restart, and the chart nereus takes the CA from it.
On AKS, the same rows are flags of az aks create: --network-plugin azure --network-policy azure, --enable-oidc-issuer and --enable-workload-identity; a build pool is a node pool labelled and tainted build.getnereus.dev/pool=build.
Check it worked¶
kubectl get storageclass shows local-path as the default, and kubectl get nodes shows every node Ready. After the install, nereus doctor says when the issuer's discovery is closed.
Clean up¶
Delete the cluster with k3d cluster delete nereus and its folder with rm -r nereus.
Next¶
- Install with Helm on this cluster.
Learn more¶
- Runtime and profiles: what the profile installs on the cluster.