Skip to content

Prepare a cluster for Nereus

Make a cluster that offers what Nereus needs, by hand, for Install with Helm; nereus init --create-cluster does the same for you.

Before you start

What any cluster must offer, and how k3d and AKS meet it. AKS is there to compare and is not tested: no installation has run on it yet.

Need k3d AKS (not tested)
Tools Docker 20.10.5+, k3d 5.0.0+, Helm 4.0.0+ az, Helm 4.0.0+
Host fs.inotify.max_user_instances=1024 the nodes are Azure's
Entrance: Gateway API kinds, a way to the Gateway no Traefik; 127.0.0.1:8080 to the load balancer's port 80 a LoadBalancer Service with a public IP, DNS and TLS in front
Images the local registry, or ghcr.io/getnereus/nereus ghcr.io, or a copy in an Azure Container Registry
Registry trust: every node trusts the local registry's CA the CA's certificate mounted on every node not needed: the nodes pull from a container registry such as ACR or ghcr.io, with credentials
API server: anonymous reads of the issuer's discovery only an AuthenticationConfiguration on the server --enable-oidc-issuer; the anonymous read is still to check
Storage: a default StorageClass local-path managed-csi
Network: NetworkPolicy enforced k3s enforces it --network-policy azure, calico or cilium

Steps

The steps make a k3d cluster; the registry CA of steps 2 and 3 is for its local registry. AKS follows after step 5.

1. Raise the host's inotify limit

With Ubuntu's default of 128, a second k3d cluster never starts. Raise it once per machine.

echo "fs.inotify.max_user_instances=1024" \
  | sudo tee /etc/sysctl.d/99-k3d-inotify.conf
sudo sysctl --system

2. Make the registry CA

A Registry answers only over TLS, with a certificate from one CA every node trusts. Its name constraints allow only localhost and names under svc.cluster.local, so trusting it trusts nothing else. Keep its files in a folder of your own, nereus/.

mkdir --parents nereus/registry
(umask 077 && openssl ecparam -name prime256v1 -genkey -noout -out nereus/ca.key)
openssl req -new -x509 -sha256 -days 3650 \
  -key nereus/ca.key \
  -out nereus/registry/ca.crt \
  -subj "/CN=Nereus registry CA" \
  -addext "basicConstraints=critical,CA:true,pathlen:0" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" \
  -addext "nameConstraints=critical,permitted;DNS:localhost,permitted;DNS:svc.cluster.local,excluded;IP:0.0.0.0/0.0.0.0,excluded;IP:::/::"
chmod 0755 nereus nereus/registry
chmod 0644 nereus/registry/ca.crt

3. Trust it for every registry

nereus/registries.yaml adds the CA to the roots of every registry the nodes pull from.

configs:
  "*":
    tls:
      ca_file: /etc/nereus/registry/ca.crt

4. Open the issuer's discovery

zot reads the cluster issuer's discovery and keys with no token, to check a build's push token. k3s turns anonymous requests off; nereus/authentication-config.yaml lets them in on those two paths only.

apiVersion: apiserver.config.k8s.io/v1
kind: AuthenticationConfiguration
anonymous:
  enabled: true
  conditions:
    - path: /.well-known/openid-configuration
    - path: /openid/v1/jwks

5. Create the cluster

No Traefik, since the profile brings Istio's Gateway; the Gateway on port 8080 of the loopback; the CA on every node and the configuration on the server. Docker mounts absolute paths.

k3d cluster create nereus \
  --k3s-arg "--disable=traefik@server:*" \
  --port "127.0.0.1:8080:80@loadbalancer" \
  --registry-config nereus/registries.yaml \
  --volume "${PWD}/nereus/registry:/etc/nereus/registry@all" \
  --volume "${PWD}/nereus/authentication-config.yaml:/etc/rancher/nereus/authentication-config.yaml@server:*" \
  --k3s-arg "--kube-apiserver-arg=authentication-config=/etc/rancher/nereus/authentication-config.yaml@server:*" \
  --wait

Keep nereus/: the nodes mount it, the API server reads its file on every restart, and the chart nereus takes the CA from it.

On AKS, the same rows are flags of az aks create: --network-plugin azure --network-policy azure, --enable-oidc-issuer and --enable-workload-identity; a build pool is a node pool labelled and tainted build.getnereus.dev/pool=build.

Check it worked

kubectl get storageclass shows local-path as the default, and kubectl get nodes shows every node Ready. After the install, nereus doctor says when the issuer's discovery is closed.

Clean up

Delete the cluster with k3d cluster delete nereus and its folder with rm -r nereus.

Next

Learn more