Collect metrics¶
See how each Nereid is doing, its passes, failures and how long a pass takes, in the metrics the platform keeps with VictoriaMetrics.
Before you start¶
- A cluster from Prepare the cluster with
--profile internal-developer-platform: it brings the metrics reef with the Nereidvictoriametrics, which scrapes every Nereid innereus-systemand each node's CPU and memory per container. kubectlreaching that cluster. The query API has no credentials yet, so only the scraper, the Nereus API and a port-forward reach it.
Steps¶
1. Open vmui¶
The Mandate victoriametrics publishes the in-cluster query address in status.query.url (nereus mandate get). From your machine, forward a port to it, then open http://localhost:8428/vmui. The --context names the cluster nereus init made.
kubectl port-forward service/vmsingle-nereus 8428:8428 \
--context k3d-nereus \
--namespace nereus-system
2. Count each Nereid's passes¶
The queries are MetricsQL, a superset of PromQL. Each Nereid built on nereid-sdk serves its metrics with the label nereid, taken from its Pod, so no Pod can pose as a Nereid. One series per Nereid: its reconcile passes since it started.
sum by (nereid) (nereus_nereid_reconcile_total)
3. See which failed, and why¶
A Nereid that never failed has no error series, so or adds a zero for it from the passes, and every Nereid shows up.
sum by (nereid) (nereus_nereid_reconcile_errors_total) or sum by (nereid) (nereus_nereid_reconcile_total) * 0
The errors carry a reason label, the cause of a failed pass: group by (nereid, reason) to see them.
4. See how long a pass takes¶
The duration is a histogram. The 95th percentile over the last five minutes, per Nereid:
histogram_quantile(0.95, sum by (nereid, le) (rate(nereus_nereid_reconcile_duration_seconds_bucket[5m])))
Check it worked¶
The same port answers the Prometheus query API, for scripts: the first query returns one series per Nereid. A target that fails shows as down, and nothing else stops.
curl --get \
--data-urlencode 'query=sum by (nereid) (nereus_nereid_reconcile_total)' \
http://localhost:8428/api/v1/query
Next¶
- Keep a secret, and release a chart that reads it.
Learn more¶
- Inside the metrics reef: what the Nereid installs, what it scrapes and who reaches it.
- Runtime and profiles: why another Nereid of the metrics reef would publish the same
status.query.url.