Nereid SDK controller metrics: assessment and proposal¶
Asked by the owner (2026-11-17), from a voice session. Assessment and proposal only; nothing implemented.
Summary¶
- The SDK exists and is custom:
lib/nereid(nereid-sdk0.2.0, #754), no Kopf, no work queue. Each Nereid runs one thread with a periodic, level-triggered pass over everything it owns (passes.forever+passes.Pace). - Metrics already exist, under Nereus names: six
nereus_nereid_*series on every Nereid'sGET /metrics, through OpenTelemetry's Prometheus exporter. 19 Nereids pinnereid-sdk==0.2.0; all 21 with a loop usetelemetry/passes. - Scraping already works: the
victoriametricsNereid'sVMPodScrape nereidspicks every pod labelledcore.<domain>/metrics: "true"innereus-system, which the CLI sets on every Nereid it deploys. A new Nereid is discovered with zero effort. - The gap is the names: none of the controller-runtime / client-go workqueue names exist, so community dashboards do not work. Several workqueue metrics have no honest meaning in a queue-less, one-worker loop and must be mapped with care.
- The portal view is already designed: #744 (each Nereid's golden signals, with sparklines on the phone), on top of #740 (home screen, one row per reef and Nereid) and #746 (baselines). No reef has an icon today.
Decisions (owner, 2026-11-17)¶
- SDK design accepted:
nereid-sdk0.3.0 emits the controller-runtime and workqueue names alongsidenereus_nereid_*, both kept. - #744 is adjusted, not replaced: its queries move to the standard names.
- Reef icon: an
iconfield on the Nereid'sGET /contract (an amendment to ADR 0022), naming an icon of an open-source set vendored in the portal,honeycomb.svgwhen missing. - Dashboards: only the portal for now; a Grafana Nereid in the metrics reef stays an idea for later.
1. The SDK¶
Location: lib/nereid/src/nereid/ — telemetry.py, passes.py, logs.py, screens.py, addresses.py.
| Question | Finding |
|---|---|
| Framework | None. Plain Python, threading, urllib against the Kubernetes API (each Nereid has its own kube.py). Kopf is not used anywhere. |
| Reconciliation | A pass: one call that reconciles the Nereid's whole world (all its objects), level-triggered, no watches (no ?watch= anywhere in controllers/). |
| Work queue | None. No per-object keys, no dedup, no rate limiter. |
| Retries | Implicit: a failed pass sets Ready=False with a reason and the next lap runs again. passes.publish retries a failed status write on the next lap. passes.forever never dies (#932). |
| Concurrency | One reconcile thread per Nereid (17× threading.Thread(target=loop, …)), so max concurrency is 1. The HTTP server runs in a separate daemon thread. |
| Pace | passes.Pace (#1297): idle seconds between laps, busy (1 s) for 30 s after a pass reported work. |
| Existing metrics | telemetry.py, OpenTelemetry SDK + PrometheusMetricReader on a private CollectorRegistry, text format 0.0.4. No process_* collector. |
The six metrics today (lib/nereid/README.md, pinned by tests/test_telemetry.py):
| Metric | Type | Labels |
|---|---|---|
nereus_nereid_reconcile_total |
counter | nereid |
nereus_nereid_reconcile_errors_total |
counter | nereid, reason |
nereus_nereid_reconcile_duration_seconds |
histogram, 5 ms – 60 s | nereid |
nereus_nereid_reconcile_period_seconds |
gauge | nereid |
nereus_nereid_last_success_timestamp_seconds |
gauge | nereid |
nereus_nereid_ready |
gauge 1/0 | nereid, reason |
2. The metrics reef and the VictoriaMetrics Nereid¶
controllers/victoriametrics/ (reef metrics), objects in nereus-system:
- VMSingle
nereus(14 d retention), VMAgentnereuswithselectAllByDefault: false(only its own scrape objects). - VMPodScrape
nereids: selectorcore.<domain>/metrics: "true", namespacenereus-systemonly, porthttp, path/metrics, labelnereidfrom the pod'sapp.kubernetes.io/name,honorLabels: false(a series cannot fake itsnereid). Built incli/src/nereus/local_nereids.py:426; the label is put on every Nereid pod incli/src/nereus/cluster.py:2683(METRICS_LABEL). - VMNodeScrape
kubelet-cadvisorfor CPU and memory per container. - ADR 0025 (metrics enter from outside only through the Nereus API, with declared sources) reserves the
nereus_prefix and thenereidlabel for the platform; scraped Nereid metrics are unaffected.
Discovery of a new Nereid: automatic. Deployed by nereus nereid apply → labelled → scraped. Nothing for the author to do.
3. Open source first¶
- controller-runtime (kubebuilder metrics reference, checked 2026-11-17) defines:
controller_runtime_reconcile_total,controller_runtime_reconcile_errors_total,controller_runtime_terminal_reconcile_errors_total,controller_runtime_reconcile_time_seconds,controller_runtime_max_concurrent_reconciles,controller_runtime_active_workers, the webhook metrics,rest_client_requests_total, and the client-go workqueue set (workqueue_depth,workqueue_adds_total,workqueue_queue_duration_seconds,workqueue_work_duration_seconds,workqueue_unfinished_work_seconds,workqueue_longest_running_processor_seconds,workqueue_retries_total). Labels (from controller-runtime's source, not the reference page, which lists none):controlleron allcontroller_runtime_*, plusresult(success,error,requeue,requeue_after) onreconcile_total;name(andcontrollerin recent versions) onworkqueue_*. To pin against the vendored version before implementing. - Kopf: exposes only a liveness/health endpoint; no Prometheus metrics (docs.kopf.dev, "Health-checks"). Adopting Kopf would bring watches and per-object handlers but not the metrics, and would mean rewriting 21 Nereids: out of scope.
- No Python library found that emits the controller-runtime names.
prometheus_client(already a dependency) is the right building block; the SDK already owns the registry.
Conclusion: adopt the names and labels (the standard), build the thin emitter in the SDK, where it already is.
4. Proposed metrics layer in the SDK¶
Principles¶
- Zero effort: emitted from
telemetry.reconcile_pass()andpasses.forever/Pace, which every Nereid already calls. Bumpnereid-sdkto 0.3.0, bump the pin in the 19requirements.txt; no Nereid code changes. - Deterministic: pure counters/gauges from the loop; no AI anywhere.
- Honest: a metric with no meaning in this model is emitted with its truthful constant value (or not at all), never faked to look like a busy queue.
controllerlabel = the Nereid's name (the value ofsetup(nereid=…)); the scrape still addsnereid.
Mapping¶
| Standard metric | Meaning in a Nereid | Source in the SDK |
|---|---|---|
controller_runtime_reconcile_total{controller,result} |
one per pass; result="error" on failure, "requeue" when the pass called Pace.busy(), else "success" |
reconcile_pass() |
controller_runtime_reconcile_errors_total{controller} |
failed passes | reconcile_pass() (keeps the Nereus reason only on nereus_nereid_reconcile_errors_total) |
controller_runtime_terminal_reconcile_errors_total{controller} |
passes failing with one of known (a refusal the Nereid raises on purpose) |
passes.run |
controller_runtime_reconcile_time_seconds{controller} |
pass duration | same histogram data, controller-runtime buckets |
controller_runtime_max_concurrent_reconciles{controller} |
constant 1 | setup() |
controller_runtime_active_workers{controller} |
1 while a pass runs, else 0 | reconcile_pass() |
workqueue_depth{name,controller} |
1 when a lap is due and not started, else 0 (practically 0) | forever |
workqueue_adds_total{name,controller} |
laps scheduled | forever |
workqueue_queue_duration_seconds{name,controller} |
lateness of a lap: actual start minus scheduled start | forever |
workqueue_work_duration_seconds{name,controller} |
pass duration | reconcile_pass() |
workqueue_unfinished_work_seconds{name,controller} |
seconds the current pass has been running | callback gauge |
workqueue_longest_running_processor_seconds{name,controller} |
same as above (one worker) — the most useful one: a stuck pass shows up | callback gauge |
workqueue_retries_total{name,controller} |
passes started right after a failed pass | forever |
process_* (CPU, RSS, fds, start time) |
prometheus_client.ProcessCollector on the SDK registry |
setup(); used by the community dashboards |
rest_client_requests_total{code,method,host} |
optional, later: needs a hook in each Nereid's kube.py; candidate for moving Kube into the SDK |
out of this step |
Migration of the Nereus names¶
The nereus_nereid_* set stays (it carries what the standard lacks: reason, period, last_success, ready). Emit both sets in 0.3.0. Then either keep both (cardinality is tiny: ~20 Nereids × ~15 series) or drop the two duplicated ones (reconcile_total, reconcile_duration_seconds) after #744's queries move to the standard names. Recommendation: keep both; drop nothing until the portal is on the standard names.
Rejected alternative: VMRule recording rules that rename nereus_* to controller_runtime_* in the store. Deterministic and no SDK change, but cannot produce active_workers, unfinished_work_seconds or process_*, and dashboards querying a raw /metrics would not see them.
Dashboards¶
The Kubebuilder Grafana plugin (grafana/v1-alpha) generates controller-runtime-metrics.json and controller-resources-metrics.json that query these names (and process_*, job). Nereus ships no Grafana; VictoriaMetrics' vmui can import Grafana-style queries but not the dashboards. Worth a follow-up decision: a Grafana Nereid in the metrics reef, or only the portal view below.
5. Portal view (RED per Nereid)¶
Already specified by #744 (each Nereid's golden signals from the metrics reef, with sparklines on the phone), on the rows of #740 (portal home screen, one row per reef and Nereid) and #746 (baseline thresholds, then 7-day baselines). #744 defines signals on each GET /v1/status row: rate, errorRatio, p95Seconds, saturation, CPU and memory, with 1 h series for sparklines, one query per signal for all Nereids (by (nereid)).
Proposed adjustments to #744, not a new issue:
- Rate:
sum by (nereid) (rate(controller_runtime_reconcile_total[5m])). - Errors:
sum by (nereid) (rate(controller_runtime_reconcile_errors_total[5m])) / R. - Duration:
histogram_quantile(0.95, sum by (nereid, le) (rate(controller_runtime_reconcile_time_seconds_bucket[5m]))). - Keep saturation (mean pass time over period) and add
workqueue_longest_running_processor_secondsas the "stuck" flag. - Layout: one compact row per Nereid (icon, name, state chip, R · E% · p95, one sparkline) on the phone; a table with all sparklines on desktop.
Reef icons: no reef has one. Only ADR 0022 screens carry an icon name (validated by /^[a-z0-9-]{1,40}$/ in static/screens.js, used today only by leagore), and the portal only sets it as data-icon with no icon set behind it. Proposal: an icon field on the Nereid's GET / contract (ADR 0022's contractVersion: 1 allows new fields), merged into its status like the screens, from one shared, open-source icon set vendored in the portal (e.g. Lucide, ISC); fallback honeycomb.svg.
Gaps¶
- No controller-runtime / workqueue names or labels: community dashboards do not work.
- No
active_workers/ in-flight gauge: a stuck pass is invisible untillast_successages. - No
process_*metrics on the SDK registry (CPU/memory come only from cAdvisor). - No
resultdimension (requeue vs success). - No API client metrics (
rest_client_requests_total): each Nereid has its ownkube.py. -
744's queries use the Nereus names; no portal signals implemented yet.¶
- No reef icon anywhere; no icon set in the portal.
Next steps¶
- Pin the exact labels from the controller-runtime version to follow (source, not the reference page), and write the mapping table into
lib/nereid/README.mdas the contract. nereid-sdk0.3.0: emit the standard set alongsidenereus_nereid_*,ProcessCollector, tests intests/test_telemetry.pypinning the exposition; bump the 19 pins.- Check
make scene-metrics(test/scene-metrics) sees the new series through the VMPodScrape. - Update #744's queries to the standard names; then deliver #739/#740/#744.
- ADR 0022 amendment: the reef
iconfield and the portal's icon set (decided). - Later: move
Kubeinto the SDK and addrest_client_requests_total; a Grafana Nereid only if the portal falls short.