fix: keep the docs in step with the manifests, stop two alerts misfiring #42

Merged

ivuorinen merged 2 commits from docs/sync-enumerations into main

2026-09-24 18:53:28 +00:00

ivuorinen commented

2026-09-24 18:48:51 +00:00

Owner

No description provided.

ivuorinen added 2 commits

2026-09-24 18:48:51 +00:00

docs: correct the enumerations the manifests already define 45046f9dbe

Five documentation claims had drifted from the manifests they describe,
each traceable to a feature commit that changed the YAML and not the prose:

- README's hostname-routing list omitted Node-RED — eight services have an
  IngressRoute, seven were named.
- README described forward-auth as protecting Longhorn alone. Five services
  carry an Authentik outpost prefix: Longhorn, AdGuard Home, Gatus, Homepage
  and Node-RED.
- docs/07-monitoring.md counted twenty-one PrometheusRule alerts. There are
  twenty-three, and the theme list omitted Node-RED flows.
- config/cluster.env.example marked Node3 as the only storage node. All four
  have carried Longhorn disks since Node4's reflash from eMMC, as
  04-label-nodes.sh and both node tables already said.
- README's script table omitted restore-test.sh, the only operator-invocable
  script it never named.

Prose that explains stayed accurate; what rotted was every place a document
indexed something the repository also enumerates in YAML. tests/docs-sync.bats
joins each of those lists against the manifests that define it, so the next
feature to land fails the suite rather than leaving a sentence behind. A route
whose host has no display_name() entry fails too, making an omission a
decision instead of a silent pass.

The bats hook's files pattern matched only tests/, scripts/lib.sh and
scripts/ci/, so version-sync.bats and renovate-annotations.bats never ran on
the files they assert about — a suite that cannot fire is worse than absent,
because it reads as coverage. The pattern now lists every input the suites
read.

fix: alert on sustained CronJob failure, not one blip

validate / hooks (pull_request) Successful in 1m23s

Details

validate / render (pull_request) Successful in 34s

Details

cef539a2e3

NodeRedFlowNeverRan used a flat `for: 1h` against a per-flow interval. It
fired on weekly-digest (interval 604800s) an hour after the Node-RED pod
moved nodes and lost its context — 168x too early, and it would have kept
firing all week until the flow next ran. It also contradicted its own
description, which promises a grace of one interval.

The grace is now the flow's own interval. Node-RED exports only the two
node_red_flow_* metrics, so kube_pod_start_time supplies the reference
point for how long the flow could have run by now. A pod restart restarts
the grace, which is the wanted behaviour: a restart is exactly when an
in-memory context is lost and the flow legitimately has no recorded run.

CronJobRepeatedlyFailing is the sustained-failure counterpart to the
chart's KubeJobFailed, which fires on a single failed Job and stays until
that Job object is removed by hand. A six-second EHOSTUNREACH to
git.esle.fi, hit while Renovate scanned an unrelated repository, failed
the 16:00 run; 17:00 and 18:00 succeeded and the alert still had to be
cleared by deleting the Job. The new rule counts distinct failed Jobs per
CronJob over the series rather than the live objects, because
failedJobsHistoryLimit is 1 everywhere here and only one failed Job
survives at a time.

Both expressions were checked against live Prometheus before being
written, each with a control that must fire: the first candidate for
NodeRedFlowNeverRan referenced process_start_time_seconds, which Node-RED
does not export, and returned empty because it evaluated to NaN rather
than because the condition was false.

ivuorinen merged commit 846a2142fb into main

2026-09-24 18:53:28 +00:00

ivuorinen deleted branch docs/sync-enumerations

2026-09-24 18:53:28 +00:00

ivuorinen referenced this pull request from a commit

2026-09-24 18:53:28 +00:00

fix: keep the docs in step with the manifests, stop two alerts misfiring (#42)