ivuorinen/turingpi-cluster
Watch
1
0
Fork
You've already forked turingpi-cluster
 0
Code Issues 1 Pull requests 6 Activity Actions

feat: back up Longhorn volumes to S3 on esle.fi #23

Merged
ivuorinen merged 1 commit from feat/longhorn-backups into main 2026-09-19 21:43:17 +00:00
Conversation 0 Commits 1 Files changed 17 +2243 -7
ivuorinen commented 2026-09-19 21:39:03 +00:00
Owner
Copy link

Longhorn keeps three replicas of every volume, which survives a failed
disk or node. It does not survive losing the chassis, a deleted volume,
or corruption the replicas copy faithfully. Nine volumes had no copy
outside the cluster, among them Authentik's PostgreSQL — users, groups
and MFA enrolments — and AdGuard's config.

The target is the SeaweedFS bucket turingpi-backup on s3.esle.fi, set
through the chart's defaultBackupStore values in the install script,
with credentials from 1Password. longhorn-manager copies a changed
longhorn-default-resource ConfigMap onto the existing BackupTarget, so
re-running the script is enough; a key left out keeps its current value,
which is why undoing it means setting the target to "" rather than
dropping the flag.

Four RecurringJobs: a nightly trim so backups carry the files in use
rather than every block ever allocated, a daily backup keeping 7 and a
weekly keeping 4, and a daily system backup keeping 7. The system backup
holds Longhorn's own state so a rebuilt cluster gets every PV/PVC back in
one restore; it runs daily because a restore uses each volume's last
backup as recorded when the system backup was taken, and a weekly one
would roll everything back up to six days.

Prometheus is trimmed but not backed up: seven days of metrics are two
thirds of the size and restoring them returns a week of graphs. Its PVC
comes from a StatefulSet template, so the labels go on by hand — adding
them to the template makes prometheus-operator delete and recreate the
StatefulSet without ever reaching the existing PVC. acme-state is backed
up but not trimmed: it is detached between CronJob runs, and trim needs
an attached filesystem. Backing up detached volumes at all needs
allowRecurringJobWhileVolumeDetached.

LonghornBackupStale fires when a volume has had no backup in 26 hours.
It is the only signal there can be: a recurring-job pod exits 0 even when
a volume's backup failed, because v1.12.1 assigns the volume jobs' error
after the return value is already set, so KubeJobFailed never sees it.
A volume never backed up counts from the later of its PVC's creation and
the backup job's, or all eight would alert the moment the rule loaded.
Gatus checks that SeaweedFS answers, which it does with 403 to anonymous
requests.

kubeconform gets a local schema directory ahead of the CRDs-catalog: the
catalog's Longhorn schemas predate 1.6 and reject task: system-backup,
which the installed CRD accepts. generate-crd-schemas.py writes them from
the pinned chart, converting exactly as the catalog's own generator does.

Longhorn keeps three replicas of every volume, which survives a failed disk or node. It does not survive losing the chassis, a deleted volume, or corruption the replicas copy faithfully. Nine volumes had no copy outside the cluster, among them Authentik's PostgreSQL — users, groups and MFA enrolments — and AdGuard's config. The target is the SeaweedFS bucket turingpi-backup on s3.esle.fi, set through the chart's defaultBackupStore values in the install script, with credentials from 1Password. longhorn-manager copies a changed longhorn-default-resource ConfigMap onto the existing BackupTarget, so re-running the script is enough; a key left out keeps its current value, which is why undoing it means setting the target to "" rather than dropping the flag. Four RecurringJobs: a nightly trim so backups carry the files in use rather than every block ever allocated, a daily backup keeping 7 and a weekly keeping 4, and a daily system backup keeping 7. The system backup holds Longhorn's own state so a rebuilt cluster gets every PV/PVC back in one restore; it runs daily because a restore uses each volume's last backup as recorded when the system backup was taken, and a weekly one would roll everything back up to six days. Prometheus is trimmed but not backed up: seven days of metrics are two thirds of the size and restoring them returns a week of graphs. Its PVC comes from a StatefulSet template, so the labels go on by hand — adding them to the template makes prometheus-operator delete and recreate the StatefulSet without ever reaching the existing PVC. acme-state is backed up but not trimmed: it is detached between CronJob runs, and trim needs an attached filesystem. Backing up detached volumes at all needs allowRecurringJobWhileVolumeDetached. LonghornBackupStale fires when a volume has had no backup in 26 hours. It is the only signal there can be: a recurring-job pod exits 0 even when a volume's backup failed, because v1.12.1 assigns the volume jobs' error after the return value is already set, so KubeJobFailed never sees it. A volume never backed up counts from the later of its PVC's creation and the backup job's, or all eight would alert the moment the rule loaded. Gatus checks that SeaweedFS answers, which it does with 403 to anonymous requests. kubeconform gets a local schema directory ahead of the CRDs-catalog: the catalog's Longhorn schemas predate 1.6 and reject task: system-backup, which the installed CRD accepts. generate-crd-schemas.py writes them from the pinned chart, converting exactly as the catalog's own generator does.
feat: back up Longhorn volumes to S3 on esle.fi
All checks were successful
validate / hooks (pull_request) Successful in 1m19s
Details
validate / render (pull_request) Successful in 31s
Details
ff13e63667 
Longhorn keeps three replicas of every volume, which survives a failed
disk or node. It does not survive losing the chassis, a deleted volume,
or corruption the replicas copy faithfully. Nine volumes had no copy
outside the cluster, among them Authentik's PostgreSQL — users, groups
and MFA enrolments — and AdGuard's config.

The target is the SeaweedFS bucket turingpi-backup on s3.esle.fi, set
through the chart's defaultBackupStore values in the install script,
with credentials from 1Password. longhorn-manager copies a changed
longhorn-default-resource ConfigMap onto the existing BackupTarget, so
re-running the script is enough; a key left out keeps its current value,
which is why undoing it means setting the target to "" rather than
dropping the flag.

Four RecurringJobs: a nightly trim so backups carry the files in use
rather than every block ever allocated, a daily backup keeping 7 and a
weekly keeping 4, and a daily system backup keeping 7. The system backup
holds Longhorn's own state so a rebuilt cluster gets every PV/PVC back in
one restore; it runs daily because a restore uses each volume's last
backup as recorded when the system backup was taken, and a weekly one
would roll everything back up to six days.

Prometheus is trimmed but not backed up: seven days of metrics are two
thirds of the size and restoring them returns a week of graphs. Its PVC
comes from a StatefulSet template, so the labels go on by hand — adding
them to the template makes prometheus-operator delete and recreate the
StatefulSet without ever reaching the existing PVC. acme-state is backed
up but not trimmed: it is detached between CronJob runs, and trim needs
an attached filesystem. Backing up detached volumes at all needs
allowRecurringJobWhileVolumeDetached.

LonghornBackupStale fires when a volume has had no backup in 26 hours.
It is the only signal there can be: a recurring-job pod exits 0 even when
a volume's backup failed, because v1.12.1 assigns the volume jobs' error
after the return value is already set, so KubeJobFailed never sees it.
A volume never backed up counts from the later of its PVC's creation and
the backup job's, or all eight would alert the moment the rule loaded.
Gatus checks that SeaweedFS answers, which it does with 403 to anonymous
requests.

kubeconform gets a local schema directory ahead of the CRDs-catalog: the
catalog's Longhorn schemas predate 1.6 and reject task: system-backup,
which the installed CRD accepts. generate-crd-schemas.py writes them from
the pinned chart, converting exactly as the catalog's own generator does.
ivuorinen deleted branch feat/longhorn-backups 2026-09-19 21:43:18 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
ivuorinen/turingpi-cluster!23
No description provided.