turingpi-kubernetes

GitOps-managed k3s cluster on a Turing Pi 2, built by adapting Turing Pi's own documentation to a 4-node cluster at 192.168.5.11-14:

• Kubernetes Installation
• Helm and Arkade
• Network Configuration
• Cluster Storage
• Argo CD
• Sample App Deployment

Detailed, per-topic write-ups adapted from each guide live in docs/. This file is the quick path from bare metal to a running, GitOps-managed sample app.

What's here vs. what the guides show

Two deliberate departures from the source guides, both explained in more detail in docs/:

1. Operated from your workstation, not from Node1 over SSH. The guides install Helm on Node1 and run every kubectl/helm command there. This repo fetches the kubeconfig to your machine once (scripts/03-fetch-kubeconfig.sh) and runs everything after that locally. SSH into a node is only used for OS-level steps: hostnames, OS packages, k3s itself, and storage packages.
2. The sample app is actually deployed via ArgoCD, not kubectl apply. The guides deploy Redis by hand and mention ArgoCD as something you could also use. Here, the Redis manifests live in gitops/apps/ and ArgoCD is what applies them — that's the GitOps part.

Everything else (node roles, install flags, MetalLB pool sizing, storage choice) mirrors the source guides directly, with IPs substituted for this cluster.

Beyond the source guides

None of this is part of the six Turing Pi guides. All of it is added the same way as everything else here: as ArgoCD-managed apps under gitops/argocd-apps/.

• Monitoring — Prometheus + Grafana for metrics, Loki + Grafana Alloy for logs, kept for 180 days. It reaches past the cluster: machines on the LAN send syslog to a receiver on 192.168.5.76, and the Home Assistant box's own system metrics are scraped and dashboarded. See 07-monitoring.md. Gatus checks every service from the outside — cluster UIs through LAN DNS and Traefik, esle.fi, home devices — and game servers are queried over Steam A2S; see 12-gatus-homepage.md.
• Secrets — every application credential the cluster uses lives in a 1Password vault and is rendered into Kubernetes Secrets by External Secrets Operator. No secret value is in git. See 10-secrets.md.
• Dependency updates — Renovate runs hourly and keeps every repository on git.esle.fi current, this one included. Here, patch and minor updates merge themselves once CI passes, in a window when someone is around; the updates that need a manual step wait for review. See 11-renovate.md.
• Backups — every Longhorn volume except Prometheus's is backed up nightly to an S3 bucket on esle.fi, with a daily system backup for restoring the whole cluster; a missed backup raises an alert. See 04-storage.md.
• Automations — Node-RED builds them as flows rather than code: it reads and acts on Home Assistant entities over its websocket API, and glues services together. See 13-node-red.md.
• Remote access — a Tailscale subnet router puts the whole LAN on a tailnet, so every service is reachable from anywhere exactly as at home, with no port opened on the router. See 14-tailscale.md.
• Shared authentication across services (Authentik), built in three phases:
  1. Local DNS + wildcard certs — AdGuard Home and an ACME wildcard cert for *.t.9b11.xyz. See 08-dns-and-certs.md.
  2. Hostname routing through Traefik — Grafana, Longhorn, ArgoCD, AdGuard Home, Authentik, Gatus, Homepage and Node-RED each get a .t.9b11.xyz route (gitops/apps/ingress-routes/). Grafana also keeps its direct MetalLB IP; Longhorn's and ArgoCD's Services are ClusterIP only.
  3. Authentik itself — native OIDC for Grafana and ArgoCD, forward-auth for the services with no native OIDC support: Longhorn, AdGuard Home, Gatus, Homepage and Node-RED. Its Providers/Applications are declared as blueprints in git (gitops/apps/authentik-blueprints/, and the OIDC blueprint in gitops/apps/external-secrets-config/authentik.yaml); the first-login setup wizard is the manual step. See 09-authentik.md.

Node reference

Node	IP	Boot disk	Role
BMC	192.168.5.10	—	Turing Pi 2 control board (not a k8s node — power control, flashing, BMC web UI)
Node1 (RK1)	192.168.5.11	NVMe, 983G (~915G free)	control-plane + worker + Longhorn storage
Node2 (RK1)	192.168.5.12	NVMe, 983G (~916G free)	worker + Longhorn storage
Node3 (RK1)	192.168.5.13	NVMe, 983G (~915G free)	worker + Longhorn storage (or NFS server)
Node4 (RK1)	192.168.5.14	NVMe, 983G (~921G free)	worker + Longhorn storage (reflashed from eMMC — see 04-storage.md)

MetalLB LoadBalancer pool: 192.168.5.20-192.168.5.200. Only .10-.14 are excluded so far — check this range against your router's DHCP lease range before applying; narrow METALLB_POOL in config/cluster.env if it overlaps.

This is not a theoretical warning on UniFi. A UniFi network's default DHCP range is .6-.254, which fully contains the pool above — MetalLB would hand out addresses the gateway also leases. In UniFi Network, go to Settings → Networks → the 192.168.5.0/24 network → DHCP Range and move it clear of .20-.200 (e.g. .201-.254) before running scripts/05-install-metallb.sh.

Prerequisites

On your workstation:

• ssh access to all four nodes (key-based, no password prompts — the scripts don't handle interactive auth)
• kubectl and helm — pinned in mise.toml (mise install); install some other way if you don't use mise
• openssl (for token generation; present by default on macOS/Linux)
• op (1Password CLI) and jq — pinned in mise.toml; used by scripts/12-seed-onepassword-vault.sh and scripts/13-create-onepassword-token.sh
• a 1Password account, to hold the vault and the read-only service account the cluster reads it with (see 10-secrets.md)

On the nodes: Ubuntu, reachable over SSH as the ubuntu user with passwordless sudo (the default for cloud-init's ubuntu user on a stock image). Every privileged remote command in scripts/ runs over a non-interactive ssh host command, which can't answer a sudo password prompt — if your image asks for one, add a NOPASSWD sudoers entry for ubuntu before running anything here.

Quick start

cp config/cluster.env.example config/cluster.env
# edit config/cluster.env if your IPs/pool/SSH user differ from the defaults

./scripts/run-all.sh

run-all.sh walks through, in order:

Script	What it does
00-generate-token.sh	Generates the k3s cluster join token (config/k3s-token)
00b-set-hostnames.sh	Sets every node's hostname to <name>.t.9b11.xyz
00c-install-prerequisites.sh	Installs OS packages k3s expects but doesn't bundle (iptables)
01-install-k3s-master.sh	Installs k3s server on Node1
02-install-k3s-workers.sh	Joins Node2-4 as workers
03-fetch-kubeconfig.sh	Pulls kubeconfig to config/kubeconfig, plus a personal copy at ~/.kube/configs/turingpi.conf
04-label-nodes.sh	Labels every node node-type=worker; gates which nodes Longhorn may use
05-install-metallb.sh	Installs MetalLB + the address pool
06-prepare-storage-nodes.sh	Installs Longhorn's node-level dependencies
07-install-longhorn.sh	Installs Longhorn (default StorageClass, all four nodes)
08-install-argocd.sh	Installs ArgoCD, prints the admin password

Not included in run-all.sh (needs manual input first):

Script	What it does
07b-setup-nfs-server.sh + 07b-install-nfs-provisioner.sh	Alternative to Longhorn — see docs/04-storage.md
09-argocd-ui.sh	Port-forwards the ArgoCD UI to localhost:8080
10-bootstrap-gitops.sh	Applies the ArgoCD "app of apps" once GIT_REPO_URL is set
12-seed-onepassword-vault.sh	Writes the vault items every Secret is rendered from, copying live values — see docs/10-secrets.md
13-create-onepassword-token.sh	Stores the 1Password service account token the operator reads the vault with — see docs/10-secrets.md
17-sync-grafana-admin-password.sh	Makes Grafana's stored admin password match its Secret, so provisioning reloads work — see docs/07-monitoring.md
generate-grafana-dashboards.sh	Regenerates the dashboards in gitops/apps/grafana-dashboards/ from pinned sources — see docs/07-monitoring.md
generate-crd-schemas.py	Regenerates schemas/, kubeconform's schemas for the Longhorn CRDs, from the chart version 07-install-longhorn.sh pins — run after bumping it
restore-test.sh	Restores a volume's latest backup into <pvc>-restore-test, verifies it, then cleans up (KEEP=1 keeps it) — see docs/04-storage.md

After the cluster is up

export KUBECONFIG=config/kubeconfig
kubectl get nodes -o wide

Then hand the sample app to ArgoCD:

git remote add origin <your-remote-url>
git push -u origin main

# add GIT_REPO_URL=<your-remote-url> to config/cluster.env, then:
./scripts/10-bootstrap-gitops.sh
./scripts/09-argocd-ui.sh   # http://localhost:8080, user: admin

The script substitutes GIT_REPO_URL into the root Application only (manifests/argocd/root-app.yaml). The child Applications in gitops/argocd-apps/ that deploy manifests from this repository name https://git.esle.fi/ivuorinen/turingpi-cluster.git directly. In a fork, change their repoURL to your remote before running the script, or set it to __GIT_REPO_URL__: the script then rewrites those files and stops so you can commit and push them, and applies root when re-run. Skip this and root reads your fork while its children keep syncing from the original repository.

Every Application (root and each one it creates from gitops/argocd-apps/) autosyncs once it exists — no manual Sync click needed — and all but argocd-config prune (see 05-argocd.md). The apps that need credentials wait on the 1Password setup in 10-secrets.md; do that before bootstrapping (see that doc for recovering if bootstrap came first). Confirm the sample app once root has had a moment to pick up its children:

kubectl -n redis-server get pvc,pods,svc
redis-cli -h 192.168.5.73 \
  -a "$(kubectl -n redis-server get secret redis-sample-auth \
        -o jsonpath='{.data.password}' | base64 -d)" ping   # PONG

For the monitoring stack, see 07-monitoring.md. For local DNS and the wildcard cert (including the bunny.net API key acme-wildcard-cert needs before it'll do anything, which comes from 1Password), see 08-dns-and-certs.md. For hostname routing and shared authentication (Authentik), including its one manual step (the first-login setup wizard), see 09-authentik.md.

Repository layout

mise.toml               # pinned tool versions
renovate.json           # this repository's Renovate config (docs/11-renovate.md)
.forgejo/workflows/     # CI: pre-commit hooks, bats tests, Helm render validation
config/
  cluster.env.example   # copy to cluster.env (gitignored) and edit
scripts/                # numbered bootstrap scripts, idempotent where possible
  ci/                   #   checkers run by the hooks and CI, not part of bootstrap
  grafana-dashboards/   #   inputs for generate-grafana-dashboards.sh
manifests/
  metallb/              # IPAddressPool + L2Advertisement (templated)
  traefik/              # HelmChartConfig pinning Traefik's IP (templated)
  argocd/               # the root "app of apps" Application (templated)
gitops/
  argocd-apps/          # one Application per app — this is what root points at
  apps/
    crds/               # CRDs ArgoCD cannot apply, installed by hook Jobs
    redis-sample/       # raw manifests for the sample app
    renovate/           # Renovate's CronJob and global config
    gatus/              # Gatus: service checks, status page, ServiceMonitor
    longhorn-backup/    # Longhorn RecurringJobs: trim, backups, system backup
    homepage/           # Homepage: the start page and its read-only RBAC
    node-red/           # Node-RED: flow editor, pinned palette, flows on a PVC
    tailscale-config/   # Tailscale: the subnet router and its ProxyClass
    acme-wildcard-cert/ # raw manifests: RBAC, PVC, CronJob for cert issuance
    ingress-routes/     # TLSStore + per-service IngressRoutes + auth Middleware
    argocd-config/      # ArgoCD's own ConfigMaps, server-side applied
    authentik-blueprints/ # ConfigMap of Authentik Providers/Applications
    cluster-dns/        # coredns-custom override for in-cluster name resolution
    external-secrets-config/ # ClusterSecretStore + ExternalSecrets from 1Password
    grafana-dashboards/ # generated dashboard ConfigMaps
    monitoring-alerts/  # PrometheusRule for this cluster's known failure modes
    monitoring-targets/ # monitors for components not installed by a chart here
tests/                  # bats suites: scripts/lib.sh, scripts/ci/, Renovate annotations, version and docs sync
schemas/                # kubeconform schemas for CRDs the catalog lags on (generate-crd-schemas.py)
docs/                   # per-topic write-ups, one per source guide, plus extras

Everything under gitops/apps/ is the path: of exactly one Application in gitops/argocd-apps/.

Docs

• 01-installation.md — k3s master/worker setup
• 02-helm-arkade.md — Helm (and optionally Arkade)
• 03-networking.md — MetalLB pool, Traefik
• 04-storage.md — Longhorn vs. NFS, and Longhorn's backups
• 05-argocd.md — ArgoCD install, UI access, sync policy
• 06-sample-app-gitops.md — the Redis app, GitOps-style
• 07-monitoring.md — Prometheus, Grafana, Loki, Alloy
• 08-dns-and-certs.md — AdGuard Home, Traefik's pinned IP, wildcard cert via acme.sh
• 09-authentik.md — hostname routing, forward-auth for Longhorn, native OIDC for Grafana/ArgoCD
• 10-secrets.md — 1Password + External Secrets Operator: where every credential lives and how it reaches the cluster
• 11-renovate.md — Renovate: what it updates, when updates merge, the PRs that need a manual step
• 12-gatus-homepage.md — Gatus service checks, game-server monitoring, and the start page
• 13-node-red.md — Node-RED: flows on a volume, the pinned palette, and the Home Assistant connection
• 14-tailscale.md — Tailscale: the subnet router, the ACL policy it depends on, and split DNS

Linting

Tool versions are pinned in mise.toml; hooks are defined in .pre-commit-config.yaml and run via prek (a faster, Rust-based pre-commit-compatible runner — reads the same config format, no Python venv bootstrap required):

mise install
prek run --all-files

What's wired up, and why:

Hook	Checks
shellcheck	Bash correctness (scripts/, .claude/hooks/, and the tests/*.bats suites)
shfmt	Bash formatting, check-only — doesn't rewrite files
yamllint	YAML style/structure
kubeconform	Kubernetes manifests validate against real API schemas — including MetalLB's and ArgoCD's CRDs via the CRDs-catalog, not just core resources
taplo (lint + format check)	mise.toml correctness and style
markdownlint	Markdown structure (line-length and table-alignment rules are relaxed for prose docs — see .markdownlint.yml)
gitleaks	Scans staged changes for hardcoded secrets/credentials before they hit git history
kube-linter	Kubernetes security/best-practice checks — non-root, read-only root filesystem, resource requests/limits, etc. (schema correctness is kubeconform's job, not this one)
pre-commit-hooks (trailing-whitespace, end-of-file-fixer, check-merge-conflict, large files, mixed-line-ending, check-yaml, check-json, shebang checks)	General file hygiene
lychee	Markdown links and #anchors resolve, offline — local targets only, external URLs are not fetched
renovate-config-validator	renovate.json against Renovate's repository-config schema (--strict --no-global)
promtool	PromQL, label templates and for: durations in PrometheusRule manifests, which kubeconform sees only as strings (scripts/ci/check-prometheus-rules.sh)
actionlint	.forgejo/workflows/ — workflow structure and the shell in run: blocks
bats	The tests/ suites, when tests/, scripts/lib.sh or scripts/ci/ change
no-secret-manifests	No kind: Secret under gitops/ or manifests/ — credentials come from 1Password through External Secrets Operator
helm-render (push only)	Renders every Helm-backed Application and validates the result: dangling volumes, kube-linter, kubeconform (scripts/ci/render-applications.sh, also CI's render job)

Install the git hooks with prek install: every hook runs on commit except helm-render, which runs only on push because it downloads every chart. Both are optional — prek run --all-files works standalone, and prek run --all-files --stage pre-push runs helm-render by hand.

Notes on secrets

config/cluster.env, config/k3s-token, and config/kubeconfig are all gitignored. Nothing sensitive is meant to reach this repo's git history — the only things that live in tracked files are cluster topology and non-secret config (IPs, pool ranges, manifests).

License

No LICENSE file, deliberately — this is a personal homelab setup, not a project intended for reuse or redistribution.