GitOps and Disaster Recovery on a Single-Node k3s

One laptop runs this site, with no second node to fail over to. How Argo CD, Sealed Secrets and WAL archiving to R2 reduce a full rebuild to two keys and a backup bucket, and the restore drill that proved it works.

One laptop, no second chance

This site's API, database, models and monitoring all run on a single laptop running k3s. There is no second node to fail over to. If the disk dies, the plan is to rebuild from scratch. The question I designed around was: what do I need, besides Git, to get everything back?

The answer today is two keys and one backup bucket.

Git is the only way in

Argo CD watches the repository that describes the cluster. I push to main, and within three minutes it applies the change. If someone edits the cluster by hand, it puts it back.

The structure is app-of-apps: a root Application creates one Application per component, in sync waves. Storage, Traefik and cert-manager come first (wave −3), then the database operator, monitoring, the database itself, and finally the backend (wave 2), so nothing starts before what it depends on.

A few decisions are about not losing data by accident:

  • Applications have no finalizer. Deleting one leaves what it deployed running. To remove something, I remove its manifest.
  • The namespace, the database cluster and its backup store are marked `Prune=false`. Even if their manifests vanish from Git, Argo CD won't delete them.
  • `root` is created once by hand, outside every Application. At first it lived in the folder that the argocd Application manages. root waited for argocd to be healthy, and argocd's health included root, which was still progressing. Each waited for the other, forever.

Deploying is a commit

When a pull request merges into the backend, CI runs lint and tests, builds an image tagged with the commit, and then commits the new tag to the cluster repository. Argo CD does the rest. On the first run I hit the health endpoint every second during the rollout: 241 requests, all 200.

The trade-off is that whoever can push to the backend's main controls what the cluster runs. The key that writes to the cluster repository is only available to workflows on main, which is acceptable for a personal project.

Secrets live in Git, encrypted

Five secrets used to exist only inside the cluster, which made them the hardest part of a rebuild. Now they are SealedSecrets: encrypted with the controller's public key, committed next to what uses them, and decrypted only inside the cluster.

That moves the risk to one place: the controller's private key. Without it, every secret in Git is useless. So I turned key rotation off (by default it adds a new key every 30 days, which means a new backup every 30 days) and keep one key, backed up outside the cluster.

The database, and proving the backup works

Postgres runs under CloudNativePG with one instance. A replica on the same disk wouldn't survive the disk, so durability comes from elsewhere: every WAL segment is archived to Cloudflare R2, plus a full backup every day, kept for 30 days. That allows restoring to any point in that window.

A backup I haven't restored isn't a backup, so I tested it before moving production onto it:

  • After a full backup, I wrote a marker table into the live database and forced its WAL to be archived.
  • I restored a temporary cluster from R2.
  • All seven tables matched row counts and content hashes, including the marker table. The marker could only have come back from the archived WAL, so that one check proved both the full backup and WAL archiving.

Moving off the managed database (Neon) also cut the median API response from 428 ms to 182 ms, since queries no longer cross the internet. It surfaced one bug: the project list had no ORDER BY, so it had always returned rows in their order on disk, which the migration changed. The list now sorts explicitly, and a test inserts rows out of order to catch it.

Rebuilding from nothing

kubectl apply -f sealed-secrets-key.yaml   # 1. the Sealed Secrets private key
kubectl -n argocd create secret ...         # 2. the read-only key for the cluster repo
kubectl apply -k infra/argocd               # 3. Argo CD
kubectl apply -f infra/argocd/root.yaml     # 4. everything else, in order

Then the database is restored from R2. Outside Git there are only a few host settings, like the NVIDIA driver and kernel limits, each written down.

Knowing when it breaks

None of this matters if I don't notice a failure. Prometheus alerts by email when backups fail or are older than 26 hours, when WAL archiving stops, when the API is down or slow, or when the laptop runs on battery. Because the cluster can't report its own death, two outside services cover that: UptimeRobot checks the API every five minutes, and Healthchecks.io emails me if Alertmanager's heartbeat stops. Either way, I find out within about ten minutes.