Backup and keep a local copy of a vaultwarden instance
  • Go 94.7%
  • Shell 4.4%
  • Dockerfile 0.7%
  • Makefile 0.2%
Find a file
grm 032aaba8cb Add compose.peek.yaml to look at the standby over HTTPS
Bitwarden clients refuse non-HTTPS server URLs, even on localhost, so
peeking at the standby through an ssh tunnel failed. A Caddy proxy with a
local-CA certificate for localhost now fronts the standby on loopback,
leaving the standby itself unpublished and syncing untouched.

README gains a "Peeking at the standby" section, covering two-factor:
TOTP works, WebAuthn does not on localhost, so use the recovery code.
Also corrects the compose comment claiming the standby is reachable only
from vaulteros: the docker host itself can reach it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 15:06:44 +03:00
cmd/vaulteros vaulteros: verified standby replica for Vaultwarden 2026-09-28 13:59:39 +03:00
internal vaulteros: verified standby replica for Vaultwarden 2026-09-28 13:59:39 +03:00
test/e2e vaulteros: verified standby replica for Vaultwarden 2026-09-28 13:59:39 +03:00
.env.example Add compose.peek.yaml to look at the standby over HTTPS 2026-09-28 15:06:44 +03:00
.gitignore vaulteros: verified standby replica for Vaultwarden 2026-09-28 13:59:39 +03:00
compose.failover.yaml vaulteros: verified standby replica for Vaultwarden 2026-09-28 13:59:39 +03:00
compose.peek.yaml Add compose.peek.yaml to look at the standby over HTTPS 2026-09-28 15:06:44 +03:00
compose.socket-proxy.yaml vaulteros: verified standby replica for Vaultwarden 2026-09-28 13:59:39 +03:00
compose.yaml Add compose.peek.yaml to look at the standby over HTTPS 2026-09-28 15:06:44 +03:00
Dockerfile vaulteros: verified standby replica for Vaultwarden 2026-09-28 13:59:39 +03:00
go.mod vaulteros: verified standby replica for Vaultwarden 2026-09-28 13:59:39 +03:00
go.sum vaulteros: verified standby replica for Vaultwarden 2026-09-28 13:59:39 +03:00
Makefile vaulteros: verified standby replica for Vaultwarden 2026-09-28 13:59:39 +03:00
README.md Add compose.peek.yaml to look at the standby over HTTPS 2026-09-28 15:06:44 +03:00
vaulteros.example.yaml vaulteros: verified standby replica for Vaultwarden 2026-09-28 13:59:39 +03:00

vaulteros

Keep a verified standby copy of a remote Vaultwarden in your own infrastructure, with docker compose.

Vaultwarden has no replication. The usual advice is either "set up database replication yourself" or "cron a tarball somewhere" — fine for an SRE, useless if you just want a second copy of your password manager you can switch DNS to. vaulteros is that second copy: it pulls consistent snapshots over SSH, checks them, keeps a deduplicated history, and serves the newest good one from a local warm Vaultwarden.

It never needs your master password, and it cannot read your secrets: vault data is end-to-end encrypted, and vaulteros only ever moves ciphertext.

Setup

Two things are needed: SSH access from the machine running vaulteros to the upstream host, and a pinned Vaultwarden version.

git clone https://git.gizmolab.net/gizmo/vaulteros && cd vaulteros
cp .env.example .env && $EDITOR .env

# 1. make the key and print the line to paste upstream
docker compose run --rm vaulteros keygen

# 2. paste that line into authorized_keys on the upstream host, then:
docker compose run --rm vaulteros doctor

# 3. go
docker compose up -d

doctor is the whole onboarding story. It probes the local tools, the SSH path, which snapshot modes work, the upstream data directory, free disk and the version ordering, and every failure names the fix.

How a sync works

  1. Consistent dump, made upstream first. Vaultwarden has shipped a backup subcommand since 1.32.1 which runs VACUUM INTO — a proper point-in-time copy that folds in pending WAL content, with no sqlite3 binary needed on the upstream host. vaulteros runs it through docker exec.
  2. Files pulled next. rsync brings attachments, sends, the JWT signing keys and config.json, hardlinking anything unchanged against the previous snapshot with --link-dest, so a long history costs almost nothing.
  3. Dump transferred last. This ordering matters: an attachment written during the run ends up as an extra file with no database row, which is harmless, rather than a row pointing at a file that does not exist.
  4. Verified. integrity_check, foreign_key_check, a check that it really is a Vaultwarden database, attachment rows reconciled against files in both directions, and the delta gate (below).
  5. Committed. The snapshot appears atomically, with a manifest.json recording where it came from, what is in it and how it was judged. Snapshots are laid out exactly like a Vaultwarden data directory, so restoring is a copy rather than a translation.
  6. Cleaned up. Upstream prunes nothing of its own, so vaulteros deletes the dump it made — and only that one.
  7. Boot-probed. A throwaway container on the standby's Vaultwarden image is given a copy of the database and signing keys and must serve /alive; with a canary account configured, it must also accept a real login. The probe gets the files through the Docker archive API rather than a bind mount, because the daemon resolves bind paths on the host, which vaulteros usually cannot see. config.json is withheld: it overrides the environment, and an upstream SMTP setting would let the probe send real mail.
  8. Served. The standby is stopped, its data replaced with a copy of the snapshot (never a hardlink — Vaultwarden writes to and migrates what it opens), stale -wal/-shm removed, restarted, and waited on until it answers.
  9. Pruned, recorded, notified. Retention runs after every commit, the run is appended to state/runs.jsonl, and notifications go out.

The delta gate

A mirror faithfully copies a disaster. If the upstream is wiped or encrypted, the next sync would replace your good copy with the damage. So every snapshot's user and cipher counts are compared with the newest verified snapshot, and a sharp drop — more than max_shrink_pct (10%) and at least min_shrink_rows (5) rows, or any drop to zero — marks the snapshot suspect. It is kept for inspection, never served, and the run exits non-zero and alerts loudly. Many attachment rows with missing files trip it too, since row counts cannot see a damaged attachment store.

The comparison is always against the last verified snapshot, never merely the last one, so a second bad snapshot cannot pass by resembling the first. If the deletion was deliberate, say so:

vaulteros accept 20260928T094824Z --reason "cleared out the old team vault"

The override is recorded in the manifest and that snapshot becomes the new baseline.

Retention

Grandfather-father-son (hourly, daily, weekly, monthly), plus last N of any status. Buckets count only verified snapshots, and count the N most recent periods that have one rather than the last N calendar periods — so weeks of suspect snapshots after an upstream wipe cannot age your pre-wipe copies out. The newest verified snapshot is never pruned.

Snapshot modes

Mode What it needs upstream What the SSH key can do
admin-api ADMIN_TOKEN set, rrsync installed, one root cron line Read the data directory. Nothing else. Recommended.
exec the SSH user in the docker group Anything root can: docker access is root-equivalent.
sqlite3 sqlite3 on the host Planned, not implemented yet.

auto tries exec first when the key has a shell (it can then also delete its own dumps, so no upstream cron is needed), then admin-api. With an rrsync key only admin-api is possible, and auto goes straight to it. doctor shows what it chose and why. The security difference comes from the key, not the mode: choose the rrsync setup.

Why admin-api is recommended. With exec, whoever compromises the backup box holds a root-equivalent key to your password server. With admin-api, Vaultwarden makes the consistent dump itself (POST /admin/config/backup_db) and the key is locked by restrict,command="rrsync -ro <data dir>": it can list and read the data directory, and every other command — a shell, a write, a path outside the directory — is refused. vaulteros then never runs a command upstream, only rsync; the version guard reads the server version from the admin diagnostics page instead of the binary.

The cost is one line in root's crontab upstream. A read-only key cannot delete the dumps Vaultwarden writes, so something upstream has to:

*/30 * * * * find /opt/vaultwarden/data -maxdepth 1 -name 'db_*.sqlite3' -mmin +60 -delete

vaulteros stays quiet while that keeps up, and fails runs with this exact line in the error once dumps start piling up. The admin token must be the plaintext one, even if upstream stores it as an Argon2 hash; it goes in ./secrets/admin_token.

Vaultwarden allows 3 admin logins per 5 minutes by default. vaulteros logs in once per run and shares that session between the version check and the dump, so scheduled syncs never come close. Several manual syncs in quick succession can hit the limit; the error says so and says to wait.

vaulteros supports upstreams on SQLite, Vaultwarden's default database. MySQL/MariaDB and Postgres upstreams are not supported (see Roadmap).

Things that will bite you if nobody says them

These are not stylistic preferences; each one is a way a standby silently fails at the moment you need it.

  • The standby's Vaultwarden must be the same version as upstream, or newer. Vaultwarden runs migrations on whatever database it opens. An older build cannot read a newer schema. vaulteros refuses to sync rather than let you find this out during a failover. Note that /api/config does not report the server version — its version field is the web vault version, something like 2025.6.0 — so vaulteros reads the real version from vaultwarden --version inside the container. With a read-only key and no container access the version is unknown, and vaulteros warns that the guard could not run instead of comparing the wrong number.
  • The standby's DOMAIN must match upstream exactly. WebAuthn, FIDO2 and passkey credentials are bound to the origin, so a standby on a different hostname breaks two-factor login. Fail over by pointing the same name at the standby, not by using a second one.
  • Retained snapshots are never booted. Because migrations mutate, booting a snapshot in place would quietly make it unrestorable to the upstream that produced it. The standby always runs on a copy.
  • The standby runs as uid 10001 on port 8080, the same user as vaulteros, so each can manage the files the other creates in their shared volume. With Vaultwarden's default root user, vaulteros could not replace the files it leaves behind.
  • The JWT signing keys are part of the backup. rsa_key.* is synced; without it, every logged-in client is signed out on failover.
  • Sync is one-way. Anything written to the standby is destroyed by the next sync, which is why it is published on no ports by default.
  • A mirror is not a backup. If you only keep the current state, a wiped or corrupted upstream propagates to your only copy on the next run. Hence the retained history, and the delta gate that refuses to materialize a snapshot whose user or cipher counts fell off a cliff.

Peeking at the standby

To check what the standby holds — say, to confirm last night's changes made it across — you can log in to it without promoting it. Syncing carries on underneath, so treat it as read-only.

On the backup host:

docker compose -f compose.yaml -f compose.peek.yaml up -d standby-peek

From your own machine:

ssh -L 8443:127.0.0.1:8443 you@backup-host

Then open https://localhost:8443 and log in as usual.

It has to be HTTPS: Bitwarden clients refuse any server URL that isn't, even localhost ("Insecure URL not allowed"). The peek puts a small Caddy in front of the standby with a certificate from its own local CA, issued for localhost. The standby itself stays unpublished, and the proxy listens on loopback only, so the tunnel is the way in. The browser will warn about the certificate the first time. Accept it, or trust the CA once:

docker compose -f compose.yaml -f compose.peek.yaml exec standby-peek \
  cat /data/caddy/pki/authorities/local/root.crt > vaulteros-peek-ca.crt
# then import vaulteros-peek-ca.crt into your browser or OS trust store

Two-factor login. Authenticator-app (TOTP) codes work as normal: the standby has the same secret as upstream. Security keys and passkeys (WebAuthn/FIDO2) do not. They are bound to your real hostname, and you are on localhost. If a key is your only second factor, log in with your recovery code instead, which you can view on the upstream at https://vault.example.com/#/settings/security/two-factor ("View recovery code"). Do that before you need it, and keep it somewhere that isn't the vault.

Using a recovery code turns two-step login off for that account. On the standby this only touches the copy: the next sync puts two-factor back, and upstream never sees it.

Also good to know:

  • Changes don't stick. Anything you edit on the standby is wiped by the next sync.
  • Sessions end at each sync. The standby restarts every time a new snapshot lands (every 6 hours by default).
  • It may send email. The standby carries upstream's config.json, SMTP settings included, so a login there can trigger a "new device" email. That's expected.

Stop peeking with:

docker compose -f compose.yaml -f compose.peek.yaml rm -sf standby-peek

Failover

docker compose run --rm vaulteros promote --reason "upstream host is dead"
docker compose -f compose.yaml -f compose.failover.yaml up -d
# point the upstream's hostname — the same one — at this host

promote makes the standby serve the newest verified snapshot (or the one you name) and pauses syncing: from that moment the standby holds writes that exist nowhere else, and a sync would destroy them. /healthz reports the promotion, since nothing is backing up the new primary.

To fail back, move the standby's data to the rebuilt upstream, then vaulteros demote. For a restore from any snapshot:

docker compose run --rm -T vaulteros restore-to-upstream latest --out - > restore.tar.gz

The archive's data/ is a complete Vaultwarden data directory. It comes with RESTORE.txt, which spells out the version and WAL rules below, and the export refuses a snapshot whose database no longer matches its recorded checksum.

Monitoring

Notifications are split by audience. A healthchecks.io-style URL is pinged on every run (/start, then success or /fail) so silence means something. ntfy is for humans and only hears about problems and recoveries — a suspect snapshot is sent at urgent priority. The webhook gets every run as JSON.

  • /healthz returns 503 once no verified snapshot is within the staleness window. Point an uptime monitor at it and "backups quietly stopped working" becomes a page.
  • /metrics exposes snapshot counts by verdict, row counts, sizes, the age of the newest good snapshot, and the last run's outcome.
  • /api/runs and /api/snapshots serve the run history and inventory as JSON.
  • / is a self-contained status page — no CDN, no external fonts, nothing that reaches out from a page that may be the only thing still running.

Security

Snapshots are sensitive even though your secrets stay encrypted. They contain the encrypted vault, master-password hashes, the JWT signing key, and config.json, which holds the admin token and SMTP credentials in plaintext. The data volume is 0700 and owned by an unprivileged user; encrypt the underlying dataset if the host is not trusted.

vaulteros needs the Docker API to restart the standby and run probes, and runs unprivileged with the socket's group (DOCKER_GID). The socket is root-equivalent. compose.socket-proxy.yaml puts a proxy in front of it that blocks volumes, networks, swarm, system and exec sessions — but vaulteros must still create containers, which is powerful by nature, so treat this as reducing surface rather than removing trust.

The SSH key should belong to a dedicated upstream account. keygen prints both setups — exec (root-equivalent) and admin-api (read-only) — with the exact commands, so the trade-off is a decision rather than an accident. Prefer admin-api.

Commands

vaulteros keygen     generate the SSH key and print the line to paste upstream
vaulteros doctor     check everything needed for a sync; changes nothing
vaulteros sync       capture, verify and commit one snapshot (--dry-run to plan)
vaulteros list       list retained snapshots
vaulteros verify     re-run every gate over a retained snapshot; changes nothing
vaulteros accept     mark a suspect snapshot good after a deliberate deletion
vaulteros prune      apply the retention policy (--dry-run to preview)
vaulteros promote    make the standby the primary and pause syncing
vaulteros demote     return the standby to being a replica
vaulteros restore-to-upstream   export a snapshot as a data directory + runbook
vaulteros serve      run the scheduler and the status server (container default)

Status

Working today, for SQLite upstreams: snapshots over admin-api (read-only key) or exec; the full verification suite with the delta gate, attachment reconciliation, boot probes and canary logins; GFS retention; run history; notifications; the warm standby with promotion; and restore export. All of it is exercised end to end against real Vaultwarden containers (make e2e), including a real upstream wipe, a restore into a blank instance, and a sync through a genuinely unprivileged rrsync -ro key.

Roadmap

Next

  • sqlite3 snapshot mode, using the host's sqlite3 .backup, as a fallback.
  • A "sync now" button on the status page.

Future goals

  • Postgres upstreams. Vaultwarden's built-in backup is SQLite-only, so this needs its own path end to end: pg_dump -Fc run inside the upstream Postgres container (so dump and server versions always match), a Postgres for the standby at least as new as upstream's, pg_restore into a scratch database that is verified and then renamed into place, a boot probe with a database beside it, and a Postgres variant of the restore export and runbook. The configuration format already reserves engine: postgres; setting it today is rejected at startup with a message saying it is not supported yet.
  • MySQL/MariaDB upstreams, along the same lines.

Development

make check    # go vet, tests, gofmt
make build    # ./bin/vaulteros
make image    # docker image

make e2e stands up a real Vaultwarden, seeds it through the API, reaches it over a real ssh hop and checks transfer, exclusion, hardlinking, cleanup, the delta gate against a real wipe, accept, history and retention.

The upstream host is behind a remote.Runner interface, so the whole snapshot pipeline is tested against a fake with no host in sight (internal/snapshot/sync_test.go), including the ordering guarantee, the hardlinking, the exclusion filters and the cleanup safety rails.