- Go 94.7%
- Shell 4.4%
- Dockerfile 0.7%
- Makefile 0.2%
Bitwarden clients refuse non-HTTPS server URLs, even on localhost, so peeking at the standby through an ssh tunnel failed. A Caddy proxy with a local-CA certificate for localhost now fronts the standby on loopback, leaving the standby itself unpublished and syncing untouched. README gains a "Peeking at the standby" section, covering two-factor: TOTP works, WebAuthn does not on localhost, so use the recovery code. Also corrects the compose comment claiming the standby is reachable only from vaulteros: the docker host itself can reach it too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|---|---|---|
| cmd/vaulteros | ||
| internal | ||
| test/e2e | ||
| .env.example | ||
| .gitignore | ||
| compose.failover.yaml | ||
| compose.peek.yaml | ||
| compose.socket-proxy.yaml | ||
| compose.yaml | ||
| Dockerfile | ||
| go.mod | ||
| go.sum | ||
| Makefile | ||
| README.md | ||
| vaulteros.example.yaml | ||
vaulteros
Keep a verified standby copy of a remote Vaultwarden
in your own infrastructure, with docker compose.
Vaultwarden has no replication. The usual advice is either "set up database replication yourself" or "cron a tarball somewhere" — fine for an SRE, useless if you just want a second copy of your password manager you can switch DNS to. vaulteros is that second copy: it pulls consistent snapshots over SSH, checks them, keeps a deduplicated history, and serves the newest good one from a local warm Vaultwarden.
It never needs your master password, and it cannot read your secrets: vault data is end-to-end encrypted, and vaulteros only ever moves ciphertext.
Setup
Two things are needed: SSH access from the machine running vaulteros to the upstream host, and a pinned Vaultwarden version.
git clone https://git.gizmolab.net/gizmo/vaulteros && cd vaulteros
cp .env.example .env && $EDITOR .env
# 1. make the key and print the line to paste upstream
docker compose run --rm vaulteros keygen
# 2. paste that line into authorized_keys on the upstream host, then:
docker compose run --rm vaulteros doctor
# 3. go
docker compose up -d
doctor is the whole onboarding story. It probes the local tools, the SSH path,
which snapshot modes work, the upstream data directory, free disk and the
version ordering, and every failure names the fix.
How a sync works
- Consistent dump, made upstream first. Vaultwarden has shipped a
backupsubcommand since 1.32.1 which runsVACUUM INTO— a proper point-in-time copy that folds in pending WAL content, with nosqlite3binary needed on the upstream host. vaulteros runs it throughdocker exec. - Files pulled next.
rsyncbrings attachments, sends, the JWT signing keys andconfig.json, hardlinking anything unchanged against the previous snapshot with--link-dest, so a long history costs almost nothing. - Dump transferred last. This ordering matters: an attachment written during the run ends up as an extra file with no database row, which is harmless, rather than a row pointing at a file that does not exist.
- Verified.
integrity_check,foreign_key_check, a check that it really is a Vaultwarden database, attachment rows reconciled against files in both directions, and the delta gate (below). - Committed. The snapshot appears atomically, with a
manifest.jsonrecording where it came from, what is in it and how it was judged. Snapshots are laid out exactly like a Vaultwarden data directory, so restoring is a copy rather than a translation. - Cleaned up. Upstream prunes nothing of its own, so vaulteros deletes the dump it made — and only that one.
- Boot-probed. A throwaway container on the standby's Vaultwarden image
is given a copy of the database and signing keys and must serve
/alive; with a canary account configured, it must also accept a real login. The probe gets the files through the Docker archive API rather than a bind mount, because the daemon resolves bind paths on the host, which vaulteros usually cannot see.config.jsonis withheld: it overrides the environment, and an upstream SMTP setting would let the probe send real mail. - Served. The standby is stopped, its data replaced with a copy of the
snapshot (never a hardlink — Vaultwarden writes to and migrates what it
opens), stale
-wal/-shmremoved, restarted, and waited on until it answers. - Pruned, recorded, notified. Retention runs after every commit, the run is
appended to
state/runs.jsonl, and notifications go out.
The delta gate
A mirror faithfully copies a disaster. If the upstream is wiped or encrypted,
the next sync would replace your good copy with the damage. So every snapshot's
user and cipher counts are compared with the newest verified snapshot, and a
sharp drop — more than max_shrink_pct (10%) and at least min_shrink_rows
(5) rows, or any drop to zero — marks the snapshot suspect. It is kept for
inspection, never served, and the run exits non-zero and alerts loudly. Many
attachment rows with missing files trip it too, since row counts cannot see a
damaged attachment store.
The comparison is always against the last verified snapshot, never merely the last one, so a second bad snapshot cannot pass by resembling the first. If the deletion was deliberate, say so:
vaulteros accept 20260928T094824Z --reason "cleared out the old team vault"
The override is recorded in the manifest and that snapshot becomes the new baseline.
Retention
Grandfather-father-son (hourly, daily, weekly, monthly), plus last N
of any status. Buckets count only verified snapshots, and count the N most
recent periods that have one rather than the last N calendar periods — so
weeks of suspect snapshots after an upstream wipe cannot age your pre-wipe
copies out. The newest verified snapshot is never pruned.
Snapshot modes
| Mode | What it needs upstream | What the SSH key can do |
|---|---|---|
admin-api |
ADMIN_TOKEN set, rrsync installed, one root cron line |
Read the data directory. Nothing else. Recommended. |
exec |
the SSH user in the docker group |
Anything root can: docker access is root-equivalent. |
sqlite3 |
sqlite3 on the host |
Planned, not implemented yet. |
auto tries exec first when the key has a shell (it can then also delete its
own dumps, so no upstream cron is needed), then admin-api. With an rrsync key
only admin-api is possible, and auto goes straight to it. doctor shows
what it chose and why. The security difference comes from the key, not the
mode: choose the rrsync setup.
Why admin-api is recommended. With exec, whoever compromises the backup
box holds a root-equivalent key to your password server. With admin-api,
Vaultwarden makes the consistent dump itself (POST /admin/config/backup_db)
and the key is locked by restrict,command="rrsync -ro <data dir>": it can list
and read the data directory, and every other command — a shell, a write, a path
outside the directory — is refused. vaulteros then never runs a command
upstream, only rsync; the version guard reads the server version from the admin
diagnostics page instead of the binary.
The cost is one line in root's crontab upstream. A read-only key cannot delete the dumps Vaultwarden writes, so something upstream has to:
*/30 * * * * find /opt/vaultwarden/data -maxdepth 1 -name 'db_*.sqlite3' -mmin +60 -delete
vaulteros stays quiet while that keeps up, and fails runs with this exact line
in the error once dumps start piling up. The admin token must be the plaintext
one, even if upstream stores it as an Argon2 hash; it goes in
./secrets/admin_token.
Vaultwarden allows 3 admin logins per 5 minutes by default. vaulteros logs in once per run and shares that session between the version check and the dump, so scheduled syncs never come close. Several manual syncs in quick succession can hit the limit; the error says so and says to wait.
vaulteros supports upstreams on SQLite, Vaultwarden's default database. MySQL/MariaDB and Postgres upstreams are not supported (see Roadmap).
Things that will bite you if nobody says them
These are not stylistic preferences; each one is a way a standby silently fails at the moment you need it.
- The standby's Vaultwarden must be the same version as upstream, or newer.
Vaultwarden runs migrations on whatever database it opens. An older build
cannot read a newer schema. vaulteros refuses to sync rather than let you find
this out during a failover. Note that
/api/configdoes not report the server version — itsversionfield is the web vault version, something like2025.6.0— so vaulteros reads the real version fromvaultwarden --versioninside the container. With a read-only key and no container access the version is unknown, and vaulteros warns that the guard could not run instead of comparing the wrong number. - The standby's
DOMAINmust match upstream exactly. WebAuthn, FIDO2 and passkey credentials are bound to the origin, so a standby on a different hostname breaks two-factor login. Fail over by pointing the same name at the standby, not by using a second one. - Retained snapshots are never booted. Because migrations mutate, booting a snapshot in place would quietly make it unrestorable to the upstream that produced it. The standby always runs on a copy.
- The standby runs as uid 10001 on port 8080, the same user as vaulteros, so each can manage the files the other creates in their shared volume. With Vaultwarden's default root user, vaulteros could not replace the files it leaves behind.
- The JWT signing keys are part of the backup.
rsa_key.*is synced; without it, every logged-in client is signed out on failover. - Sync is one-way. Anything written to the standby is destroyed by the next sync, which is why it is published on no ports by default.
- A mirror is not a backup. If you only keep the current state, a wiped or corrupted upstream propagates to your only copy on the next run. Hence the retained history, and the delta gate that refuses to materialize a snapshot whose user or cipher counts fell off a cliff.
Peeking at the standby
To check what the standby holds — say, to confirm last night's changes made it across — you can log in to it without promoting it. Syncing carries on underneath, so treat it as read-only.
On the backup host:
docker compose -f compose.yaml -f compose.peek.yaml up -d standby-peek
From your own machine:
ssh -L 8443:127.0.0.1:8443 you@backup-host
Then open https://localhost:8443 and log in as usual.
It has to be HTTPS: Bitwarden clients refuse any server URL that isn't, even
localhost ("Insecure URL not allowed"). The peek puts a small Caddy in front
of the standby with a certificate from its own local CA, issued for
localhost. The standby itself stays unpublished, and the proxy listens on
loopback only, so the tunnel is the way in. The browser will warn about the
certificate the first time. Accept it, or trust the CA once:
docker compose -f compose.yaml -f compose.peek.yaml exec standby-peek \
cat /data/caddy/pki/authorities/local/root.crt > vaulteros-peek-ca.crt
# then import vaulteros-peek-ca.crt into your browser or OS trust store
Two-factor login. Authenticator-app (TOTP) codes work as normal: the
standby has the same secret as upstream. Security keys and passkeys
(WebAuthn/FIDO2) do not. They are bound to your real hostname, and you are
on localhost. If a key is your only second factor, log in with your
recovery code instead, which you can view on the upstream at
https://vault.example.com/#/settings/security/two-factor ("View recovery
code"). Do that before you need it, and keep it somewhere that isn't the vault.
Using a recovery code turns two-step login off for that account. On the standby this only touches the copy: the next sync puts two-factor back, and upstream never sees it.
Also good to know:
- Changes don't stick. Anything you edit on the standby is wiped by the next sync.
- Sessions end at each sync. The standby restarts every time a new snapshot lands (every 6 hours by default).
- It may send email. The standby carries upstream's
config.json, SMTP settings included, so a login there can trigger a "new device" email. That's expected.
Stop peeking with:
docker compose -f compose.yaml -f compose.peek.yaml rm -sf standby-peek
Failover
docker compose run --rm vaulteros promote --reason "upstream host is dead"
docker compose -f compose.yaml -f compose.failover.yaml up -d
# point the upstream's hostname — the same one — at this host
promote makes the standby serve the newest verified snapshot (or the one you
name) and pauses syncing: from that moment the standby holds writes that
exist nowhere else, and a sync would destroy them. /healthz reports the
promotion, since nothing is backing up the new primary.
To fail back, move the standby's data to the rebuilt upstream, then
vaulteros demote. For a restore from any snapshot:
docker compose run --rm -T vaulteros restore-to-upstream latest --out - > restore.tar.gz
The archive's data/ is a complete Vaultwarden data directory. It comes with
RESTORE.txt, which spells out the version and WAL rules below, and the export
refuses a snapshot whose database no longer matches its recorded checksum.
Monitoring
Notifications are split by audience. A healthchecks.io-style URL is pinged on
every run (/start, then success or /fail) so silence means something. ntfy is
for humans and only hears about problems and recoveries — a suspect snapshot is
sent at urgent priority. The webhook gets every run as JSON.
/healthzreturns 503 once no verified snapshot is within the staleness window. Point an uptime monitor at it and "backups quietly stopped working" becomes a page./metricsexposes snapshot counts by verdict, row counts, sizes, the age of the newest good snapshot, and the last run's outcome./api/runsand/api/snapshotsserve the run history and inventory as JSON./is a self-contained status page — no CDN, no external fonts, nothing that reaches out from a page that may be the only thing still running.
Security
Snapshots are sensitive even though your secrets stay encrypted. They contain the
encrypted vault, master-password hashes, the JWT signing key, and config.json,
which holds the admin token and SMTP credentials in plaintext. The data volume
is 0700 and owned by an unprivileged user; encrypt the underlying dataset if
the host is not trusted.
vaulteros needs the Docker API to restart the standby and run probes, and runs
unprivileged with the socket's group (DOCKER_GID). The socket is
root-equivalent. compose.socket-proxy.yaml puts a proxy in front of it that
blocks volumes, networks, swarm, system and exec sessions — but vaulteros must
still create containers, which is powerful by nature, so treat this as reducing
surface rather than removing trust.
The SSH key should belong to a dedicated upstream account. keygen prints both
setups — exec (root-equivalent) and admin-api (read-only) — with the exact
commands, so the trade-off is a decision rather than an accident. Prefer
admin-api.
Commands
vaulteros keygen generate the SSH key and print the line to paste upstream
vaulteros doctor check everything needed for a sync; changes nothing
vaulteros sync capture, verify and commit one snapshot (--dry-run to plan)
vaulteros list list retained snapshots
vaulteros verify re-run every gate over a retained snapshot; changes nothing
vaulteros accept mark a suspect snapshot good after a deliberate deletion
vaulteros prune apply the retention policy (--dry-run to preview)
vaulteros promote make the standby the primary and pause syncing
vaulteros demote return the standby to being a replica
vaulteros restore-to-upstream export a snapshot as a data directory + runbook
vaulteros serve run the scheduler and the status server (container default)
Status
Working today, for SQLite upstreams: snapshots over admin-api (read-only key)
or exec; the full verification suite with the delta gate, attachment
reconciliation, boot probes and canary logins; GFS retention; run history;
notifications; the warm standby with promotion; and restore export. All of it
is exercised end to end against real Vaultwarden containers (make e2e),
including a real upstream wipe, a restore into a blank instance, and a sync
through a genuinely unprivileged rrsync -ro key.
Roadmap
Next
sqlite3snapshot mode, using the host'ssqlite3 .backup, as a fallback.- A "sync now" button on the status page.
Future goals
- Postgres upstreams. Vaultwarden's built-in backup is SQLite-only, so this
needs its own path end to end:
pg_dump -Fcrun inside the upstream Postgres container (so dump and server versions always match), a Postgres for the standby at least as new as upstream's,pg_restoreinto a scratch database that is verified and then renamed into place, a boot probe with a database beside it, and a Postgres variant of the restore export and runbook. The configuration format already reservesengine: postgres; setting it today is rejected at startup with a message saying it is not supported yet. - MySQL/MariaDB upstreams, along the same lines.
Development
make check # go vet, tests, gofmt
make build # ./bin/vaulteros
make image # docker image
make e2e stands up a real Vaultwarden, seeds it through the API, reaches it
over a real ssh hop and checks transfer, exclusion, hardlinking, cleanup, the
delta gate against a real wipe, accept, history and retention.
The upstream host is behind a remote.Runner interface, so the whole snapshot
pipeline is tested against a fake with no host in sight
(internal/snapshot/sync_test.go), including the ordering guarantee, the
hardlinking, the exclusion filters and the cleanup safety rails.