Use this runbook before changing a machine or upgrading Sonny 0.12.7. It covers local files and SQLite. A managed PostgreSQL deployment, Lanning, remote services and Desktop identity recovery each have their own coordinated recovery requirements. The privacy guide inventories the stores and explains why a copied database is sensitive.
The executed Linux fixture recovers a real TUI transcript and closed files, preserves committed SQLite WAL data after an abrupt process interruption, and replays restored TUI history to a controlled model. It uses the same binary, workspace and private identity environment with an explicitly selected restored session database. It does not certify relocation of a complete managed profile, Desktop recovery import, live provider access or cross-platform recovery.
1. Define the recovery set
Record the binary version, account, workspace, sonny config path output and
actual launch options. Include database and trace overrides, service manifests,
environment-selected paths and project state. Do not put credential values in
the inventory. Check for older personal profiles whose manifest still names
<profile>/control.db rather than the fresh <data>/control.db default.
Preserve configuration, authoritative memory/Recipes, required sessions,
control state, encryption/identity material and any required traces/exports.
Record source revisions so a later restore does not silently load newer Recipe
or model definitions. Review delegated worktrees and unpublished Git changes;
copying only .git or only its linked worktree is not a complete Git recovery
plan. Derived memory indexes can be regenerated from the preserved YAML.
Runtime encrypted blobs need their matching key generations. The local runtime
key normally lives at <data>/runtime/blob-vault.key; configured
SONNY_RUNTIME_BLOB_KEYRING/SONNY_RUNTIME_BLOB_KEY and shared stores require
the deployment’s key custodian. Do not invent a replacement key and expect it
to decrypt old data. A local profile’s signed manifest, policy, keyring,
namespace, root bundle and unseal material are related authority, not disposable
cache. Protect them with the recovery set and restrict access to it.
2. Stop writers and protect the destination
Close all clients using the selected stores. Stop the relevant supervisor, mesh/service instance, background workers and other writers through the deployment’s normal controls. Confirm those specific instances are stopped; closing one terminal is insufficient. Record interrupted work and any external effects already performed. Stop scheduling new work before the backup.
Use a new private destination on an approved storage device. For the following
Unix example, replace every /absolute/... value with a reviewed path.
RECOVERY_DIR must not exist yet; the source database paths must already exist.
The commands do not stop services or choose the correct stores for you.
umask 077
RECOVERY_DIR=/absolute/private/recovery-set
SESSIONS_DB=/absolute/active/sessions.db
CONTROL_DB=/absolute/active/control.db
mkdir -m 700 "$RECOVERY_DIR"
Keep the restrictive umask for the rest of this procedure. It protects newly created files; review permissions of existing sources and files copied with preserved modes separately. The tested release can create its session database with a mode inherited from the launching umask. On Windows use a destination ACL restricted to the responsible account and your approved recovery operators; the Unix permission checks here are not a Windows ACL verification.
3. Snapshot each SQLite store
SQLite may hold committed data in a -wal file even after a process crashes.
A copy of the main .db alone can omit it. Do not remove -wal/-shm files to
make a database look closed, and do not copy a changing database and its sidecars
at unrelated times.
Use SQLite’s backup API. The downloadable
snapshot helper
uses that API, checks
integrity, creates a new 0600 destination, and refuses an existing destination.
It never restores or replaces the source. Review the downloaded helper before
running it:
curl --fail --location 'https://sonny.sh/examples/recovery/sqlite_snapshot.py' -o "$RECOVERY_DIR/sqlite_snapshot.py"
python3 "$RECOVERY_DIR/sqlite_snapshot.py" "$SESSIONS_DB" "$RECOVERY_DIR/sessions.db"
python3 "$RECOVERY_DIR/sqlite_snapshot.py" "$CONTROL_DB" "$RECOVERY_DIR/control.db"
Snapshot any other required SQLite stores individually. One valid database snapshot does not establish consistency across several databases, YAML files and keyrings. Keep all application writers stopped while collecting the whole recovery set. A helper failure leaves that store unqualified; preserve the original and investigate its error before proceeding.
4. Collect closed files and record the set
Copy the reviewed configuration, identity/key material, authoritative memory, required traces, state, project files and unpublished work using your approved backup tool. Preserve ownership and intended private permissions. Record the version, source and destination mapping, file hashes and completion time in a private manifest. Keep it with the set; filenames and paths can also be sensitive.
For an interrupted process, preserve its available records and working tree. Do not expect a database snapshot to complete the interrupted task or undo a tool’s external effect. Mesh restart has a separate interrupted-session procedure .
Encrypt the backup using the organization’s approved mechanism and keep recovery keys/passphrases separately. Owner-only permissions protect local access; the snapshot helper does not encrypt its output. Copying a sealed-file credential store and its local plaintext fallback master key together does not provide independent protection from someone who can read that backup.
5. Verify in a disposable restore location
Retain the original set unchanged. Restore into new private locations and compare file hashes before starting Sonny. Run SQLite integrity checks on the restored copies and confirm expected session IDs, required files and key generations. Do not overwrite an active database to test whether a backup works.
For a local TUI session-store test, launch the same release with
--session-db pointing at the restored copy and --session naming a full ID
from that copy. Keep the intended workspace, reviewed Recipe/model configuration
and required control/identity state available. Select that session when the TUI
asks. Inspect its original messages before allowing a new task. Exercise a
small approved request against an appropriate test provider if replay itself
needs validation.
The fixture verified original TUI text reaching the synthetic provider after
restore. It also preserved the abruptly interrupted session row, but its
headless execution positive was refused by runtime authority before a model
request. Restoring that row is not a successful continuation. A one-shot
--prompt call with the same ID does not establish restored transcript replay
in this release. An integrity check, a visible session ID and a successful
provider login answer different questions.
For a complete control/profile recovery, also verify signed manifests and absolute paths, identity/tenant, workspace trust, runtime decryption, permissions, isolation, client compatibility and required service connectivity. Use the deployment’s supported recovery procedure for those components. A session-database test alone does not approve starting a relocated managed node.
6. Activate only the verified recovery
Keep writers stopped while the operator installs the verified set at its intended paths. Preserve the previous set for rollback and keep its keys until all required retained data is accounted for. Start the intended instances through normal service controls, check their version/readiness and inspect interrupted work before permitting further changes. If validation fails, stop the new writers and retain both sets for diagnosis.
On a different machine, reauthenticate providers through the credential flow and review MCP approvals, executable hashes and permissions. OS keychain entries, remote access and provider tokens have separate migration/revocation rules. Do not infer portable credentials from a copied session database.
The tagged Desktop has a separate personal identity recovery flow: backup exports a passphrase-encrypted root bundle and verifies it by import; restore requires an empty personal session projection and explicit replacement confirmation, stages the identity, and promotes it after restart. No hosted escrow or account recovery is provided. Keep the bundle and passphrase separately. Those release-source contracts are not an executed Desktop restore test here and do not replace the data backup above.
When recovery does not validate
| Observation | Next action |
|---|---|
Missing recent messages after a plain .db copy | Preserve the source and sidecars; take a SQLite backup snapshot of the original store rather than deleting WAL files |
| Integrity failure or missing store | Stop activation; keep original and failed copies and investigate the exact selected paths |
| History visible but provider/tool unavailable | Review credentials, selected endpoints, executable hashes and approval scope separately |
| Encrypted runtime data cannot be opened | Check the original keyring and retained decrypt-only generations; do not generate replacements over them |
| Signed profile or absolute path mismatch | Use the deployment’s profile recovery controls; do not hand-edit signatures or widen trust to force startup |
| Session interrupted at shutdown | Inspect task records, files and remote effects before explicitly continuing or terminating it |
Deleting a failed test copy does not delete the original backup or external copies. Apply the organization’s retention and deletion policy to both after the recovery decision is complete.