> For the complete documentation index, see [llms.txt](https://docs.bitsafe.finance/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bitsafe.finance/decentralization-manager/operations-and-reference/backup-recovery-and-upgrades.md). # Backup, recovery, and upgrades ## Purpose Use this runbook to back up and restore one Decentralization Manager node. Use it also to upgrade every node in a peer mesh. This runbook covers the data directory of the node and its configuration. It does not recover a lost Canton signing key, a Decentralized Party key, or an identity-provider tenant. Use the Canton, membership, or identity recovery procedure for those failures. ## Before you begin Confirm that you have: * Administrative access to the node's runtime environment and its persistent volume. * A backup destination with encryption, access control, retention, and deletion policies. * The tag and the digest of the deployed image. * The `DECPM_DB_ENCRYPTION_KEY` value from your secret store, if the node uses one. * A record of the Noise public key of this node, kept outside the node. * A list of peers, Decentralized Parties, expected packages, and active workflow runs. * A maintenance window that every peer operator agreed to, when the procedure affects the mesh. {% hint style="info" %} Treat every backup as sensitive. The SQLite database holds the party credentials. The node encrypts the stored client IDs, client secrets, usernames, and passwords only when `DECPM_DB_ENCRYPTION_KEY` is set. A restore of an encrypted database needs the same key. {% endhint %} ## What to back up The node writes its state to the `data/` directory under its root directory. The `--dir` (`-d`) flag or `DECPM_DIR` sets the root directory. | Start command or image | Root directory | Data directory | | ---------------------------------------------------------------------- | --------------------- | -------------------- | | `dec-party-manager -d /app serve`, as in the deployment guide manifest | `/app` | `/app/data` | | Root image `…:` without `-d` | `/` | `/data` | | Nonroot image `…:-nonroot` without `-d` | `/home/nonroot` | `/home/nonroot/data` | | Binary without `--dir` and without `DECPM_DIR` | The current directory | `./data` | The root directory has this layout: ``` / ├── .env # optional; see the .env item below for where the node reads it └── data/ ├── noise.key # Noise private key, hex-encoded, mode 0600 ├── decpm.db # SQLite database ├── decpm.db-wal # SQLite write-ahead log, when present └── decpm.db-shm # SQLite shared-memory file, when present ``` Back up the complete data directory. Do not select single files. * `noise.key` is the Noise identity of the node. Peers store the matching public key and use it to authenticate this node. * `decpm.db` is the SQLite database. The database runs in write-ahead-log mode, so the `-wal` file can hold recent writes. The database holds these records: * Peer records and party credentials. * The cached membership of each Decentralized Party. * The governance audit trail. * Pending and handled invitations. * Workflow runs, and the artifacts that a resume or a retry needs. * ACS import quarantine records. Two more items can belong to the backup: * `DECPM_DB_PATH`, or the `--db` flag, moves the database. Back up that file and its `-wal` and `-shm` files. * A `.env` file can hold secrets. The node reads it from the `--dir` (`-d`) directory, or from the working directory without that flag. `DECPM_DIR` does not change this location. Back it up under the controls of your secret store. {% hint style="info" %} The node keeps no DAR files on its volume. `POST /dars/upload` and DAR distribution send each DAR to the Canton participant. Back up DARs with the participant, or keep them in your release source. {% endhint %} Record these items with each backup: * The image tag and digest, and the `build_version` that `GET /node-config` returns. * The node configuration, without secret values in the ordinary copy. The [Configuration reference](/decentralization-manager/operations-and-reference/configuration-reference.md) lists every variable. * The Canton network, synchronizer, Admin API, and Ledger API settings, including the TLS certificate files. * The identity-provider type and the client IDs. * Whether database encryption is on. * The AWS credentials setup, if the node signs with AWS KMS. See [Sign with AWS KMS](/decentralization-manager/operations-and-reference/sign-with-aws-kms.md). * The backup time, operator, storage location, checksum, and retention date. * The workflow state and the reason for the maintenance. {% hint style="warning" %} Keep `noise.key` through every restore. A node that finds no key file at startup generates a new key. It logs `No Noise keypair found, generating new one`. A new key gives the node a new peer identity. Every peer operator must then replace the stored public key through an authenticated channel. {% endhint %} ## Create a cold backup Take a cold backup. A copy of a running SQLite database in write-ahead-log mode can be inconsistent. The container image has no shell and no `sqlite3` tool, so you cannot make an online backup with it. 1. Notify the peer operators. Start no new workflow runs. 2. Finish or cancel every in-flight run. At startup, the node resumes every run that the database records as `inprogress`. 3. Record the image tag, the Noise public key, the peer status, the package state, and the workflow state. 4. Stop the node. ```bash kubectl -n scale deploy/dec-party-manager --replicas=0 kubectl -n get pods ``` 5. Confirm that no process writes to the persistent volume. 6. Create an encrypted volume snapshot, or an archive of the complete data directory. The image has no shell, so copy the files from a helper pod that mounts the volume. 7. Create a checksum for the backup artifact. ```bash sha256sum ``` 8. Verify that the artifact contains `noise.key` and `decpm.db`. 9. Start the same release again. ```bash kubectl -n scale deploy/dec-party-manager --replicas=1 kubectl -n rollout status deploy/dec-party-manager ``` 10. Verify authentication, Canton connectivity, the Noise identity, peers, packages, and the workflow state. **Expected result:** the node starts with the same Noise public key and the same local state. ## Restore a node Restore into an isolated environment first. Do not open the Noise port to peers until the identity and state checks pass. 1. Declare the incident and record the reason for the restore. 2. Make sure that the failed node and the replacement never run at the same time with the same identity. 3. Select the newest valid backup from before the failure. 4. Verify the backup checksum and the access record. 5. Provision one replacement node with the same data path and the image tag from the backup record. 6. Restore the complete data directory, including `noise.key`, `decpm.db`, and any `-wal` and `-shm` files. 7. Change the file owner to uid 65532 if the replacement runs the `-nonroot` image and the files come from the root image. See [Move a node to the nonroot image](#move-a-node-to-the-nonroot-image). 8. Restore the node configuration. Supply the original `DECPM_DB_ENCRYPTION_KEY` before startup, if the node used database encryption. 9. Start the replacement without peer access. 10. Read the startup logs. The node runs SQL migrations at startup. Stop if a migration fails. 11. Compare the `Noise public key: ` log line with your record. `GET /keys/status` returns the same key in `public_key`. 12. Test the stored party credentials. The node requests a fresh token for each party and reports `success` for each one. Set `DECMAN_URL` and `DECMAN_TOKEN` as in [API and workflow reference](/decentralization-manager/operations-and-reference/api-and-workflow-reference.md#authentication). ```bash curl --fail-with-body -X POST \ -H "Authorization: Bearer ${DECMAN_TOKEN}" \ "${DECMAN_URL}/auth/test" ``` 13. Verify the identity-provider login. `GET /node-health` reports whether the Admin API and the Ledger API answer. 14. Compare the peers, party credentials, packages, and workflow state with the recovery record. 15. Open the Noise port. Ask the peer operators to verify connectivity in both directions. 16. Run read-only checks, such as peer status and contract queries. 17. Run one low-risk governance action only after the read-only checks pass. 18. Record the recovery evidence. Close or hand off the incident. **Stop and escalate if one of these conditions is true:** * The Noise public key changed. * The credential test fails with the original encryption key. * A migration fails. * Peer records are missing. * Package IDs differ. * The workflow state is unclear. * The failed node can still start. {% hint style="warning" %} A node with no stored party credentials accepts the first `PUT /party-config` without a token. This bootstrap rule lets an operator configure a new node. Keep the HTTP port of a replacement node private until its database is in place. See [Authentication and operator roles](/decentralization-manager/operations-and-reference/authentication-and-operator-roles.md). {% endhint %} {% hint style="info" %} A wrong `DECPM_DB_ENCRYPTION_KEY` does not stop startup. The node then uses each stored ciphertext as the secret. Every party token request then fails. The credential test in step 12 detects this condition. {% endhint %} ## Recover from common failure scenarios | Scenario | Response | | ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | The pod or container failed, and the volume is intact | Start the same image against the existing volume. Verify the Noise key and the local state before you resume workflows. | | The volume is lost, and a valid backup exists | Restore the data directory and the matching release in isolation. Follow the full restore procedure. | | `noise.key` is lost or changed | Do not let the node run with a new key unnoticed. If you cannot restore the original key, every peer must authenticate and approve the new public key. | | `decpm.db` is lost, and `noise.key` is intact | The node keeps its peer identity. It loses the peer records, the party credentials, and the workflow state. Restore the database. Without a backup, enter the peers and the party credentials again. | | The database encryption key is lost | Do not set a new key against the existing database. Recover the original key from your secret store, or escalate to your security team. | | A failure happens during an active run | Read the run state and the ledger state before you retry, cancel, or start another run. A failed coordinator does not prove that the governed action did not advance. | | A run that adds a member failed | Retry the run on the coordinator. Do not cancel it. Do not reconnect the participant of the new member by hand. See [Add a member to a Decentralized Party](/decentralization-manager/operations-and-reference/add-a-member.md). | | Canton or Decentralized Party signing material is lost | This backup does not hold that material. Use the membership and Canton key-recovery procedure. | ## Move a node to the nonroot image Each release from v1.7.0 has two images. Both images hold the same binary and take the same configuration. | Tag | Runs as | `DECPM_DIR` default | | ---------------------------------------------------------------- | ------------ | ------------------- | | `public.ecr.aws/dlc-link/decentralization-manager:` | uid 0 (root) | `/` | | `public.ecr.aws/dlc-link/decentralization-manager:-nonroot` | uid 65532 | `/home/nonroot` | A volume that the root image wrote holds root-owned files. `fsGroup` alone does not fix this problem. `fsGroup` changes the group owner and adds group write permission. The user owner stays root. The node sets mode 0600 on `data/noise.key` at every start. Uid 65532 cannot change the mode of a file that it does not own. With `fsGroup`, the node logs `Failed to chmod key file ''`. Without `fsGroup`, it logs `Failed to read key file ''`. The node does not stop after either error. It runs without the Noise listener, so it cannot exchange messages with its peers. Change the owner of the volume once. Add a temporary root init container before the existing init containers: ```yaml initContainers: - name: chown-data image: busybox:latest command: ["sh", "-c", "chown -R 65532:65532 /app"] securityContext: runAsUser: 0 runAsNonRoot: false volumeMounts: - name: data mountPath: /app ``` Remove this init container after one start without a key file error. From then on, the node owns every file that it writes. * A namespace under the `restricted` Pod Security Standard rejects this container. Run the same `chown -R` from a one-off root pod that mounts the volume. * A `docker run` with a bind mount needs the same change of owner on the host directory. Check the data path before you switch images. The deployment guide manifest passes `-d /app`, so its data path stays the same. A node that uses the image default keeps its data in `/data` on the root image. The nonroot image expects it in `/home/nonroot/data`. Move the files, or set `DECPM_DIR`. {% hint style="info" %} Releases up to and including v1.6.3 have only the root image. To harden one of them without an upgrade, change the owner once as above. Then set `runAsUser: 65532` and the same `fsGroup` on the pod. {% endhint %} ## Upgrade a peer mesh All nodes of a Decentralized Party must run the same release. The code itself accepts any peer that has the same wire version and reports version `0.1.9` or later. Treat version compatibility as a property of the whole mesh. Each Noise frame starts with a wire-version byte. A node rejects every frame whose byte differs from its own. ### Wire-format boundaries A change of the wire version forces a lockstep upgrade. Nodes on different wire versions cannot exchange any peer message. The log on each side shows `Noise protocol version mismatch`. | Change | Wire version | First release | What changes | | -------------------------------------------- | ------------------------ | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | Concurrent workflows | No version byte → `0xD1` | v1.4.0 | Every frame gets the version byte and a run-routing field. Migrations `000013` to `000015` apply. | | ACS block streaming | `0xD1` → `0xD2` | v1.8.0 | The ACS transfer of an add-member run moves to block streaming. Migrations `000016` to `000018` apply. | | Deployment intents and legacy-key retirement | `0xD2` → `0xD3` | Releases after v1.13.0 | Contracts invitations carry deployment-intent commitments. The membership workflows retire legacy keys. Migrations `000020`, `000022`, and `000023` apply. | Releases before v1.4.0, such as v1.0.0, have no version byte. A coordinator also checks the version of each invitee before a run starts. It sends a health probe and expects version `0.1.9` or later in the reply. Otherwise it refuses the start with HTTP `409` and names the blocking peers. A node on another wire version cannot answer the probe, so a mixed mesh cannot start a run. A release that keeps the wire version can run next to the previous release. You can then upgrade one node at a time. Release notes from v1.8.0 state whether the release changes `WIRE_VERSION`. See [Compatibility, releases, and troubleshooting](/decentralization-manager/operations-and-reference/compatibility-releases-and-troubleshooting.md). ### Plan the change 1. Read the release notes. Find wire-version changes, new migrations, package changes, and rollback limits. 2. Record the current release of each peer. `GET /participants-status` returns `version` and `build_version` for each peer. 3. Plan a lockstep window if the release changes `WIRE_VERSION`. Plan a rolling upgrade only when the release notes say that the wire version is unchanged. 4. Finish or cancel every run on every node. 5. Create and verify a cold backup of every node. 6. Define the go and no-go checks, the rollback authority, the communications, and the maximum window. ### Lockstep upgrade Use this procedure when a release changes the wire version. 1. Confirm that every peer operator is ready. Confirm that no run is in flight on any node. 2. Stop every Decentralization Manager node in the mesh. 3. Set the new image tag on every node. 4. Start the nodes. Each node applies the new migrations at startup. 5. Read the startup log and the migration result of each node. 6. Verify the Noise public key on every node. 7. Verify peer connectivity across the full mesh. 8. Verify authentication, Canton connections, party credentials, and package IDs. 9. Run read-only contract and status queries. 10. Run one low-risk governance action. 11. Resume normal workflows and record the result. ### Upgrade to wire version 0xD3 Releases after v1.13.0 use wire version `0xD3`. This version adds deployment-intent verification and legacy-key retirement. Older peers cannot accept the new invitations or the new signing commands. The version byte also rejects retries of runs that started before the upgrade. 1. Finish or cancel every active run on every node. 2. Stop every node in the mesh. 3. Upgrade the whole mesh to the same build. 4. Start the mesh again. 5. Start a new Contracts run for each deployment whose invitation a peer accepted before the upgrade. Those invitations carry no deployment-intent commitments. After this upgrade, three workflows end with a `RetireLegacyKeys` step: add-member (`AddParty`), remove-member (`Kick`), and change-threshold (`ChangeThreshold`). * The step starts after the inline signing keys in `PartyToParticipant` take effect. * The namespace owners then authorize the removal of the old `PartyToKeyMapping` under the current namespace threshold. * The run completes only after Canton shows the removal. * A party without a legacy mapping skips the step. Upgrade the participating nodes before you run these workflows. If `RetireLegacyKeys` times out, bring enough upgraded namespace owners online. Then click **Retry** on the run card of the coordinator. The run resumes at the cleanup step and does not repeat the topology change. The inline keys stay authoritative while the cleanup is pending. ### Rolling upgrade Use a rolling upgrade only when the release notes state that `WIRE_VERSION` is unchanged. 1. Back up the node. 2. Finish or cancel the runs that this node coordinates or takes part in. 3. Update the image tag of the node. Wait for the rollout. 4. Verify the startup log, the Noise public key, and peer connectivity. 5. Repeat the steps for the next node. Some features need the new release on several nodes. The release notes name those features. ## Roll back an upgrade A rollback is a restore, not an image downgrade. * Each release note names the migrations that the release adds. After they apply, a downgrade on the same volume is not supported. * An older binary refuses to start on a database that a newer release migrated. Its log shows `migration was previously applied but is missing in the resolved migrations`. * The node runs forward migrations only. Do not reverse migrations by hand. To roll back: 1. Stop workflows and isolate the affected mesh. 2. Compare the failure with the rollback conditions that you defined. 3. Check whether the upgrade crossed a wire-format boundary. If it did, roll back every node in the mesh. 4. Stop every affected node. 5. Restore the pre-upgrade backup and the pre-upgrade image together. 6. Run the restore checks and the mesh checks again before you resume work. {% hint style="info" %} A restore brings back the database as it was at backup time. Runs, audit entries, and invitations from after the backup are not in it. The ledger keeps every change that Canton accepted after the backup. Compare the Decentralized Party topology and the contracts with the restored state. {% endhint %} ## Test cadence A production recovery program includes: * A checksum and a restore verification for every backup. * A scheduled restore test in isolation. * A mesh-upgrade rehearsal before each release that changes the wire version or adds migrations. * A peer-key replacement exercise. * A disaster exercise for volume loss, lost credentials, and failure during an active run. * Evidence for each backup, restore, and upgrade test. ## Technical sources * [Deployment guide](https://github.com/DLC-link/decentralization-manager/blob/main/docs/DEPLOYMENT_GUIDE.md) * [Adding a member to a Decentralized Party](https://github.com/DLC-link/decentralization-manager/blob/main/docs/ADDING_A_MEMBER_TO_A_DECENTRALIZED_PARTY.md) * [Data directory names](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/consts.rs) * [Data paths](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/config.rs) * [Command-line flags](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/cli.rs) * [Startup sequence](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/main.rs) * [Noise wire version and key file](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/noise/mod.rs) * [Noise listener startup](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/server/mod.rs) * [SQLite connection and migrations](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/db/sqlite.rs) * [Encryption of stored secrets](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/db/crypto.rs) * [Container image](https://github.com/DLC-link/decentralization-manager/blob/main/Dockerfile) * [Release notes](https://github.com/DLC-link/decentralization-manager/releases) ## Related pages * [Production deployment](/decentralization-manager/get-started/production-deployment.md) * [Authentication and operator roles](/decentralization-manager/operations-and-reference/authentication-and-operator-roles.md) * [Membership and governance changes](/decentralization-manager/operations-and-reference/membership-and-governance-changes.md) * [Add a member to a Decentralized Party](/decentralization-manager/operations-and-reference/add-a-member.md) * [Security architecture](/decentralization-manager/operations-and-reference/security-architecture.md) * [Compatibility, releases, and troubleshooting](/decentralization-manager/operations-and-reference/compatibility-releases-and-troubleshooting.md) ## Next step 👉 **Check the release facts before an upgrade:** [Compatibility, releases, and troubleshooting](/decentralization-manager/operations-and-reference/compatibility-releases-and-troubleshooting.md). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.bitsafe.finance/decentralization-manager/operations-and-reference/backup-recovery-and-upgrades.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.