> For the complete documentation index, see [llms.txt](https://docs.bitsafe.finance/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bitsafe.finance/decentralization-manager/operations-and-reference/backup-recovery-and-upgrades.md).

# Backup, recovery, and upgrades

### Purpose

Use this runbook to protect and restore one Decentralization Manager instance, or to upgrade every Decentralization Manager instance in a connected peer mesh.

This runbook covers the application host and its local persistent state. It does not recover a lost Canton signing key, Decentralized Party key share, or external identity-provider tenant. Escalate those failures to the applicable Canton, membership, or identity recovery procedure.

### Before you begin

Confirm that you have:

* Administrative access to the deployment and its persistent volume.
* An approved backup destination with encryption, access control, retention, and deletion policies.
* The exact deployed image digest, tag, or commit.
* The active `DECPM_DB_ENCRYPTION_KEY`, if configured, available through the approved secret-recovery process.
* An out-of-band record of this instance's Noise public key and peer identity.
* A current list of peers, Decentralized Parties, expected packages, and active workflows.
* A maintenance window coordinated with every peer operator when the procedure affects the mesh.

{% hint style="info" %}
Treat the backup as sensitive. The SQLite database can contain party credentials. If `DECPM_DB_ENCRYPTION_KEY` is unset, those secrets are stored in plaintext. If it is set, restoration requires the same key.
{% endhint %}

### What to back up

Back up the complete persistent data directory rather than selecting individual files. The default layout is:

```
<DECPM_DIR>/
└── data/
    ├── noise.key
    ├── decpm.db
    └── dars/
```

The data protects:

* `noise.key`: the persistent Noise identity used by peers to recognize this instance.
* `decpm.db`: peer records, party credentials, workflow state, and application metadata.
* `dars/`: local DAR artifacts used for contract deployment.

Record these items separately with each backup:

* Image digest, release tag, and source commit.
* Deployment configuration, excluding secret values from the ordinary manifest copy.
* Canton network, synchronizer, Admin API, and Ledger API connection settings.
* Identity-provider type and client identifiers.
* Whether database encryption is enabled.
* Backup time, operator, storage location, checksum, and retention date.
* The workflow state and maintenance reason at the time of backup.

{% hint style="info" %}
Do not regenerate `noise.key` during a normal restore. A new key changes the peer identity and requires every peer operator to replace the stored public key through an authenticated channel.
{% endhint %}

### Create an application-consistent backup

The conservative baseline is a cold backup. Stop the only active instance before copying or snapshotting the persistent volume. Do not use an online SQLite backup unless the supported release documents a consistent procedure.

1. Notify peer operators and stop new workflows.
2. Finish, cancel, or dismiss every in-flight workflow according to its supported procedure.
3. Record the current image digest, Noise public key, peer status, package state, and workflow state.
4. Stop the Decentralization Manager instance.

```bash
kubectl -n <namespace> scale deploy/dec-party-manager --replicas=0
kubectl -n <namespace> get pods
```

1. Confirm that no process can write to the persistent volume.
2. Create an encrypted storage snapshot or archive of the complete persistent data directory.
3. Create and store a checksum for the backup artifact.

```bash
sha256sum <backup-artifact>
```

1. Verify that the artifact contains `noise.key`, `decpm.db`, and `dars/`.
2. Restart the same approved release.

```bash
kubectl -n <namespace> scale deploy/dec-party-manager --replicas=1
kubectl -n <namespace> rollout status deploy/dec-party-manager
```

1. Verify authentication, Canton connectivity, Noise identity, peers, packages, and workflow state.

**Expected result:** the instance returns with the same Noise public key and the same local application state.

### Restore an instance

Restore into an isolated replacement environment first. Do not expose the Noise port to peers until identity and state checks pass.

1. Declare the incident and record why restoration is required.
2. Prevent the failed and replacement instances from running at the same time with the same identity.
3. Select the newest valid backup that predates the failure.
4. Verify the backup checksum and access record.
5. Provision a single replacement instance with the same persistent path and the exact compatible image digest.
6. Restore the complete persistent data directory, including `noise.key`, `decpm.db`, and `dars/`.
7. Restore the approved deployment configuration and make the original `DECPM_DB_ENCRYPTION_KEY` available before startup, if database encryption was enabled.
8. Start the replacement without public peer exposure.
9. Check startup logs. SQL migrations run automatically on startup. Stop if a migration fails or if the selected release does not match the backup's compatibility record.
10. Use the authenticated `/keys/status` endpoint or the Network panel to compare the Noise public key with the out-of-band record.
11. Verify identity-provider login and Canton Admin and Ledger API connectivity.
12. Compare the peer list, party credentials, DAR inventory, package IDs, and workflow state with the recovery record.
13. Expose the Noise port and ask peer operators to verify mutual connectivity.
14. Run read-only health checks, including peer status and contract queries.
15. Run one approved low-risk governance action only after read-only checks pass.
16. Record recovery evidence and close or hand off the incident.

**Stop and escalate if:** the Noise public key changed, the database cannot be decrypted, migrations fail, peer records are missing, package IDs differ, workflow state is ambiguous, or the failed instance may still be running.

### Recover from common failure scenarios

<table data-search="false"><thead><tr><th>Scenario</th><th>Required response</th></tr></thead><tbody><tr><td>Pod or container failed, persistent volume intact</td><td>Redeploy the same approved image against the existing volume. Verify the Noise key and application state before resuming workflows.</td></tr><tr><td>Persistent volume lost, valid backup available</td><td>Restore the complete data directory and the compatible release in isolation. Follow the full restore procedure.</td></tr><tr><td><code>noise.key</code> lost or changed</td><td>Do not silently create a replacement identity. If the original key cannot be restored, every peer must authenticate and approve the new public key through the incident-recovery process.</td></tr><tr><td><code>decpm.db</code> lost but Noise key survives</td><td>The instance can retain its peer identity, but peer records, party credentials, and workflow state may be unavailable. Restore the database. If no backup exists, follow the supported state-reconstruction procedure.</td></tr><tr><td>Database encryption key unavailable</td><td>Do not rotate the key against the existing encrypted database. Recover the original key through the secret-management procedure or escalate to Security.</td></tr><tr><td>Failure during an active workflow</td><td>Determine the workflow and on-ledger state before retrying, cancelling, or starting another workflow. Do not assume that a failed coordinator means the governed action did not advance.</td></tr><tr><td>Lost Canton or Decentralized Party signing material</td><td>This backup does not recover it. Use the membership and Canton key-recovery process documented for the supported release.</td></tr></tbody></table>

### Upgrade a peer mesh

Treat version compatibility as a mesh-wide property. Do not assume that mixed versions can communicate.

#### Plan the change

1. Read the release notes and identify database migrations, Noise protocol changes, package changes, and rollback limits.
2. Build a compatibility record for every peer: current release, target release, image digest, Canton version, package IDs, and backup reference.
3. Confirm whether the release explicitly supports a rolling upgrade. If it does not, use a lockstep maintenance window.
4. Finish, cancel, or dismiss all workflows.
5. Create and verify a cold backup for every node.
6. Define go or no-go checks, rollback authority, communications, and the maximum maintenance window.

#### Lockstep upgrade

Use this procedure when a release changes the peer wire format or when rolling compatibility is not explicitly documented.

1. Confirm that every peer operator is ready and that no workflow is active.
2. Stop all Decentralization Manager instances in the mesh.
3. Update every deployment to the approved image digest.
4. Start the instances in the agreed order.
5. Check each instance's startup logs and migration result.
6. Verify the Noise public key on every instance.
7. Verify mutual peer connectivity across the full mesh.
8. Verify authentication, Canton connections, party credentials, DAR inventory, and package IDs.
9. Run read-only contract and status queries.
10. Run one approved low-risk governance action.
11. Resume normal workflows and record the result.

{% hint style="info" %}
**Known compatibility boundary at reviewed commit `76d7156`:** the concurrent-workflow release changed the Noise wire format. A mixed-version mesh does not degrade gracefully. Nodes on version `0.1.9` or later reject older peers, and older builds cannot parse the newer frames. This transition requires a lockstep upgrade with no workflow in flight. Confirm the mapping between repository versions and published image tags before applying it to production.
{% endhint %}

### Roll back an upgrade

Rollback is a release-specific recovery action, not a simple image downgrade.

1. Stop workflows and isolate the affected mesh.
2. Compare the observed failure with the approved rollback conditions.
3. Confirm whether database, Noise protocol, or Daml package state changed.
4. If the release procedure permits rollback, stop every affected node and restore the pre-upgrade application backup and release together.
5. Do not start an older binary against a database that has completed incompatible migrations.
6. Do not attempt to reverse SQL migrations manually unless the release procedure and Engineering explicitly require it.
7. Re-run the restore verification and mesh checks before resuming work.

At the reviewed commit, migrations `000013` and `000014` support concurrent workflows. The repository notes that downgrading requires reversing both migrations, and that the `000013` reversal fails while concurrent `InProgress` rows exist. Treat rollback as a controlled release-recovery procedure.

### Test cadence

A production recovery program should include:

* A checksum and restore verification for every backup.
* A scheduled isolated restore test.
* A mesh-upgrade rehearsal before a release with protocol or database changes.
* A documented peer-key replacement exercise.
* A disaster exercise for volume loss, unavailable credentials, and failure during an active workflow.
* Evidence retention for each backup, restore, and upgrade test.

### Technical sources

* [Repository README at the reviewed commit](https://github.com/DLC-link/decentralization-manager/blob/76d7156c85e605abc008e5fadf8aaa97f4706471/README.md)
* [Deployment guide at the reviewed commit](https://github.com/DLC-link/decentralization-manager/blob/76d7156c85e605abc008e5fadf8aaa97f4706471/docs/DEPLOYMENT_GUIDE.md)
* [Security policy at the reviewed commit](https://github.com/DLC-link/decentralization-manager/blob/76d7156c85e605abc008e5fadf8aaa97f4706471/docs/SECURITY.md)

### Related pages

* [Decentralization Manager production deployment](/decentralization-manager/get-started/production-deployment.md)
* [Decentralization Manager authentication and operator roles](/decentralization-manager/operations-and-reference/authentication-and-operator-roles.md)
* [Decentralization Manager membership and governance changes](/decentralization-manager/operations-and-reference/membership-and-governance-changes.md)
* [Decentralization Manager security architecture](/decentralization-manager/operations-and-reference/security-architecture.md)
* [Decentralization Manager compatibility, releases, and troubleshooting](/decentralization-manager/operations-and-reference/compatibility-releases-and-troubleshooting.md)

### Next step

👉 **Confirm the supported recovery path:** [Decentralization Manager compatibility, releases, and troubleshooting](/decentralization-manager/operations-and-reference/compatibility-releases-and-troubleshooting.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.bitsafe.finance/decentralization-manager/operations-and-reference/backup-recovery-and-upgrades.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.