Backup, recovery, and upgrades

Purpose

Use this runbook to protect and restore one Decentralization Manager instance, or to upgrade every Decentralization Manager instance in a connected peer mesh.

This runbook covers the application host and its local persistent state. It does not recover a lost Canton signing key, Decentralized Party key share, or external identity-provider tenant. Escalate those failures to the applicable Canton, membership, or identity recovery procedure.

Before you begin

Confirm that you have:

• Administrative access to the deployment and its persistent volume.
• An approved backup destination with encryption, access control, retention, and deletion policies.
• The exact deployed image digest, tag, or commit.
• The active DECPM_DB_ENCRYPTION_KEY, if configured, available through the approved secret-recovery process.
• An out-of-band record of this instance's Noise public key and peer identity.
• A current list of peers, Decentralized Parties, expected packages, and active workflows.
• A maintenance window coordinated with every peer operator when the procedure affects the mesh.

Treat the backup as sensitive. The SQLite database can contain party credentials. If DECPM_DB_ENCRYPTION_KEY is unset, those secrets are stored in plaintext. If it is set, restoration requires the same key.

What to back up

Back up the complete persistent data directory rather than selecting individual files. The default layout is:

<DECPM_DIR>/
└── data/
    ├── noise.key
    ├── decpm.db
    └── dars/

The data protects:

• noise.key: the persistent Noise identity used by peers to recognize this instance.
• decpm.db: peer records, party credentials, workflow state, and application metadata.
• dars/: local DAR artifacts used for contract deployment.

Record these items separately with each backup:

• Image digest, release tag, and source commit.
• Deployment configuration, excluding secret values from the ordinary manifest copy.
• Canton network, synchronizer, Admin API, and Ledger API connection settings.
• Identity-provider type and client identifiers.
• Whether database encryption is enabled.
• Backup time, operator, storage location, checksum, and retention date.
• The workflow state and maintenance reason at the time of backup.

Do not regenerate noise.key during a normal restore. A new key changes the peer identity and requires every peer operator to replace the stored public key through an authenticated channel.

Create an application-consistent backup

The conservative baseline is a cold backup. Stop the only active instance before copying or snapshotting the persistent volume. Do not use an online SQLite backup unless the supported release documents a consistent procedure.

1. Notify peer operators and stop new workflows.
2. Finish, cancel, or dismiss every in-flight workflow according to its supported procedure.
3. Record the current image digest, Noise public key, peer status, package state, and workflow state.
4. Stop the Decentralization Manager instance.

kubectl -n <namespace> scale deploy/dec-party-manager --replicas=0
kubectl -n <namespace> get pods

1. Confirm that no process can write to the persistent volume.
2. Create an encrypted storage snapshot or archive of the complete persistent data directory.
3. Create and store a checksum for the backup artifact.

sha256sum <backup-artifact>

1. Verify that the artifact contains noise.key, decpm.db, and dars/.
2. Restart the same approved release.

kubectl -n <namespace> scale deploy/dec-party-manager --replicas=1
kubectl -n <namespace> rollout status deploy/dec-party-manager

1. Verify authentication, Canton connectivity, Noise identity, peers, packages, and workflow state.

Expected result: the instance returns with the same Noise public key and the same local application state.

Restore an instance

Restore into an isolated replacement environment first. Do not expose the Noise port to peers until identity and state checks pass.

1. Declare the incident and record why restoration is required.
2. Prevent the failed and replacement instances from running at the same time with the same identity.
3. Select the newest valid backup that predates the failure.
4. Verify the backup checksum and access record.
5. Provision a single replacement instance with the same persistent path and the exact compatible image digest.
6. Restore the complete persistent data directory, including noise.key, decpm.db, and dars/.
7. Restore the approved deployment configuration and make the original DECPM_DB_ENCRYPTION_KEY available before startup, if database encryption was enabled.
8. Start the replacement without public peer exposure.
9. Check startup logs. SQL migrations run automatically on startup. Stop if a migration fails or if the selected release does not match the backup's compatibility record.
10. Use the authenticated /keys/status endpoint or the Network panel to compare the Noise public key with the out-of-band record.
11. Verify identity-provider login and Canton Admin and Ledger API connectivity.
12. Compare the peer list, party credentials, DAR inventory, package IDs, and workflow state with the recovery record.
13. Expose the Noise port and ask peer operators to verify mutual connectivity.
14. Run read-only health checks, including peer status and contract queries.
15. Run one approved low-risk governance action only after read-only checks pass.
16. Record recovery evidence and close or hand off the incident.

Stop and escalate if: the Noise public key changed, the database cannot be decrypted, migrations fail, peer records are missing, package IDs differ, workflow state is ambiguous, or the failed instance may still be running.

Recover from common failure scenarios

Scenario	Required response
Pod or container failed, persistent volume intact	Redeploy the same approved image against the existing volume. Verify the Noise key and application state before resuming workflows.
Persistent volume lost, valid backup available	Restore the complete data directory and the compatible release in isolation. Follow the full restore procedure.
noise.key lost or changed	Do not silently create a replacement identity. If the original key cannot be restored, every peer must authenticate and approve the new public key through the incident-recovery process.
decpm.db lost but Noise key survives	The instance can retain its peer identity, but peer records, party credentials, and workflow state may be unavailable. Restore the database. If no backup exists, follow the supported state-reconstruction procedure.
Database encryption key unavailable	Do not rotate the key against the existing encrypted database. Recover the original key through the secret-management procedure or escalate to Security.
Failure during an active workflow	Determine the workflow and on-ledger state before retrying, cancelling, or starting another workflow. Do not assume that a failed coordinator means the governed action did not advance.
Lost Canton or Decentralized Party signing material	This backup does not recover it. Use the membership and Canton key-recovery process documented for the supported release.

Upgrade a peer mesh

Treat version compatibility as a mesh-wide property. Do not assume that mixed versions can communicate.

Plan the change

1. Read the release notes and identify database migrations, Noise protocol changes, package changes, and rollback limits.
2. Build a compatibility record for every peer: current release, target release, image digest, Canton version, package IDs, and backup reference.
3. Confirm whether the release explicitly supports a rolling upgrade. If it does not, use a lockstep maintenance window.
4. Finish, cancel, or dismiss all workflows.
5. Create and verify a cold backup for every node.
6. Define go or no-go checks, rollback authority, communications, and the maximum maintenance window.

Lockstep upgrade

Use this procedure when a release changes the peer wire format or when rolling compatibility is not explicitly documented.

1. Confirm that every peer operator is ready and that no workflow is active.
2. Stop all Decentralization Manager instances in the mesh.
3. Update every deployment to the approved image digest.
4. Start the instances in the agreed order.
5. Check each instance's startup logs and migration result.
6. Verify the Noise public key on every instance.
7. Verify mutual peer connectivity across the full mesh.
8. Verify authentication, Canton connections, party credentials, DAR inventory, and package IDs.
9. Run read-only contract and status queries.
10. Run one approved low-risk governance action.
11. Resume normal workflows and record the result.

Known compatibility boundary at reviewed commit 76d7156: the concurrent-workflow release changed the Noise wire format. A mixed-version mesh does not degrade gracefully. Nodes on version 0.1.9 or later reject older peers, and older builds cannot parse the newer frames. This transition requires a lockstep upgrade with no workflow in flight. Confirm the mapping between repository versions and published image tags before applying it to production.

Roll back an upgrade

Rollback is a release-specific recovery action, not a simple image downgrade.

1. Stop workflows and isolate the affected mesh.
2. Compare the observed failure with the approved rollback conditions.
3. Confirm whether database, Noise protocol, or Daml package state changed.
4. If the release procedure permits rollback, stop every affected node and restore the pre-upgrade application backup and release together.
5. Do not start an older binary against a database that has completed incompatible migrations.
6. Do not attempt to reverse SQL migrations manually unless the release procedure and Engineering explicitly require it.
7. Re-run the restore verification and mesh checks before resuming work.

At the reviewed commit, migrations 000013 and 000014 support concurrent workflows. The repository notes that downgrading requires reversing both migrations, and that the 000013 reversal fails while concurrent InProgress rows exist. Treat rollback as a controlled release-recovery procedure.

Test cadence

A production recovery program should include:

• A checksum and restore verification for every backup.
• A scheduled isolated restore test.
• A mesh-upgrade rehearsal before a release with protocol or database changes.
• A documented peer-key replacement exercise.
• A disaster exercise for volume loss, unavailable credentials, and failure during an active workflow.
• Evidence retention for each backup, restore, and upgrade test.

Technical sources

• Repository README at the reviewed commit
• Deployment guide at the reviewed commit
• Security policy at the reviewed commit

Related pages

• Decentralization Manager production deployment
• Decentralization Manager authentication and operator roles
• Decentralization Manager membership and governance changes
• Decentralization Manager security architecture
• Decentralization Manager compatibility, releases, and troubleshooting

Next step

👉 Confirm the supported recovery path: Decentralization Manager compatibility, releases, and troubleshooting.