> For the complete documentation index, see [llms.txt](https://docs.bitsafe.finance/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bitsafe.finance/decentralization-manager/operations-and-reference/compatibility-releases-and-troubleshooting.md). # Compatibility, releases, and troubleshooting ## Where releases are published | Artifact | Location | | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | Release notes | [GitHub releases](https://github.com/DLC-link/decentralization-manager/releases). Each tag has the form `v`. | | Container images | `public.ecr.aws/dlc-link/decentralization-manager:v` and `public.ecr.aws/dlc-link/decentralization-manager:v-nonroot` | | Daml packages (DARs) | The [`releases/`](https://github.com/DLC-link/decentralization-manager/tree/main/releases) folder of the repository | | Source code | The tagged commit in the repository | * The release workflow refuses a tag that does not match the version in `crates/decman/Cargo.toml`. * A GitHub release has release notes only. It has no binary assets. * Each release has two images with the same binary. The `-nonroot` image exists from v1.7.0 and runs as uid 65532. See the Move a node to the nonroot image section of [Backup, recovery, and upgrades](/decentralization-manager/operations-and-reference/backup-recovery-and-upgrades.md#move-a-node-to-the-nonroot-image). * Releases before v1.6.0 used the image repository `public.ecr.aws/dlc-link/canton-decparty-manager`. Newer releases go only to `decentralization-manager`. * Pin an exact image tag. Do not use `latest`. ### Daml packages The binary does not contain the Daml packages. Upload or distribute the DARs yourself. See [Package and contract operations](/decentralization-manager/operations-and-reference/package-and-contract-operations.md). | Folder | Contents | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `releases/v1/` | The version 1 packages: `governance-action-v1`, `governance-core-v1`, `governance-rewards-v1`, `governance-rewards-automation-v1`, `governance-token-custody-v1`, `governance-utility-credential-v1`, `governance-utility-onboarding-v1`, and `orphan-marker` | | `releases/v0/rc1` to `releases/v0/rc4` | Earlier release-candidate packages | The file name of each DAR ends with its package version, for example `governance-utility-onboarding-v1-0.5.0.dar`. A folder can hold several versions of one package. ## What the release notes state From v1.8.0, each release note states these facts at the top: * Whether the release changes `WIRE_VERSION` or `MIN_PEER_VERSION`. * Which database migrations the release adds. The notes then list new features, bug fixes, API changes, and operations changes. ## Version facts ### Versions that a node reports | Field | Value | Use | | --------------- | --------------------------------------------------------------------------------------------------- | ---------------------------------------- | | `version` | The Cargo package version of the build | Peers compare it with `MIN_PEER_VERSION` | | `build_version` | The image tag on a release image. A commit SHA on other CI builds. `-dev` on a local build | Display only | `GET /node-config` returns both fields for this node. `GET /participants-status` returns both fields for each peer. {% hint style="info" %} The Cargo version does not always change together with the wire version. Compare the release tags in `build_version` across the mesh. {% endhint %} ### Peer compatibility All nodes of a Decentralized Party must run the same release. The code itself accepts any peer that has the same wire version and reports version `0.1.9` or later. These two checks work as follows. The first check is the Noise wire version. Each frame starts with a version byte. A node rejects a frame with another byte. | Releases | Wire version | | -------------------------------------- | --------------- | | Releases before v1.4.0, such as v1.0.0 | No version byte | | v1.4.0 to v1.7.0 | `0xD1` | | v1.8.0 to v1.13.0 | `0xD2` | | Releases after v1.13.0 | `0xD3` | A change of the wire version needs a lockstep upgrade of the whole mesh. [Backup, recovery, and upgrades](/decentralization-manager/operations-and-reference/backup-recovery-and-upgrades.md#upgrade-a-peer-mesh) has the procedure for each boundary. The second check is `MIN_PEER_VERSION`, which is `0.1.9`. Before a run starts, the coordinator probes each invitee. It refuses the start unless each invitee reports version `0.1.9` or later. Every release from v1.4.0 reports a 1.x version. So the check fails only for a peer that cannot answer the probe. ### Canton and Daml * The node uses Canton protocol version 35 for key export and topology operations. This value is a constant in the code. * The Decentralized Party topology puts Daml signing keys inside the `PartyToParticipant` mapping. This feature needs Canton 3.4 or later. * Every Daml package in the repository builds with Daml SDK 3.4.11. * The integration tests run against Splice LocalNet 0.6.12. * `DECPM_CANTON_NETWORK` takes `mainnet`, `testnet`, or `devnet`. * `GET /node-health` returns the Canton version of the participant in `participant.version`. The repository does not publish a support matrix of Canton or Splice releases. ### Database migrations The node applies new migrations at startup. | Release | New migrations | | ---------------------- | -------------------------------- | | v1.4.0 | `000013` to `000015` | | v1.8.0 | `000016` to `000018` | | v1.9.0 | `000019` | | Releases after v1.13.0 | `000020`, `000022`, and `000023` | v1.10.0 to v1.13.0 add no migration. After a release applies its migrations, a downgrade on the same volume is not supported. See the Roll back an upgrade section of [Backup, recovery, and upgrades](/decentralization-manager/operations-and-reference/backup-recovery-and-upgrades.md#roll-back-an-upgrade). ## Before you deploy or upgrade Record these facts for every node in the mesh: * The release tag and the image digest. * The wire version of the current release and of the target release. * The migrations that the target release adds. * The Canton version of the participant, from `GET /node-health`. * The packages that the participant vets. The **Packages** page and `GET /packages/compare-peers` compare them across peers. * The identity provider and the deployment type. * A verified cold backup. ## Troubleshooting order 1. Confirm the release of every peer. `GET /participants-status` lists `version` and `build_version`. 2. Check `GET /node-health` and the logs. 3. Verify the identity provider and the Canton connections. 4. Check the persistent volume and the Noise identity. 5. Inspect the workflow runs and the contract state. 6. Collect sanitized evidence before you change the configuration. ## Troubleshooting The messages in this table come from the node. Most of them appear in the log. HTTP errors appear in the `error` field of the response. | Message or symptom | Cause | Action | | ---------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `Noise protocol version mismatch: got 0x…, expected 0x… — the peer is running an incompatible dec-party-manager build` | The two nodes use different wire versions. | Upgrade the mesh in lockstep. See [Backup, recovery, and upgrades](/decentralization-manager/operations-and-reference/backup-recovery-and-upgrades.md#upgrade-a-peer-mesh). | | HTTP `409` `Cannot start the workflow: every invited peer must run dec-party-manager >= 0.1.9 … Blocking peer(s): …` | The coordinator could not read a version from the named peers. The peer is down, unreachable, or on another wire version. | Read the reason next to each peer. Upgrade the peer to the release of the other nodes, bring it online, or deselect it. | | Peer status `HandshakeFailed` | One side stores a wrong Noise public key, or the peer runs another wire version. | Exchange the peer records again. See [Connect peers](/decentralization-manager/get-started/connect-peers.md). | | Peer status `Unreachable` | The TCP connection to the Noise port fails. | Check the Noise port, `DECPM_PUBLIC_ADDRESS`, and the firewall. | | `Failed to chmod key file ''` in the startup log | The nonroot image runs on a volume that the root image wrote. The node keeps running, but it starts no Noise listener and cannot exchange messages with its peers. | Change the owner of the volume to uid 65532. See [Backup, recovery, and upgrades](/decentralization-manager/operations-and-reference/backup-recovery-and-upgrades.md#move-a-node-to-the-nonroot-image). Then restart the node. | | `Failed to read key file ''` in the startup log | The runtime user cannot read `noise.key`. The node keeps running, but it starts no Noise listener and cannot exchange messages with its peers. | Change the owner of the volume to the runtime user. `fsGroup` alone does not fix this failure. Then restart the node. | | `--insecure / DECPM_INSECURE is only permitted on the devnet network; refusing to start on ` | Insecure mode is on for a network other than devnet. | Unset `DECPM_INSECURE` and configure an identity provider. | | `connecting to : the admin API channel is plaintext. …` | The Admin API serves TLS, and the node connects without TLS. | Set `DECPM_CANTON_ADMIN_TLS=true`. Use the `LEDGER` variable when the message names the ledger API. | | `connecting to : TLS is enabled for the ledger API. …` | The endpoint serves plaintext, or a private CA issued its certificate. | Unset `DECPM_CANTON_LEDGER_TLS`, or point `DECPM_CANTON_LEDGER_TLS_CA_CERT` at the CA. Use the `ADMIN` variables for the admin API. | | `connecting to : nothing accepts TCP connections there. …` | The Canton API is down, is still starting, or has a wrong host or port. | Check the host and the port. The TLS settings do not cause this failure. | | `admin API mTLS is half-configured: …` or `ledger API mTLS is half-configured: …` | Only one of the client certificate and the client key is set. | Set both, or set neither. | | `migration 4 was previously applied but has been modified` | A release from v1.4.0 to v1.6.1 started on a database that an earlier build, such as v1.0.0, created. | Run v1.6.2 or later. These releases repair the stored checksum at startup. | | `migration was previously applied but is missing in the resolved migrations` | An older binary started on a database that a newer release migrated. | Run the newer release, or restore the pre-upgrade backup. | | HTTP `401` `missing bearer token`, `token issuer not trusted: …`, or `token is not active` | The operator token is missing, from another issuer, or expired. | Check the identity-provider settings. See [Authentication and operator roles](/decentralization-manager/operations-and-reference/authentication-and-operator-roles.md). | | HTTP `403` `missing required role: ` | The token lacks the role in `DECPM_ADMIN_ROLE`. | Grant the role, or fix the role claim. | | `metrics_port collides with another listener; this node reports no metrics` | `DECPM_METRICS_PORT` equals the HTTP port or the Noise port. | Choose another port. See [Monitoring and logging](/decentralization-manager/operations-and-reference/monitoring-and-logging.md). | [API and workflow reference](/decentralization-manager/operations-and-reference/api-and-workflow-reference.md#troubleshooting) lists more API and workflow errors. ## Technical sources * [Release notes](https://github.com/DLC-link/decentralization-manager/releases) * [Release workflow](https://github.com/DLC-link/decentralization-manager/blob/main/.github/workflows/release.yml) * [Container image](https://github.com/DLC-link/decentralization-manager/blob/main/Dockerfile) * [Released DARs](https://github.com/DLC-link/decentralization-manager/tree/main/releases) * [Version constants](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/consts.rs) * [Build identity](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/build_info.rs) * [Noise wire version](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/noise/mod.rs) * [Noise listener startup](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/server/mod.rs) * [Peer version check](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/server/handlers/workflows.rs) * [Canton connection errors](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/config.rs) * [Database migrations](https://github.com/DLC-link/decentralization-manager/tree/main/crates/decman/migrations) * [Architecture reference](https://github.com/DLC-link/decentralization-manager/blob/main/docs/ARCHITECTURE.md) * [Deployment guide](https://github.com/DLC-link/decentralization-manager/blob/main/docs/DEPLOYMENT_GUIDE.md) ## Related pages * [Backup, recovery, and upgrades](/decentralization-manager/operations-and-reference/backup-recovery-and-upgrades.md) * [API and workflow reference](/decentralization-manager/operations-and-reference/api-and-workflow-reference.md) * [Monitoring and logging](/decentralization-manager/operations-and-reference/monitoring-and-logging.md) * [Production deployment](/decentralization-manager/get-started/production-deployment.md) * [Authentication and operator roles](/decentralization-manager/operations-and-reference/authentication-and-operator-roles.md) * [Security architecture](/decentralization-manager/operations-and-reference/security-architecture.md) * [Support and developer resources](https://docs.bitsafe.finance/support-and-developer-resources/) ## Next step 👉 **Still blocked?** [Support and developer resources](https://docs.bitsafe.finance/support-and-developer-resources/). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.bitsafe.finance/decentralization-manager/operations-and-reference/compatibility-releases-and-troubleshooting.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.