> For the complete documentation index, see [llms.txt](https://docs.bitsafe.finance/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bitsafe.finance/decentralization-manager/get-started/production-deployment.md).

# Production deployment

[Run with Docker](/decentralization-manager/get-started/run-with-docker.md) · [Configuration reference](/decentralization-manager/operations-and-reference/configuration-reference.md)

## Production architecture

Each participating organization deploys one Decentralization Manager node. A node has three listeners. Each listener needs a different exposure:

| Listener | Default port | Protocol                                                         | Expose to                                                                  |
| -------- | ------------ | ---------------------------------------------------------------- | -------------------------------------------------------------------------- |
| HTTP     | `8080`       | Plain HTTP: operator UI, operator API, tenant API and `/healthz` | Operators and wallet providers, through a TLS-terminating ingress or proxy |
| Noise    | `9000`       | The Noise protocol (`NN_PSK2`) over raw TCP                      | Other nodes, through a TCP pass-through load balancer or a direct port     |
| Metrics  | `9464`       | Plain HTTP at `/metrics`, with no authentication                 | The metrics collector, inside the private network only                     |

The node also needs private network access to the Admin API and the Ledger API of its Canton participant. It needs persistent storage, an identity provider, a secret store, log collection, metrics collection and backups.

## Peer traffic does not use TLS

Peer traffic uses the Noise protocol directly over TCP. Noise encrypts each connection and authenticates the peer with keys from the peer records. A TLS-terminating proxy or an HTTP load balancer in front of the Noise port breaks peering.

* Expose the Noise port through a layer 4 (TCP) load balancer, or through a direct port.
* Set `DECPM_PUBLIC_ADDRESS` to the host name or IP address that peers dial.
* Restart the node after you change `DECPM_PUBLIC_ADDRESS`.

The repository deployment guide uses two Kubernetes Services. A `ClusterIP` Service carries the HTTP port behind the Ingress. A `LoadBalancer` Service carries the Noise port over TCP. Neither Service exposes the metrics port.

## Choose the image

Each release has two images on Amazon ECR Public:

| Image                                                            | Runs as   | `DECPM_DIR` default |
| ---------------------------------------------------------------- | --------- | ------------------- |
| `public.ecr.aws/dlc-link/decentralization-manager:<tag>`         | uid 0     | `/`                 |
| `public.ecr.aws/dlc-link/decentralization-manager:<tag>-nonroot` | uid 65532 | `/home/nonroot`     |

* Both images hold the same binary and take the same configuration.
* Prefer the `-nonroot` image. An admission policy with `runAsNonRoot: true` and no `runAsUser` rejects the root image, because that image declares uid 0.
* Pin an explicit release tag, such as `v1.13.0`, or an image digest. Do not use `latest`.
* The images are `linux/amd64` images.
* The images are distroless and contain no shell. Run init containers and debug tools from a separate utility image.
* Releases before v1.6.0 used the repository name `public.ecr.aws/dlc-link/canton-decparty-manager`.

Neither image needs root. The binary binds ports above 1024 and writes only under its data directory. Give the pod `fsGroup: 65532`, so that uid 65532 can write to a new volume.

## Move an existing node to uid 65532

A volume that the root image wrote holds root-owned files. The node sets the mode of `data/noise.key` to 0600 at every start when the mode differs. `fsGroup` adds group permissions, so the mode differs. Uid 65532 cannot change the mode of a file that it does not own. The node then logs `Failed to chmod key file` and keeps running without the Noise listener. A node without the Noise listener cannot peer.

Change the owner of the volume once, before the first start as uid 65532. On Kubernetes, add a temporary root init container ahead of the other init containers. This example mounts the volume at `/app`:

```yaml
initContainers:
  - name: chown-data
    image: busybox:latest
    command: ["sh", "-c", "chown -R 65532:65532 /app"]
    securityContext:
      runAsUser: 0
      runAsNonRoot: false
    volumeMounts:
      - name: data
        mountPath: /app
```

Remove the init container after one successful start. A namespace under the `restricted` Pod Security Standard rejects a root init container. In that case, run the same `chown -R` from a one-off root pod that mounts the volume. For a Docker bind mount, run `sudo chown -R 65532:65532` on the host directory.

## Required controls

* Pin a release tag or an image digest.
* Run as uid 65532, with `allowPrivilegeEscalation: false` and all capabilities dropped.
* Run one replica per node. The repository manifest uses `replicas: 1` with a `ReadWriteOnce` volume for the Noise key and the SQLite database.
* Never set `--insecure` or `DECPM_INSECURE`. Insecure mode starts only on `devnet`. The node refuses to start in insecure mode on `testnet` or `mainnet`.
* Set `DECPM_CANTON_NETWORK` explicitly. The network defaults to `devnet` when `DECPM_CANTON_NETWORK` is unset.
* Configure Keycloak or Auth0 for the operator UI. See [Authentication and operator roles](/decentralization-manager/operations-and-reference/authentication-and-operator-roles.md).
* Set `DECPM_ADMIN_ROLE`. Without it, every authenticated user is an admin.
* Set `DECPM_DB_ENCRYPTION_KEY`. Without it, the node stores party credentials in plaintext.
* Terminate TLS for the HTTP listener at the ingress or proxy. The node itself serves plain HTTP.
* Use TLS or mTLS to the Canton APIs when the participant serves it. Otherwise, keep the Canton APIs on a private network.
* Keep the metrics port inside the private network, or set `DECPM_METRICS_PORT=0`.
* Set `DECPM_TENANT_API_KEYS` only on a node that hosts an external party or adds hosts to a local party.
* Keep the encryption key, the tenant API keys and the Canton client keys in a secret store, not in the image.
* Persist the data directory on a volume, and back it up.

{% hint style="warning" %}
A fresh node, with no stored party credentials, accepts the first `PUT /party-config` without a token. Keep the HTTP listener unreachable for other users until you save the first party configuration.
{% endhint %}

{% hint style="warning" %}
The metrics listener has no authentication and binds to `DECPM_HOST`. It answers any caller that reaches it. The HTTP listener binds to the same address. Do not publish port `9464` through an Ingress, a load balancer or a public security group.
{% endhint %}

The `/v0/tenant/*` endpoints accept a tenant API key instead of an identity-provider token. Wallet providers that call them must reach the HTTP listener.

## Health checks

`GET /healthz` on the HTTP port returns `200` with `{"status":"ok"}`. It needs no token and does no I/O. Use it as a liveness probe. It does not check Canton, the peers or the database.

The HTTP listener starts after the node reads its participant ID from the Admin API and runs the database migrations. If the Admin API is unreachable at startup, the process exits. The container then restarts.

The repository manifest defines no probes. This example adds them:

```yaml
startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 5
  failureThreshold: 60
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 10
```

Use `httpGet` probes. The image contains no shell for `exec` probes. `GET /node-health` reports whether the Admin API and the Ledger API are reachable. It needs a token.

## Deployment sequence

1. Choose the release tag and the image variant.
2. Provision the namespace, the secret store, the persistent volume and the network policy.
3. Configure the identity-provider client for the operator UI.
4. Create the HTTP Service behind the Ingress and the TCP load balancer for the Noise port.
5. Set `DECPM_PUBLIC_ADDRESS` to the Noise address, and deploy one replica.
6. Verify the login, `/healthz`, the Canton connection, persistence, metrics and logs.
7. On an AWS KMS participant, grant `kms:Sign` on the party keys. See [Sign with AWS KMS](/decentralization-manager/operations-and-reference/sign-with-aws-kms.md).
8. Save the first party configuration.
9. Exchange peer records with the other operators. See [Connect peers](/decentralization-manager/get-started/connect-peers.md).
10. Run one low-risk workflow, and record the deployment evidence.

The [repository deployment guide](https://github.com/DLC-link/decentralization-manager/blob/main/docs/DEPLOYMENT_GUIDE.md) has complete Kubernetes manifests and the identity-provider setup.

## Persistent data

The volume holds `data/noise.key` and `data/decpm.db`. A deleted volume gives the node a new Noise key. Every peer must then update its record for this node. A deleted volume also removes the peer list and the party credentials.

Back up the volume with an application-consistent procedure, and test the restore. Protect the backups as sensitive material. [Backup, recovery, and upgrades](/decentralization-manager/operations-and-reference/backup-recovery-and-upgrades.md) gives the procedure.

## Logs and metrics

The node writes one JSON object per line to stdout. `DECPM_LOG_FORMAT=text` gives the console format. `RUST_LOG` sets the log filter. The metrics listener serves Prometheus metrics at `/metrics` on `DECPM_METRICS_PORT`. [Monitoring and logging](/decentralization-manager/operations-and-reference/monitoring-and-logging.md) lists the metrics and the alerts.

## Upgrades

The node runs its SQLite migrations at startup. For most releases, a new image tag is the only upgrade step.

A coordinator invites only peers that use the same Noise wire version and report version `0.1.9` or later. All nodes of a Decentralized Party must still run the same release.

A release that changes the Noise wire version needs a lockstep upgrade of the whole mesh. Nodes on different wire versions reject each other's frames. A mixed mesh fails the health preflight of every workflow run.

| Wire change               | First release          | What changed                                                                                                                       |
| ------------------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| No version byte to `0xD1` | v1.4.0                 | Every frame starts with a protocol version byte, for concurrent workflows. Earlier releases, such as v1.0.0, have no version byte. |
| `0xD1` to `0xD2`          | v1.8.0                 | The ACS moves from an inline snapshot to block-by-block streaming.                                                                 |
| `0xD2` to `0xD3`          | Releases after v1.13.0 | Contracts invitations need deployment-intent commitments. Membership workflows end with a `RetireLegacyKeys` phase.                |

[Backup, recovery, and upgrades](/decentralization-manager/operations-and-reference/backup-recovery-and-upgrades.md) gives the lockstep upgrade procedure and the extra steps for the `0xD3` upgrade.

## Production checklist

* [ ] A pinned release tag or digest, preferably the `-nonroot` image
* [ ] One replica and a persistent volume owned by uid 65532
* [ ] The identity provider, `DECPM_ADMIN_ROLE` and operator roles
* [ ] `DECPM_DB_ENCRYPTION_KEY` in the secret store
* [ ] TLS at the ingress, and TLS or mTLS to Canton where the participant serves it
* [ ] A raw TCP path to the Noise port, and `DECPM_PUBLIC_ADDRESS` set
* [ ] The metrics port kept private, or turned off
* [ ] The first party configuration saved before the HTTP listener opens to other users
* [ ] `kms:Sign` granted on AWS KMS participants
* [ ] Liveness probes on `/healthz`, logs, metrics, alerts and resource limits
* [ ] A tested backup and restore
* [ ] A lockstep upgrade and rollback plan for wire-version releases
* [ ] Incident and support contacts

## Technical sources

* [Deployment guide](https://github.com/DLC-link/decentralization-manager/blob/main/docs/DEPLOYMENT_GUIDE.md)
* [Release Dockerfile](https://github.com/DLC-link/decentralization-manager/blob/main/Dockerfile)
* [Release workflow](https://github.com/DLC-link/decentralization-manager/blob/main/.github/workflows/release.yml)
* [Architecture: Noise protocol server](https://github.com/DLC-link/decentralization-manager/blob/main/docs/ARCHITECTURE.md)
* [Noise wire version and key file](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/noise/mod.rs)
* [Minimum peer version](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/consts.rs)
* [Server startup and listeners](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/server/mod.rs)
* [Liveness endpoint](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/server/handlers/config.rs)
* [Authentication middleware](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/server/middleware/auth.rs)
* [Signing with AWS KMS party keys](https://github.com/DLC-link/decentralization-manager/blob/main/docs/KMS_SIGNING.md)

## Related pages

* [Configuration reference](/decentralization-manager/operations-and-reference/configuration-reference.md)
* [Authentication and operator roles](/decentralization-manager/operations-and-reference/authentication-and-operator-roles.md)
* [Security architecture](/decentralization-manager/operations-and-reference/security-architecture.md)
* [Monitoring and logging](/decentralization-manager/operations-and-reference/monitoring-and-logging.md)
* [Backup, recovery, and upgrades](/decentralization-manager/operations-and-reference/backup-recovery-and-upgrades.md)
* [Compatibility, releases, and troubleshooting](/decentralization-manager/operations-and-reference/compatibility-releases-and-troubleshooting.md)

## Next step

👉 **Lock down operator access:** [Authentication and operator roles](/decentralization-manager/operations-and-reference/authentication-and-operator-roles.md).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.bitsafe.finance/decentralization-manager/get-started/production-deployment.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.