> For the complete documentation index, see [llms.txt](https://docs.bitsafe.finance/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.bitsafe.finance/decentralization-manager/operations-and-reference/monitoring-and-logging.md). # Monitoring and logging Health endpoints, the Prometheus metrics listener and its metrics, and the log format and log levels of a Decentralization Manager node. A Decentralization Manager node reports its state in three ways: * Health endpoints on the HTTP port. * Prometheus metrics on a separate metrics port. * Logs on standard output, in JSON by default. ## Health endpoints | Route | Access | What it reports | | -------------------------- | -------------- | ------------------------------------------------------------------- | | `GET /healthz` | Public | Liveness only. The route returns `{"status":"ok"}` and does no I/O. | | `GET /node-health` | Operator token | The health of each hop from the node to Canton. | | `GET /participants-status` | Operator token | The connection status, latency, and version of each peer. | ### Liveness `/healthz` needs no token and sends `Cache-Control: no-store`. Use it as the liveness probe of the container: ```yaml livenessProbe: httpGet: path: /healthz port: 8080 ``` A `200` from `/healthz` proves only that the HTTP server answers. It does not check Canton or the peers. ### Node health `GET /node-health` returns these fields: | Field | Content | | ------------------------- | ------------------------------------------------------------------------------------------------- | | `status` | `Healthy`, `Degraded`, or `Down` | | `checked_at` | The time of the probe, in RFC 3339 format | | `admin_api`, `ledger_api` | `reachable`, `latency_ms`, and `error` for each Canton API | | `participant` | Initialization, active state, uptime, Canton version, topology queue depths, and component states | | `synchronizers` | Each connected synchronizer and whether its connection is healthy | The node derives `status` from the probes: * `Down`: the Admin API or the Ledger API did not answer. * `Degraded`: both APIs answered, and the participant reports a problem. Examples are an uninitialized or passive participant, a component that is not OK, or a topology queue deeper than 100. A missing or unhealthy synchronizer connection also counts. * `Healthy`: both APIs answered, and the participant reports no problem. The node serves the same snapshot for 5 seconds. When the metrics port is not `0` and does not collide with another listener, a background task also refreshes the snapshot every 30 seconds. ### Peer status `GET /participants-status` returns a `statuses` array. Each entry has `id`, `status`, `latency_ms`, `workflow`, `version`, and `build_version`. The `status` values are: | Value | Meaning | | ----------------- | ---------------------------------------------------------------------------------------------------------------------------- | | `CurrentNode` | This node. | | `Connected` | The node reached the peer over Noise. | | `Unreachable` | The TCP connection to the peer failed. | | `HandshakeFailed` | The Noise handshake or the reply failed. One side probably stores a wrong public key, or the peer runs another wire version. | ## Metrics listener The node serves Prometheus metrics in text format at `/metrics` on its own listener. | Setting | Value | | -------------- | ---------------------------------------------------------------------------------------- | | Port | `DECPM_METRICS_PORT` or `--metrics-port`. The default is `9464`. | | Turn off | Set the port to `0`. | | Bind address | The same address as the HTTP server: `DECPM_HOST` or `--host`. The default is `0.0.0.0`. | | Path | `/metrics` | | Authentication | None | {% hint style="danger" %} **Hardening.** The metrics listener has no authentication. It answers every caller that can reach the port. The metric labels contain participant IDs and Decentralized Party IDs. Keep port `9464` off public networks. Do not add it to a public Service, load balancer, or Ingress. Block it in the host firewall or security group. Set `DECPM_METRICS_PORT=0` on a node without a metrics collector. {% endhint %} The listener has no separate bind address. It binds to the same address as the HTTP server, `DECPM_HOST`. Use the network controls of your platform to keep it private. The node does not stop when the listener cannot start. It logs one of these lines at `ERROR` and serves no metrics: * `metrics_port collides with another listener; this node reports no metrics`. The metrics port equals the HTTP port or the Noise port. * `binding the metrics port failed; this node reports no metrics`. Another process holds the port, for example a second local node. On a successful start, the node logs `Serving metrics on :/metrics`. With port `0`, it logs `Metrics endpoint disabled (metrics_port = 0)`. ### Scrape the listener The deployment guide manifest uses Prometheus scrape annotations on the pod. It also declares the metrics port on the container: ```yaml template: metadata: annotations: prometheus.io/scrape: "true" prometheus.io/path: "/metrics" prometheus.io/port: "9464" spec: containers: - name: dec-party-manager ports: - name: metrics containerPort: 9464 protocol: TCP env: - name: DECPM_METRICS_PORT value: "9464" ``` Read the metrics by hand from a host on the same private network: ```bash curl -s "http://:9464/metrics" | grep '^decman_' ``` ## Metrics reference The node registers only the metrics in this section. Every name starts with `decman_`. The node exports no process or runtime metrics. ### Node health | Metric | Type | Labels | Meaning | | -------------------------------------- | ----- | ------ | ------------------------------------------------------------------------------------------ | | `decman_node_health_status` | Gauge | None | Node health: `2` healthy, `1` degraded, `0` down. | | `decman_node_health_probed_at_seconds` | Gauge | None | Unix time of the last health probe. | | `decman_canton_admin_api_latency_ms` | Gauge | None | Round-trip time of the last Admin API probe, in milliseconds. `NaN` after a failed probe. | | `decman_canton_ledger_api_latency_ms` | Gauge | None | Round-trip time of the last Ledger API probe, in milliseconds. `NaN` after a failed probe. | ### Peers | Metric | Type | Labels | Meaning | | ----------------------------- | ------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `decman_peer_reachable` | Gauge | `peer` | `1` when this node can reach the peer, `0` when it cannot. The liveness check sets it every 5 seconds. Traffic from the peer in the last 15 seconds also counts as reachable. | | `decman_peer_ping_loop_total` | Counter | None | Ticks of the liveness check, one every 5 seconds. A flat rate means that `decman_peer_reachable` is stale. | The `peer` label holds the participant ID of the peer. A peer that you remove from the peer list loses its series. ### Workflow runs | Metric | Type | Labels | Meaning | | ------------------------------------- | ------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------- | | `decman_workflow_runs_started_total` | Counter | `kind` | Runs that this node started as coordinator. A retry counts as a new start. A run that resumes after a restart does not. | | `decman_workflow_runs_finished_total` | Counter | `kind`, `outcome` | Runs that this node coordinated and that ended. | The `kind` values are `onboarding`, `kick`, `contracts`, `dars`, `add_party`, and `change_threshold`. The `outcome` values are `completed`, `failed`, and `cancelled`. The counters cover coordinator runs only. ### Governance | Metric | Type | Labels | Meaning | | ----------------------------------------------------- | ------- | ------------------- | ----------------------------------------------------------------------------------------- | | `decman_governance_actions_total` | Counter | `step`, `outcome` | Governance write attempts of this node. | | `decman_governance_pending` | Gauge | `decparty`, `state` | Open governance actions at the last successful read. | | `decman_governance_oldest_pending_expires_in_seconds` | Gauge | `decparty` | Seconds until the earliest live confirmation on an open action expires. | | `decman_governance_queue_truncated` | Gauge | `decparty` | `1` when the last successful read covered only part of the open proposals. `0` otherwise. | | `decman_governance_poll_total` | Counter | `outcome` | Passes of the governance poller, one each minute. | Label values and rules: * `step` is `propose`, `confirm`, `execute`, `expire`, `cancel_confirmation`, or `cancel_proposal`. * `outcome` is `ok` or `error`. For `decman_governance_actions_total`, `error` covers failures before and during the ledger submission. A request that fails validation does not count. * `state` is `awaiting_approvals` or `executable`. An executable action has enough confirmations. * `decparty` holds the Decentralized Party ID. The governance poller reads the queue of each party that has party credentials on this node. It reads up to 1,000 proposals for each party. With more open proposals, `decman_governance_queue_truncated` is `1` and `decman_governance_pending` is a lower bound. The expiry gauge is then absent. The expiry gauge is also absent when no open action has a live confirmation. It keeps falling while reads fail. A failed read counts as `outcome="error"` in `decman_governance_poll_total`. ### Reward automation | Metric | Type | Labels | | ---------------------------------------------------- | ------- | ---------- | | `decman_reward_heartbeat_total` | Counter | None | | `decman_reward_sweep_total` | Counter | `decparty` | | `decman_reward_coupons_assigned_total` | Counter | `decparty` | | `decman_reward_coupons_skipped_total` | Counter | `decparty` | | `decman_reward_sweep_zero_assigned_total` | Counter | `decparty` | | `decman_reward_sweep_failed_total` | Counter | `decparty` | | `decman_reward_expiry_read_total` | Counter | `decparty` | | `decman_reward_credentials_unavailable_total` | Counter | `decparty` | | `decman_reward_auth_unregistered_total` | Counter | `decparty` | | `decman_reward_oldest_unassigned_expires_in_seconds` | Gauge | `decparty` | [Reward automation](/decentralization-manager/operations-and-reference/reward-automation.md#reward-metrics) explains each reward metric. ### Alert conditions The metric descriptions in the code support these alert conditions: * `decman_node_health_status` is below `2`. Check that `decman_node_health_probed_at_seconds` is recent, so that a stale value cannot look healthy. * `decman_peer_reachable` is `0` for a peer. * `decman_peer_ping_loop_total` stops increasing. * `decman_workflow_runs_finished_total{outcome="failed"}` increases. * `decman_governance_actions_total{outcome="error"}` increases. * `decman_governance_oldest_pending_expires_in_seconds` falls below a margin that you choose. * `decman_governance_poll_total{outcome="error"}` increases, or `decman_governance_poll_total` stops increasing. * `decman_reward_heartbeat_total` stops increasing on a node that assigns rewards. * Every `decman_` series disappears while the node still answers `/healthz`. ## Logs ### Log format The node writes one JSON object per line to standard output. The object has these keys: | Key | Content | | ----------- | --------------------------------------------------------- | | `timestamp` | The event time | | `level` | `ERROR`, `WARN`, `INFO`, `DEBUG`, or `TRACE` | | `target` | The Rust module that logged the event | | `fields` | The event fields. `fields.message` holds the message text | The code convention records values as separate fields, not inside the message text. A log pipeline that parses JSON can then query each field. Numeric fields stay numbers. Set `DECPM_LOG_FORMAT=text` for the readable console format during local work. The comparison ignores case and surrounding spaces. Every other value, and an unset variable, gives JSON. Leave the variable unset when a log pipeline parses JSON. ### Log levels `RUST_LOG` sets the log levels. It uses the filter syntax of the Rust `tracing` library. Without `RUST_LOG`, the node uses this filter: ``` dec_party_manager=info,tokio_noise=error,hyper_noise=error ``` To see debug output from the node, set: ```bash RUST_LOG=dec_party_manager=debug,tokio_noise=error,hyper_noise=error ``` The default filter keeps the `tokio_noise` and `hyper_noise` libraries at `error`. ### Log lines to watch | Level | Message | Meaning | | ------- | -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | `INFO` | `Noise public key: ` | The Noise public key that the node uses. Compare it with the key that the peers store. | | `INFO` | `No Noise keypair found, generating new one` | The node created a new Noise identity. Peers must update their records. | | `WARN` | `INSECURE MODE ENABLED: …` | Insecure mode is on. Authentication is effectively off. | | `WARN` | `peer became unreachable` | A peer stopped answering. The `peer` and `error` fields name the peer and the cause. | | `INFO` | `peer became reachable` | A peer answers again. | | `ERROR` | ` loop returned; ` or ` task died; ` | A background task stopped. The `task` field names it. The node does not restart the task. | | `ERROR` | `dec-party-manager exited with an error` | The process stopped with a fatal error. The `error` field holds the cause. | | `ERROR` | `dec-party-manager panicked` | The process panicked. The `panic` field holds the details. | The background tasks are `heartbeat`, `peer listener`, `noise listener`, `reward automation`, `governance poller`, `metrics server`, `node health refresh`, and `ACS relay session reaper`. Restart the node to bring a stopped task back. The node logs a fatal error and a panic through the same JSON format. A log pipeline can count them like any other `ERROR` line. {% hint style="info" %} Treat logs as operational data with access control. Log lines contain participant IDs, party IDs, and contract IDs. {% endhint %} ## Troubleshooting | Symptom | Cause | Action | | ------------------------------------------------------- | -------------------------------------------------------------------- | -------------------------------------------------------------------- | | The metrics port does not answer | The port is `0`, collides with another listener, or could not bind. | Read the startup log for the metrics lines above. | | Two local nodes report one set of metrics | Both nodes use the default port `9464`. The second bind fails. | Give each node its own `DECPM_METRICS_PORT`, or set `0`. | | Every `decman_` series is empty, and `/healthz` answers | The `metrics server` task stopped. | Read the task log line. Restart the node. | | `decman_peer_reachable` does not change | The `heartbeat` task stopped. `decman_peer_ping_loop_total` is flat. | Restart the node. | | The governance gauges do not change | The `governance poller` task stopped, or reads fail. | Check `decman_governance_poll_total` and the task log line. | | `decman_node_health_status` is `0` | The Admin API or the Ledger API does not answer. | Read `admin_api.error` and `ledger_api.error` in `GET /node-health`. | | The log pipeline shows plain text lines | `DECPM_LOG_FORMAT=text` is set. | Unset `DECPM_LOG_FORMAT`. | ## Technical sources * [Logging setup](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/main.rs) * [Command-line flags](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/cli.rs) * [Metrics listener and background tasks](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/server/mod.rs) * [Health and metrics handlers](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/server/handlers/config.rs) * [Node health probes](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/server/node_health.rs) * [Peer reachability](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/server/peer_status.rs) * [Governance poller](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/server/governance_poller.rs) * [Governance action counter](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/server/handlers/governance.rs) * [Workflow run counters](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/server/handlers/workflows.rs) * [Reward automation metrics](https://github.com/DLC-link/decentralization-manager/blob/main/crates/decman/src/server/reward_automation.rs) * [Deployment guide](https://github.com/DLC-link/decentralization-manager/blob/main/docs/DEPLOYMENT_GUIDE.md) * [Contributing guide, logging and metrics](https://github.com/DLC-link/decentralization-manager/blob/main/docs/CONTRIBUTING.md) ## Related pages * [Production deployment](/decentralization-manager/get-started/production-deployment.md) * [Configuration reference](/decentralization-manager/operations-and-reference/configuration-reference.md) * [Reward automation](/decentralization-manager/operations-and-reference/reward-automation.md) * [Security architecture](/decentralization-manager/operations-and-reference/security-architecture.md) * [Compatibility, releases, and troubleshooting](/decentralization-manager/operations-and-reference/compatibility-releases-and-troubleshooting.md) ## Next step 👉 **Plan backups and upgrades:** [Backup, recovery, and upgrades](/decentralization-manager/operations-and-reference/backup-recovery-and-upgrades.md). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.bitsafe.finance/decentralization-manager/operations-and-reference/monitoring-and-logging.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.