Monitoring and logging

A Decentralization Manager node reports its state in three ways:

• Health endpoints on the HTTP port.
• Prometheus metrics on a separate metrics port.
• Logs on standard output, in JSON by default.

Health endpoints

Route	Access	What it reports
GET /healthz	Public	Liveness only. The route returns {"status":"ok"} and does no I/O.
GET /node-health	Operator token	The health of each hop from the node to Canton.
GET /participants-status	Operator token	The connection status, latency, and version of each peer.

Liveness

/healthz needs no token and sends Cache-Control: no-store. Use it as the liveness probe of the container:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080

A 200 from /healthz proves only that the HTTP server answers. It does not check Canton or the peers.

Node health

GET /node-health returns these fields:

Field	Content
status	Healthy, Degraded, or Down
checked_at	The time of the probe, in RFC 3339 format
admin_api, ledger_api	reachable, latency_ms, and error for each Canton API
participant	Initialization, active state, uptime, Canton version, topology queue depths, and component states
synchronizers	Each connected synchronizer and whether its connection is healthy

The node derives status from the probes:

• Down: the Admin API or the Ledger API did not answer.
• Degraded: both APIs answered, and the participant reports a problem. Examples are an uninitialized or passive participant, a component that is not OK, or a topology queue deeper than 100. A missing or unhealthy synchronizer connection also counts.
• Healthy: both APIs answered, and the participant reports no problem.

The node serves the same snapshot for 5 seconds. When the metrics port is not 0 and does not collide with another listener, a background task also refreshes the snapshot every 30 seconds.

Peer status

GET /participants-status returns a statuses array. Each entry has id, status, latency_ms, workflow, version, and build_version. The status values are:

Value	Meaning
CurrentNode	This node.
Connected	The node reached the peer over Noise.
Unreachable	The TCP connection to the peer failed.
HandshakeFailed	The Noise handshake or the reply failed. One side probably stores a wrong public key, or the peer runs another wire version.

Metrics listener

The node serves Prometheus metrics in text format at /metrics on its own listener.

Setting	Value
Port	DECPM_METRICS_PORT or --metrics-port. The default is 9464.
Turn off	Set the port to 0.
Bind address	The same address as the HTTP server: DECPM_HOST or --host. The default is 0.0.0.0.
Path	/metrics
Authentication	None

Danger

Hardening. The metrics listener has no authentication. It answers every caller that can reach the port. The metric labels contain participant IDs and Decentralized Party IDs. Keep port 9464 off public networks. Do not add it to a public Service, load balancer, or Ingress. Block it in the host firewall or security group. Set DECPM_METRICS_PORT=0 on a node without a metrics collector.

The listener has no separate bind address. It binds to the same address as the HTTP server, DECPM_HOST. Use the network controls of your platform to keep it private.

The node does not stop when the listener cannot start. It logs one of these lines at ERROR and serves no metrics:

• metrics_port collides with another listener; this node reports no metrics. The metrics port equals the HTTP port or the Noise port.
• binding the metrics port failed; this node reports no metrics. Another process holds the port, for example a second local node.

On a successful start, the node logs Serving metrics on <host>:<port>/metrics. With port 0, it logs Metrics endpoint disabled (metrics_port = 0).

Scrape the listener

The deployment guide manifest uses Prometheus scrape annotations on the pod. It also declares the metrics port on the container:

template:
  metadata:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/path: "/metrics"
      prometheus.io/port: "9464"
  spec:
    containers:
      - name: dec-party-manager
        ports:
          - name: metrics
            containerPort: 9464
            protocol: TCP
        env:
          - name: DECPM_METRICS_PORT
            value: "9464"

Read the metrics by hand from a host on the same private network:

curl -s "http://<node-address>:9464/metrics" | grep '^decman_'

Metrics reference

The node registers only the metrics in this section. Every name starts with decman_. The node exports no process or runtime metrics.

Node health

Metric	Type	Labels	Meaning
decman_node_health_status	Gauge	None	Node health: 2 healthy, 1 degraded, 0 down.
decman_node_health_probed_at_seconds	Gauge	None	Unix time of the last health probe.
decman_canton_admin_api_latency_ms	Gauge	None	Round-trip time of the last Admin API probe, in milliseconds. NaN after a failed probe.
decman_canton_ledger_api_latency_ms	Gauge	None	Round-trip time of the last Ledger API probe, in milliseconds. NaN after a failed probe.

Peers

Metric	Type	Labels	Meaning
decman_peer_reachable	Gauge	peer	1 when this node can reach the peer, 0 when it cannot. The liveness check sets it every 5 seconds. Traffic from the peer in the last 15 seconds also counts as reachable.
decman_peer_ping_loop_total	Counter	None	Ticks of the liveness check, one every 5 seconds. A flat rate means that decman_peer_reachable is stale.

The peer label holds the participant ID of the peer. A peer that you remove from the peer list loses its series.

Workflow runs

Metric	Type	Labels	Meaning
decman_workflow_runs_started_total	Counter	kind	Runs that this node started as coordinator. A retry counts as a new start. A run that resumes after a restart does not.
decman_workflow_runs_finished_total	Counter	kind, outcome	Runs that this node coordinated and that ended.

The kind values are onboarding, kick, contracts, dars, add_party, and change_threshold. The outcome values are completed, failed, and cancelled. The counters cover coordinator runs only.

Governance

Metric	Type	Labels	Meaning
decman_governance_actions_total	Counter	step, outcome	Governance write attempts of this node.
decman_governance_pending	Gauge	decparty, state	Open governance actions at the last successful read.
decman_governance_oldest_pending_expires_in_seconds	Gauge	decparty	Seconds until the earliest live confirmation on an open action expires.
decman_governance_queue_truncated	Gauge	decparty	1 when the last successful read covered only part of the open proposals. 0 otherwise.
decman_governance_poll_total	Counter	outcome	Passes of the governance poller, one each minute.

Label values and rules:

• step is propose, confirm, execute, expire, cancel_confirmation, or cancel_proposal.
• outcome is ok or error. For decman_governance_actions_total, error covers failures before and during the ledger submission. A request that fails validation does not count.
• state is awaiting_approvals or executable. An executable action has enough confirmations.
• decparty holds the Decentralized Party ID.

The governance poller reads the queue of each party that has party credentials on this node. It reads up to 1,000 proposals for each party. With more open proposals, decman_governance_queue_truncated is 1 and decman_governance_pending is a lower bound. The expiry gauge is then absent.

The expiry gauge is also absent when no open action has a live confirmation. It keeps falling while reads fail. A failed read counts as outcome="error" in decman_governance_poll_total.

Reward automation

Metric	Type	Labels
decman_reward_heartbeat_total	Counter	None
decman_reward_sweep_total	Counter	decparty
decman_reward_coupons_assigned_total	Counter	decparty
decman_reward_coupons_skipped_total	Counter	decparty
decman_reward_sweep_zero_assigned_total	Counter	decparty
decman_reward_sweep_failed_total	Counter	decparty
decman_reward_expiry_read_total	Counter	decparty
decman_reward_credentials_unavailable_total	Counter	decparty
decman_reward_auth_unregistered_total	Counter	decparty
decman_reward_oldest_unassigned_expires_in_seconds	Gauge	decparty

Reward automation explains each reward metric.

Alert conditions

The metric descriptions in the code support these alert conditions:

• decman_node_health_status is below 2. Check that decman_node_health_probed_at_seconds is recent, so that a stale value cannot look healthy.
• decman_peer_reachable is 0 for a peer.
• decman_peer_ping_loop_total stops increasing.
• decman_workflow_runs_finished_total{outcome="failed"} increases.
• decman_governance_actions_total{outcome="error"} increases.
• decman_governance_oldest_pending_expires_in_seconds falls below a margin that you choose.
• decman_governance_poll_total{outcome="error"} increases, or decman_governance_poll_total stops increasing.
• decman_reward_heartbeat_total stops increasing on a node that assigns rewards.
• Every decman_ series disappears while the node still answers /healthz.

Logs

Log format

The node writes one JSON object per line to standard output. The object has these keys:

Key	Content
timestamp	The event time
level	ERROR, WARN, INFO, DEBUG, or TRACE
target	The Rust module that logged the event
fields	The event fields. fields.message holds the message text

The code convention records values as separate fields, not inside the message text. A log pipeline that parses JSON can then query each field. Numeric fields stay numbers.

Set DECPM_LOG_FORMAT=text for the readable console format during local work. The comparison ignores case and surrounding spaces. Every other value, and an unset variable, gives JSON. Leave the variable unset when a log pipeline parses JSON.

Log levels

RUST_LOG sets the log levels. It uses the filter syntax of the Rust tracing library. Without RUST_LOG, the node uses this filter:

dec_party_manager=info,tokio_noise=error,hyper_noise=error

To see debug output from the node, set:

RUST_LOG=dec_party_manager=debug,tokio_noise=error,hyper_noise=error

The default filter keeps the tokio_noise and hyper_noise libraries at error.

Log lines to watch

Level	Message	Meaning
INFO	Noise public key: <hex>	The Noise public key that the node uses. Compare it with the key that the peers store.
INFO	No Noise keypair found, generating new one	The node created a new Noise identity. Peers must update their records.
WARN	INSECURE MODE ENABLED: …	Insecure mode is on. Authentication is effectively off.
WARN	peer became unreachable	A peer stopped answering. The peer and error fields name the peer and the cause.
INFO	peer became reachable	A peer answers again.
ERROR	<task> loop returned; <consequence> or <task> task died; <consequence>	A background task stopped. The task field names it. The node does not restart the task.
ERROR	dec-party-manager exited with an error	The process stopped with a fatal error. The error field holds the cause.
ERROR	dec-party-manager panicked	The process panicked. The panic field holds the details.

The background tasks are heartbeat, peer listener, noise listener, reward automation, governance poller, metrics server, node health refresh, and ACS relay session reaper. Restart the node to bring a stopped task back.

The node logs a fatal error and a panic through the same JSON format. A log pipeline can count them like any other ERROR line.

Note

Treat logs as operational data with access control. Log lines contain participant IDs, party IDs, and contract IDs.

Troubleshooting

Symptom	Cause	Action
The metrics port does not answer	The port is 0, collides with another listener, or could not bind.	Read the startup log for the metrics lines above.
Two local nodes report one set of metrics	Both nodes use the default port 9464. The second bind fails.	Give each node its own DECPM_METRICS_PORT, or set 0.
Every decman_ series is empty, and /healthz answers	The metrics server task stopped.	Read the task log line. Restart the node.
decman_peer_reachable does not change	The heartbeat task stopped. decman_peer_ping_loop_total is flat.	Restart the node.
The governance gauges do not change	The governance poller task stopped, or reads fail.	Check decman_governance_poll_total and the task log line.
decman_node_health_status is 0	The Admin API or the Ledger API does not answer.	Read admin_api.error and ledger_api.error in GET /node-health.
The log pipeline shows plain text lines	DECPM_LOG_FORMAT=text is set.	Unset DECPM_LOG_FORMAT.

Technical sources

• Logging setup
• Command-line flags
• Metrics listener and background tasks
• Health and metrics handlers
• Node health probes
• Peer reachability
• Governance poller
• Governance action counter
• Workflow run counters
• Reward automation metrics
• Deployment guide
• Contributing guide, logging and metrics

Related pages

• Production deployment
• Configuration reference
• Reward automation
• Security architecture
• Compatibility, releases, and troubleshooting

Next step

→ Plan backups and upgrades: Backup, recovery, and upgrades.