Monitoring

This page is for an operator whose validator is signing behind sentries (Validator security). At the end Prometheus scrapes the validator, and a short set of alert rules pages you before the downtime window closes. Journald keeps logs bounded, and you have a fixed checklist to run after every restart.

Enable the CometBFT metrics endpoint

gaiad config set config instrumentation.prometheus true --skip-validate

The config.toml key is prometheus in the [instrumentation] table (default false). The listener is prometheus_listen_addr = ":26660". The metric namespace is cometbft, so every metric below is exported as cometbft_<subsystem>_<name>. On gaia v28.3.1 gaiad config set config needs --skip-validate. Confirm with gaiad config get config instrumentation.prometheus, then restart the service.

curl -s localhost:26660/metrics | grep '^cometbft_consensus_height'

The command prints cometbft_consensus_height{chain_id="cosmoshub-4"} <height>. The endpoint binds all interfaces by default. Firewall 26660 to your Prometheus host, or set prometheus_listen_addr to a private address. Application metrics (bank, staking, Cosmos SDK telemetry) are separate and optional. They need gaiad config set app telemetry.enabled true, gaiad config set app telemetry.prometheus-retention-time 60, and gaiad config set app api.enable true. They are served by the API server at http://localhost:1317/metrics?format=prometheus. Keep the API bound to localhost on a validator.

Metrics that matter for a validator

All names have the cometbft_ prefix and a chain_id label. The full list is in the CometBFT metrics reference.

Metric	What it tells you
`cometbft_consensus_height`	The height this node has committed. Compare with a public node or your sentries; a gap means you are behind.
`cometbft_consensus_validator_last_signed_height`	The last height this node’s consensus key signed. Should trail `height` by at most a block or two.
`cometbft_consensus_validator_missed_blocks`	Blocks this node was expected to sign and did not, counted since start. Any increase is a missed signature. The series only exists after the first missed block.
`cometbft_consensus_validator_power`	This node’s voting power. `0` means not in the active set, or signing with a key that is not the validator’s.
`cometbft_consensus_validators` and `cometbft_consensus_missing_validators`	Validators in the set and how many missed the last block. A rising missing count network-wide points at a chain-level problem, not yours.
`cometbft_consensus_rounds`	Rounds needed to commit the last block. More than 0 for several blocks means the network is struggling to reach quorum.
`cometbft_consensus_block_interval_seconds`	Time between blocks, a histogram. The 2026-10-15 average was 5.7 seconds.
`cometbft_p2p_peers`	Connected peers. On the validator this should equal your sentry count; `0` means isolated.
`cometbft_mempool_size`	Pending transactions. Sustained growth with no decrease means the node cannot keep up.
`cometbft_blocksync_syncing` and `cometbft_statesync_syncing`	Listed in the CometBFT reference as `1` while syncing, but gaia v28.3.1 did not export either series on any node checked, including one that state synced. Use `catching_up` from `curl -s localhost:26657/status` to see whether a node has fallen behind.
`cometbft_state_block_processing_time`	Time to execute a block. Rising values warn of disk or CPU pressure before you start missing blocks.

Add node_exporter on every host for disk, memory, and clock drift. A full disk is the most common reason a healthy validator stops.

Scrape and alert

A minimal Prometheus scrape config:

scrape_configs:
  - job_name: cosmoshub-validator
    static_configs:
      - targets: ['<validator-private-ip>:26660']
        labels: {role: validator}
      - targets: ['<sentry1-private-ip>:26660', '<sentry2-private-ip>:26660']
        labels: {role: sentry}

Alert rules, tuned to the Hub’s 10,000-block window with the 5 percent floor. The window closes about 15 hours after signing stops, so page on minutes, not hours.

groups:
  - name: cosmoshub-validator
    rules:
      - alert: ValidatorNotSigning
        expr: (cometbft_consensus_height{role="validator"} - ignoring(validator_address) cometbft_consensus_validator_last_signed_height{role="validator"}) > 5
        for: 1m
        labels: {severity: page}
        annotations: {summary: "Validator has not signed for more than 5 blocks"}
      - alert: ValidatorMissedBlocks
        expr: increase(cometbft_consensus_validator_missed_blocks{role="validator"}[10m]) > 10
        for: 0m
        labels: {severity: page}
        annotations: {summary: "More than 10 blocks missed in 10 minutes"}
      - alert: HeightStalled
        expr: increase(cometbft_consensus_height[2m]) == 0
        for: 2m
        labels: {severity: page}
        annotations: {summary: "No new block committed in 2 minutes"}
      - alert: VotingPowerZero
        expr: (up{role="validator"} == 1) unless on (instance) cometbft_consensus_validator_power
        for: 5m
        labels: {severity: page}
        annotations: {summary: "Validator has no voting power (jailed, unbonded, or wrong key)"}
      - alert: ValidatorFewPeers
        expr: cometbft_p2p_peers{role="validator"} < 2
        for: 5m
        labels: {severity: page}
        annotations: {summary: "Validator connected to fewer than 2 sentries"}
      - alert: NodeDown
        expr: up == 0
        for: 2m
        labels: {severity: page}
        annotations: {summary: "Metrics endpoint unreachable"}
      - alert: MempoolBacklog
        expr: cometbft_mempool_size > 5000
        for: 10m
        labels: {severity: warn}
        annotations: {summary: "Mempool backlog, node may be falling behind"}
      - alert: DiskLow
        expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.10
        for: 15m
        labels: {severity: warn}
        annotations: {summary: "Less than 10 percent disk free"}

Two of the expressions look odd on purpose. The ValidatorNotSigning rule needs ignoring(validator_address) because cometbft_consensus_validator_last_signed_height has a validator_address label that cometbft_consensus_height does not. Without it the subtraction returns nothing. The VotingPowerZero rule tests for an absent gauge instead of == 0 because CometBFT stops exporting cometbft_consensus_validator_power (and validator_last_signed_height) when the validator leaves the set. As a result, == 0 never matches. For the same reason ValidatorNotSigning only covers a node that is up and still in the set. The HeightStalled rule firing on the validator alone means your node stopped. Firing on validator and sentries together, it means the chain halted (an upgrade height, usually). Route page alerts to something that wakes you. See the Prometheus alerting rules and Alertmanager docs for routing and silences, and Grafana for dashboards.

A second signal from outside

Metrics from the validator cannot tell you the validator is down when the whole host is gone. Nor can up == 0 tell you whether you are also being slashed. From a machine that is not part of the validator setup, poll the chain:

gaiad query slashing signing-info <cosmosvalcons> --node https://cosmos-rpc.publicnode.com:443 --output json

The missed_blocks_counter field is the chain’s own count inside the current 10,000-block window. Alert when it rises over a few minutes. A jailed_until in the future means you are already jailed. Get your cosmosvalcons address with gaiad comet show-address on the host that holds the key, or from gaiad query staking validator <cosmosvaloper> (consensus_pubkey). Many operators also use a third-party validator alerting bot for the same data. Treat it as the third signal, not the first.

Logs with journald

A systemd service (Run gaiad as a service) sends gaiad output to journald. The commands name the plain gaiad unit. Use -u cosmovisor if the node runs under the Cosmovisor unit.

journalctl -u gaiad -f
journalctl -u gaiad --since "1 hour ago" | grep -iE 'error|panic|missed|double'

Keep the journal bounded in /etc/systemd/journald.conf, then systemctl restart systemd-journald:

[Journal]
SystemMaxUse=4G
MaxRetentionSec=14day

The journalctl --disk-usage command shows what the journal holds. The journalctl --vacuum-size=2G command trims it immediately. See journald.conf(5) and journalctl(1). Log volume is set in config.toml. The default is log_level = "info". The value *:error,consensus:info,p2p:error cuts noise while keeping what you need. Use log_format = "plain", or "json" for a log shipper. Set them with gaiad config set config log_level "<value>" --skip-validate.

After every restart

Run these in order each time gaiad or the signer restarts, before you walk away:

Run curl -s localhost:26657/status | jq '.result.sync_info.catching_up'. It prints false.
Run curl -s localhost:26657/status | jq '.result.validator_info'. It shows the expected pub_key and a non-zero voting_power.
The signer’s state file (or data/priv_validator_state.json for a local key) shows a height that increases every few seconds.
Run curl -s localhost:26660/metrics | grep -E '^cometbft_consensus_(height|validator_last_signed_height)[{ ]'. The two values are within a block or two of each other.
Run gaiad query slashing signing-info <cosmosvalcons> --output json | jq .val_signing_info.missed_blocks_counter. It prints null (the JSON output omits zero fields) or a number that stops increasing.
Run journalctl -u gaiad --since "5 minutes ago" | grep -ciE 'panic|found signature from the same key'. It prints 0. An exit with found signature from the same key (the double_sign_check_height check) means the key signed recently somewhere. Stop and find out where before starting again.

Verify

Prometheus shows the validator and sentry targets as UP.
Stopping the node on the testnet fires NodeDown within two minutes, and the chain-side missed_blocks_counter rises. The ValidatorMissedBlocks alert fires once the node is back, because the node counts the blocks it was expected to sign while it catches up. Stopping only the signer, with the node up and in the set, is the case ValidatorNotSigning is written for. Starting everything again clears the alerts.
journalctl --disk-usage stays under the limit you set after a week.

Rehearse a governance upgrade: Upgrade with Cosmovisor. Keep the Validator FAQ for the questions delegators ask.

Cosmos Hub

Enable the CometBFT metrics endpoint

Metrics that matter for a validator

Scrape and alert

A second signal from outside

Logs with journald

After every restart

Verify

Next

​Enable the CometBFT metrics endpoint

​Metrics that matter for a validator

​Scrape and alert

​A second signal from outside

​Logs with journald

​After every restart

​Verify

​Next

Enable the CometBFT metrics endpoint

Metrics that matter for a validator

Scrape and alert

A second signal from outside

Logs with journald

After every restart

Verify

Next