> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cosmos.network/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring

> How a validator operator turns on the CometBFT Prometheus endpoint, alerts on the metrics that mean missed blocks, keeps logs under control, and checks signing after every restart.

Commands on this page were run on gaia v28.3.1 (Cosmos SDK v0.53.4, CometBFT v0.38.23) on 2026-10-15 by agent testing on a local devnet.

This page is for an operator whose validator is signing behind sentries ([Validator security](https://docs.cosmos.network/hub/latest/validators/security.md)). At the end Prometheus scrapes the validator, and a short set of alert rules pages you before the downtime window closes. Journald keeps logs bounded, and you have a fixed checklist to run after every restart.

## Enable the CometBFT metrics endpoint

```bash theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
gaiad config set config instrumentation.prometheus true --skip-validate
```

The `config.toml` key is `prometheus` in the `[instrumentation]` table (default `false`). The listener is `prometheus_listen_addr = ":26660"`. The metric namespace is `cometbft`, so every metric below is exported as `cometbft_<subsystem>_<name>`. On gaia v28.3.1 `gaiad config set config` needs `--skip-validate`. Confirm with `gaiad config get config instrumentation.prometheus`, then restart the service.

```bash theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
curl -s localhost:26660/metrics | grep '^cometbft_consensus_height'
```

The command prints `cometbft_consensus_height{chain_id="cosmoshub-4"} <height>`. The endpoint binds all interfaces by default. Firewall 26660 to your Prometheus host, or set `prometheus_listen_addr` to a private address.

Application metrics (bank, staking, Cosmos SDK telemetry) are separate and optional. They need `gaiad config set app telemetry.enabled true`, `gaiad config set app telemetry.prometheus-retention-time 60`, and `gaiad config set app api.enable true`. They are served by the API server at `http://localhost:1317/metrics?format=prometheus`. Keep the API bound to localhost on a validator.

## Metrics that matter for a validator

All names have the `cometbft_` prefix and a `chain_id` label. The full list is in the [CometBFT metrics reference](https://docs.cosmos.network/cometbft/latest/docs/core/metrics.md).

| Metric | What it tells you |
| - | - |
| `cometbft_consensus_height` | The height this node has committed. Compare with a public node or your sentries; a gap means you are behind. |
| `cometbft_consensus_validator_last_signed_height` | The last height this node's consensus key signed. Should trail `height` by at most a block or two. |
| `cometbft_consensus_validator_missed_blocks` | Blocks this node was expected to sign and did not, counted since start. Any increase is a missed signature. The series only exists after the first missed block. |
| `cometbft_consensus_validator_power` | This node's voting power. `0` means not in the active set, or signing with a key that is not the validator's. |
| `cometbft_consensus_validators` and `cometbft_consensus_missing_validators` | Validators in the set and how many missed the last block. A rising missing count network-wide points at a chain-level problem, not yours. |
| `cometbft_consensus_rounds` | Rounds needed to commit the last block. More than 0 for several blocks means the network is struggling to reach quorum. |
| `cometbft_consensus_block_interval_seconds` | Time between blocks, a histogram. The 2026-10-15 average was 5.7 seconds. |
| `cometbft_p2p_peers` | Connected peers. On the validator this should equal your sentry count; `0` means isolated. |
| `cometbft_mempool_size` | Pending transactions. Sustained growth with no decrease means the node cannot keep up. |
| `cometbft_blocksync_syncing` and `cometbft_statesync_syncing` | Listed in the CometBFT reference as `1` while syncing, but gaia v28.3.1 did not export either series on any node checked, including one that state synced. Use `catching_up` from `curl -s localhost:26657/status` to see whether a node has fallen behind. |
| `cometbft_state_block_processing_time` | Time to execute a block. Rising values warn of disk or CPU pressure before you start missing blocks. |

Add [node\_exporter](https://github.com/prometheus/node_exporter) on every host for disk, memory, and clock drift. A full disk is the most common reason a healthy validator stops.

## Scrape and alert

A minimal Prometheus scrape config:

```yaml theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
scrape_configs:
  - job_name: cosmoshub-validator
    static_configs:
      - targets: ['<validator-private-ip>:26660']
        labels: {role: validator}
      - targets: ['<sentry1-private-ip>:26660', '<sentry2-private-ip>:26660']
        labels: {role: sentry}
```

Alert rules, tuned to the Hub's 10,000-block window with the 5 percent floor. The window closes about 15 hours after signing stops, so page on minutes, not hours.

```yaml theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
groups:
  - name: cosmoshub-validator
    rules:
      - alert: ValidatorNotSigning
        expr: (cometbft_consensus_height{role="validator"} - ignoring(validator_address) cometbft_consensus_validator_last_signed_height{role="validator"}) > 5
        for: 1m
        labels: {severity: page}
        annotations: {summary: "Validator has not signed for more than 5 blocks"}
      - alert: ValidatorMissedBlocks
        expr: increase(cometbft_consensus_validator_missed_blocks{role="validator"}[10m]) > 10
        for: 0m
        labels: {severity: page}
        annotations: {summary: "More than 10 blocks missed in 10 minutes"}
      - alert: HeightStalled
        expr: increase(cometbft_consensus_height[2m]) == 0
        for: 2m
        labels: {severity: page}
        annotations: {summary: "No new block committed in 2 minutes"}
      - alert: VotingPowerZero
        expr: (up{role="validator"} == 1) unless on (instance) cometbft_consensus_validator_power
        for: 5m
        labels: {severity: page}
        annotations: {summary: "Validator has no voting power (jailed, unbonded, or wrong key)"}
      - alert: ValidatorFewPeers
        expr: cometbft_p2p_peers{role="validator"} < 2
        for: 5m
        labels: {severity: page}
        annotations: {summary: "Validator connected to fewer than 2 sentries"}
      - alert: NodeDown
        expr: up == 0
        for: 2m
        labels: {severity: page}
        annotations: {summary: "Metrics endpoint unreachable"}
      - alert: MempoolBacklog
        expr: cometbft_mempool_size > 5000
        for: 10m
        labels: {severity: warn}
        annotations: {summary: "Mempool backlog, node may be falling behind"}
      - alert: DiskLow
        expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} < 0.10
        for: 15m
        labels: {severity: warn}
        annotations: {summary: "Less than 10 percent disk free"}
```

Two of the expressions look odd on purpose. The `ValidatorNotSigning` rule needs `ignoring(validator_address)` because `cometbft_consensus_validator_last_signed_height` has a `validator_address` label that `cometbft_consensus_height` does not. Without it the subtraction returns nothing. The `VotingPowerZero` rule tests for an absent gauge instead of `== 0` because CometBFT stops exporting `cometbft_consensus_validator_power` (and `validator_last_signed_height`) when the validator leaves the set. As a result, `== 0` never matches. For the same reason `ValidatorNotSigning` only covers a node that is up and still in the set. The `HeightStalled` rule firing on the validator alone means your node stopped. Firing on validator and sentries together, it means the chain halted (an upgrade height, usually). Route `page` alerts to something that wakes you. See the Prometheus [alerting rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/) and [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) docs for routing and silences, and [Grafana](https://grafana.com/docs/grafana/latest/) for dashboards.

## A second signal from outside

Metrics from the validator cannot tell you the validator is down when the whole host is gone. Nor can `up == 0` tell you whether you are also being slashed. From a machine that is not part of the validator setup, poll the chain:

```bash theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
gaiad query slashing signing-info <cosmosvalcons> --node https://cosmos-rpc.publicnode.com:443 --output json
```

The `missed_blocks_counter` field is the chain's own count inside the current 10,000-block window. Alert when it rises over a few minutes. A `jailed_until` in the future means you are already jailed. Get your `cosmosvalcons` address with `gaiad comet show-address` on the host that holds the key, or from `gaiad query staking validator <cosmosvaloper>` (`consensus_pubkey`). Many operators also use a third-party validator alerting bot for the same data. Treat it as the third signal, not the first.

## Logs with journald

A systemd service ([Run gaiad as a service](https://docs.cosmos.network/hub/latest/node/service.md)) sends gaiad output to journald. The commands name the plain `gaiad` unit. Use `-u cosmovisor` if the node runs under the Cosmovisor unit.

```bash theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
journalctl -u gaiad -f
journalctl -u gaiad --since "1 hour ago" | grep -iE 'error|panic|missed|double'
```

Keep the journal bounded in `/etc/systemd/journald.conf`, then `systemctl restart systemd-journald`:

```ini theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
[Journal]
SystemMaxUse=4G
MaxRetentionSec=14day
```

The `journalctl --disk-usage` command shows what the journal holds. The `journalctl --vacuum-size=2G` command trims it immediately. See [journald.conf(5)](https://man7.org/linux/man-pages/man5/journald.conf.5.html) and [journalctl(1)](https://man7.org/linux/man-pages/man1/journalctl.1.html).

Log volume is set in `config.toml`. The default is `log_level = "info"`. The value `*:error,consensus:info,p2p:error` cuts noise while keeping what you need. Use `log_format = "plain"`, or `"json"` for a log shipper. Set them with `gaiad config set config log_level "<value>" --skip-validate`.

## After every restart

Run these in order each time gaiad or the signer restarts, before you walk away:

1. Run `curl -s localhost:26657/status | jq '.result.sync_info.catching_up'`. It prints `false`.
2. Run `curl -s localhost:26657/status | jq '.result.validator_info'`. It shows the expected `pub_key` and a non-zero `voting_power`.
3. The signer's state file (or `data/priv_validator_state.json` for a local key) shows a `height` that increases every few seconds.
4. Run `curl -s localhost:26660/metrics | grep -E '^cometbft_consensus_(height|validator_last_signed_height)[{ ]'`. The two values are within a block or two of each other.
5. Run `gaiad query slashing signing-info <cosmosvalcons> --output json | jq .val_signing_info.missed_blocks_counter`. It prints `null` (the JSON output omits zero fields) or a number that stops increasing.
6. Run `journalctl -u gaiad --since "5 minutes ago" | grep -ciE 'panic|found signature from the same key'`. It prints `0`. An exit with `found signature from the same key` (the `double_sign_check_height` check) means the key signed recently somewhere. Stop and find out where before starting again.

## Verify

* Prometheus shows the validator and sentry targets as `UP`.
* Stopping the node on the testnet fires `NodeDown` within two minutes, and the chain-side `missed_blocks_counter` rises. The `ValidatorMissedBlocks` alert fires once the node is back, because the node counts the blocks it was expected to sign while it catches up. Stopping only the signer, with the node up and in the set, is the case `ValidatorNotSigning` is written for. Starting everything again clears the alerts.
* `journalctl --disk-usage` stays under the limit you set after a week.

## Next

Rehearse a governance upgrade: [Upgrade with Cosmovisor](https://docs.cosmos.network/hub/latest/hub-tutorials/upgrade-node.md). Keep the [Validator FAQ](https://docs.cosmos.network/hub/latest/validators/validator-faq.md) for the questions delegators ask.