> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cosmos.network/llms.txt
> Use this file to discover all available pages before exploring further.
# Telemetry
Gather relevant insights about your application and modules with custom metrics and telemetry.
## Overview
The `telemetry` package provides observability tooling for Cosmos SDK applications using [OpenTelemetry](https://opentelemetry.io/docs/). It offers a unified initialization point for traces, metrics, and logs via the OpenTelemetry declarative configuration API.
This package:
* Initializes OpenTelemetry SDK using YAML configuration files
* Provides backward compatibility with Cosmos SDK's legacy `go-metrics` wrapper API
* Includes built-in instrumentation for host, runtime, and disk I/O metrics
## Quick Start
### 1. Start a Local Telemetry Backend
```bash theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
docker run -p 3000:3000 -p 4317:4317 -p 4318:4318 --rm -ti grafana/otel-lgtm
```
### 2. Create Configuration File
Create an `otel.yaml` file:
```yaml theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
file_format: "1.0-rc.3"
resource:
attributes:
- name: service.name
value: my-cosmos-app
tracer_provider:
processors:
- batch:
exporter:
otlp_grpc:
endpoint: http://localhost:4317
meter_provider:
readers:
- pull:
exporter:
prometheus/development:
host: 0.0.0.0
port: 9464
logger_provider:
processors:
- batch:
exporter:
otlp_grpc:
endpoint: http://localhost:4317
extensions:
instruments:
host: {}
runtime: {}
diskio: {}
propagators:
- tracecontext
```
### 3. Initialize Telemetry
**Option A: Environment Variable (Recommended)**
Set `OTEL_EXPERIMENTAL_CONFIG_FILE` to your config path. This initializes the SDK before any meters/tracers are created, avoiding atomic load overhead.
```bash theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
export OTEL_EXPERIMENTAL_CONFIG_FILE=/path/to/otel.yaml
```
**Option B: Node Config Directory**
An empty `otel.yaml` will now be generated in `~/./config/`. Place the desired configuration in `otel.yaml`.
**Option C: Programmatic Initialization**
The SDK will first attempt to initialize via env var, then using the config in the node's home directory.
You may optionally initialize telemetry yourself using the `telemetry.InitializeOpenTelemetry` function:
```go theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
err := telemetry.InitializeOpenTelemetry("/path/to/otel.yaml")
if err != nil {
log.Fatal(err)
}
defer telemetry.Shutdown(context.Background())
```
## Configuration
### OpenTelemetry Configuration
The package uses the [OpenTelemetry declarative configuration spec](https://opentelemetry.io/docs/languages/sdk-configuration/declarative-configuration/). Key sections:
| Section | Purpose |
| ----------------- | ------------------------------- |
| `resource` | Service identity and attributes |
| `tracer_provider` | Trace export configuration |
| `meter_provider` | Metrics export configuration |
| `logger_provider` | Log export configuration |
For examples containing available options, see the [OpenTelemetry configuration examples](https://github.com/open-telemetry/opentelemetry-configuration/tree/main/examples).
### Extensions
The `extensions` section of the `otel.yaml` configuration file provides additional features not yet supported by the standard otelconf:
```yaml theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
extensions:
# Optional file-based exporters
trace_file: "/path/to/traces.json"
metrics_file: "/path/to/metrics.json"
metrics_file_interval: "10s"
logs_file: "/path/to/logs.json"
# Custom instrumentation additions
instruments:
host: {}
runtime: {}
diskio:
disable_virtual_device_filter: true # removes the automatic filtering of virtual disks. Operating systems such as Linux typically add virtual disks, which can add duplication to disk io data. These disks usually take the form of loopback, RAID, partitions, etc.
# Trace context propagation
propagators:
- tracecontext
- baggage
- b3
- jaeger
```
## Custom Instruments
### Host Instrumentation (`host`)
Reports host-level metrics using `go.opentelemetry.io/contrib/instrumentation/host`:
* CPU usage
* Memory usage
* Network I/O
```yaml theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
extensions:
instruments:
host: {}
```
### Runtime Instrumentation (`runtime`)
Reports Go runtime metrics using `go.opentelemetry.io/contrib/instrumentation/runtime`:
* Goroutine count
* GC statistics
* Memory allocations
```yaml theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
extensions:
instruments:
runtime: {}
```
### Disk I/O Instrumentation (`diskio`)
Reports disk I/O metrics using gopsutil:
| Metric | Description |
| ---------------------------- | ----------------------------- |
| `system.disk.io` | Bytes read/written |
| `system.disk.operations` | Read/write operation counts |
| `system.disk.io_time` | Time spent on I/O operations |
| `system.disk.operation_time` | Time per read/write operation |
| `system.disk.merged` | Merged read/write operations |
```yaml theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
extensions:
instruments:
diskio: {}
# Or with options:
diskio:
disable_virtual_device_filter: true # Include loopback, RAID, partitions on Linux
```
By default, virtual devices (loopback, RAID, partitions) are filtered out on Linux to avoid double-counting I/O.
## Propagators
Configure trace context propagation for distributed tracing:
| Propagator | Description |
| -------------- | --------------------------- |
| `tracecontext` | W3C Trace Context (default) |
| `baggage` | W3C Baggage |
| `b3` | Zipkin B3 single header |
| `b3multi` | Zipkin B3 multi-header |
| `jaeger` | Jaeger propagation |
## Developer Usage
### Using Meters and Tracers
After initialization, use standard OpenTelemetry APIs:
```go theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
import (
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/metric"
)
var (
tracer = otel.Tracer("my-package")
meter = otel.Meter("my-package")
myCounter metric.Int64Counter
)
func init() {
var err error
myCounter, err = meter.Int64Counter("my.counter")
if err != nil {
panic(err)
}
}
func MyFunction(ctx context.Context) error {
ctx, span := tracer.Start(ctx, "MyFunction")
defer span.End()
myCounter.Add(ctx, 1)
// ... your code
return nil
}
```
### Shutdown
Always call `Shutdown()` when the application exits:
```go theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
func (a *App) Close() {
telemetry.Shutdown(ctx)
}
```
## Legacy API (Deprecated)
The package provides backward-compatible wrappers for `github.com/hashicorp/go-metrics`. These are **deprecated** and users should migrate to OpenTelemetry APIs directly.
### OpenTelemetry Bridge
Cosmos SDK v0.54.0+ provides a bridge to send existing go-metrics to the meter provider defined in your OpenTelemetry config.
To bridge your metrics, set the `metrics-sink` in `app.toml` to "otel".
```go theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
###############################################################################
### Telemetry Configuration ###
###############################################################################
[telemetry]
# other fields...
metrics-sink = "otel"
```
### Legacy Configuration
```go theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
cfg := telemetry.Config{
ServiceName: "my-service",
Enabled: true,
EnableHostname: true,
EnableHostnameLabel: true,
EnableServiceLabel: true,
PrometheusRetentionTime: 60, // seconds
GlobalLabels: [][]string{{"chain_id", "cosmoshub-1"}},
MetricsSink: "otel", // "mem", "statsd", "dogstatsd", "otel"
StatsdAddr: "localhost:8125",
}
m, err := telemetry.New(cfg)
```
### Legacy Metrics Functions
All are deprecated; prefer OpenTelemetry:
```go theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
// Counters
telemetry.IncrCounter(1.0, "tx", "count")
telemetry.IncrCounterWithLabels([]string{"tx", "count"}, 1.0, labels)
// Gauges
telemetry.SetGauge(42.0, "mempool", "size")
telemetry.SetGaugeWithLabels([]string{"mempool", "size"}, 42.0, labels)
// Timing
start := telemetry.Now()
// ... operation
telemetry.MeasureSince(start, "tx", "process_time")
// Module-specific helpers
telemetry.ModuleMeasureSince("bank", start, "send", "time")
telemetry.ModuleSetGauge("bank", 100.0, "balance", "total")
```
## Metrics Sink Types
| Sink | Description |
| ----------- | --------------------------------------------------- |
| `mem` | In-memory sink with SIGUSR1 dump support (default) |
| `statsd` | StatsD protocol |
| `dogstatsd` | Datadog DogStatsD |
| `otel` | OpenTelemetry (bridges to configured MeterProvider) |
## Best Practices
1. **Use environment variable initialization** for production to avoid atomic load overhead
2. **Always call `Shutdown()`** to ensure metrics/traces are flushed
3. **Thread `context.Context`** properly for correct span correlation
## Viewing Telemetry Data
With Grafana LGTM running:
1. Open [http://localhost:3000](http://localhost:3000)
2. Use the Drilldown views to explore:
* **Traces**: Distributed trace visualization
* **Metrics**: Query and dashboard metrics
* **Logs**: Structured log search
## Related Documentation
* [OpenTelemetry Go SDK](https://opentelemetry.io/docs/languages/go/)
* [OpenTelemetry Configuration Spec](https://opentelemetry.io/docs/languages/sdk-configuration/declarative-configuration/)
* [otelconf Go Package](https://pkg.go.dev/go.opentelemetry.io/contrib/otelconf)
## Cosmos SDK Metrics
The following metrics are emitted from the Cosmos SDK.
| Metric | Description | Unit | Type |
| :------------------------------ | :---------------------------------------------------------------------------------------- | :-------------- | :------ |
| `tx_count` | Total number of txs processed via `FinalizeBlock` | tx | counter |
| `tx_successful` | Total number of successful txs processed via `FinalizeBlock` | tx | counter |
| `tx_failed` | Total number of failed txs processed via `FinalizeBlock` | tx | counter |
| `tx_gas_used` | The total amount of gas used by a tx | gas | gauge |
| `tx_gas_wanted` | The total amount of gas requested by a tx | gas | gauge |
| `tx_msg_send` | The total amount of tokens sent in a `MsgSend` (per denom) | token | gauge |
| `tx_msg_withdraw_reward` | The total amount of tokens withdrawn in a `MsgWithdrawDelegatorReward` (per denom) | token | gauge |
| `tx_msg_withdraw_commission` | The total amount of tokens withdrawn in a `MsgWithdrawValidatorCommission` (per denom) | token | gauge |
| `tx_msg_delegate` | The total amount of tokens delegated in a `MsgDelegate` | token | gauge |
| `tx_msg_begin_unbonding` | The total amount of tokens undelegated in a `MsgUndelegate` | token | gauge |
| `tx_msg_begin_begin_redelegate` | The total amount of tokens redelegated in a `MsgBeginRedelegate` | token | gauge |
| `tx_msg_ibc_transfer` | The total amount of tokens transferred via IBC in a `MsgTransfer` (source or sink chain) | token | gauge |
| `ibc_transfer_packet_receive` | The total amount of tokens received in a `FungibleTokenPacketData` (source or sink chain) | token | gauge |
| `new_account` | Total number of new accounts created | account | counter |
| `gov_proposal` | Total number of governance proposals | proposal | counter |
| `gov_vote` | Total number of governance votes for a proposal | vote | counter |
| `gov_deposit` | Total number of governance deposits for a proposal | deposit | counter |
| `staking_delegate` | Total number of delegations | delegation | counter |
| `staking_undelegate` | Total number of undelegations | undelegation | counter |
| `staking_redelegate` | Total number of redelegations | redelegation | counter |
| `ibc_transfer_send` | Total number of IBC transfers sent from a chain (source or sink) | transfer | counter |
| `ibc_transfer_receive` | Total number of IBC transfers received to a chain (source or sink) | transfer | counter |
| `ibc_client_create` | Total number of clients created | create | counter |
| `ibc_client_update` | Total number of client updates | update | counter |
| `ibc_client_upgrade` | Total number of client upgrades | upgrade | counter |
| `ibc_client_misbehaviour` | Total number of client misbehaviors | misbehaviour | counter |
| `ibc_connection_open-init` | Total number of connection `OpenInit` handshakes | handshake | counter |
| `ibc_connection_open-try` | Total number of connection `OpenTry` handshakes | handshake | counter |
| `ibc_connection_open-ack` | Total number of connection `OpenAck` handshakes | handshake | counter |
| `ibc_connection_open-confirm` | Total number of connection `OpenConfirm` handshakes | handshake | counter |
| `ibc_channel_open-init` | Total number of channel `OpenInit` handshakes | handshake | counter |
| `ibc_channel_open-try` | Total number of channel `OpenTry` handshakes | handshake | counter |
| `ibc_channel_open-ack` | Total number of channel `OpenAck` handshakes | handshake | counter |
| `ibc_channel_open-confirm` | Total number of channel `OpenConfirm` handshakes | handshake | counter |
| `ibc_channel_close-init` | Total number of channel `CloseInit` handshakes | handshake | counter |
| `ibc_channel_close-confirm` | Total number of channel `CloseConfirm` handshakes | handshake | counter |
| `tx_msg_ibc_recv_packet` | Total number of IBC packets received | packet | counter |
| `tx_msg_ibc_acknowledge_packet` | Total number of IBC packets acknowledged | acknowledgement | counter |
| `ibc_timeout_packet` | Total number of IBC timeout packets | timeout | counter |
| `store_iavl_get` | Duration of an IAVL `Store#Get` call | ms | summary |
| `store_iavl_set` | Duration of an IAVL `Store#Set` call | ms | summary |
| `store_iavl_has` | Duration of an IAVL `Store#Has` call | ms | summary |
| `store_iavl_delete` | Duration of an IAVL `Store#Delete` call | ms | summary |
| `store_iavl_commit` | Duration of an IAVL `Store#Commit` call | ms | summary |
| `store_iavl_query` | Duration of an IAVL `Store#Query` call | ms | summary |