> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cosmos.network/llms.txt
> Use this file to discover all available pages before exploring further.

# Upgrade Handlers

> Understanding and performing coordinated chain upgrades

## Overview

Upgrade handlers enable coordinated on-chain upgrades across all validators at specific block heights via governance proposals. They provide a mechanism for chains to perform data migrations, parameter updates, and module upgrades in a deterministic way.

<Info>
  Upgrade handlers are critical for maintaining consensus during chain upgrades. All validators must run the same upgrade logic at the same height.
</Info>

## When to Use Upgrade Handlers

Upgrade handlers are required when:

* **Breaking state changes**: Modifying storage formats or data structures
* **Module migrations**: Updating module versions or parameters
* **Protocol upgrades**: Implementing new features that require state transitions
* **Data migrations**: Moving data between different storage locations

## Basic Structure

### Registering Upgrade Handlers

Upgrade handlers are registered in your app's `RegisterUpgradeHandlers()` method:

<Expandable title="View upgrade handler registration example">
  ```go theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
  // app/upgrades.go
  package app

  import (
      "context"

      upgradetypes "cosmossdk.io/x/upgrade/types"
      sdk "github.com/cosmos/cosmos-sdk/types"
      "github.com/cosmos/cosmos-sdk/types/module"
  )

  func (app *App) RegisterUpgradeHandlers() {
      app.UpgradeKeeper.SetUpgradeHandler(
          "v1.0.0", // upgrade name (must match governance proposal)
          func(ctx context.Context, plan upgradetypes.Plan, fromVM module.VersionMap) (module.VersionMap, error) {
              sdkCtx := sdk.UnwrapSDKContext(ctx)
              sdkCtx.Logger().Info("Starting upgrade", "name", plan.Name)

              // Run module migrations
              migrations, err := app.ModuleManager.RunMigrations(ctx, app.configurator, fromVM)
              if err != nil {
                  return nil, err
              }

              // Add custom migration logic here

              sdkCtx.Logger().Info("Upgrade complete", "name", plan.Name)
              return migrations, nil
          },
      )
  }
  ```
</Expandable>

### Module Version Management

The upgrade handler receives and returns a `module.VersionMap` that tracks module versions:

```go theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
// fromVM contains the module versions before the upgrade
// The returned VersionMap contains the new versions after migration
migrations, err := app.ModuleManager.RunMigrations(ctx, app.configurator, fromVM)
```

## Organizing Upgrade Code

For better maintainability, organize upgrades in separate packages:

### Directory Structure

```
app/
├── upgrades/
│   ├── v1_0_0/
│   │   ├── constants.go    # Upgrade name and configuration
│   │   ├── handler.go      # Main upgrade handler
│   │   └── migrations.go   # Migration logic
│   ├── v1_1_0/
│   │   ├── constants.go
│   │   ├── handler.go
│   │   └── migrations.go
│   └── types.go            # Shared types
└── upgrades.go             # RegisterUpgradeHandlers
```

### constants.go

<Expandable title="View upgrade constants definition">
  ```go theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
  // app/upgrades/v1_0_0/constants.go
  package v1_0_0

  import (
      storetypes "cosmossdk.io/store/types"
      "github.com/yourchain/app/upgrades"
  )

  const UpgradeName = "v1.0.0"

  var Upgrade = upgrades.Upgrade{
      UpgradeName:          UpgradeName,
      CreateUpgradeHandler: CreateUpgradeHandler,
      StoreUpgrades: storetypes.StoreUpgrades{
          Added:   []string{}, // New modules
          Deleted: []string{}, // Removed modules
      },
  }
  ```
</Expandable>

### handler.go

<Expandable title="View upgrade handler implementation">
  ```go theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
  // app/upgrades/v1_0_0/handler.go
  package v1_0_0

  import (
      "context"

      storetypes "cosmossdk.io/store/types"
      upgradetypes "cosmossdk.io/x/upgrade/types"
      sdk "github.com/cosmos/cosmos-sdk/types"
      "github.com/cosmos/cosmos-sdk/types/module"
  )

  func CreateUpgradeHandler(
      mm *module.Manager,
      configurator module.Configurator,
      keepers *upgrades.UpgradeKeepers,
      storeKeys map[string]*storetypes.KVStoreKey,
  ) upgradetypes.UpgradeHandler {
      return func(c context.Context, plan upgradetypes.Plan, vm module.VersionMap) (module.VersionMap, error) {
          ctx := sdk.UnwrapSDKContext(c)

          // Run module migrations
          vm, err := mm.RunMigrations(c, configurator, vm)
          if err != nil {
              return nil, err
          }

          // Custom migrations
          if err := runCustomMigrations(ctx, keepers, storeKeys); err != nil {
              return nil, err
          }

          return vm, nil
      }
  }
  ```
</Expandable>

## Migration Patterns

### Parameter Migrations

Migrating module parameters to new formats:

```go theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
func migrateParams(ctx sdk.Context, keeper paramskeeper.Keeper) error {
    // Get old params
    var oldParams v1.Params
    keeper.GetParamSet(ctx, &oldParams)

    // Convert to new format
    newParams := v2.Params{
        Field1: oldParams.Field1,
        Field2: convertField(oldParams.Field2),
        // New field with default value
        Field3: "default",
    }

    // Set new params
    keeper.SetParams(ctx, newParams)
    return nil
}
```

### State Migrations

Moving data between different storage locations:

```go theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
func migrateState(ctx sdk.Context, storeKey storetypes.StoreKey) error {
    store := ctx.KVStore(storeKey)

    // Iterate over old storage
    iterator := storetypes.KVStorePrefixIterator(store, oldPrefix)
    defer iterator.Close()

    for ; iterator.Valid(); iterator.Next() {
        oldKey := iterator.Key()
        value := iterator.Value()

        // Transform key/value if needed
        newKey := transformKey(oldKey)
        newValue := transformValue(value)

        // Write to new location
        store.Set(newKey, newValue)

        // Delete old entry
        store.Delete(oldKey)
    }

    return nil
}
```

### Module Addition/Removal

Adding or removing modules during upgrade:

```go theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
// In constants.go
var Upgrade = upgrades.Upgrade{
    UpgradeName: "v2.0.0",
    CreateUpgradeHandler: CreateUpgradeHandler,
    StoreUpgrades: storetypes.StoreUpgrades{
        Added:   []string{"newmodule"},
        Deleted: []string{"oldmodule"},
    },
}

// In handler.go
func CreateUpgradeHandler(...) upgradetypes.UpgradeHandler {
    return func(c context.Context, plan upgradetypes.Plan, vm module.VersionMap) (module.VersionMap, error) {
        // Delete old module version
        delete(vm, "oldmodule")

        // Initialize new module
        if err := newModuleKeeper.InitGenesis(ctx, defaultGenesis); err != nil {
            return nil, err
        }

        // Run migrations
        return mm.RunMigrations(c, configurator, vm)
    }
}
```

## Best Practices

<Warning>
  Always test upgrade handlers thoroughly on testnets before mainnet deployment.
</Warning>

### Idempotency

Make migrations idempotent when possible:

```go theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
func migrateSomething(ctx sdk.Context, store sdk.KVStore) error {
    // Check if migration already done
    if store.Has(migrationCompleteKey) {
        ctx.Logger().Info("Migration already completed, skipping")
        return nil
    }

    // Perform migration
    // ...

    // Mark as complete
    store.Set(migrationCompleteKey, []byte{1})
    return nil
}
```

### Error Handling

Use comprehensive error handling and logging:

```go theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
func migrate(ctx sdk.Context, keeper Keeper) error {
    ctx.Logger().Info("Starting migration", "module", "mymodule")

    count := 0
    iterator := keeper.IterateAllRecords(ctx)
    defer iterator.Close()

    for ; iterator.Valid(); iterator.Next() {
        if err := processRecord(iterator.Key(), iterator.Value()); err != nil {
            ctx.Logger().Error("Failed to migrate record",
                "key", iterator.Key(),
                "error", err,
            )
            return fmt.Errorf("migration failed at record %d: %w", count, err)
        }
        count++

        // Log progress for long migrations
        if count%1000 == 0 {
            ctx.Logger().Info("Migration progress", "processed", count)
        }
    }

    ctx.Logger().Info("Migration complete", "total_migrated", count)
    return nil
}
```

### Testing

Create comprehensive tests for upgrade handlers:

```go theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
func TestUpgradeHandler(t *testing.T) {
    app := setupApp(t)
    ctx := app.NewContext(false, tmproto.Header{Height: 1})

    // Setup pre-upgrade state
    setupOldState(t, ctx, app)

    // Run upgrade handler
    _, err := v1_0_0.CreateUpgradeHandler(
        app.ModuleManager,
        app.configurator,
        &upgrades.UpgradeKeepers{
            // ... keepers
        },
        app.keys,
    )(ctx, upgradetypes.Plan{Name: "v1.0.0"}, app.ModuleManager.GetVersionMap())

    require.NoError(t, err)

    // Verify post-upgrade state
    verifyNewState(t, ctx, app)
}
```

## Upgrade Process

### Create Upgrade Proposal

Submit a governance proposal with the upgrade details:

```bash theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
mantrachaind tx gov submit-proposal software-upgrade v1.0.0 \
    --title "Upgrade to v1.0.0" \
    --description "Upgrade description" \
    --upgrade-height 1000000 \
    --from validator \
    --deposit 10000000stake
```

### Vote on Proposal

Validators and delegators vote on the upgrade:

```bash theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
mantrachaind tx gov vote 1 yes --from validator
```

### Prepare Binary

Build and distribute the new binary with the upgrade handler:

```bash theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
# Build new binary
make build

# Test upgrade on local network
./scripts/test-upgrade.sh

# Distribute to validators
# Use Cosmovisor for automated upgrades
```

### Monitor Upgrade

Watch logs during the upgrade height:

```bash theme={"theme":{"light":"github-light-high-contrast","dark":"github-dark-high-contrast"}}
# Monitor upgrade logs
tail -f ~/.mantrachaind/logs/upgrade.log

# Verify upgrade success
mantrachaind query upgrade applied v1.0.0
```

## Cosmos EVM Specific Migrations

For Cosmos EVM chains, specific migrations include:

* **[ERC20 Precompiles Migration](./erc20-precompiles-migration)**: Required for v0.3.x to v0.4.0
* **Fee Market Parameters**: Updating EIP-1559 parameters
* **Custom Precompiles**: Registering new precompiled contracts
* **EVM State**: Migrating account balances or contract storage

## Troubleshooting

### Consensus Failure

**Symptom:** Chain halts with consensus failure at upgrade height

**Causes:**

* Validators running different binary versions
* Upgrade handler not registered
* Non-deterministic migration logic

**Solution:**

* Ensure all validators have the same binary
* Verify upgrade handler is registered
* Review migration logic for non-determinism

### Upgrade Panic

**Symptom:** Node panics during upgrade

**Causes:**

* Unhandled error in migration
* Missing required state
* Invalid type assertions

**Solution:**

* Add comprehensive error handling
* Validate state before migration
* Use safe type conversions

### State Corruption

**Symptom:** Invalid state after upgrade

**Causes:**

* Partial migration completion
* Incorrect data transformation
* Missing cleanup of old data

**Solution:**

* Make migrations atomic
* Thoroughly test transformations
* Ensure old data is properly cleaned up

## References

* [Cosmos SDK Upgrade Module](/sdk/v0.53/build/modules/upgrade)
* [Cosmovisor Documentation](/sdk/v0.53/build/tooling/cosmovisor)