Upgrade a node

You have a node running as a service from Run gaiad as a service. At the end the node runs the new gaia version from the exact height the network switched, without missing more blocks than the upgrade itself costs. Every upgrade is announced on the gaia releases page and in the validator announcements channel on Discord. The release notes name the pattern, the height, and the binary.

Two upgrade patterns

The Hub uses two patterns and the release notes say which one applies. A governance upgrade is a passed MsgSoftwareUpgrade proposal with a plan name and a plan height. At that height the old binary stops with ERR UPGRADE "<name>" NEEDED at height: <height>. The new binary’s upgrade handler then runs the migrations. Releases v28.0.0 (plan v28.0.0, height 32785900) and v28.2.0 (plan v28.2.0, height 33073000) were governance upgrades. Cosmovisor handles the swap on its own if the binary is in place. A halt-height upgrade is a coordinated restart without a plan. Every operator sets halt-height to the announced height. The node stops there with CONSENSUS FAILURE!!! err="failed to apply block; error halt per configuration height <height> time 0", and the operator starts the new binary. Releases v28.1.1 (height 32830000) and v28.3.1 (height 33086740) were halt-height upgrades. Cosmovisor does not switch on its own unless you register the height with add-upgrade --upgrade-height. Upgrade history lists every upgrade since genesis with its pattern.

Before upgrade day

Read the release notes and confirm the height

gaiad query upgrade plan

For a governance upgrade this prints the plan name, height, and info once the proposal has passed. Before that it prints an empty {}. For a halt-height upgrade it stays {}. The height comes from the release notes. On 2026-10-15 there was no plan scheduled.

Rehearse on the testnet

The testnet runs the release candidate one or two weeks ahead of mainnet. Upgrading a testnet node through the same height is the only way to find out, while it is still cheap, that your machine lacks RAM for a migration or that your unit file is wrong.

Get and verify the new binary

Install the new release with the same method and verification as Install gaiad: signature and checksum for a release binary, the tag’s commit for a source build. Do not install it over the running binary yet.

/tmp/gaiad-new version --long | grep -E '^(version|commit):'

The version must match the release notes exactly, including the patch number. Keep the file somewhere the service user can read.

Check memory, disk, and a backup

Migrations can need far more memory than steady state. Confirm 32 GB or swap from System requirements. Confirm free disk of at least the size of data/. Take a snapshot or filesystem backup of data/ the day before. With UNSAFE_SKIP_BACKUP=true in the unit, Cosmovisor does not do this for you.

Stage the binary for Cosmovisor

For a governance upgrade the directory name is the plan name, character for character:

sudo -u gaia env DAEMON_NAME=gaiad DAEMON_HOME=/var/lib/gaia/.gaia cosmovisor add-upgrade v28.2.0 /tmp/gaiad-new
ls -l /var/lib/gaia/.gaia/cosmovisor/upgrades/v28.2.0/bin/gaiad

The add-upgrade subcommand copies the binary to cosmovisor/upgrades/<name>/bin/gaiad and makes it executable. It is the same as creating the directory and copying by hand. Cosmovisor reads the plan name from data/upgrade-info.json, which the running node writes at the upgrade height. It then switches current to that directory.For a halt-height upgrade, add --upgrade-height:

sudo -u gaia env DAEMON_NAME=gaiad DAEMON_HOME=/var/lib/gaia/.gaia cosmovisor add-upgrade v28.3.1 /tmp/gaiad-new --upgrade-height 33086740 --force
cat /var/lib/gaia/.gaia/data/upgrade-info.json

This also writes data/upgrade-info.json with the name and height, so Cosmovisor switches binaries at that height on its own. Do not set halt-height as well when you use this. The two mechanisms stop the node for different reasons, and Cosmovisor only restarts for its own. On a node that has already been through a governance upgrade under Cosmovisor, data/upgrade-info.json still holds that old plan. The command then fails with Error: file already exists at .../data/upgrade-info.json. The --force flag overwrites it, and the height-based switch then works. Run cosmovisor show-upgrade-info first, because --force also discards a plan that is still pending.

Or set the halt height without Cosmovisor

gaiad config set app halt-height 33086740
gaiad config get app halt-height
sudo systemctl restart gaiad

This writes halt-height in app.toml. The running process reads it at start, so restart the node. At that height the node commits the block, logs the consensus failure quoted above, and stops. Set it back to 0 after the upgrade, or the node refuses to pass that height again.

On upgrade day

Do not start the new binary before the network reaches the height. A node that applies a migration early produces a different app hash and stops with wrong Block.Header.AppHash. It must then be rolled back or restored. A validator in that state misses blocks until it is fixed and risks jail for downtime.

With Cosmovisor and a staged binary there is nothing to do at the height except watch:

journalctl -u cosmovisor -f -o cat

Expect UPGRADE "v28.2.0" NEEDED at height: 33073000 (governance) or the halt per configuration line (halt-height). Then come Cosmovisor’s own lines about the upgrade and the binary switch. Then the new binary prints its applying upgrade line, followed by block execution. Upgrades with migrations can sit in applying upgrade for minutes. A quiet log is not a hung node until memory or CPU drop to zero. Without Cosmovisor, at the height:

sudo systemctl stop gaiad
sudo install -m 0755 /tmp/gaiad-new /usr/local/bin/gaiad
gaiad version
gaiad config set app halt-height 0
sudo systemctl start gaiad

The stop is clean because the node has already halted. The install command replaces the binary atomically. Running gaiad version proves which one is on the path. Resetting halt-height lets the node continue. With the container image the same sequence is docker stop, change the image tag, then docker run as on Install gaiad.

If something goes wrong

The node stopped at the height but Cosmovisor did not switch: check ls -l cosmovisor/upgrades/ for a directory name that matches the plan name exactly. Check cat data/upgrade-info.json. Check journalctl for a permissions error on the binary. Fix the directory, then run systemctl start cosmovisor. Cosmovisor reads upgrade-info.json on start and switches. The new binary panics during the migration: the usual cause is memory. Stop, add swap, and start again. The migration restarts from the beginning. If it panics on something else, the chain’s Discord channel is where the fix is coordinated. Your data/ backup is what you restore if a patched binary requires it. The node reports an app hash mismatch after the upgrade: a binary was started one height early or a wrong patch was used. The gaiad rollback command undoes the last committed height on both CometBFT and the application:

sudo systemctl stop cosmovisor
gaiad rollback --home /var/lib/gaia/.gaia

It removes no blocks. On restart the node re-executes the block with the correct binary. Run it once per bad height. On a validator, the restart does not change priv_validator_state.json, so there is no double sign risk from the rollback itself. The validator will not sign the rolled-back height again at the same round (error signing vote: step regression at height <height> round 0). On mainnet it misses that one signature and signs from the next round or height. Running gaiad start --unsafe-skip-upgrades <height> tells the node to ignore a scheduled plan at that height. It exists for coordinated recovery from a broken upgrade and nothing else. Using it alone forks your node off the network.

After the upgrade

sudo -u gaia /var/lib/gaia/.gaia/cosmovisor/current/bin/gaiad version
gaiad query upgrade applied v28.2.0
gaiad status | jq '{height: .sync_info.latest_block_height, catching_up: .sync_info.catching_up}'
sudo ls -l /var/lib/gaia/.gaia/cosmovisor/current

The version command prints the new tag. It runs the binary under cosmovisor/current, because Cosmovisor switches that symlink and leaves /usr/local/bin/gaiad alone. A plain gaiad version therefore still prints the old tag after a successful Cosmovisor upgrade. On a node upgraded by hand, gaiad version is the right check. The upgrade applied <name> query prints the block header at which a governance plan was executed, which is the proof the handler ran. It has no record of halt-height upgrades. For those, version plus a climbing height is the proof. The current symlink now points at upgrades/<name>. A validator also checks that it is signing again on an explorer or with the queries on Monitoring. Clean up when the node is stable. Remove binaries under cosmovisor/upgrades/ older than the previous upgrade. If you set halt-height, confirm it is 0 again.

Verify

sudo -u gaia /var/lib/gaia/.gaia/cosmovisor/current/bin/gaiad version
gaiad query upgrade module-versions | head

The first prints the version the release notes named. On a node upgraded by hand, run gaiad version instead. The second lists each module’s consensus version after the upgrade, which is what a migration changes. Compare it with another operator’s output if you suspect a partial migration.

Upgrade history for the record of past upgrades, and Monitoring so the next halt is something you are told about rather than find out.

Cosmos Hub

Two upgrade patterns

Before upgrade day

On upgrade day

If something goes wrong

After the upgrade

Verify

Next

​Two upgrade patterns

​Before upgrade day

​On upgrade day

​If something goes wrong

​After the upgrade

​Verify

​Next

Two upgrade patterns

Before upgrade day

On upgrade day

If something goes wrong

After the upgrade

Verify

Next