> Source: [/docs/guide/node/validator-failover](https://tempo.xyz/developers/docs/guide/node/validator-failover.md) > Section: [Tempo EVM](https://tempo.xyz/developers/docs/development.md). Build on Tempo EVM: wallets and keys, payments, transactions and fees, TIP-20 Tokens, contracts, exchange, network details, nodes, protocol specifications, and changelog. > [Documentation index](https://tempo.xyz/developers/llms.txt) · [Docs skill](https://tempo.xyz/developers/SKILL.md) > Tempo MCP: Use `search`, `find_pages`, `read_page`, and `code` at `https://mcp.tempo.xyz` for Tempo and related documentation. > If an indexed page is missing or reflects an older site, read this site’s [documentation index](https://tempo.xyz/developers/llms.txt) and Markdown pages directly. # Validator failover Move your validator to a replacement host while preserving its identity and consensus state. Choose the procedure that matches the state you have available: | Deployment | Recovery path | | --- | --- | | Bare metal | [Rotate or transfer the consensus directory](https://tempo.xyz/developers/docs/guide/node/validator-failover#bare-metal-validator) | | Cloud Provider | [Transfer the consensus directory](https://tempo.xyz/developers/docs/guide/node/validator-failover#cloud-validator) | :::danger[Prevent double-signing] Only one node may run with a validator signing key at a time. Before starting the replacement, fence the old validator: stop it and prevent automatic restarts. ::: ## Bare-metal validator If the bare metal host is not ephemeral, failover to a new host by following the [validator rotation](https://tempo.xyz/developers/docs/guide/node/validator-rotation) runbook which promotes a standby follower node into validator whilst deactivating the running one. If the setup allows for a shared disks, the existing validator's identity can be retained restoring it's consensus directly on a new host as described in the steps [below](https://tempo.xyz/developers/docs/guide/node/validator-failover#restore-the-replacement). If the new host is registered under a different ingress or egress ips, make sure they are [updated onchain](https://tempo.xyz/developers/docs/guide/node/validator-rotation#update-ip-addresses). ## Cloud validator Reuse the validator's signing key, latest complete consensus directory, and registered ingress and egress addresses. If the addresses change, [update them onchain](https://tempo.xyz/developers/docs/guide/node/validator-rotation#update-ip-addresses) and wait for finalization before starting the replacement. If consensus state is lost, follow [resetting validator data](https://tempo.xyz/developers/docs/guide/node/validator-lifecycle#resetting-your-validators-data). ### Prepare cloud consensus storage Keep execution data on [local NVMe storage](https://tempo.xyz/developers/docs/guide/node/system-requirements) and consensus data on a persistent network-attached volume that you can detach and attach to a replacement host. Use `--consensus.datadir` to select the consensus directory on that volume. Provision and mount the volume before starting the service. Keep the other flags from your existing validator configuration. The Bash/Zsh command below assumes an encrypted key: replace `` with the command that retrieves its encryption secret. Omit `--consensus.secret` only for an unencrypted key. ```bash tempo node \ --chain mainnet \ --datadir \ --consensus.datadir \ --consensus.signing-key \ --consensus.secret <() \ --consensus.listen-address 0.0.0.0: ``` To migrate an existing validator to this layout, stop it, copy its complete consensus directory to the mounted volume, update `--consensus.datadir`, and restart. Configure the service to require the volume mount so it cannot start against an empty local directory when the volume is unavailable. Allow only the active validator to write to this volume. ### Execution recovery window The consensus volume retains recent finalized blocks that the replacement can replay into its execution database. This matters when local execution disks are lost and the replacement starts from a snapshot or an older execution backup. A synced follower already has recent execution state, so it avoids most or all of this replay. The default retention window is about ~3 epochs worth of blocks, roughly ~9hrs worth of data (500ms block times). This means the execution disk can be 9 hours behind the previously active validator and startup fine. Any longer would and the failover node would fail to startup. If your failover procedure always involves using a standby follower node that is in sync, the retention window can be shortened to accomadate a smaller consensus disk via the `--consensus.finalized-blocks-retention` flag. A reasonable default in this circumstance is `16_384`, leaving a ~2hr window to succesfully failover. ### Restore the replacement ::::steps #### Fence the old validator Fence the old validator and disable any automation that could restart it. If transferring a network-attached volume, detach it using your cloud provider's procedure once the old host can no longer write. #### Prepare execution state on the replacement Use the same node version and chain configuration as the old validator. **For a [synced standby](https://tempo.xyz/developers/docs/guide/node/rpc):** stop the follower cleanly and wait for it to exit. The follower's local execution state, `--datadir` will continue in use as a promoted validator. ```bash sudo systemctl stop tempo-follower ``` If the follower is not managed by systemd, send `SIGINT` or `SIGTERM` and wait for the process to exit. **For a fresh replacement:** restore a recent execution backup or download a minimal snapshot into a new local data directory. ```bash tempo download --chain (mainnet|testnet) --minimal --datadir ``` Download the snapshot before attaching the preserved consensus volume. The download will installs its own consensus data under the local data directory. However the validator will use the preserved volume instead when started with `--consensus.datadir`. :::danger[Protect the preserved consensus state] Do not point `tempo download --consensus.datadir` at the preserved volume. In particular, `tempo download --force` removes existing consensus data at its destination. If replacing a failed download, use a separate local directory and leave the preserved volume untouched. ::: #### Transfer the validator's consensus directory Attach and mount the preserved volume, or restore the previous validator's complete consensus directory. Keep it separate from any consensus directory created by the follower or snapshot; do not merge their files. A follower's own consensus archive does not contain the active validator's private signing state. Using it can delay signing participation until the node recovers a share. Transfer the active validator's latest complete directory to preserve that state. #### Start with both data paths Start the validator with the [storage configuration above](https://tempo.xyz/developers/docs/guide/node/validator-failover#prepare-cloud-consensus-storage), using the replacement's local `--datadir`, the transferred `--consensus.datadir`, and the old validator's signing key. Remove `--follow` if reusing a follower's service configuration. If startup fails with an error reporting insufficient execution state, restore fresher execution state, either by [syncing a follower](https://tempo.xyz/developers/docs/guide/node/rpc) or by downloading a fresh snapshot. Verify [consensus participation](https://tempo.xyz/developers/docs/guide/node/validator-monitoring#consensus-states) on the replacement. :::: ## After failover Confirm [validator status](https://tempo.xyz/developers/docs/guide/node/validator-status) and [consensus participation](https://tempo.xyz/developers/docs/guide/node/validator-monitoring#consensus-states) on the replacement. Treat the replacement as the active validator until you intentionally move the role again. Prevent the old host from restarting in validator mode. You can repurpose it as a follower and future standby by configuring `--follow` without `--consensus.signing-key` or `--consensus.secret`. If the old host's local consensus datadir is empty, follow [Running RPC and Standby Nodes](https://tempo.xyz/developers/docs/guide/node/rpc) to initially boostrap it again as a follower. Keep the active validator's volume attached exclusively to the active instance.