> Source: [/docs/guide/node/validator-monitoring](https://tempo.xyz/developers/docs/guide/node/validator-monitoring.md) > Section: [Tempo EVM](https://tempo.xyz/developers/docs/development.md). Build on Tempo EVM: wallets and keys, transactions and fees, TIP-20 Tokens, contracts, exchange, network details, nodes, protocol specifications, and changelog. > [Documentation index](https://tempo.xyz/developers/llms.txt) · [Docs skill](https://tempo.xyz/developers/SKILL.md) > Tempo MCP: Use `search`, `find_pages`, `read_page`, and `code` at `https://mcp.tempo.xyz` for Tempo and related documentation. > If an indexed page is missing or reflects an older site, read this site’s [documentation index](https://tempo.xyz/developers/llms.txt) and Markdown pages directly. # Monitoring a validator Monitor your Tempo validator's health, diagnose issues, and manage logs. These examples assume execution metrics are enabled with `--metrics 9000`. Consensus metrics listen on `127.0.0.1:8001` by default; use `--consensus.metrics-address` to change that address. ## Consensus states ### Proposer Your validator is currently the leader and proposing blocks. Check proposal activity: ```bash # Number of blocks your node has built and resolved curl -s localhost:9000/metrics | grep reth_payloads_resolved_block ``` If this counter is increasing, your validator is actively proposing blocks. ### Voter To confirm your validator has started voting, check its outbound notarize and finalize votes: ```bash # Individual votes sent by your node curl -s localhost:8001/metrics \ | grep '^consensus_engine_epoch_manager_simplex_voter_outbound_messages_total{' \ | grep -E 'message="(Notarize|Finalize)"' ``` Increasing counters confirm your node is signing and attempting to broadcast votes. The same outbound metric counts certificates under `message="Notarization"`, `"Finalization"`, and `"Nullification"`. A node observing an epoch without a signing share can broadcast certificates built from other validators' votes, so keep the individual-vote filter above. ## Execution states ### Catching up Your node is syncing historical blocks. ```bash # Check sync stage curl -s localhost:9000/metrics | grep reth_sync_checkpoint ``` If `reth_sync_checkpoint` shows stages other than `Finish`, you're still syncing. ### Up to sync Your node is fully synced and processing new blocks in real-time. ```bash # Processed height should match or be close to finalized height curl -s localhost:8001/metrics | grep -E "marshal_finalized_height|marshal_processed_height" ``` Both values should be nearly equal and increasing together. ## Metrics glossary | Metric Name | Description | When to alert? | Meaning | | --- | --- | --- | --- | | `consensus_engine_dkg_manager_ceremony_successes_total` | Number of successful DKG ceremonies | Critical when it hasn't increased in 12 hours | Your node has participated in a successful DKG ceremony | | `consensus_engine_dkg_manager_ceremony_failures_total` | Number of failed DKG ceremonies | Warning when it increases | Your node has failed to participate in a DKG ceremony | | `consensus_engine_dkg_manager_how_often_dealer` | How many times your node has been a dealer (distributing shares) | Warning when it hasn't increased in 6 hours | Your node is distributing signing shares to other validators | | `consensus_engine_dkg_manager_how_often_player` | How many times your node has been a player (receiving shares) | Warning when it hasn't increased in 6 hours | Your node is receiving signing shares from dealers | | `consensus_engine_marshal_finalized_height` | Latest finalized height your node is aware of | Warning when it hasn't increased in 1 hour, critical when it hasn't increased in 3 hours | Your node is aware of the latest finalized height | | `consensus_engine_marshal_processed_height` | Latest height your node has processed | Critical when it hasn't increased in an hour, warning when it's behind finalized height | Your node is processing blocks | | `consensus_engine_peer_manager_peers` | Number of peers registered with the consensus peer manager | Warning when it's below the expected validator count, critical when it's 0 | Your node is registered with consensus peers | | `consensus_engine_epoch_manager_simplex_batcher_inbound_messages_total` | Number of inbound messages related to voting on the consensus layer | Warning when it's not increasing and your node is synced | Your node is receiving voting messages from the consensus layer | | `consensus_engine_epoch_manager_simplex_voter_outbound_messages_total` | Number of outbound votes and certificates, distinguished by `message` | Warning when `Notarize` and `Finalize` counters stop increasing while your node is synced and expected to vote | Your node is attempting to broadcast votes when filtered to `Notarize` and `Finalize`; certificate broadcasts also occur in observer mode | | `consensus_application_parent_ahead_of_local_time` | Number of times the parent block timestamp was ahead of local time | Warning when it increases frequently | Your node's clock may be out of sync — check NTP configuration | | `reth_sync_checkpoint` | Current sync progress | Warning on when there are no changes in the `Finish` stage | Your node is syncing with the network | | `reth_payloads_resolved_block` | Number of built and resolved payloads | Warning when it hasn't increased in 12 hours | Your node has built and resolved blocks | ## Grafana dashboard We provide a pre-built Grafana dashboard for monitoring your validator. It visualizes key metrics including node status, sync progress, voting activity, consensus latency, and execution performance. ### Importing the dashboard 1. Download the dashboard JSON from the [Tempo repository](https://github.com/tempoxyz/tempo/blob/main/contrib/grafana/dashboards/validator-health.json) 2. In Grafana, go to **Dashboards → Import** 3. Either paste the JSON content or upload the file 4. Select your Prometheus and Loki datasources when prompted 5. Click **Import** ### Dashboard variables The dashboard uses these template variables: | Variable | Description | | --- | --- | | `datasource` | Prometheus datasource | | `loki_ds` | Loki datasource (for log-based panels) | | `network_name` | Filter by network/validator job name | Make sure your Prometheus is scraping metrics from your validator's metrics endpoint (default: `localhost:8001/metrics`). ## Log management ### Parsing logs Tempo logs include ANSI escape codes for colors. To strip them for grep/awk: ```bash # Strip colors when searching sudo journalctl -u tempo | sed 's/\x1b\[[0-9;]*m//g' | grep "error" # Or disable colors at runtime RUST_LOG_STYLE=never tempo node ... ``` If you're using Loki, you can also use the `decolorize` filter to strip colors: ``` {job="tempo-node"} | decolorize ``` ### Log levels Control verbosity with `RUST_LOG`: ```bash # Default (info) RUST_LOG=info # Debug consensus only RUST_LOG=info,tempo_commonware_node::consensus=debug # Quiet mode (warnings and errors only) RUST_LOG=warn ```