Monitoring validator performance

A healthy validator attests in every epoch, proposes when chosen, and takes part in sync committees when selected. This page shows how to confirm that, using Grandine's logs, its performance reports, its Prometheus metrics, the Beacon Node API and an external explorer. It ends with a checklist.

For setting up Prometheus and Grafana in general, see Metrics. For log levels and formats, see Logging.

Duty log lines

Grandine logs each duty at the info level. With validators attached, you should see lines like these every slot or epoch:

loaded 4 validator key(s)
validators [4242, 4243] attesting in slot 123456
validators [4242] aggregating in slot 123456
validators [4243] participating in sync subcommittee 2 in slot 123457
validators [4243] aggregating in sync committee in slot 123457
starting block proposal task for validator 4242 at slot 123460
validator 4242 proposing beacon block with root 0x... in slot 123460

• loaded N validator key(s) appears once at startup. If you expected keys and see failed to load validator keys instead, check the keystore and password options.
• attesting in slot lists the validator indices that attested in that slot. Each active validator should appear once per epoch.
• A proposal ends with proposing beacon block with root .... With a builder it says using the builder API (before Gloas) or using builder N (from Gloas on).

Warnings to look for:

Performance reports

With the built-in beacon node, Grandine can print a summary of how its validators did. Start it with --report-validator-performance:

grandine                                  \
  --network mainnet                       \
  --jwt-secret /path/to/jwt.hex           \
  --eth1-rpc-urls http://EL-HOST:8551     \
  --keystore-dir /path/to/validator_keys  \
  --keystore-password-file /path/to/password.txt \
  --report-validator-performance

Once per epoch, while the node is synced, it logs:

attestation performance for epoch: 1000, expected: 4, produced: 4, correct target: 4 (100.00%), correct head: 4 (100.00%), included: 4 (100.00%), inclusion delays: ...
sync committee message performance for epoch: 1001, expected: 32, produced: 32, correct: 32, included: 32 (100.00%)

• The attestation report covers the epoch before the previous one, because attestations can still be included one epoch later.
• expected is the number of validators with an attestation duty, produced the number of attestations seen, and included the number found in blocks.
• correct target and correct head show how many votes matched the canonical chain. A low head rate often means late blocks or a slow node.
• inclusion delays summarizes how many slots it took for attestations to be included. A delay of 1 is ideal.

The report also warns about individual votes, for example:

validators [4242] voted for outdated head 0x... (by 2 slots) at slot 123456 (expected to vote for block 0x...)
validators [4242] voted for non-canonical block 0x... at slot 123456 (expected to vote for block 0x...)

If no validator is in a sync committee, the sync report says no sync committee members for epoch N with M registered validators.

Note: --report-validator-performance reads blocks from the built-in beacon node. It is not accepted with grandine bn, and grandine vc refuses it with --report-validator-performance reads blocks from the built-in beacon node. With a separate validator client, use an explorer or the Beacon Node API instead.

Prometheus metrics

Enable metrics with --metrics. Grandine serves them at http://127.0.0.1:5054/metrics by default. Change the address and port with --metrics-address and --metrics-port.

grandine ... --metrics
curl -s http://127.0.0.1:5054/metrics | grep -i validator

A minimal Prometheus scrape configuration:

scrape_configs:
  - job_name: grandine
    static_configs:
      - targets: ["127.0.0.1:5054"]

Validator metrics (also exported by a separate grandine vc):

Beacon node metrics that matter for validators (not exported by grandine vc):

Histograms appear with _bucket, _sum and _count suffixes. For example, a Prometheus alert on a stalled head:

- alert: GrandineHeadStalled
  expr: beacon_clock_slot - beacon_head_slot > 4
  for: 5m

Grandine does not export per-validator balances or effectiveness metrics. Use an explorer or the Beacon Node API for those.

Beacon Node API

The Beacon Node API (127.0.0.1:5052 by default) answers most "is it working?" questions directly:

# Sync status: is_syncing should be false, is_optimistic false, el_offline false
curl -s http://127.0.0.1:5052/eth/v1/node/syncing

# Peer count
curl -s http://127.0.0.1:5052/eth/v1/node/peer_count

# Validator status and balance (by public key or index)
curl -s http://127.0.0.1:5052/eth/v1/beacon/states/head/validators/0xPUBKEY

# Upcoming proposals in the current epoch
curl -s http://127.0.0.1:5052/eth/v1/validator/duties/proposer/EPOCH

A rising balance from one epoch to the next means the validator earns rewards.

Liveness tracking

With --track-liveness, the beacon node records which validators it has seen attesting or proposing in the current and the previous epoch, and serves the standard liveness endpoint:

curl -s -X POST -H "Content-Type: application/json" \
  -d '["4242", "4243"]' \
  http://127.0.0.1:5052/eth/v1/validator/liveness/EPOCH

Only the current and the previous epoch can be queried. is_live: true means the validator was seen on the network. Liveness tracking is also what doppelganger protection uses. Without --track-liveness, a Grandine beacon node returns an error for this endpoint.

beaconcha.in and other explorers

Block explorers such as beaconcha.in show each validator's attestations, proposals, sync committee duties and rewards, and can send alerts by email or through their mobile app. Look up your validators by public key or index.

Grandine can also push its own status to beaconcha.in's machine metrics once a minute. Create an API key at beaconcha.in/api/key-management, then pass the endpoint with your key, and optionally a machine name, to --remote-metrics-url:

grandine ... --remote-metrics-url "https://beaconcha.in/api/v1/client/metrics?apikey=YOUR_API_KEY&machine=MACHINE_NAME"

Grandine sends beacon node, validator (number of active and total validators) and system metrics. The beaconcha.in mobile app then shows the machine and can alert you when it goes offline. See Metrics.

Is everything fine? A checklist

Run through this list after setting up, after every upgrade, and whenever something looks wrong.

1. Version. The startup log line client version: ... shows the release you expect.
2. Execution client. It is synced, and Grandine reaches it. GET /eth/v1/node/syncing reports el_offline: false. See Connecting an execution client.
3. Beacon node synced. is_syncing: false and is_optimistic: false, and beacon_head_slot stays within a slot or two of beacon_clock_slot.
4. Peers. The peer count is stable and near the target (--target-peers). See Networking and peers.
5. Finality. beacon_finalized_epoch advances every epoch, and the log shows new finalized checkpoint regularly.
6. Keys loaded. The log shows loaded N validator key(s) with the expected N, and no failed to load validator keys.
7. Fee recipient. The startup line suggested fee recipient: 0x... shows your address, not the Grandine donation address. See Fee recipient, graffiti and gas limit.
8. Validators active. Their status is active_ongoing in the Beacon Node API or on an explorer.
9. Attesting. Each validator index appears in a validators [...] attesting in slot line once per epoch. With --report-validator-performance, included and correct target are close to 100%.
10. No warnings. No repeated cannot perform duties, slashing protector rejected or failed to publish messages.
11. Balances. Balances rise from epoch to epoch on an explorer.
12. Slashing protection backed up. You have a recent export. See Slashing protection.
13. Monitoring and alerts. Prometheus scrapes the metrics endpoint, or beaconcha.in monitoring is set up, and alerts reach you.
14. Service. Grandine runs as a service that restarts after a reboot. See Running as a service.

Some missed attestations are normal: late blocks, reorgs and restarts all cause them. A validator that misses most of its attestations for several epochs needs attention. See Troubleshooting.

Command line options

• --metrics (default: disabled): serve Prometheus metrics.
• --metrics-address (default: 127.0.0.1): address of the metrics endpoint.
• --metrics-port (default: 5054): port of the metrics endpoint.
• --remote-metrics-url (default: none): URL to push metrics to once a minute, for example beaconcha.in.
• --report-validator-performance (default: disabled): log attestation and sync committee performance once per epoch. Requires the built-in beacon node.
• --track-liveness (default: disabled): beacon node option. Record validator liveness and serve POST /eth/v1/validator/liveness/{epoch}.

See the command line reference for the full list.