feat(validator): add VC DB metrics and slashing protection timing to dashboard

[TechFusionData]

Motivation: Resolves remaining items in #5663. The validator client exposes DB metrics (vc_db_read_req_total, vc_db_write_req_total, vc_db_read_items_total, vc_db_write_items_total, vc_db_size_bytes_total, vc_db_approximate_size_time_seconds) but these weren't visualised in the VC dashboard — only in lodestar_vm_host.json. Slashing protection check latency also had no visibility.

Changes:

Dashboard (dashboards/lodestar_validator_client.json):

• New "DB Metrics" row with 6 panels (IDs 52–57): read/write req rates by bucket (log scale), read/write item rates by bucket (log scale), DB size in bytes, approximate size duration
• New "Slashing Protection Timing" row with 2 panels (IDs 59–60): average block check and attestation check duration
• All rate panels use the existing $rate_interval template variable

Source (packages/validator/src/):

• metrics.ts: Two new histograms — vc_slashing_protection_block_check_seconds and vc_slashing_protection_attestation_check_seconds with buckets [0.0001, 0.001, 0.01, 0.1]
• services/validatorStore.ts: Wraps checkAndInsertBlockProposal and checkAndInsertAttestation with startTimer() / finally for the new histograms

Why slashing protection timing matters: These checks run on every block proposal and attestation. If the slashing protection DB grows large or encounters lock contention, check latency can silently increase and eat into the validator's signing window. Having it on the dashboard lets operators spot degradation before it causes missed duties.

Checklist:

• JSON is valid, panel IDs are unique (51–60)
• Follows existing panel style conventions from lodestar_vm_host.json
• New metrics use standard Prometheus histogram patterns
• Timing instrumentation uses try/finally to record on error paths
• AI tools were used to assist with this PR

[TechFusionData]

AI Disclosure: This PR was developed with AI assistance (Claude) for code exploration, panel structure, and JSON authoring, per Lodestar contribution policy.

[gemini-code-assist]

Code Review

This pull request adds a new 'DB Metrics' row to the Lodestar Validator Client dashboard, featuring panels for tracking database read/write request rates, item processing rates, total database size, and approximate size calculation durations. The review feedback suggests replacing the built-in $__rate_interval variable with the dashboard's custom $rate_interval variable across several new panels to maintain consistency and ensure proper integration with dashboard controls.

[TechFusionData]

Added a second commit with Part 2 — slashing protection timing metrics:

New Prometheus metrics (packages/validator/src/metrics.ts):

• vc_slashing_protection_block_check_seconds — histogram of checkAndInsertBlockProposal execution time
• vc_slashing_protection_attestation_check_seconds — histogram of checkAndInsertAttestation execution time
• Buckets: [0.1ms, 1ms, 10ms, 100ms] — aligned with expected DB read latency

Instrumentation (packages/validator/src/services/validatorStore.ts):

• Both calls wrapped with startTimer() / finally { timer?.() } so timing is always recorded, including on the error path

Dashboard (dashboards/lodestar_validator_client.json):

• New "Slashing Protection Timing" row section with two avg-duration panels (block check + attestation check, IDs 58–60)

AI-assisted implementation

[TechFusionData]

Added a second commit with Part 2 — slashing protection timing metrics:

New Prometheus metrics (packages/validator/src/metrics.ts):

• vc_slashing_protection_block_check_seconds — histogram of checkAndInsertBlockProposal execution time
• vc_slashing_protection_attestation_check_seconds — histogram of checkAndInsertAttestation execution time
• Buckets: [0.1ms, 1ms, 10ms, 100ms]

Instrumentation (packages/validator/src/services/validatorStore.ts):

• Both calls wrapped with startTimer() / finally { timer?.() } so timing is always recorded, including on the error path

Dashboard (dashboards/lodestar_validator_client.json):

• New Slashing Protection Timing row section with two avg-duration panels (IDs 58-60)

AI-assisted implementation

[TechFusionData]

Addressed bot review feedback: replaced $__rate_interval with the dashboard custom variable $rate_interval across all 7 new panel expressions (IDs 52–60). The built-in Grafana variable bypasses the dashboard dropdown, whereas $rate_interval is the existing convention throughout this dashboard.

[nazarhussain]

• The PR title says DB metrics only, but the branch now also adds:

  • 2 new validator metrics
  • 2 new slashing-protection dashboard panels
  • runtime instrumentation in validatorStore

• The checklist/body still contains “No source code changes (dashboard JSON only)”, which is no longer true

Please update the PR title/description and particularly add details why these metrics are useful.

[TechFusionData]

Good catch — updated the title, description, and checklist to cover the full scope (DB metrics panels + slashing protection timing metrics and instrumentation). Added rationale for the timing metrics.

[nazarhussain]

@TechFusionData PR looks good. Please fix the linting issue with the generated dashboard json, so we can merge it.