Main issue with prometheus metrics endpoint ports

[phoenixanand]

I recently deployed Arc Localnet on an AWS EC2 instance and ran into several issues that I believe affect anyone deploying on a remote Linux server (not Docker Desktop). I've documented them all and would love to contribute fixes.

Prometheus Scrape Targets Point to Wrong Ports

Problem: The default prometheus.yml targets ports 6060-6065 for validator metrics, but the actual Prometheus metrics endpoint is on port 9101 (and similar for other validators). All validator targets show health: down out of the box.

# Current - wrong
- targets: ['host.docker.internal:6060']

# Should be
- targets: ['host.docker.internal:9101']

Suggested Fix: Update prometheus.yml with the correct metrics ports that match the actual validator port mappings.

---

Monitoring Stack Not Started by make testnet

Problem: Grafana and Prometheus live in .quake/monitoring/compose.yaml but make testnet only starts .quake/localdev/compose.yaml. New users have no indication that monitoring needs to be started separately with a different command.
Suggested Fix: Either include monitoring in make testnet, or add a make monitoring target and document it clearly in the README.

---

Issues Found During Remote EC2 Deployment

1. Missing Directory Initialization Script

Problem: Docker bind mounts fail on first run because host directories don't exist.

Error: bind source path does not exist: .../blockscout/logs
Error: bind source path does not exist: .../blockscout/dets

Suggested Fix: Add a make setup or scripts/init.sh that pre-creates all required bind mount directories before make testnet.

---

2. Blockscout dets Directory Permission Error (Backend Crash)

Problem: The Blockscout backend crashes immediately on Linux because the ./dets/ directory doesn't have write permissions for the container user. This cascades into all PostgreSQL connections being dropped.

{file_error, "./dets/queue_storage", eacces}

Suggested Fix: Either create the directory with correct permissions in an init script, or add a user: override in the blockscout service in compose.yaml.

---

3. Hardcoded localhost in Frontend Environment

Problem: NEXT_PUBLIC_API_HOST=localhost and NEXT_PUBLIC_APP_HOST=localhost are hardcoded in the frontend config. This works on Docker Desktop (where localhost = the machine) but completely breaks on any remote server , the browser tries to call the user's own machine instead of the server.

# deployments/blockscout.yaml
NEXT_PUBLIC_API_HOST: localhost  # ❌ breaks on remote servers

Suggested Fix: Replace with a configurable environment variable, e.g.:

NEXT_PUBLIC_API_HOST: ${BLOCKSCOUT_HOST:-localhost}

Then document that remote deployments need to set BLOCKSCOUT_HOST=<server-ip>.

---

4. Monitoring Ports Bound to 127.0.0.1

Problem: Grafana is bound to 127.0.0.1:3000:3000 and cAdvisor to 127.0.0.1:8080:8080, making them inaccessible on remote servers even after opening AWS Security Group rules.
Suggested Fix: Change to 0.0.0.0:3000:3000 or document that users need to change this for remote access.

---

5. No Documentation for Remote/Cloud Deployment

Problem: The README and setup docs appear to assume Docker Desktop on a local machine. There is no guide for deploying on a remote Linux server (EC2, GCP, etc.).
Suggested Fix: Add a "Remote Deployment" section to the README covering:

• Setting BLOCKSCOUT_HOST to the server's public IP
• Running make setup before make testnet
• Opening required ports in cloud security groups (80, 3000, 9090)
• Starting the monitoring stack separately

---

[giwaov]

I’d be happy to help with a scoped part of this if it’s available, especially the monitoring/compose setup pieces.

Could this, or a specific subtask from it, be assigned to me before I start? I’ll wait for confirmation before opening a PR.