Skip to content
BranchesTags

Repository files navigation

seisstream

Seisstream streams MiniSEED from SeedLink into RabbitMQ, stores waveform samples in TimescaleDB, and runs event/phase detection from AMQP. The core components are C services (connector, consumer) plus a Python earthquake detector.

Architecture

%%{init: {"theme":"neutral","themeVariables":{"fontSize":"18px","primaryTextColor":"#000","lineColor":"#000"}}}%%
flowchart TB

  subgraph BG[" "]
    direction TB

    subgraph SeedLink Servers
      SL1[SeedLink Server #1]
      SL2[SeedLink Server #2]
      SL3[SeedLink Server #3]
    end

    SL1 -->|SeedLink/MiniSEED| CON1[Connector #1<br/>libslink -> AMQP]
    SL2 -->|SeedLink/MiniSEED| CON2[Connector #2<br/>libslink -> AMQP]
    SL3 -->|SeedLink/MiniSEED| CON3[Connector #3<br/>libslink -> AMQP]

    CON1 -->|AMQP publish| MQ[(AMQP Broker<br/>RabbitMQ)]
    CON2 -->|AMQP publish| MQ
    CON3 -->|AMQP publish| MQ

    MQ -->|AMQP consume| CNS1[Consumer #1<br/>AMQP -> libmseed]
    MQ -->|AMQP consume| CNS2[Consumer #2<br/>AMQP -> libmseed]
    MQ -->|AMQP consume| CNS3[Consumer #3<br/>AMQP -> libmseed]
    MQ -->|AMQP consume| DET[EQ Detector<br/>AMQP ->  EQ detections / phase picks]

    CNS1 -->|bulk load| PG[(Timescale DB)]
    CNS2 -->|bulk load| PG
    CNS3 -->|bulk load| PG
    DET -->| push EQ detections + phase picks| PG

    LOC[EQ Locator<br/> Calculate EQ locations ]
    GRAF[Grafana<br/>Dashboards/Alerts]

    PG -->|query phase picks| LOC
    LOC -->|push EQ origins| PG
    PG --> | query waveforms| GRAF
    LOC -->| query hypocenters | GRAF

    class CON1,CON2,CON3 connector;
    class CNS1,CNS2,CNS3 consumer;
    class DET detector;
    class LOC locator;
  end

  classDef connector fill:#e8f7ef,stroke:#2f855a,color:#111;
  classDef consumer fill:#fff7e6,stroke:#b7791f,color:#111;
  classDef detector fill:#fdecef,stroke:#c53030,color:#111;
  classDef locator fill:#edf2ff,stroke:#5a67d8,color:#111;
  style BG fill:#ffffff,stroke:#cccccc,stroke-width:2px,rx:12,ry:12;

  classDef connector fill:#e8f7ef,stroke:#2f855a,color:#111;
  classDef consumer fill:#fff7e6,stroke:#b7791f,color:#111;
  classDef detector fill:#fdecef,stroke:#c53030,color:#111;
  classDef locator fill:#edf2ff,stroke:#5a67d8,color:#111;

  class CON1,CON2,CON3 connector;
  class CNS1,CNS2,CNS3 consumer;
  class DET detector;
Loading

Repository Layout

  • connector/: SeedLink client that forwards packets to RabbitMQ.
  • consumer/: AMQP consumer that parses MiniSEED (libmseed) and bulk-loads samples into TimescaleDB.
  • detector/: Python detector that consumes MiniSEED from AMQP and writes event_detections and phase_picks.
  • locator/: Python locator that calculates origins from phase picks and writes origins + origin_arrivals.
  • tools/publish_mseed/: synthetic MiniSEED publisher for functional testing.

Quick Start (Docker)

Prerequisites:

  • Docker 20.10+ and Docker Compose v2+
  • Minimum 4GB RAM (8GB recommended for ML detector mode)
  • 2+ CPU cores

  1. Create a local environment file and set deployment values:

    cp .env.example .env

  2. Create the stream list file used by the connector container:

    cp connector/streamlist.conf.example streamlist.conf

    Then edit streamlist.conf (selectors/stations) and set SEEDLINK_HOST in .env as needed.

  3. Start core services:

    docker compose up -d

Notes:

  • Detector image build can take significantly longer than connector/consumer because it installs heavy ML dependencies such as torch and seisbench.
  • Validate rendered compose config with: 
    docker compose config

Verify It Works

Check service status and logs after startup:

docker compose ps
docker compose logs -f connector consumer detector locator

Expected behavior:

  • connector logs show packets received from SeedLink and AMQP publishes.
  • consumer logs show AMQP consumption and inserts/bulk loads into PostgreSQL.
  • detector logs show detector startup and detections/picks (depending on mode and incoming data).
  • locator logs show pick associations and origin writes.

Grafana:

  • Default URL: http://localhost:3000
  • Credentials: GRAFANA_USER / GRAFANA_PASSWORD from .env (or defaults in docker-compose.yml)

Demos

System run demo:

real_data.mp4

Synthetic testing demo (STA/LTA detector mode):

synthetic_detections.mp4

Real Event Detection Demo

Real event detection example from station GE.PSZ, using SeisBench EQTransformer (--detector-mode seisbench --sb-pretrained original). The video is shown at 2x speed, and the event is correctly detected. Purple annotations indicate the first P and S wave arrivals for the main event of the Szarvas, Hungary earthquake swarm on 19 August 2023.

real_data_detection.mp4

Monitoring (RabbitMQ Metrics)

This project supports RabbitMQ metrics via Prometheus and Grafana using a Compose override.

Monitoring files:

  • docker-compose.monitoring.yml: monitoring services and overrides.
  • monitoring/rabbitmq/rabbitmq.conf: RabbitMQ Prometheus settings.
  • monitoring/rabbitmq/enabled_plugins: enables rabbitmq_prometheus plugin.
  • monitoring/prometheus/prometheus.yml: Prometheus scrape configuration.
  • grafana/provisioning/datasources/prometheus.yml: Grafana Prometheus datasource.

Start with monitoring enabled:

docker compose -f docker-compose.yml -f docker-compose.monitoring.yml up -d

Monitoring endpoints:

  • RabbitMQ metrics: http://localhost:15692/metrics
  • Prometheus: http://localhost:9090
  • Grafana: http://localhost:3000

Verify scrape targets:

curl -s http://localhost:15692/metrics | head
curl -s http://localhost:9090/api/v1/targets | jq .

Notes:

  • The monitoring override extends services from docker-compose.yml; core app behavior is unchanged.
  • Queue-level metrics are enabled (prometheus.return_per_object_metrics = true), which can increase cardinality in very large deployments.
  • No alert rules or preloaded RabbitMQ dashboard are included. Create these in Prometheus/Grafana as needed.

Configuration

The Docker setup uses these environment variable groups:

  • RabbitMQ: RABBITMQ_USER, RABBITMQ_PASS
  • TimescaleDB/PostgreSQL: PGUSER, PGPASSWORD, PGDATABASE
  • AMQP routing: AMQP_EXCHANGE, CONSUMER_AMQP_BINDING_KEY, DETECTOR_AMQP_BINDING_KEY
  • SeedLink source: SEEDLINK_HOST
  • Detector runtime: DETECTOR_MODE, DETECTOR_SB_PRETRAINED
  • Locator runtime: LOCATOR_POLL_SECONDS, LOCATOR_LOOKBACK_SECONDS, LOCATOR_ASSOCIATION_WINDOW_SECONDS, LOCATOR_MIN_STATIONS, LOCATOR_MIN_PICK_SCORE, LOCATOR_VP_KM_S, LOCATOR_VS_KM_S, LOCATOR_TRAVEL_TIME_MODEL, LOCATOR_TAUP_MODEL, LOCATOR_TAUP_P_PHASES, LOCATOR_TAUP_S_PHASES, LOCATOR_MAX_RESIDUAL_SECONDS, LOCATOR_LOG_LEVEL
  • Grafana admin: GRAFANA_USER, GRAFANA_PASSWORD

Template (.env.example):

# RabbitMQ
RABBITMQ_USER=guest
RABBITMQ_PASS=guest

# TimescaleDB/PostgreSQL
PGUSER=seis
PGPASSWORD=seis
PGDATABASE=seismic

# AMQP routing
AMQP_EXCHANGE=stations
CONSUMER_AMQP_BINDING_KEY=#
DETECTOR_AMQP_BINDING_KEY=GE.#

# SeedLink source (host:port)
SEEDLINK_HOST=geofon.gfz-potsdam.de:18000

# Detector runtime (Docker Compose detector service)
DETECTOR_MODE=seisbench
DETECTOR_SB_PRETRAINED=original

# Locator
LOCATOR_POLL_SECONDS=5.0
LOCATOR_LOOKBACK_SECONDS=600
LOCATOR_ASSOCIATION_WINDOW_SECONDS=8.0
LOCATOR_MIN_STATIONS=4
LOCATOR_MIN_PICK_SCORE=0.0
LOCATOR_VP_KM_S=6.0
LOCATOR_VS_KM_S=3.5
LOCATOR_TRAVEL_TIME_MODEL=taup
LOCATOR_TAUP_MODEL=/app/locator/models/graczer_weber_prem_hybrid.npz
LOCATOR_TAUP_P_PHASES=P,p
LOCATOR_TAUP_S_PHASES=S,s
LOCATOR_MAX_RESIDUAL_SECONDS=3.0
LOCATOR_LOG_LEVEL=INFO

# Grafana admin
GRAFANA_USER=admin
GRAFANA_PASSWORD=admin

Detector Modes

  • sta_lta: classic trigger detector, outputs event windows.
  • seisbench: SeisBench EQTransformer (pretrained), outputs event windows and phase picks.

Synthetic Testing

You can publish synthetic MiniSEED into RabbitMQ to exercise consumer and detector without SeedLink.

python3 tools/publish_mseed/publish_mseed.py --host 127.0.0.1 --exchange stations --event --event-probability 0.1 --event-amplitude 2500 --event-duration 20 --event-frequency 0.6

Docker option:

COMPOSE_PROFILES=tools docker compose run --rm publisher --host rabbitmq --exchange stations --count 3

Native Development

Build

Prerequisites: libslink, librabbitmq, libmseed, and libpq headers/libs available to your compiler.

make            # builds connector and consumer into ./build
make connector  # builds only connector
make consumer   # builds only consumer

Detector Native Run

Use this when running the detector directly on the host instead of Compose.

cd detector
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m detector.main --host 127.0.0.1 --exchange stations --pg-host 127.0.0.1 --detector-mode sta_lta

For SeisBench mode:

python -m detector.main --host 127.0.0.1 --exchange stations --pg-host 127.0.0.1 --detector-mode seisbench --sb-pretrained original

Locator Native Run

cd locator
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python main.py --pg-host 127.0.0.1 --pg-user seis --pg-password seis --pg-db seismic --log-level INFO

CLI Reference

Connector (SeedLink -> AMQP)

./build/connector [options] host[:port]
  -V                 report version
  -h                 show help
  -v                 increase verbosity (repeatable)
  -p                 print packet details
  -Ap                prompt for SeedLink user/password
  -At                prompt for SeedLink token
  -nd <secs>         reconnect delay (default 30)
  -nt <secs>         idle timeout (default 600)
  -k <secs>          keepalive interval
  -l <listfile>      stream list file (multi-station)
  -s <selectors>     selectors for all-station/default
  -S <streams>       NET_STA[:selectors], comma-separated
  -x <statefile>     save/restore sequence state
  --amqp-host host   AMQP host (default 127.0.0.1)
  --amqp-port port   AMQP port (default 5672)
  --amqp-user user   AMQP user (default guest)
  --amqp-password pw AMQP password (default guest)
  --amqp-vhost vhost AMQP vhost (default /)
  --amqp-exchange ex AMQP exchange (default empty)
  --amqp-routing-key k AMQP routing key/queue (default binq)

Consumer (AMQP -> TimescaleDB)

./build/consumer [opts]
  -h <amqp-host>         (default 127.0.0.1)
  -p <amqp-port>         (default 5672)
  -u <amqp-user>         (default guest)
  -P <amqp-pass>         (default guest)
  -v <amqp-vhost>        (default /)
  --amqp-exchange <ex>   Exchange to consume from (default empty = AMQP default)
  --amqp-binding-key <k> Key to bind queue when using an exchange (default queue name)
  -q <queue>             (default binq)
  --prefetch <n>         (default 10)
  --pg-host <host>       (default localhost)
  --pg-port <port>       (default 5432)
  --pg-user <user>       (default seis)
  --pg-password <pw>     (default seis)
  --pg-db <name>         (default seismic)

Detector (AMQP -> Detections + Picks)

python -m detector.main [opts]
  --host <amqp-host>             (default 127.0.0.1)
  --port <amqp-port>             (default 5672)
  --user <amqp-user>             (default guest)
  --password <amqp-pass>         (default guest)
  --vhost <amqp-vhost>           (default /)
  --exchange <amqp-exchange>     (default stations)
  --queue <queue>                (default empty for exclusive)
  --binding-key <key>            (repeatable, default "#")
  --prefetch <n>                 (default 50)
  --buffer-seconds <secs>        (default 120)
  --detect-every-seconds <secs>  (default 15)
  --preprocess-fmin <hz>         (default 0.1)
  --preprocess-fmax <hz>         (default 10.0)
  --sta-seconds <secs>           (default 6.0)
  --lta-seconds <secs>           (default 20.0)
  --trigger-on <v>               (default 2.5)
  --trigger-off <v>              (default 0.5)
  --pick-filter-seconds <secs>   (default 2.0)
  --detector-mode <mode>         (sta_lta or seisbench; default sta_lta)
  --sb-pretrained <name>         (default original)
  --sb-threshold-p <value>       (default 0.3)
  --sb-threshold-s <value>       (default 0.3)
  --sb-detection-threshold <v>   (default 0.3)
  --sb-device <cpu|cuda>         (default cpu)
  --log-level <level>            (default INFO)
  --pg-host <host>               (default localhost)
  --pg-port <port>               (default 5432)
  --pg-user <user>               (default seis)
  --pg-password <pw>             (default seis)
  --pg-db <name>                 (default seismic)

Database Schema

db/init/01_schema.sql defines three TimescaleDB hypertables:

  • seismic_samples: raw waveform samples (network, station, location, channel, sample_rate, samples, start_time)
  • event_detections: event windows (network, station, location, channel, ts_on, ts_off, trigger_value)
  • phase_picks: phase picks (network, station, location, channel, ts, phase, confidence)

See db/init/01_schema.sql for full schema definitions.

Troubleshooting

  • Connector exits quickly: verify SeedLink credentials and SEEDLINK_HOST.
  • Consumer cannot connect: check PGUSER, PGPASSWORD, and PGDATABASE.
  • No data in DB: confirm streamlist.conf, AMQP_EXCHANGE, CONSUMER_AMQP_BINDING_KEY, and DETECTOR_AMQP_BINDING_KEY.
  • Use docker compose logs -f connector consumer detector locator to inspect runtime errors.

TODO

  • Locator: add persisted incremental cursor/state so cycles process strictly new picks (not only lookback scans).
  • Locator: implement origin finalization policy (promote preliminary -> final) based on stability/quality criteria.
  • Locator: add multi-event and noisy-pick system tests (late picks, outliers, overlapping events).
  • Locator: improve association quality checks beyond pure time-window grouping.
  • Locator: add uncertainty metrics and better residual/outlier handling in the solver.

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages