Market Forge

A resumable Yahoo Finance historical market data downloader designed for deployment with docker and docker compose.

Plombery provides the Web UI, manual execution, execution logs, and scheduled triggers. PostgreSQL stores symbols, download checkpoints, chunk-level audit records, and normalized OHLCV bars.

Features and Guarantees

• Reads symbols from a CSV file containing a symbol column.
• Normalizes common ticker formats for Yahoo Finance:
  • US.AAPL → AAPL
  • NASDAQ:AAPL → AAPL
  • BRK.B → BRK-B
• Uses 2000-01-01 as the default start date.
• Uses the current date in America/New_York as the default end date.
• Tracks download progress by symbol, interval, and next_start.
• Downloads up to 10,000 calendar days per chunk by default.
• Commits bar writes and checkpoint advancement in the same PostgreSQL transaction.
• Uses (symbol_id, interval, bar_time) as the idempotency key for bar UPSERT operations.
• Uses FOR UPDATE SKIP LOCKED and database leases to prevent duplicate task claims and recover interrupted work.
• Isolates symbol-level failures so one failed symbol does not stop the rest of the batch.
• Automatically continues historical backfill every 10 minutes.
• Prepares a seven-day overlapping incremental refresh window at 20:15 Eastern Time on US weekdays.
• Stores unadjusted Yahoo Finance prices by default with AUTO_ADJUST=false.

The project currently pins Plombery to version 0.5.2.

Plombery is suitable for single-node Python scheduling and operational monitoring. This deployment should not be horizontally scaled into a large number of Plombery replicas.

Architecture

symbols.csv
    │
    ▼
Plombery Pipelines
    │
    ├── seed_market_symbols
    ├── download_market_data
    └── prepare_daily_market_refresh
    │
    ▼
Yahoo Finance / yfinance
    │
    ▼
PostgreSQL
    ├── md_symbols
    ├── md_download_checkpoints
    ├── md_download_chunks
    ├── md_bars
    └── md_bars_yahoo

Requirements

• Docker
• Docker Compose
• A cloud server with persistent storage
• Outbound access to Yahoo Finance
• At least 30–50 GB of available disk space for a large symbol universe

Configuration

Copy the example environment file:

cp .env.example .env

Example configuration:

POSTGRES_DB=market_data
POSTGRES_USER=market_data
POSTGRES_PASSWORD=replace-with-a-strong-password

PLOMBERY_PORT=8000

DEFAULT_START_DATE=2000-01-01
DEFAULT_INTERVAL=1d

DOWNLOAD_BATCH_SIZE=50
CHUNK_DAYS=10000
MAX_ATTEMPTS=3

REQUEST_PAUSE_MIN=0.5
REQUEST_PAUSE_MAX=1.5
DOWNLOAD_TIMEOUT_SECONDS=30

DAILY_OVERLAP_DAYS=7
AUTO_ADJUST=false

Configuration Reference

Variable	Default	Description
POSTGRES_DB	market_data	PostgreSQL database name
POSTGRES_USER	market_data	PostgreSQL user
POSTGRES_PASSWORD	—	PostgreSQL password; must be changed for production
PLOMBERY_PORT	8000	Host port used for the Plombery Web UI
DEFAULT_START_DATE	2000-01-01	Default historical download start date
DEFAULT_INTERVAL	1d	Default Yahoo Finance bar interval
DOWNLOAD_BATCH_SIZE	50	Maximum number of symbol checkpoints claimed per run
CHUNK_DAYS	10000	Maximum calendar-day range downloaded per symbol and chunk
MAX_ATTEMPTS	3	Maximum download attempts before a task remains failed
REQUEST_PAUSE_MIN	0.5	Minimum random pause between symbol requests, in seconds
REQUEST_PAUSE_MAX	1.5	Maximum random pause between symbol requests, in seconds
DOWNLOAD_TIMEOUT_SECONDS	30	Timeout for a Yahoo Finance request
DAILY_OVERLAP_DAYS	7	Number of recent days re-downloaded during daily refresh
AUTO_ADJUST	false	Whether yfinance should automatically adjust OHLC prices

For a production deployment, use a strong PostgreSQL password and store secrets through an appropriate secret-management mechanism.

Start the Services

Build and start the containers:

docker compose up -d --build

Check service status:

docker compose ps

Open the Plombery Web UI:

http://<server-ip>:8000

Follow application logs:

docker compose logs -f plombery

Follow PostgreSQL logs:

docker compose logs -f postgres

Symbol Input

Edit:

data/symbols.csv

The file must contain a symbol column:

symbol
AAPL
INTC
QQQ
US.NVDA
BRK.B

Blank values are ignored. Symbols that normalize to the same Yahoo Finance ticker should be treated as duplicates.

Initial Backfill

After the services are running:

1. Open the Plombery Web UI.
2. Run the seed_market_symbols pipeline manually.
3. Run the download_market_data pipeline once manually.
4. Allow the continue_backfill trigger to continue processing every 10 minutes.

seed_market_symbols

This pipeline:

• Reads data/symbols.csv.
• Normalizes each symbol into Yahoo Finance format.
• Inserts or updates records in md_symbols.
• Creates or updates symbol-level checkpoints.
• Uses DEFAULT_START_DATE as the initial download start date.
• Uses the current date in America/New_York when no explicit end date is supplied.

download_market_data

This pipeline:

• Claims up to DOWNLOAD_BATCH_SIZE checkpoints.
• Uses PostgreSQL row locking with FOR UPDATE SKIP LOCKED.
• Assigns a lease to each claimed checkpoint.
• Downloads one date chunk for each claimed symbol.
• Writes bars using idempotent UPSERT operations.
• Advances the checkpoint in the same transaction as the bar write.
• Records chunk-level success or failure.
• Continues processing other symbols when one symbol fails.

With the supplied configuration, one run claims up to 50 symbols, and each symbol can download up to 10,000 calendar days in one chunk.

continue_backfill

The scheduled backfill trigger runs every 10 minutes.

It continues processing symbols whose checkpoint status indicates that more data remains to be downloaded.

A completed symbol is not downloaded again unless:

• Its target end date is extended.
• A daily overlapping refresh is prepared.
• Its checkpoint is reset.
• A manual rebuild is requested.

Daily Incremental Refresh

The daily refresh preparation runs at:

20:15 America/New_York
Monday through Friday

It does not directly download market data. Instead, it:

1. Updates the target end date to the current Eastern Time date.
2. Moves each checkpoint back by DAILY_OVERLAP_DAYS.
3. Marks the checkpoint as requiring additional processing.
4. Allows subsequent download_market_data runs to re-download the overlapping range.

With the default configuration:

DAILY_OVERLAP_DAYS=7

The downloader re-fetches the latest seven days and writes them using UPSERT operations. This allows recent Yahoo Finance corrections to replace previously stored values without creating duplicate bars.

Data Consistency

Checkpoint Transaction

For each successfully downloaded symbol chunk, the following operations are committed together:

UPSERT md_bars
UPDATE md_download_chunks
ADVANCE md_download_checkpoints.next_start
UPDATE md_download_checkpoints.last_success_date
RELEASE checkpoint lease
COMMIT

If the process stops before the transaction commits, neither the bars nor the checkpoint advancement become visible.

The next run therefore downloads the same chunk again safely.

If the transaction has already committed, the next run continues from the updated next_start.

Idempotency

Bars use the following logical key:

symbol_id + interval + bar_time

The database performs an UPSERT when the same bar is downloaded again.

This supports:

• Process restarts
• Overlapping date windows
• Yahoo Finance corrections
• Manual retries
• Daily incremental updates

Database Tables

md_symbols

Stores original and normalized ticker identifiers.

Typical fields include:

symbol_id
source_symbol
provider
provider_symbol
enabled
created_at
updated_at

md_download_checkpoints

Stores symbol-level download progress.

Typical fields include:

symbol_id
interval
requested_start
target_end
next_start
last_success_date
status
attempts
downloaded_rows
last_error
lease_owner
lease_expires_at

md_download_chunks

Stores a detailed audit record for each downloaded date range.

Typical fields include:

checkpoint_id
chunk_start
chunk_end
status
attempts
row_count
first_data_date
last_data_date
error_message
started_at
completed_at

md_bars

Stores normalized OHLCV bars.

The primary key is:

symbol_id + interval + bar_time

With AUTO_ADJUST=false, OHLC fields contain Yahoo Finance unadjusted prices.

md_bars_yahoo

Provides a convenient view that joins bars with Yahoo Finance ticker information.

Query Examples

Query Recent Bars

SELECT *
FROM md_bars_yahoo
WHERE yahoo_symbol = 'AAPL'
  AND interval = '1d'
ORDER BY bar_time DESC
LIMIT 100;

Query Download Progress

SELECT
    status,
    COUNT(*) AS symbols,
    SUM(downloaded_rows) AS rows,
    MIN(next_start) AS earliest_next_start,
    MAX(last_success_date) AS latest_success_date
FROM md_download_checkpoints
GROUP BY status
ORDER BY status;

Query Failed Symbols

SELECT
    s.provider_symbol,
    c.next_start,
    c.target_end,
    c.attempts,
    c.last_error,
    c.updated_at
FROM md_download_checkpoints AS c
INNER JOIN md_symbols AS s
    ON s.symbol_id = c.symbol_id
WHERE c.status = 'failed'
ORDER BY c.updated_at DESC;

Query Bar Counts by Symbol with a Total Row

SELECT
    CASE
        WHEN GROUPING(s.provider_symbol) = 1 THEN 'TOTAL'
        ELSE s.provider_symbol
    END AS symbol,
    COUNT(*) AS bar_count
FROM md_bars AS b
INNER JOIN md_symbols AS s
    ON s.symbol_id = b.symbol_id
GROUP BY ROLLUP (s.provider_symbol)
ORDER BY
    GROUPING(s.provider_symbol),
    s.provider_symbol;

Query Symbols That Still Require Downloading

SELECT
    s.provider_symbol,
    c.status,
    c.next_start,
    c.target_end,
    c.last_success_date
FROM md_download_checkpoints AS c
INNER JOIN md_symbols AS s
    ON s.symbol_id = c.symbol_id
WHERE c.next_start <= c.target_end
ORDER BY
    c.next_start,
    s.provider_symbol;

TradeComposer Integration

When this service and TradeComposer share the same PostgreSQL instance, a TradeComposer importer can read directly from:

md_symbols
md_bars

The importer should map:

md_symbols.provider_symbol
    ↓
TradeComposer instrument_id

The TradeComposer destination table should use an idempotency key equivalent to:

instrument_id + timeframe + datetime

and write data using UPSERT semantics.

The downloader staging schema should remain separate from TradeComposer business tables. This separation provides:

• Safe reprocessing
• Independent auditing
• Easier schema migration
• Clear data ownership
• Reduced coupling between ingestion and trading logic

A recommended flow is:

Yahoo Finance
    ↓
Downloader staging tables
    ↓
Validated TradeComposer importer
    ↓
TradeComposer market-data tables

Operations

Open a PostgreSQL Shell

docker compose exec postgres \
  psql \
  -U market_data \
  -d market_data

When custom credentials are configured, use the corresponding values from .env.

Rebuild the Application Container

docker compose up -d --build plombery

Restart Services

docker compose restart

Stop Services

docker compose down

Delete All Stored Data

Warning: this removes the PostgreSQL volume, including all bars and checkpoints.

docker compose down -v
docker compose up -d --build

Back Up PostgreSQL

docker compose exec -T postgres \
  pg_dump \
  -U market_data \
  -d market_data \
  --format=custom \
  > market_data.dump

Restore PostgreSQL

cat market_data.dump | docker compose exec -T postgres \
  pg_restore \
  -U market_data \
  -d market_data \
  --clean \
  --if-exists

Production Considerations

Protect the Web UI

Do not expose port 8000 directly to the public Internet without authentication.

Use one or more of the following:

• Cloud firewall rules
• A VPN
• An authenticated reverse proxy
• Plombery OAuth support
• IP allowlisting

PostgreSQL Initialization

The initialization script under:

sql/001_init.sql

runs automatically only when PostgreSQL creates a new, empty data directory.

It does not run again for an existing Docker volume.

When using an external PostgreSQL instance, execute the script manually:

psql "$DATABASE_URL" -f sql/001_init.sql

For later schema changes, use explicit versioned migration scripts rather than editing only 001_init.sql.

Large Symbol Universes

For approximately 10,000 symbols:

• Use a single Plombery application instance.
• Keep Yahoo Finance request concurrency conservative.
• Monitor the failed-task rate.
• Monitor PostgreSQL disk usage.
• Monitor checkpoint progress.
• Avoid running multiple uncontrolled backfill workers.
• Consider PostgreSQL table partitioning or TimescaleDB if the dataset expands to intraday bars.

The supplied configuration claims up to 50 symbols per batch:

DOWNLOAD_BATCH_SIZE=50

This is more aggressive than a 20-symbol batch and should be monitored for Yahoo Finance throttling.

Storage

For a large daily-bar universe, reserve at least:

30–50 GB

Actual usage depends on:

• Number of valid symbols
• Listing history
• Number of intervals
• PostgreSQL index size
• Retained audit history
• WAL and backup retention

Yahoo Finance Limitations

Yahoo Finance and yfinance are not official exchange data sources.

Possible issues include:

• HTTP throttling
• Temporary empty responses
• Delisted or renamed tickers
• Historical corrections
• Corporate-action revisions
• API behavior changes
• Incomplete coverage for some securities

The checkpoint and UPSERT design protects against interrupted processing, but it cannot guarantee the correctness or completeness of Yahoo Finance source data.

Default Scheduling Summary

Pipeline or Trigger	Schedule	Purpose
seed_market_symbols	Manual	Import symbols and initialize checkpoints
download_market_data	Manual	Run an immediate download batch
continue_backfill	Every 10 minutes	Continue unfinished historical downloads
Daily refresh preparation	Weekdays at 20:15 ET	Reopen the latest seven-day update window

License

Apache 2.0 License