This program converts a full GFS run (GRIB2 files) into SQLite.

Features

• Querying all parameters for any grid cell takes ~60 ms—fast enough for APIs while staying light on memory/CPU; for analytics I'm working on grib_to_duckdb.
• Each file is processed in seconds; units are converted, wind speed/direction are computed, and hourly precipitation is derived.

Why this program is fast

• Written in Go with minimal overhead.
• Processes GRIB files in memory (little disk I/O).
• Uses stdin for bulk inserts into SQLite.

Dependencies

• GDAL (extract grid layout and values)
• QGIS (polygonize grid)
• SpatiaLite (geospatial types and queries)
• sqinn (write to SQLite via stdin)
• mattn/go-sqlite3 (work around sqinn limitations)

Schema

• [table] grid_data
  • [column] date
  • [column] x
  • [column] y
  • [column] parameter1
  • [column] parameter2
  • [column] parameter...
• [table] grid_layout
  • [column] x
  • [column] y
  • [column] lat
  • [column] lon
  • [column] geom

Try it out

1. Download GFS data

# you might need to update the url; check https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/
mkdir -p data/gfs && cd data/gfs
for x in $(seq 0  384); wget https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20260212/18/atmos/gfs.t18z.pgrb2.1p00.f$(printf "%03d" $x);

2. Convert GRIB to SQLite

docker compose up -d                 
docker compose exec grib2-sqlite bash
# Output is stored in the Docker volume for performance; feel free to mount elsewhere.
time go run grib2_to_sqlite.go -in ./data/gfs -out ./output/tmp.sqlite -workers 8

3. Testing Queries

sqlite3 output/tmp.sqlite
.timer on

SELECT load_extension('/opt/conda/lib/mod_spatialite.so.8.1.0');

-- EXPLAIN QUERY PLAN
WITH pt AS (
  SELECT MakePoint(41.235, 23.237, 4326) AS g
)
SELECT gd.*
FROM idx_grid_layout_geom r
JOIN grid_layout gl ON gl.ROWID = r.pkid
JOIN grid_data gd   ON gd.x = gl.x AND gd.y = gl.y
JOIN pt
WHERE r.xmin <= MbrMaxX(pt.g)
  AND r.xmax >= MbrMinX(pt.g)
  AND r.ymin <= MbrMaxY(pt.g)
  AND r.ymax >= MbrMinY(pt.g)
  AND ST_Intersects(gl.geom, pt.g);

Benchmark

# query_1.go measures the time to query a random grid point (all parameters × all times)
go run query_1.go -db output/tmp.sqlite -n 100 -v