JSON to Parquet on a single EC2 box

Convert the Spotify Million Playlist Dataset (MPD) ~156 GB of pretty-printed JSON - into compact ZSTD Parquet on one large EC2 instance, using single-node engines (DuckDB, PyArrow, polars). No cluster, no Spark.

The headline: a single m6id.8xlarge with local NVMe converted 256 GB of JSON to Parquet in 99 seconds with DuckDB, using ~5 GB of RAM. For data that still fits one machine's throughput, a fast single-node engine beats a distributed one on time, cost, and setup.

What it produces

Each MPD slice is one big pretty-printed JSON object ({info, playlists:[…]}), not newline-delimited. The output is one row per playlist, with the nested tracks array preserved as a LIST<STRUCT> column:

pid BIGINT, name VARCHAR, collaborative BOOLEAN, modified_at BIGINT,
modified_at_ts TIMESTAMPTZ, num_tracks INT, num_albums INT, num_followers INT,
num_edits INT, duration_ms BIGINT, num_artists INT, description VARCHAR,
tracks STRUCT(pos, artist_name, track_uri, artist_uri, track_name,
              album_uri, duration_ms, album_name)[]

All three engines emit this identical schema, so they can be compared directly on the same input and output.

The three converters

Script	Engine	Notes
src/json_to_parquet.py	DuckDB	Streaming; ~7 GB RAM regardless of dataset size. Fastest.
src/json_to_parquet_pyarrow.py	PyArrow	Streams batches into write_dataset.
src/json_to_parquet_polars.py	polars	Materializes each folder before writing (higher RAM).

Each runs one worker per source folder (ProcessPoolExecutor), so the 1,000 independent slices convert fully in parallel.

Quick start (local)

python -m venv .venv && . .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install -r requirements.txt

# unzip the MPD so slices land in ./data/*.json, then:
python src/json_to_parquet.py --per-folder --file-size-mb 128 \
    --data-dir data --out-dir parq --memory-limit-gb 100

DuckDB streams — peak RAM is a few GB even on hundreds of GB of input — so a modest 32–64 GB machine with a fast NVMe SSD can run this locally. You do not need the cloud (or a cluster) for this at all.

Running it on EC2

The full walkthrough — launching the instance, staging the data on local NVMe, running each engine, pushing Parquet to S3, and tearing down — is in docs/ec2_conversion.md.

Benchmark numbers and the run cost are in docs/benchmark_results.md.

Requirements

• Python ≥ 3.11
• duckdb, pyarrow, polars (pinned in requirements.txt / pyproject.toml)

License

MIT — see LICENSE.