Local-Level Data Guide — Pipeline

Reproducible Python pipeline behind the illustrated guide to creating sub-county geographic datasets (Arlington County: broadband × income → civic associations).

Setup

uv venv --python 3.12 && uv sync
export CENSUS_API_KEY=your_key   # free: https://api.census.gov/data/key_signup.html
# or put CENSUS_API_KEY=your_key in a .env file (gitignored)

R (optional)

The redistribution algebra is also available in R via the sdc.redistribute package, which powers the parallel pipeline in pipeline-r/:

# released package from CRAN:
install.packages("sdc.redistribute")
# or the development version from the repository:
# install.packages("pak"); pak::pak("dads2busy/sdc.redistribute")

Then run the R pipeline from the repository root with Rscript pipeline-r/run.R.

Run

uv run python -m pipeline.run

Outputs land in data/:

• civic_income.csv, civic_broadband.csv — redistributed measures
• civic_combined.geojson — joined + derived metrics (ratio, bivariate class)
• civic_income_parcels.csv — parcel-centroid income redistribution via pipeline.acquire_parcels + redistribute_income_parcels
• civic_income_comparison.geojson — side-by-side comparison of area-weighted vs parcel-based estimates (compare_methods)

Method

Counts (aggregate income, households, broadband tests) are redistributed area-weighted with sdc-redistribute; intensive rates (mean income, mean speed) are derived from count ratios. The guide now includes a second, parcel-based approach that assigns block-group income to civic associations using parcel centroids as a spatial proxy, with a comparison chapter showing where the two methods agree and diverge. See docs/superpowers/specs/.

Figures

Regenerate every map, bivariate chart, scatter, and flowchart used in the guide:

uv run python -m pipeline.build_figures   # writes figures/ and figures/diagrams/

Requires the Graphviz dot binary (brew install graphviz) for the flowcharts.

Tests

uv run pytest -m "not network"   # fast, deterministic
uv run pytest -m network         # live Census/Ookla acquisition

The Guide

The illustrated guide lives in guide/ (Quarto book). Render it:

cd guide && quarto render          # PDF (primary) + HTML site -> guide/_output/

Figures come from pipeline.build_figures; the data from pipeline.run.

The final chapter is the full pipeline source, generated from pipeline/*.py. Regenerate it after changing pipeline code:

uv run python scripts/build_code_appendix.py   # rewrites guide/08-code.qmd

Published site: https://uva-bi-sdad.github.io/local-level-data-guide/