Automate menu-spending PDF pipeline (fetch → extract → validate → geocode) + 2024–2026 data#47
Open
HarryBrisson wants to merge 14 commits into
Open
Automate menu-spending PDF pipeline (fetch → extract → validate → geocode) + 2024–2026 data#47HarryBrisson wants to merge 14 commits into
HarryBrisson wants to merge 14 commits into
Conversation
…als from a PDF scripts_automation/validate_extraction.py runs Sean's extract_pdf_data on a CIP menu PDF, cleans cost, aggregates per-ward totals, and compares to the matching year in '2005-2023 menu spending.csv'. Proves the automated extraction is faithful before trusting it on newly-published years (2024-H2, 2025+). Needs a local PDF (the CIP archive WAFs scripted access).
The city retired the HTML index page (cip-archive.html -> CapitalPublications.html) and WAFs scripted access to it, and PDF filenames are inconsistent across years/formats -- so URLs can't be guessed. But the /content/dam/ asset files aren't gated, and Wayback keeps un-gated snapshots of the index pages that LIST the assets. fetch_menu_pdfs.py discovers URLs from those snapshots + a verified manifest, normalizes (percent-encodes spaces), and downloads directly. Pulls the full span 2012->2025 Q2 incl current-term data. Coverage: Sean's extractor (2019+ layout) extracts the 2019-2023 annual postings AND the 2024/2025 quarterly 'AMR'/'Menu Report' formats cleanly (50 wards, sane totals); pre-2019 PDFs use an older layout that needs a separate extractor (but Sean's 2005-2023 CSV already covers them).
…ck lag)
Recent quarterly menu reports use 3 inconsistent filename styles ('Q1 2025 Aldermanic Menu Program
Report', 'Menu Report 2025 Q2', '2026 Q1 Menu Report'), so probe_recent_quarters() tries each style
per (year, quarter) from 2024 forward via cheap HEAD-style GETs. Auto-discovers current-term reports
without waiting for a Wayback snapshot. Now pulls 2012 -> 2026 Q1 incl full-year 2025 (Menu Report
2025 Q4). Confirmed the /content/dam/ assets need no session cookie -- just a browser UA.
…'s data to current term process_gap_years.py runs the fetched quarterly reports through Sean's own extractor + categorizer (schema-matched), tagging each with (year, period). 6,895 categorized line items: 2024 thru Q3 ($69.9M), 2025 full-year ($70.4M), 2026 Q1 ($11M). The 2025 Q1->Q2->Q4 cumulative sequence enables a spending-rate metric. Resets extract_pdf_data's module-global state between PDFs.
spending_metrics.py computes, with no geocoding needed: spending RATE (from cumulative quarterly reports -- how fast a ward deploys its $1.5M + live budget-left), project COUNT/SIZE, and FOCUS (Herfindahl category concentration). Real 2025/2026 results surface big variation: front-loaded full-spenders (Wards 41/7/5 at ~100% by mid-year) vs slow/under-spenders (Ward 33: 0% by mid, 56% used). Spread-vs-centralized metric still needs geocoding (next).
Centroid of each project's geocoded geometry -> per-ward spatial standard deviation of dollars (km). Prototyped on Sean's 2019-2022 geocoded geojson (no geocoding needed): compact lakefront wards cluster (48: 0.79km), sprawling SW-side wards fan out (12: 7.65km); median 1.82km. Caveat noted: raw distance partly tracks ward area; normalize by ward extent for a pure choice signal.
data/2024-2026 menu spending.csv extends Sean's series to the current term (annual cumulative snapshot per year: 2024 thru Q3, 2025 full-year, 2026 thru Q1), Sean's column schema. geocode_gap_years.py geocodes the gap locations via the API geocoder with a resumable JSONL cache + emits a geojson matching the 2019-2022 schema so spread_metric runs on it. Spread + spending metrics consume these.
…ted ward totals) Cross-checks two independent reads of each PDF: our pipeline summed per ward vs the 'WARD COMMITTED <year> TOTAL' line the city prints. Self-referential, so it works on ANY report regardless of layout; if the printed-total marker itself vanishes (deeper format change) it reports UNVALIDATABLE loudly instead of passing bad data. Grades by fidelity (FLOOR 98%) since the coordinate extractor drops ~0.5% of line items. Across data/pdf/: 11 pass (2019-2026, 98.95-99.83% captured), 1 fail (2017 layout our extractor can't read), 7 unvalidatable (pre-2019 / no marker). Non-zero exit gates CI / post-fetch. Finding: our extraction is closer to the city's true totals than Sean's CSV (2019: ours 99.65% vs his 98.9%).
geocode_gap_years.py: ThreadPoolExecutor (8 workers, I/O-bound) -> finishes in ~2min foreground instead of ~50min; resumable JSONL cache unchanged. Geocoded 99% of gap-year locations (4858/4902); the ~1% misses are PDF artifacts (e.g. 'WLUNT AVE' = 'W LUNT'). spread_metric.py: trims points >12km from the ward MEDIAN center before computing standard distance (one bad geocode otherwise dominates -- Ward 42 read 40km, now correctly 0.74km as the most-concentrated/Loop ward); parameterized --geojson/--year. 2025 current-term spread: 0.74-5.30km, median 1.92km.
normalize_location() repairs the geocoder's known rejections (DuSable LSD rename, SB/NB bound suffix, Broadway's missing street-type) and a repair pass retries only the NULL locations (can't regress good geocodes). Recovers 23/36 -> 99.7% geocoded (4887/4902). The residual 14 are written to data/geocode/ungeocoded.txt and are the verbose 'ON X FROM Y TO Z (range)' segment format the geocoder can't parse (a candidate for a future segment-parser extension) plus 2 named places.
2005-2025 (complete years only; 2024-Q3/2026-Q1 partials skipped so utilization + latest-year scoring stay honest). active_transport_share/budget_utilization/project_diversity per year from Sean's 2005-2023 + our 2025; spending_spread current-era (2025) only (historical needs era-matched ward boundaries). budget_utilization uses per-year allotment. Output keyed by year for period-anchored ingestion.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Automates a pipeline around the existing menu-spending extraction so the dataset can be kept current without manual PDF downloads, plus the 2024–2026 data it produces. Everything lives in a new
scripts_automation/dir and builds on the existingward_spendingextractor + geocoder rather than replacing them.Context: the city retired the old CIP archive index page (
cip-archive.html→CapitalPublications.html) and WAF-blocks scripted access to it, but the/content/dam/PDF assets themselves are open. These tools work around that.What's included
fetch_menu_pdfs.py: discovers PDF URLs from Wayback snapshots of the (un-gated) index pages + a verified manifest, and probes recent quarterly reports across their inconsistent naming styles. Pulls 2012 → 2026 Q1.process_gap_years.py: runs the fetched quarterly reports through the existingextract_pdf_data+post_processor, schema-matched to2005-2023 menu spending.csv(resets the extractor's module-global state between PDFs).validate_extraction.pyreproduces published per-ward totals (2019 within 0.81%);validate_self_consistency.pycross-checks our summed line items against the city's own printedWARD COMMITTED <year> TOTALlines — format-robust, and reports UNVALIDATABLE loudly if that marker ever disappears (11 pass / 1 fail / 7 unvalidatable across the corpus).geocode_gap_years.py: parallelized + resumable JSONL cache via the existingGeoCoderAPI, with anormalize_location()repair pass (DuSable LSD rename, SB/NB suffix, Broadway's missing street-type). 99.7% geocoded; residuals documented.spending_metrics.py,spread_metric.py.data/2024-2026 menu spending.csv(annual cumulative snapshot per year).Notes for review
2005-2023 menu spending.csvhas ~$827k. Could be a real underspend or a gap in the existing data — worth a look.ON STREET FROM CROSS TO CROSS (range)segment format (recent reports) isn't parsed — 12 of 14 residual locations. Afrom…tosegment parser would close it.