An interactive choropleth visualization of U.S. domestic migration patterns using IRS Statistics of Income (SOI) tax data spanning 33 tax years, 1990–91 through 2022–23. Explore population, household, and income flows at the state and county level through a D3.js-powered map and time-series chart.
AGI availability: the two oldest years (1990–91 and 1991–92) have no adjusted gross income column in the original IRS files. AGI is stored as
0for these years, and the app automatically excludes them from the AGI category's charts and map slider (Population and Households are unaffected) — see Data Notes.
- Step 1 — Place raw source files
- Step 2 — Parse FIPS lookups
- Step 3 — Enrich migration CSVs
- Step 4 — Validate enriched files
- Step 5 — Build & chunk the SQLite database
| Requirement | Version | Notes |
|---|---|---|
| Python | 3.10 or later | Only the standard library is used — no pip install required |
| A modern web browser | Chrome, Firefox, Safari, or Edge | D3.js v7 is loaded from a CDN |
| A local HTTP server | One that supports HTTP Range requests (see below) |
Required to load the SQLite database — opening index.html directly with file:// will be blocked by CORS, and servers without Range support (e.g. Python's http.server) silently corrupt county-level data |
IRSMigrationDataProject/
├── index.html # Visualization entry point
├── styles.css # Design system (Milestone 2.2)
├── script.js # D3 logic (Phases 3–7, 11)
│
├── scripts/
│ ├── download_raw_data.py # Step 1: fetch 2008+ CSVs directly from irs.gov
│ ├── convert_legacy_xls_data.py # Step 1: fetch + convert pre-2008 XLS/legacy ZIPs to CSV
│ ├── parse_fips.py # Step 2: build FIPS lookup CSVs
│ ├── enrich_state_data.py # Step 3a: enrich state migration files
│ ├── enrich_county_data.py # Step 3b: enrich county migration files
│ ├── enrich_data.py # Convenience wrapper: runs steps 2–4 in one command
│ ├── validate_data.py # Step 4: validate all enriched outputs
│ ├── build_sqlite_db.py # Step 5a: consolidate enriched CSVs into database.sqlite
│ └── chunk_database.py # Step 5b: split database.sqlite for sql.js-httpvfs
│
└── data/
├── fips/
│ ├── all-geocodes-v2021.csv # Census geocode source (pre-2022 definitions)
│ ├── all-geocodes-v2025.csv # Census geocode source (2022+ CT planning regions)
│ ├── state_fips.csv # Generated — do not edit manually
│ └── county_fips.csv # Generated — do not edit manually
│
├── original/
│ ├── state_inflow/ # stateinflow9091.csv ... 2223.csv (33 files)
│ ├── state_outflow/ # stateoutflow9091.csv ... 2223.csv (33 files)
│ ├── county_inflow/ # countyinflow9091.csv ... 2223.csv (33 files)
│ └── county_outflow/ # countyoutflow9091.csv ... 2223.csv (33 files)
│
├── enriched/
│ ├── state_inflow/ # stateinflow*_enriched.csv — generated
│ ├── state_outflow/ # stateoutflow*_enriched.csv — generated
│ ├── county_inflow/ # countyinflow*_enriched.csv — generated
│ └── county_outflow/ # countyoutflow*_enriched.csv — generated
│
├── database.sqlite # Generated by build_sqlite_db.py — not committed
└── db_chunks/ # Generated by chunk_database.py — served to the browser
├── config.json # sql.js-httpvfs chunk config (size, count, cache-bust)
└── database.sqlite.000 ... # 40MB chunks of database.sqlite
All scripts must be run from the project root (the directory containing index.html).
The 2008–09 through 2022–23 tax years are published as flat per-direction CSVs; fetch them directly from IRS SOI Migration Data:
python scripts/download_raw_data.pyThis writes stateinflow0809.csv ... stateinflow2223.csv (and the state_outflow/
county_inflow/county_outflow equivalents) into data/original/.
Years before 2008–09 (back to 1990–91) are published in older per-state XLS workbooks (and, for
1990–91/1991–92 counties, fixed-width .txt files) bundled into year-specific ZIPs, with several
one-off column-layout quirks across the decades. convert_legacy_xls_data.py downloads and
normalizes each of these into the same flat CSV shape:
python scripts/convert_legacy_xls_data.py <year_tag> # e.g. 9091, 9192, ..., 0708Run it once per legacy year tag; see LEGACY_ZIPS in the script for the full supported list.
No AGI (income) column exists in the original 1990–91/1991–92 files — AGI is written as 0 for
those two years only (see Data Notes).
Download from Census Gazetteer / Geocodes (or the Census FTP archive):
| File | Destination | Notes |
|---|---|---|
all-geocodes-v2021.csv |
data/fips/ |
Pre-2022 county definitions |
all-geocodes-v2025.csv |
data/fips/ |
Includes CT planning regions |
It is possible to prepare the data in just one command:
python scripts/enrich_data.pyHowever, steps 2, 3, and 4 explain the role of the individual commands executed in the aforementioned one-command script.
python scripts/parse_fips.pyWhat it does: Reads both Census geocode CSVs and writes two unified lookup files:
data/fips/state_fips.csv—fips_code,state_name,state_postaldata/fips/county_fips.csv—state_fips,county_fips,county_name,state_name,state_postal(combines all pre-2022 counties plus Connecticut's 9 planning regions from the 2025 vintage)
Expected output:
Parsing data/fips/all-geocodes-v2021.csv …
52 state entries, 3,221 county entries
...
Wrote 52 rows → data/fips/state_fips.csv
Wrote 3,230 rows → data/fips/county_fips.csv
✓ Both pre-2022 and 2022+ CT geographies are present
Note: You will see a warning that Puerto Rico (FIPS 72) has no postal code. This is expected — Puerto Rico is not included in the IRS migration data.
Run the two enrichment scripts in either order:
python scripts/enrich_state_data.py
python scripts/enrich_county_data.pyEach script reads the raw IRS files from data/original/ and writes enriched versions to
data/enriched/, adding the missing state name, state postal code, and county name columns
derived from the FIPS lookup files.
Enriched state schema:
y2_state, y2_state_name, y2_statefips,
y1_statefips, y1_state, y1_state_name,
n1, n2, AGI
Enriched county schema:
y2_state, y2_state_name, y2_statefips, y2_countyfips, y2_county_name,
y1_statefips, y1_countyfips, y1_state, y1_state_name, y1_county_name,
n1, n2, AGI
All FIPS codes in the enriched files are zero-padded strings (2 digits for state, 3 for county), not integers.
python scripts/validate_data.pyChecks a batch of enriched files (currently 56 — not yet extended to every Phase 11 year) against five criteria and prints a pass/fail report:
- Row counts match raw originals
- No unexpected empty values in key columns
- Special aggregate FIPS codes (96, 97, 98) are present
- All FIPS codes are correctly zero-padded
- All Connecticut county rows resolve to a non-empty name
A clean run ends with:
Result: 56/56 files passed all checks
✓ All validations passed.
For a faster row-count-only check:
python scripts/validate_data.py --quickpython scripts/build_sqlite_db.py
python scripts/chunk_database.pybuild_sqlite_db.py consolidates every enriched CSV into a single indexed data/database.sqlite.
chunk_database.py then splits it into 40MB chunks under data/db_chunks/ and writes
config.json, which the front-end's sql.js-httpvfs loader uses to fetch only the byte ranges
it needs via HTTP Range requests (see below) rather than downloading the whole database.
Whenever the database is rebuilt, re-run
chunk_database.pytoo —script.js'sDB_LENGTH_BYTESconstant must match the freshly built database's exact byte size (it doubles as both a correctness check and a cache-buster for the chunk files). A stale value silently corrupts reads instead of failing loudly.
Because script.js loads data via fetch(), you need a local HTTP server — opening
index.html directly with file:// will fail due to browser CORS restrictions.
Important: the app queries data/database.sqlite remotely via sql.js-httpvfs,
which reads the database in chunks using HTTP Range requests. Your local server
must support Range requests (return 206 Partial Content), or county-level
data will silently fail to load (state data still works, since its reads happen to
land near the start of the file). Python's built-in http.server does not
support Range requests — avoid it for this project.
Option A — Node.js serve (recommended):
npx serve .Option B — VS Code Live Server extension:
Right-click index.html → Open with Live Server.
| Column | Meaning |
|---|---|
y1 |
Receiving geography (inflow files) / origin geography (outflow files) |
y2 |
Sending geography (inflow files) / destination geography (outflow files) |
n1 |
Number of households |
n2 |
Number of individuals |
AGI |
Adjusted gross income (thousands of dollars) |
Special aggregate FIPS codes used by the IRS (not real geographies):
| State FIPS | Meaning |
|---|---|
96 |
Total migration — U.S. and Foreign |
97 |
Total migration — U.S. only |
98 |
Total migration — Foreign only |
57 |
Foreign |
58 |
Same-state aggregate (no label; pass-through from raw file) |
59 |
Different-state aggregate (no label; pass-through from raw file) |
Connecticut geography change: Starting with the 2021–22 IRS files, Connecticut's
8 traditional counties were replaced by 9 planning regions in the Census FIPS system.
The unified county_fips.csv includes both sets of geographies so all year-ranges
resolve correctly without any per-file handling.
AGI availability (1990–91 and 1991–92): these two years' original IRS files have no AGI
column at all. AGI is stored as 0 for both years rather than left blank, on the condition
that the UI never actually surfaces it: script.js's YEARS_WITHOUT_AGI set and
getYearsForCategory() helper exclude both years from the Individual Trend and Pairwise
Comparison charts' x-axes, and from the Map's year slider (both its selectable range and its
visible ticks), whenever the AGI category is selected. Population and Households are
unaffected. If the Map is sitting on one of these years and the category switches to AGI, the
year snaps forward to 1992–93 in the same interaction.
The visualization supports 32 distinct metrics across different categories:
| Group | Metrics |
|---|---|
| Population | Inflow · Outflow · Net · Inflow rate · Outflow rate · Net rate · Inbound Rate · Outbound Rate |
| Households | Inflow · Outflow · Net · Inflow rate · Outflow rate · Net rate · Inbound Rate · Outbound Rate |
| AGI | Inflow · Outflow · Net · Inflow rate · Outflow rate · Net rate · Inbound Rate · Outbound Rate |
| Average AGI | Avg per individual moving in · Avg per household moving in · Avg per individual moving out · Avg per household moving out |
| AGI Ratio | Ratio of Avg In-Migrant Ind. to Out-Migrant Ind. · Ratio of Avg In-Migrant HH to Out-Migrant HH · Ratio of Avg Out-Migrant Ind. to In-Migrant Ind. · Ratio of Avg Out-Migrant HH to In-Migrant HH |
"Share" metrics express a region's flow as a fraction of its total flow (IRS aggregate code 96).