Hh backfill adhoc new by huangh · Pull Request #755 · mbta/lamp

huangh · 2026-05-26T19:22:35Z

Asana Task: <asana_ticket_url>

What changes does this PR propose?

Describe the changes, details you want to call attention to, and any context you want to add.

How were these changes validated?

New tests?
Used a runner?
Doesn't need validation? Maybe it's just documentation.

What questions should reviewers consider?

Focus on this thing, even though it seems minor!

… including rowgroups and size of files after changes. includes some scrips and runners

…on code that should probably be rehomed

github-actions · 2026-05-28T15:34:33Z

LCOV of commit `6649de6` during Continuous Integration (Python) #1951

Summary coverage rate:
  lines......: 65.0% (3629 of 5580 lines)
  functions..: 29.9% (275 of 920 functions)
  branches...: no data found

Files changed coverage rate:
                                                                                     |Lines       |Functions  |Branches    
  Filename                                                                           |Rate     Num|Rate    Num|Rate     Num
  =========================================================================================================================
  src/lamp_py/ad_hoc/backfill_runner_terminal_predictions.py                         | 0.0%     18|    -     0|    -      0
  src/lamp_py/ad_hoc/pipeline.py                                                     | 0.0%     14| 0.0%     2|    -      0
  src/lamp_py/aws/s3.py                                                              |52.2%    320|23.8%    42|    -      0
  src/lamp_py/ingestion/backfill/delta_reingestion.py                                |60.7%     61|25.0%     4|    -      0
  src/lamp_py/ingestion/compress_gtfs/gtfs_schema_map.py                             |97.7%     44|50.0%     4|    -      0
  src/lamp_py/ingestion/compress_gtfs/gtfs_to_parquet.py                             |69.0%    100|40.0%    10|    -      0
  src/lamp_py/ingestion/compress_gtfs/pq_to_sqlite.py                                |87.8%     49|50.0%     6|    -      0
  src/lamp_py/ingestion/compress_gtfs/schedule_details.py                            |81.2%     96|50.0%     8|    -      0
  src/lamp_py/ingestion/config_busloc_trip.py                                        |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/config_rt_trip.py                                            |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/convert_gtfs.py                                              |79.2%     48|37.5%     8|    -      0
  src/lamp_py/ingestion/convert_gtfs_rt.py                                           |88.6%    237|50.0%    26|    -      0
  src/lamp_py/ingestion/convert_gtfs_rt_fullset.py                                   |45.6%    114|16.7%    18|    -      0
  src/lamp_py/ingestion/converter.py                                                 |95.9%     49|35.0%    20|    -      0
  src/lamp_py/ingestion/daily/config.py                                              | 100%      2|    -     0|    -      0
  src/lamp_py/ingestion/daily/trip_updates.py                                        |75.0%     12|25.0%     4|    -      0
  src/lamp_py/ingestion/glides.py                                                    |95.6%    182|45.5%    44|    -      0
  src/lamp_py/ingestion/gtfs_rt_detail.py                                            |94.1%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/ingest_gtfs.py                                               | 0.0%     61| 0.0%    10|    -      0
  src/lamp_py/ingestion/pipeline.py                                                  | 0.0%     38| 0.0%     4|    -      0
  src/lamp_py/ingestion/utils.py                                                     |63.1%    111|35.0%    20|    -      0

huangh · 2026-05-28T15:39:11Z

github-actions · 2026-05-28T15:39:43Z

LCOV of commit `36c6804` during Continuous Integration (Python) #1952

Summary coverage rate:
  lines......: 65.2% (3637 of 5580 lines)
  functions..: 29.9% (275 of 920 functions)
  branches...: no data found

Files changed coverage rate:
                                                                                     |Lines       |Functions  |Branches    
  Filename                                                                           |Rate     Num|Rate    Num|Rate     Num
  =========================================================================================================================
  src/lamp_py/ad_hoc/backfill_runner_terminal_predictions.py                         | 0.0%     18|    -     0|    -      0
  src/lamp_py/ad_hoc/pipeline.py                                                     | 0.0%     14| 0.0%     2|    -      0
  src/lamp_py/aws/s3.py                                                              |52.2%    320|23.8%    42|    -      0
  src/lamp_py/ingestion/backfill/delta_reingestion.py                                |60.7%     61|25.0%     4|    -      0
  src/lamp_py/ingestion/compress_gtfs/gtfs_schema_map.py                             |97.7%     44|50.0%     4|    -      0
  src/lamp_py/ingestion/compress_gtfs/gtfs_to_parquet.py                             |77.0%    100|40.0%    10|    -      0
  src/lamp_py/ingestion/compress_gtfs/pq_to_sqlite.py                                |87.8%     49|50.0%     6|    -      0
  src/lamp_py/ingestion/compress_gtfs/schedule_details.py                            |81.2%     96|50.0%     8|    -      0
  src/lamp_py/ingestion/config_busloc_trip.py                                        |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/config_rt_trip.py                                            |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/convert_gtfs.py                                              |79.2%     48|37.5%     8|    -      0
  src/lamp_py/ingestion/convert_gtfs_rt.py                                           |88.6%    237|50.0%    26|    -      0
  src/lamp_py/ingestion/convert_gtfs_rt_fullset.py                                   |45.6%    114|16.7%    18|    -      0
  src/lamp_py/ingestion/converter.py                                                 |95.9%     49|35.0%    20|    -      0
  src/lamp_py/ingestion/daily/config.py                                              | 100%      2|    -     0|    -      0
  src/lamp_py/ingestion/daily/trip_updates.py                                        |75.0%     12|25.0%     4|    -      0
  src/lamp_py/ingestion/glides.py                                                    |95.6%    182|45.5%    44|    -      0
  src/lamp_py/ingestion/gtfs_rt_detail.py                                            |94.1%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/ingest_gtfs.py                                               | 0.0%     61| 0.0%    10|    -      0
  src/lamp_py/ingestion/pipeline.py                                                  | 0.0%     38| 0.0%     4|    -      0
  src/lamp_py/ingestion/utils.py                                                     |63.1%    111|35.0%    20|    -      0

huangh · 2026-05-28T15:41:11Z

+        self,
+        config_type: ConfigType,
+        metadata_queue: Queue[Optional[str]],
+        max_workers: int = 8,


max worker 8 matches cpu count now. can lower this back if we decide to shrink ingestion ecs back

huangh · 2026-05-28T15:43:00Z


    try:
-        files = file_list_from_s3(bucket_name=S3_INCOMING, file_prefix=bucket_filter, max_list_size=10000)
+        files = file_list_from_s3(bucket_name=S3_INCOMING, file_prefix=bucket_filter, max_list_size=50000)


grab more so we get >15 minutes of data each time. processing still remains under memory threshold with about 50% margin

github-actions · 2026-05-28T15:57:05Z

LCOV of commit `9027712` during Continuous Integration (Python) #1954

Summary coverage rate:
  lines......: 65.2% (3637 of 5580 lines)
  functions..: 29.9% (275 of 920 functions)
  branches...: no data found

Files changed coverage rate:
                                                                                     |Lines       |Functions  |Branches    
  Filename                                                                           |Rate     Num|Rate    Num|Rate     Num
  =========================================================================================================================
  src/lamp_py/ad_hoc/backfill_runner_terminal_predictions.py                         | 0.0%     18|    -     0|    -      0
  src/lamp_py/ingestion/backfill/delta_reingestion.py                                |60.7%     61|25.0%     4|    -      0
  src/lamp_py/ingestion/compress_gtfs/gtfs_schema_map.py                             |97.7%     44|50.0%     4|    -      0
  src/lamp_py/ingestion/compress_gtfs/gtfs_to_parquet.py                             |77.0%    100|40.0%    10|    -      0
  src/lamp_py/ingestion/compress_gtfs/pq_to_sqlite.py                                |87.8%     49|50.0%     6|    -      0
  src/lamp_py/ingestion/compress_gtfs/schedule_details.py                            |81.2%     96|50.0%     8|    -      0
  src/lamp_py/ingestion/config_busloc_trip.py                                        |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/config_rt_trip.py                                            |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/convert_gtfs.py                                              |79.2%     48|37.5%     8|    -      0
  src/lamp_py/ingestion/convert_gtfs_rt.py                                           |88.6%    237|50.0%    26|    -      0
  src/lamp_py/ingestion/convert_gtfs_rt_fullset.py                                   |45.6%    114|16.7%    18|    -      0
  src/lamp_py/ingestion/converter.py                                                 |95.9%     49|35.0%    20|    -      0
  src/lamp_py/ingestion/daily/config.py                                              | 100%      2|    -     0|    -      0
  src/lamp_py/ingestion/daily/trip_updates.py                                        |75.0%     12|25.0%     4|    -      0
  src/lamp_py/ingestion/glides.py                                                    |95.6%    182|45.5%    44|    -      0
  src/lamp_py/ingestion/gtfs_rt_detail.py                                            |94.1%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/ingest_gtfs.py                                               | 0.0%     61| 0.0%    10|    -      0
  src/lamp_py/ingestion/pipeline.py                                                  | 0.0%     38| 0.0%     4|    -      0
  src/lamp_py/ingestion/utils.py                                                     |63.1%    111|35.0%    20|    -      0

…nd vehicle positions for now

github-actions · 2026-05-28T18:21:56Z

LCOV of commit `84da10e` during Continuous Integration (Python) #1961

Summary coverage rate:
  lines......: 65.2% (3637 of 5582 lines)
  functions..: 29.9% (275 of 920 functions)
  branches...: no data found

Files changed coverage rate:
                                                                                     |Lines       |Functions  |Branches    
  Filename                                                                           |Rate     Num|Rate    Num|Rate     Num
  =========================================================================================================================
  src/lamp_py/ad_hoc/backfill_runner_terminal_predictions.py                         | 0.0%     18|    -     0|    -      0
  src/lamp_py/ingestion/backfill/delta_reingestion.py                                |60.7%     61|25.0%     4|    -      0
  src/lamp_py/ingestion/compress_gtfs/gtfs_schema_map.py                             |97.7%     44|50.0%     4|    -      0
  src/lamp_py/ingestion/compress_gtfs/gtfs_to_parquet.py                             |77.0%    100|40.0%    10|    -      0
  src/lamp_py/ingestion/compress_gtfs/pq_to_sqlite.py                                |87.8%     49|50.0%     6|    -      0
  src/lamp_py/ingestion/compress_gtfs/schedule_details.py                            |81.2%     96|50.0%     8|    -      0
  src/lamp_py/ingestion/config_busloc_trip.py                                        |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/config_rt_trip.py                                            |82.4%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/convert_gtfs.py                                              |79.2%     48|37.5%     8|    -      0
  src/lamp_py/ingestion/convert_gtfs_rt.py                                           |88.6%    237|50.0%    26|    -      0
  src/lamp_py/ingestion/convert_gtfs_rt_fullset.py                                   |45.6%    114|16.7%    18|    -      0
  src/lamp_py/ingestion/converter.py                                                 |95.9%     49|35.0%    20|    -      0
  src/lamp_py/ingestion/daily/config.py                                              | 100%      2|    -     0|    -      0
  src/lamp_py/ingestion/daily/trip_updates.py                                        |75.0%     12|25.0%     4|    -      0
  src/lamp_py/ingestion/glides.py                                                    |95.6%    182|45.5%    44|    -      0
  src/lamp_py/ingestion/gtfs_rt_detail.py                                            |94.1%     17|12.5%     8|    -      0
  src/lamp_py/ingestion/ingest_gtfs.py                                               | 0.0%     63| 0.0%    10|    -      0
  src/lamp_py/ingestion/pipeline.py                                                  | 0.0%     38| 0.0%     4|    -      0
  src/lamp_py/ingestion/utils.py                                                     |63.1%    111|35.0%    20|    -      0

huangh added 6 commits May 26, 2026 15:23

split off all ingestion refactor logic to a new branch

13169aa

analysis scripts to look at ingestion refactor and backfill results -…

59a2c98

… including rowgroups and size of files after changes. includes some scrips and runners

backfill runner using the new ingestion method - some delta_reingesti…

48a97a3

…on code that should probably be rehomed

revert this - don't need to change

025ca24

move validation script to validate directory

a961038

clened up linting and typing

8f628ac

huangh force-pushed the hh-backfill-adhoc-new branch from 8a49f67 to 8f628ac Compare May 26, 2026 19:25

huangh added 3 commits May 28, 2026 11:29

correct all pylint/mypy. uv ruff/ty capitalization as well

ba83f3a

final pylint disables in tests

fa1af08

black

6649de6

huangh added 2 commits May 28, 2026 11:36

pylint ignore

36c6804

reverting changes in pipeline that dont matter for this pr

2453b20

huangh commented May 28, 2026

View reviewed changes

Comment thread src/lamp_py/ingestion/compress_gtfs/gtfs_schema_map.py

huangh May 28, 2026

Copy link
Copy Markdown

Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ignore

huangh commented May 28, 2026

View reviewed changes

huangh added 2 commits May 28, 2026 11:45

revert some changes that are unnecessary

785ad64

final blintest

9027712

huangh added 3 commits May 28, 2026 11:59

only use the 15 minute increment partitioned format for tripupdates a…

3b2ecd7

…nd vehicle positions for now

blk

2d52b5b

add marimo used investigation dependency

84da10e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Hh backfill adhoc new#755

Hh backfill adhoc new#755
huangh wants to merge 16 commits into
mainfrom
hh-backfill-adhoc-new

huangh commented May 26, 2026

Uh oh!

github-actions Bot commented May 28, 2026

Uh oh!

huangh May 28, 2026

Uh oh!

github-actions Bot commented May 28, 2026

Uh oh!

huangh May 28, 2026

Uh oh!

huangh May 28, 2026

Uh oh!

github-actions Bot commented May 28, 2026

Uh oh!

github-actions Bot commented May 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

huangh commented May 26, 2026

What changes does this PR propose?

How were these changes validated?

What questions should reviewers consider?

Uh oh!

github-actions Bot commented May 28, 2026

LCOV of commit 6649de6 during Continuous Integration (Python) #1951

Uh oh!

huangh May 28, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 28, 2026

LCOV of commit 36c6804 during Continuous Integration (Python) #1952

Uh oh!

huangh May 28, 2026

Choose a reason for hiding this comment

Uh oh!

huangh May 28, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 28, 2026

LCOV of commit 9027712 during Continuous Integration (Python) #1954

Uh oh!

github-actions Bot commented May 28, 2026

LCOV of commit 84da10e during Continuous Integration (Python) #1961

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

LCOV of commit `6649de6` during Continuous Integration (Python) #1951

LCOV of commit `36c6804` during Continuous Integration (Python) #1952

LCOV of commit `9027712` during Continuous Integration (Python) #1954

LCOV of commit `84da10e` during Continuous Integration (Python) #1961