[Bug]: Determine if the use of XCOM to hold successful features is causing failures when running DAGs

[nikki-t]

It seems like the modular Airflow DAG will fail if the successful features file produced from the execution of the stage out task is too large.

[nikki-t]

Looking at the issue, here is the size of the stage output from the logs:

[2025-07-07, 19:20:09 UTC] {pod_manager.py:471} INFO - [base] Stage Out output:
[2025-07-07, 19:20:09 UTC] {pod_manager.py:471} INFO - [base] {
[2025-07-07, 19:20:09 UTC] {pod_manager.py:471} INFO - [base]   "failed_features": {
[2025-07-07, 19:20:09 UTC] {pod_manager.py:471} INFO - [base]     "location": "file:///data/stage_out/failed_features.json",
[2025-07-07, 19:20:09 UTC] {pod_manager.py:471} INFO - [base]     "basename": "failed_features.json",
[2025-07-07, 19:20:09 UTC] {pod_manager.py:471} INFO - [base]     "class": "File",
[2025-07-07, 19:20:09 UTC] {pod_manager.py:471} INFO - [base]     "checksum": "sha1$e9b73c133ee6531ae67bea35e739b18c09efab9b",
[2025-07-07, 19:20:09 UTC] {pod_manager.py:471} INFO - [base]     "size": 45,
[2025-07-07, 19:20:09 UTC] {pod_manager.py:471} INFO - [base]     "path": "/data/stage_out/failed_features.json"
[2025-07-07, 19:20:09 UTC] {pod_manager.py:471} INFO - [base]   },
[2025-07-07, 19:20:09 UTC] {pod_manager.py:471} INFO - [base]   "successful_features": {
[2025-07-07, 19:20:09 UTC] {pod_manager.py:471} INFO - [base]     "location": "file:///data/stage_out/successful_features.json",
[2025-07-07, 19:20:09 UTC] {pod_manager.py:471} INFO - [base]     "basename": "successful_features.json",
[2025-07-07, 19:20:09 UTC] {pod_manager.py:471} INFO - [base]     "class": "File",
[2025-07-07, 19:20:09 UTC] {pod_manager.py:471} INFO - [base]     "checksum": "sha1$094666e9b287ea7e644d2d0eca28966e7205c64d",
[2025-07-07, 19:20:09 UTC] {pod_manager.py:471} INFO - [base]     "size": 165195,
[2025-07-07, 19:20:09 UTC] {pod_manager.py:471} INFO - [base]     "path": "/data/stage_out/successful_features.json"
[2025-07-07, 19:20:09 UTC] {pod_manager.py:471} INFO - [base]   }
[2025-07-07, 19:20:09 UTC] {pod_manager.py:471} INFO - [base] }

I think the size may be in bytes and the successful_feature.json file is "165195" bytes which is 165.195 KB. It looks like the XCOM limit varies based on the database back end. My guess is we are bumping up against the limit.