Pointed out by @IrisdeRuiter - apparently for large runs, recalc_sources_df results in an OOM when running on jupyterhub (only 8GB RAM per user).
I haven't looked into it yet, but I suspect the offending line is here
|
measurements_df_temp = measurements_df[[ |
|
'flux_int', 'flux_int_err', 'flux_peak', 'flux_peak_err', 'source' |
|
]].extract().to_pandas_df() |
There is no trivial way to convert the measurements dataframe from vaex to dask without either re-loading or writing to a temporary file. Re-loading the data isn't trivial because the pipeline run path isn't actually stored within the PipeRun object - the relevant data is loaded and passed to it (in hindsight this was probably stupid).
I am fairly hesitant to touch this given that it will be fixed in V4, but if there's a big need then I can probably put together a hacky temp file solution.
Pointed out by @IrisdeRuiter - apparently for large runs,
recalc_sources_dfresults in an OOM when running on jupyterhub (only 8GB RAM per user).I haven't looked into it yet, but I suspect the offending line is here
vast-tools/vasttools/pipeline.py
Lines 1192 to 1194 in b320d43
There is no trivial way to convert the measurements dataframe from vaex to dask without either re-loading or writing to a temporary file. Re-loading the data isn't trivial because the pipeline run path isn't actually stored within the PipeRun object - the relevant data is loaded and passed to it (in hindsight this was probably stupid).
I am fairly hesitant to touch this given that it will be fixed in V4, but if there's a big need then I can probably put together a hacky temp file solution.