Unit tests test_experiment.py fails

[gadieid]

Reason 1:

/home/gadi/Production/pyargos/argos/experimentSetup/dataObjects.py:548: in refresh
    experimentDict = getattr(self, f"fix_json_version{fileVersion}")(experimentDict)
E   AttributeError: 'TestExp' object has no attribute '_fix_json_version_1_0_0'

The DataObjects has method for
fix_json_version_1_0_0_

Reason 2 after duplicating the method above:

/home/gadi/Production/pyargos/argos/experimentSetup/dataObjects.py:964: in init
    properties = propertiesPandas.merge(trialSet.propertiesTable, left_index=True, right_index=True)[

... pandas internal traceback ...

E   KeyError: "['type', 'label', 'description'] not in index"

Indeed, after printing availables columns there are just
AVAILABLE COLUMNS: ['val', 'type_x', 'type_y', 'name']

Seems to be a data problem??

Probably much more problems

[ilayfalach]

Unit Test Stabilization and Data Infrastructure
We have addressed the specific errors reported in this issue and simultaneously upgraded the entire testing workflow to prevent similar regressions. All changes are currently consolidated in the branch: issue857.

1. Direct Resolution of Reported Errors (Reason 1 & 2)
   AttributeError (fix_json_version): The error was caused by a naming mismatch during the experiment loading phase. We updated dataObjects.py to correctly identify the method fix_json_version_1_0_0_ (including the trailing underscore) as defined in the class.

Pandas Merge KeyError: The "not in index" error was triggered by changes in how newer Pandas versions handle index merging and column naming. We refactored the merge logic to ensure the required columns (type, label, description) are preserved and correctly identified post-merge.

2. General Infrastructure Improvements & Stabilization
   In addition to the specific fixes, we performed a comprehensive "alignment" of the project environment to ensure consistency across the team:

Core Compatibility (Pandas & Dask):

Resolved multiple errors stemming from the transition to Pandas 3.x, including updating time-frequency strings from '1S' to '1s'.

Updated dask.dataframe calls to align with current API standards.

S3 Bootstrap Infrastructure:

Implemented an automated mechanism to retrieve test data directly from the cloud.

Data is pulled from the public S3 bucket (hera.kaplanopensource.co.il) and undergoes full SHA256 integrity verification against the manifest.json.

Workspace Maintenance:

Redirected all cache and log artifacts to a dedicated directory (cache/test_results/) to maintain a clean project root.

Updated .gitignore to prevent temporary test directories from being tracked.

3. The New "One-Command" Workflow
   To simplify the process for all developers, a single command now handles data synchronization, environment setup, and test execution:

make test

What this command does:

Triggers Bootstrap: Checks for new data in S3 and downloads it to the local directory.

Environment Setup: Automatically links the tests to the data path (TEST_HERA).

Full Pytest Execution: Runs the complete test suite (192 passing tests) and generates a detailed report.

4. Current Status in Branch issue857
   192 Tests Passing: Core logic for the Data Layer, Meteorology, and Repository modules is fully verified.

GDAL Handling: GIS/Topography tests will now gracefully Skip if the system GDAL library is missing, rather than failing the build.

Intentional Errors: Currently, "Errors" only persist for files not included in the initial POC subset (e.g., slicedYamim). These will resolve once the remaining data is uploaded to S3.

@yehudarav @gadieid @lior-antonov

[lior-antonov]

Do all tests pass when the required packages are installed now?

[ilayfalach]

Yes. All logic-related tests pass (192 tests). The only remaining "Errors" are strictly due to the POC data subset (missing specific .parquet and .shp files) or optional proprietary packages (like argos).

[lior-antonov]

@ilayfalach Please open a pull request for this.
@erasta @yehudarav what do we do next to integrate it with github CICD. More specifically, where will we pull the testing data to? what do we need to setup for that?