[BUG] CI does not manage component test dependencies

[hbelmiro]

Problem

Components that declare runtime dependencies via packages_to_install in their @dsl.component decorator (e.g., sdg-hub, pandas, datasets) have those packages installed inside the container at execution time. However, the component's unit tests need those same packages to be importable in the test environment -- either to mock them (@mock.patch("sdg_hub.core.flow.base.Flow.from_yaml")) or to call component.python_func() directly in local runner tests.

Currently, there is no mechanism for CI to automatically install component-specific test dependencies before running a component's test suite. The test runner (scripts/tests/run_component_tests.py) discovers and executes tests, but only the project-wide test extra from pyproject.toml is installed. Components with additional runtime dependencies have their tests fail due to missing imports.

Current workaround

Add component specific dependencies to the general test dependencies.

Impact

• Component tests that depend on external packages cannot run in CI without adding those packages to the global [project.optional-dependencies] test list in pyproject.toml.
• Adding every component's runtime dependencies to the global test extra does not scale as the catalog grows -- it increases install time for all contributors and risks dependency conflicts between unrelated components.

Desired outcome

A convention and supporting automation that allows components to declare their own test-time dependencies in a way that CI can discover and install them dynamically, without hardcoding component-specific logic in workflows or polluting the global dependency list.

---

Impacted by this bug? Give it a 👍.

[prachimaskar]

Hi @hbelmiro 👋

I'd like to work on this issue!

After looking at the codebase, here's my plan:

Root cause confirmed: run_local.py and the unit tests in components/data_processing/sdg/shared/ import pandas and sdg-hub directly, which are currently hardcoded into the global test extra in pyproject.toml as a workaround.

My proposed fix:

1. Add a test-requirements.txt file inside each component directory that needs extra test dependencies (starting with the SDG component)
2. Update scripts/tests/run_component_tests.py to auto-detect and install a component's test-requirements.txt before running its tests
3. Remove component-specific packages (sdg-hub) from the global [project.optional-dependencies] test in pyproject.toml

This keeps dependencies scoped per component, avoids polluting the global list, and scales cleanly as new components are added.

Could you please assign this issue to me?