GPSampler as Default Optuna Sampler

[thomasATbayer]

Branch: 46-make-gp-sampler-the-standard-optuna-sampler-when-torch-is-installed

Overview

This PR changes MotherML's default Optuna sampler selection when users do not pass a custom sampler.

• If torch is available: use GPSampler
• If torch is not available: use TPESampler
• If GPSampler initialization fails for any reason: fall back to TPESampler

The goal is to get stronger optimization in low-budget, expensive-trial settings, while keeping behavior robust across environments.

Sampler decision logic (plain-English)

When sampler=None, MotherML now does the following:

1. If torch is available, try to build GPSampler.
2. If GP creation fails (dependency/runtime/init issue), fall back to TPESampler.
3. If torch is not available, use TPESampler directly.

This gives a safe default that prefers GP when possible but never hard-fails the run.

Why this helps

MotherML tuning commonly has:

• relatively few trials
• non-trivial cost per trial
• mixed search spaces (shared parameters + conditional parameters)

GPSampler is often effective on the stable shared search space in this regime. But GP does not fully cover dynamic/conditional spaces. To avoid low-quality fallback behavior there, this PR configures a hybrid:

• GP for relative/shared space
• TPE as the independent sampler for startup and out-of-relative-space parameters

Why GP + TPE together (and not GP alone)

GPSampler is strongest on a stable relative search space. In real MotherML tuning, many models (especially CatBoost-style spaces) include branching/conditional parameters.

For those parameters, GP does not always provide relative/joint suggestions. Optuna then delegates to the sampler's independent_sampler.

If that independent sampler is random, quality can drop on expensive low-budget runs. Using TPE here gives a history-informed fallback after startup instead of staying purely random.

What changed in code

File: src/mother/optimization/core.py

1. Added explicit sampler builders:

• _build_tpe_sampler(...)
• _build_gp_sampler(...)

2. Default sampler selection logic in MotherTuner.__init__ now:

• tries GP when torch is available
• catches GP initialization errors and falls back to TPE
• falls back to TPE when torch is unavailable

3. GPSampler is configured with:

• independent_sampler=optuna.samplers.TPESampler(...)
• the same n_startup_trials value as GP

4. Runtime safety for default path:

• GP init errors are caught and logged
• default path recovers to TPE rather than failing tuning setup

Important behavior (reviewer-friendly)

This is the key runtime behavior for startup and conditional parameters:

1. Startup phase is controlled by global completed-trial count (study-level), not by per-parameter occurrence count.
2. During startup, independent sampling is used.
3. Because independent sampler is TPE, and TPE itself has a startup phase, TPE uses random sampling during its own startup window.
4. After startup, conditional/branch-only parameters are sampled by history-informed TPE instead of pure random.

Detailed runtime flow for dynamic/conditional spaces

For each trial, sampling effectively works like this:

1. If the trial count is still in startup, independent sampling is used.
2. Once startup is over, GP proposes parameters in its relative/shared space.
3. Parameters not covered by GP's relative space (common in conditional branches) are sampled via the independent sampler.
4. Because that independent sampler is TPE, those uncovered parameters transition from random startup sampling to history-informed TPE sampling after startup.

Important reviewer note: startup cutoff is study-level trial count, not "how many times a specific conditional parameter appeared."

In short:

• Shared stable space: GP-optimized after startup
• Conditional/out-of-relative-space params: TPE, with random startup then history-informed behavior

What this PR does not change

To keep scope focused, this PR is about default sampler strategy and safe fallback behavior.

It does not change broader optimization workflow semantics beyond sampler choice and GP/TPE wiring.

It also does not claim full joint GP modeling over all conditional branches. This is a pragmatic hybrid to improve defaults in mixed spaces.

Test coverage

File: test/unit/test_model_tuner.py

Covered in TestDefaultSamplerSelection:

• GP selected when torch is available
• TPE selected when torch is unavailable
• GP init failure falls back to TPE
• GP dynamic-search-space warning behavior
• custom sampler passthrough

Also validated:

• callback behavior remains tested in TestGetCallbacks

Trade-off summary

Pros:

• better default for small trial budgets on shared/stable spaces
• safer behavior for conditional spaces than GP + random fallback
• resilient fallback when GP stack is unavailable or fails

Caveat:

• this is a pragmatic hybrid, not full joint modeling of all conditional interactions by GP
• when large parts of the important search space are highly conditional/tree-structured, a fully TPE-centric strategy may still outperform GP-first defaults

Suggested PR wording

This PR makes GPSampler the default Optuna sampler in MotherML when torch is available, with TPESampler as the independent sampler and as full fallback when GP cannot be initialized. The hybrid keeps GP benefits for low-budget optimization on stable shared spaces, while using history-informed TPE for startup and conditional parameters that fall outside GP's relative search space.

[Copilot]

Pull request overview

Updates MotherML’s Optuna tuning behavior by switching the default sampler selection logic and making Optuna early-termination safer for hold-out validation setups.

Changes:

• Make MotherTuner default to optuna.samplers.GPSampler when torch is available, otherwise fall back to TPESampler.

• Bump mother-ml version in uv.lock to 1.0.1.

Reviewed changes

Copilot reviewed 1 out of 2 changed files in this pull request and generated 2 comments.

File	Description
uv.lock	Updates locked package version for mother-ml to 1.0.1.
src/mother/optimization/core.py	Changes default Optuna sampler selection and adds hold-out detection in early-stopping callback setup.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/mother/optimization/core.py:152

• Default sampler selection switches to GPSampler whenever torch_available is true, but torch_available only checks for torch and does not handle missing GP dependencies (e.g. scipy) or other instantiation-time import errors. In environments where torch is installed but scipy is not, optuna.samplers.GPSampler(...) will raise at runtime instead of falling back to TPE. This also regresses the prior ability to override TPESampler's multivariate via kwargs.

Consider wrapping GPSampler construction in a try/except and falling back to TPESampler, while preserving multivariate=kwargs.get("multivariate", True).

            if torch_available:
                module_logger.debug("torch available — using GPSampler as default")
                self.sampler = optuna.samplers.GPSampler(
                    seed=seed,
                    n_startup_trials=n_startup_trials,

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

src/mother/optimization/core.py:152

• Default sampler selection only checks torch_available, but GPSampler requires additional optional deps (e.g. SciPy) and can raise ImportError at instantiation time even when torch is installed. This also contradicts the PR description which says it should fall back to TPESampler when GP dependencies are unavailable. Consider wrapping GPSampler initialization in try/except ImportError and falling back, and restore the prior multivariate override via kwargs.get("multivariate", True) for backward compatibility.

            if torch_available:
                module_logger.debug("torch available — using GPSampler as default")
                self.sampler = optuna.samplers.GPSampler(
                    seed=seed,
                    n_startup_trials=n_startup_trials,

src/mother/optimization/core.py:179

• get_callbacks now has a cross_validation parameter and can skip early stopping for hold-out (n_splits < 2), but the docstring still only describes the torch availability behavior. Updating the docstring will help callers understand why None may be returned even when early stopping is enabled.

        """
        Prepares and returns a list of callbacks for early stopping in Optuna optimization.

        If early stopping with Optuna is enabled and PyTorch is available, this method
        will return a list containing a TerminatorCallback instance. If PyTorch is not

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

test/unit/test_model_tuner.py:252

• This test forces torch_available=True but does not ensure torch/scipy are actually installed. Since torch is an optional extra (pyproject.toml [project.optional-dependencies].torch), GPSampler initialization can fail in CI and the tuner will correctly fall back to TPESampler, making this assertion flaky. Consider stubbing GPSampler to a lightweight fake so the test only validates branch selection logic.

        monkeypatch.setattr("mother.optimization.core.torch_available", True)
        tuner = MotherTuner(
            scorer=make_scorer(mean_squared_error, greater_is_better=False),
        )
        assert isinstance(tuner.sampler, optuna.samplers.GPSampler)

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (2)

test/unit/test_model_tuner.py:248

• This test forces torch_available=True but does not ensure the GP stack is actually installed. In environments where the optional torch/scipy dependencies are not installed (they are not in the default dev dependency group), MotherTuner will fall back to TPESampler and this assertion will fail. Skip this test when torch/scipy are unavailable (similar to the later GP test).

    def test_gp_sampler_when_torch_available(self, monkeypatch):
        """GPSampler should be selected when torch is available."""
        monkeypatch.setattr("mother.optimization.core.torch_available", True)

src/mother/optimization/core.py:138

• _build_tpe_sampler hard-codes multivariate=True/group=True/constant_liar=True. If callers previously relied on tuning these (e.g., disabling multivariate TPE for certain search spaces), there’s no longer a supported way to do so when sampler=None. Consider either (a) exposing these as explicit MotherTuner parameters, or (b) documenting that the defaults are now fixed and sampler= must be provided for customization.

            multivariate=True,
            group=True,
            constant_liar=True,
            seed=seed,
            n_startup_trials=n_startup_trials,

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (2)

src/mother/optimization/core.py:151

• _build_gp_sampler configures independent_sampler as a plain TPESampler(seed=..., n_startup_trials=...), which drops the stronger default TPE configuration used elsewhere (multivariate=True, group=True, constant_liar=True). That means parameters sampled outside the GP relative search space (and startup trials) may behave differently/worse than the intended TPE fallback.

        return optuna.samplers.GPSampler(
            seed=seed,
            independent_sampler=optuna.samplers.TPESampler(
                seed=seed,
                n_startup_trials=n_startup_trials,
            ),

test/unit/test_model_tuner.py:254

• This test forces torch_available=True without ensuring the optional GP stack is actually installed. If torch (and any GPSampler runtime deps) are not present in the test environment, MotherTuner will correctly fall back to TPESampler and this assertion will fail. Skip when torch isn't installed (and consider also skipping when GPSampler deps are missing).

    def test_gp_sampler_when_torch_available(self, monkeypatch):
        """GPSampler should be selected when torch is available."""
        monkeypatch.setattr("mother.optimization.core.torch_available", True)
        tuner = MotherTuner(

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

src/mother/optimization/core.py:181

• MotherTuner.__init__ still accepts **kwargs, but the default TPESampler configuration no longer honors the previously-supported multivariate override (it is now always True). This is a silent behavioral/API change for callers that pass multivariate= expecting it to affect the default sampler.

        if sampler is None:
            if torch_available:
                try:
                    module_logger.debug("torch available — using GPSampler as default")
                    self.sampler = self._build_gp_sampler(

src/mother/optimization/core.py:185

• The PR description says hold-out (single-fold) validation disables TerminatorCallback, but optimize() still unconditionally passes callbacks=self.get_callbacks() whenever early_stopping_optuna is enabled and torch_available is true. With a 1-split CV object, this can still attach the terminator and potentially crash/behave incorrectly when only one CV score is reported.

            if torch_available:
                try:
                    module_logger.debug("torch available — using GPSampler as default")
                    self.sampler = self._build_gp_sampler(
                        seed=seed,
                        n_startup_trials=n_startup_trials,
                    )
                except Exception as gpsampler_error:

test/unit/test_model_tuner.py:216

• This test forces torch_available=True via monkeypatch but does not skip when PyTorch is actually not installed. In environments without torch, constructing TerminatorCallback() may fail (or the import path may differ), making the test suite brittle.

    def test_early_stopping_enabled_returns_callback_when_torch_available(self, monkeypatch):
        """get_callbacks should return terminator callback when enabled and torch is available."""
        monkeypatch.setattr("mother.optimization.core.torch_available", True)
        tuner = MotherTuner(