Enable Users to provide their own cv function and/or tuning function to optuna

[thomasATbayer]

Feature Request: Custom CV Scorer & Custom Optuna Objective in MotherTuner

Summary

Two related extension points to make MotherTuner more flexible — particularly for deep learning workflows, probabilistic scoring, and early stopping:

1. Custom CV scorer — allow passing a callable that replaces cross_val_score internally, giving full control over how folds are scored (e.g. NLL, CRPS, distribution scoring).
2. Custom Optuna objective function — allow bypassing MotherTuner's built-in objective() closure entirely, enabling full control over training, validation, early stopping, and what value is reported to Optuna.

---

Motivation

1. Custom CV Scorer

MotherTuner currently accepts a scorer argument that is either a string (sklearn scorer name) or a sklearn-compatible callable passed to cross_val_score. This covers most classical ML use cases, but falls short when:

• You want to score over predicted distributions (e.g. negative log-likelihood, CRPS, energy score) rather than point predictions, which require access to the raw distribution object rather than a scalar prediction.
• You need a scorer that is aware of the CV fold structure, e.g. calibration-aware metrics that need both in-fold and out-of-fold data simultaneously.
• Your model produces multiple output heads and you want a combined custom metric that cannot be expressed as a standard sklearn scorer.
• You use a deep learning model where validation is performed inside fit() via callbacks, and the relevant metric is stored on the estimator after fitting rather than computed post-hoc by sklearn.

The proposal is to accept a cv_scorer_fn parameter on MotherTuner that replaces the cross_val_score call entirely. The function would receive the estimator, data, CV splitter, and current Optuna trial, and return an array of per-fold scores. MotherTuner would still own the Optuna study, reporting, and pruning logic.

---

2. Custom Optuna Objective Function

The objective() closure inside MotherTuner.optimize() currently samples hyperparameters from the trial, clones and fits the estimator, runs cross-validation, and returns the mean score to Optuna. For deep learning, this pattern breaks down because:

• Early stopping must happen inside fit() per fold, not after cross_val_score finishes. Optuna's native pruning integrations (e.g. PyTorch Lightning, Keras, XGBoost callbacks) require direct access to the trial object inside the training loop.
• Hyperparameter sampling may need to interact with architecture decisions that cannot be expressed as sklearn set_params() strings (e.g. dynamic layer counts, conditional parameter blocks).
• Some frameworks report their own validation metric internally (e.g. via eval_set callbacks), and the user wants to read that metric back from the fitted model rather than calling predict() again.

The proposal is to accept an objective_fn parameter on MotherTuner that replaces the entire inner objective() closure. The function would receive the Optuna trial, the base estimator, the data, and the CV splitter, and return a scalar directly to the Optuna study. MotherTuner would still own study creation, the sampler, n_trials, direction, and callbacks.

---

Proposed Behaviour

• Both parameters default to None — fully backward compatible, existing behaviour unchanged.
• When cv_scorer_fn is provided, it replaces the cross_val_score call but the returned per-fold array is still fed into the existing report_cross_validation_scores logic for Optuna pruning support.
• When objective_fn is provided, the entire inner objective() closure is replaced. The user is responsible for cloning, fitting, pruning, and returning the trial value.
• If both are provided simultaneously, a clear ValueError is raised — they are mutually exclusive.
• The default_parameters first-trial enqueue logic could be preserved optionally via a flag when using a custom objective.

---

Affected Areas

Area	Change
src/mother/optimization/core.py	Add cv_scorer_fn / objective_fn params to MotherTuner.__init__, branch logic in optimize()
src/mother/optimization/config.py	Add cv_scorer_fn and objective_fn as optional callable fields to MotherTunerConfig
src/mother/settings.py	Expose new params through the public settings surface if applicable
examples/	Add a notebook demonstrating NLL/distribution scoring and/or deep learning early stopping
mkdocs/docs/	Document both new parameters

---

Related

• Optuna pruning integrations: https://optuna.readthedocs.io/en/stable/reference/integration.html
• sklearn make_scorer for custom scoring: https://scikit-learn.org/stable/modules/model_evaluation.html#defining-your-scoring-strategy-from-metric-functions