Added custom eval metric feature

[ozguraslank]

Resolved #83

[Copilot]

Pull request overview

This PR adds a custom evaluation metric feature to the flexml library, allowing users to define and use their own scoring functions for model evaluation and tuning. The implementation introduces a new CustomScore class to wrap custom metric functions and integrates this functionality across the supervised learning pipeline.

Key changes:

• New CustomScore class to validate and wrap custom metric functions with support for probability vs label predictions and maximize/minimize optimization directions
• Extended start_experiment, tune_model, and related methods to accept callable custom metrics alongside standard string-based metrics
• Comprehensive test suite covering regression, binary classification, multiclass classification, and tuning scenarios

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 20 comments.

Show a summary per file

File	Description
tests/test_custom_metrics.py	Comprehensive test suite with 9 test cases covering custom metrics for regression, classification (binary/multiclass), tuning (GridSearch, RandomizedSearch, Optuna), and error handling
flexml/structures/custom_score.py	New class to wrap and validate custom scoring functions with parameter validation, sklearn scorer integration, and support for probability/label-based metrics
flexml/structures/supervised_base.py	Updates to handle custom metrics in experiment workflow including parameter handling, model selection logic for minimize vs maximize, and leaderboard sorting
flexml/helpers/validators.py	Type hint updates to support CustomScore objects in addition to string-based metrics
flexml/helpers/supervised_helpers.py	Enhanced evaluate_model_perf to compute custom metrics alongside standard metrics with proper handling of probabilities vs labels
flexml/_model_tuner.py	Support for custom metrics in all tuning methods (grid search, randomized search, Optuna) with proper scorer integration

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.