| Overview | |
|---|---|
| CI/CD |   |
| Code |   |
skordinal is an experimental framework built on Python that integrates with scikit-learn to automate machine learning experiments through simple JSON configuration files. Initially designed for ordinal classification, it supports regular classification algorithms as long as they are compatible with scikit-learn, making it easy to run reproducible experiments across multiple datasets and classification methods.
skordinal requires Python 3.10 or higher and is tested on Python 3.10, 3.11, 3.12, 3.13, and 3.14.
All dependencies are managed through pyproject.toml and include:
- numpy (>=1.21)
- pandas (>=1.0.1)
- scikit-learn (>=1.3.0)
- scipy (>=1.7)
-
Clone the repository:
git clone https://github.com/ayrna/skordinal cd skordinal -
Install the framework:
pip install .For development purposes, use editable installation:
pip install -e .Optional dependencies for development:
pip install -e .[dev]
Test your installation with the provided example:
python examples/run_recipe.py examples/recipes/full_demo.pyskordinal includes sample datasets with pre-partitioned train/test splits using a 30-holdout experimental design.
Basic experiment configuration:
A recipe is a Python file that defines a top-level RECIPE dict whose keys
map directly to Benchmark constructor parameters. The required keys are
models, datasets, eval_metrics, and results_path; all other keys are
optional and fall back to the Benchmark defaults.
from sklearn.svm import SVC
from skordinal.classifiers import OrdinalDecomposition
from skordinal.experiments import ModelConfig
RECIPE = {
"datasets": ["balance_scale", "era", "esl"],
"cv": 3,
"n_jobs": 1,
"input_preprocessing": "std",
"results_path": "results/",
"eval_metrics": [
"accuracy_score",
"mean_absolute_error",
"average_mean_absolute_error",
"mean_zero_one_error",
],
"tuning_metric": "neg_mean_absolute_error",
"models": {
"SVM": ModelConfig(
SVC(),
param_grid={"C": [0.001, 0.1, 1, 10, 100], "gamma": [0.1, 1, 10]},
),
"SVMOP": ModelConfig(
OrdinalDecomposition(
dtype="ordered_partitions",
decision_method="frank_hall",
base_classifier=SVC(probability=True),
),
param_grid={
"base_classifier__C": [0.01, 0.1, 1, 10],
"base_classifier__gamma": [0.01, 0.1, 1, 10],
},
),
},
}Run the experiment:
python examples/run_recipe.py my_experiment.pyResults are saved in the results/ folder with performance metrics for each
dataset-classifier combination. The framework automatically performs
cross-validation, hyperparameter tuning, and evaluation on test sets.
Each classifier-dataset pair gets its own directory holding a report.csv
with per-resample metrics, a hyperparameter_configuration.csv recording the
best hyperparameters chosen for each seed, and a predictions_by_seed/
folder grouping the outputs of every resample:
results/
└── <classifier>/
└── <dataset>/
├── report.csv
├── hyperparameter_configuration.csv
└── predictions_by_seed/
└── seed_<N>/
├── train_predictions.csv
├── test_predictions.csv
├── train_confusion_matrix.txt
└── test_confusion_matrix.txt
Each predictions CSV lists the Pattern ID, the zero-based Target class
index, a Prediction probabilities column with the per-class probabilities
(for estimators that support predict_proba), and the zero-based
Prediction class index — the argmax of the probabilities when they are
present, or the estimator's own prediction otherwise. Confusion matrices are
written alongside as plain-text .txt files, the fitted estimator for each
seed is stored under a models/ subfolder, and aggregated
train_summary.csv and test_summary.csv files are produced across all
runs.
Experiments are defined as Python recipe files — a module that exposes a
top-level RECIPE dict. Every key in RECIPE corresponds to a Benchmark
constructor parameter. Recipes can be run from the command line or loaded
programmatically via Benchmark.from_recipe("path/to/recipe.py").
These keys control how the benchmark is executed.
Required:
datasets: list of dataset names. A loader or subfolder with each name must be available underdata_home.eval_metrics: list of metric names computed on train and test for every partition.results_path: folder where result files are written.
Optional:
tuning_metric(default"neg_mean_absolute_error"): scoring criterion passed toGridSearchCVto select the best hyperparameters.cv(default3): number of cross-validation folds.n_jobs(default1): parallel jobs forGridSearchCV.input_preprocessing(defaultNone):"std"for standardisation,"norm"for normalisation,Nonefor no scaling.resamples(default30): number of train/test resamples per dataset.data_home(defaultNone): base directory for dataset files;Noneuses the bundled datasets.random_state(defaultNone): integer seed for reproducibility.verbose(defaultTrue): print progress during the run.
Required. A dict mapping a label (str) to a ModelConfig instance.
ModelConfig binds a scikit-learn-compatible estimator to an optional
param_grid.
ModelConfig(estimator, param_grid=None): wraps any estimator that implements the scikit-learn estimator interface.param_gridis a dict of hyperparameter name → list of values forGridSearchCV. For pipeline-style estimators (e.g.OrdinalDecomposition) use the double-underscore syntax ("base_classifier__C") to target nested parameters.
python examples/run_recipe.py experiment_file.pyResults are stored in the specified output folder with detailed performance metrics and hyperparameter information for each dataset and configuration combination.