Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@

## Unreleased

## 1.0.1

- Fresh GMM studies now use the held-out-calibrated defaults: twice the raw
warm-up rule before power-of-two rounding, a 12.5% elite fraction, no ongoing
post-warm-up Sobol' cadence, one mixture component, and a five-sample elite
floor. Direct low-level GMM constructors and refit configuration use the same
component cap. Checkpoints that predate these explicit fields continue to
resolve to the historical defaults when loaded.
- Multi-group GMM refits now select elites by Pareto rank and crowding
distance, and grouped objectives apply priority weights only within their
explicit group.
Expand All @@ -18,6 +26,8 @@
seeds, immutable provenance manifests, failure-visible reporting, fixed
metric scales, and strict result-coverage validation. The suite also adds
analytic grouped-TLP and sealed-test mixed-space HPO capability studies.
- Updated the documentation dependency lockfile to resolve CVE-2026-61632 in
`pymdown-extensions`.

## 1.0.1-rc8

Expand Down
8 changes: 4 additions & 4 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

26 changes: 17 additions & 9 deletions docs/cli-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,8 @@ objectives:
priority: 0.5
group: cost

max_trials: 200 # trial cap and S for the automatic GMM warm-up

strategy:
type: gmm
refit_interval: 20
Expand All @@ -81,6 +83,11 @@ strategy:
# max_checkpoints: 5
```

`max_trials` is optional. When set, it caps dispatched trials and supplies the
total budget `S` used by the automatic GMM warm-up. If neither `max_trials` nor
`strategy.total_budget` is set, the warm-up calculation uses `S=200` without
imposing a trial cap.

### Space Configuration

Each parameter in the `space:` section has a `type` and
Expand Down Expand Up @@ -174,10 +181,10 @@ strategy:
refit_interval: 20 # how often GMM refits (used by "gmm")
seed: 42 # optional seed for reproducible runs
exploration_budget: 50 # number of Sobol asks before switching to GMM
elite_fraction: 0.25 # fraction of top trials used for GMM fitting (default: 0.25)
ongoing_exploration_period: 5 # every Nth post-warmup ask is Sobol (0 disables)
max_components: 3 # maximum fitted GMM components
min_elite_samples: 1 # minimum feasible elite workset before fitting
elite_fraction: 0.125 # fraction of top trials used for GMM fitting
ongoing_exploration_period: 0 # every Nth post-warmup ask is Sobol (0 disables)
max_components: 1 # maximum fitted GMM components
min_elite_samples: 5 # minimum feasible elite workset before fitting
max_refit_samples: 4096 # maximum elite samples passed to one GMM fit
max_refit_candidates: 16384 # maximum trials ranked to choose elites
```
Expand All @@ -187,11 +194,12 @@ strategy:
| `type` | `"gmm"` | Strategy type: `"gmm"`, `"sobol"`, or `"random"` |
| `refit_interval` | `20` | How often the GMM refits (only used by `"gmm"`) |
| `seed` | none | Seed for reproducible runs. When omitted, HOLA draws one seed once and records it in full checkpoints. |
| `exploration_budget` | none | Number of issued Sobol exploration suggestions before switching to GMM exploitation. Pending asks count against this budget. When omitted, we use a formula based on `total_budget` and the search dimension. |
| `elite_fraction` | `0.25` | Fraction of top trials used for GMM refitting. Must be in (0.0, 1.0]. |
| `ongoing_exploration_period` | `5` | Continue global Sobol' exploration every Nth post-warmup suggestion. Use `0` to disable; explicit periods must be at least 2. |
| `max_components` | `3` | Maximum fitted GMM components. The effective count can be lower for small elite sets. |
| `min_elite_samples` | `1` | Minimum feasible elite workset required before fitting. Must not exceed `max_refit_samples`. |
| `total_budget` | none | Alternative source for `S` and the trial cap when top-level `max_trials` is omitted. |
| `exploration_budget` | none | Number of issued Sobol exploration suggestions before switching to GMM exploitation. Pending asks count against this budget. When omitted, HOLA doubles `min(floor(S/5), 50 + 2n)` and then rounds down to a power of two, for total budget `S` and dimension `n`; `S=200` when neither budget field is set. |
| `elite_fraction` | `0.125` | Fraction of top trials used for GMM refitting. Must be in (0.0, 1.0]. |
| `ongoing_exploration_period` | `0` | Continue global Sobol' exploration every Nth post-warmup suggestion. The default `0` disables it; explicit periods must be at least 2. |
| `max_components` | `1` | Maximum fitted GMM components. The effective count can be lower for small elite sets. |
| `min_elite_samples` | `5` | Minimum feasible elite workset required before fitting. Must not exceed `max_refit_samples`. |
| `max_refit_samples` | `4096` | Maximum elite samples passed to one GMM fit. Must be at least 1. |
| `max_refit_candidates` | `16384` | Maximum retained trials ranked to choose elites. Must be at least `max_refit_samples`; longer histories use deterministic stratified coverage of the full retained history. |

Expand Down
16 changes: 10 additions & 6 deletions docs/concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,15 +113,19 @@ rank followed by descending crowding distance.
The lifecycle follows three phases.

1. **Warmup.** The first trials build up the leaderboard. HOLA continues
Sobol' sampling until the first empirical fit is installed.
Sobol' sampling until the first empirical fit is installed. With no explicit
exploration budget, the implementation doubles
`min(floor(S/5), 50 + 2n)` and rounds down to a power of two, for total
budget `S` and dimension `n`. The warm-up calculation uses `S=200` when no
total trial budget is configured; this fallback does not impose a trial cap.
2. **Refit.** Every `refit_interval` trials (default 20), we refit
the GMM to the top 25% of trials, subject to the configured minimum
the GMM to the top 12.5% of trials, subject to the configured minimum
feasible elite workset. If the first scheduled fit lacks that workset,
each subsequent completion retries until the first empirical model is
installed; later refits return to the configured cadence.
3. **Exploit.** New samples are drawn from the updated GMM,
focusing on promising regions. By default, every fifth post-warmup
suggestion remains a global Sobol' exploration point.
focusing on promising regions. Ongoing post-warmup Sobol' exploration is
disabled by default and can be enabled with an explicit period.

GMM exploitation uses seeded Owen-scrambled Gauss–Sobol' points.
One Sobol' coordinate selects the mixture component, and inverse-normal
Expand All @@ -136,9 +140,9 @@ on completed trials in the leaderboard. Pending asks that cross the nominal
warmup boundary before the first completed-data fit remain Sobol' suggestions;
the unfitted prior is never used as exploitation.

`max_components` bounds mixture complexity (default 3), while
`max_components` bounds mixture complexity (default 1), while
`min_elite_samples` can delay fitting until a feasible elite workset reaches a
requested size (default 1). The effective component count may be smaller: the
requested size (default 5). The effective component count may be smaller: the
implementation requires enough elite samples to support each component.

Two implementation limits keep refitting bounded on unusually long studies.
Expand Down
14 changes: 7 additions & 7 deletions docs/python-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,7 @@ study = Study(
| `objectives` | `list` | required | List of `Minimize` / `Maximize` objectives (at least one) |
| `strategy` | `str` or strategy class | `"gmm"` | Search strategy. Pass a string (`"gmm"`, `"sobol"`, `"random"`) for defaults, or a configuration class (`Gmm(...)`, `Sobol()`, `Random()`) for fine-grained control. |
| `seed` | `int` or `None` | `None` | Random seed for reproducibility. When set, the same seed produces the same candidate sequence. |
| `max_trials` | `int` or `None` | `None` | Maximum number of trials. When set, `ask()` raises after this many trials have been dispatched. |
| `max_trials` | `int` or `None` | `None` | Maximum number of trials. When set, `ask()` raises after this many trials have been dispatched and the value supplies `S` for the automatic GMM warm-up. When omitted, the warm-up calculation uses `S=200` without imposing a trial cap. |

## The Ask/Tell Loop

Expand Down Expand Up @@ -458,7 +458,7 @@ Study(strategy=Gmm(refit_interval=10, elite_fraction=0.1), ...)

Gaussian Mixture Model strategy. Uses Sobol exploration followed
by GMM exploitation. Refits a GMM to the top `elite_fraction`
(default 25%) of trials every `refit_interval` (default 20)
(default 12.5%) of trials every `refit_interval` (default 20)
completed trials. With multiple objective groups, elites are ordered
by non-domination rank and then descending crowding distance. The
exploration budget counts issued `ask` suggestions, including pending
Expand Down Expand Up @@ -488,11 +488,11 @@ Study(strategy=Gmm(refit_interval=10, elite_fraction=0.1), ...)
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `refit_interval` | `int` or `None` | 20 | How often the GMM is refit, in completed trials |
| `elite_fraction` | `float` or `None` | 0.25 | Fraction of top trials used for refitting. Must be in (0, 1]. |
| `exploration_budget` | `int` or `None` | auto | Number of issued Sobol exploration suggestions before GMM exploitation begins. Pending asks count against this budget. When omitted, computed automatically from the total budget and number of dimensions. |
| `ongoing_exploration_period` | `int` or `None` | 5 | Continue global Sobol' exploration every Nth post-warmup suggestion. Use `0` to disable; explicit periods must be at least 2. |
| `max_components` | `int` or `None` | 3 | Maximum fitted mixture components. The effective count can be lower when the elite set is small. |
| `min_elite_samples` | `int` or `None` | 1 | Minimum feasible elite workset required before fitting. Must not exceed `max_refit_samples`. |
| `elite_fraction` | `float` or `None` | 0.125 | Fraction of top trials used for refitting. Must be in (0, 1]. |
| `exploration_budget` | `int` or `None` | auto | Number of issued Sobol exploration suggestions before GMM exploitation begins. Pending asks count against this budget. When omitted, HOLA doubles `min(floor(S/5), 50 + 2n)` and then rounds down to a power of two, for total budget `S` and dimension `n`; `S=200` when `max_trials` is unset. |
| `ongoing_exploration_period` | `int` or `None` | 0 | Continue global Sobol' exploration every Nth post-warmup suggestion. The default `0` disables it; explicit periods must be at least 2. |
| `max_components` | `int` or `None` | 1 | Maximum fitted mixture components. The effective count can be lower when the elite set is small. |
| `min_elite_samples` | `int` or `None` | 5 | Minimum feasible elite workset required before fitting. Must not exceed `max_refit_samples`. |
| `max_refit_samples` | `int` or `None` | 4096 | Maximum elite samples used by one GMM fit. Must be at least 1. |
| `max_refit_candidates` | `int` or `None` | 16384 | Maximum retained trials ranked during elite selection. Must be at least `max_refit_samples`; longer histories use deterministic stratified coverage. |

Expand Down
4 changes: 2 additions & 2 deletions hola-cli/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "hola-cli"
version = "1.0.1-rc8"
version = "1.0.1"
edition = "2024"
description = "Command-line server and distributed worker for HOLA"
documentation = "https://github.com/blackrock/HOLA/blob/main/docs/cli-guide.md"
Expand All @@ -19,7 +19,7 @@ path = "src/main.rs"
doc = false

[dependencies]
hola = { path = "../hola", features = ["server"], version = "1.0.1-rc8" }
hola = { path = "../hola", features = ["server"], version = "1.0.1" }
clap = { version = "4", features = ["derive"] }
# Keep the established module name while using the maintained, API-compatible fork.
serde_yaml = { package = "serde_yaml_ng", version = "0.10" }
Expand Down
19 changes: 10 additions & 9 deletions hola-cli/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1526,16 +1526,17 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
.checkpoint
.as_ref()
.map(|checkpoint| PathBuf::from(&checkpoint.directory));
let engine = HolaEngine::from_config(study_config)
.map_err(|e| format!("Failed to create engine: {e}"))?;

if let Some(path) = load_from {
let checkpoint_kind = engine
.load_checkpoint_with_fallback(&path)
.await
.map_err(|e| format!("Failed to load checkpoint '{path}': {e}"))?;
let engine = if let Some(path) = load_from {
let (engine, checkpoint_kind) =
HolaEngine::load_configured_checkpoint(study_config, &path)
.await
.map_err(|e| format!("Failed to load checkpoint '{path}': {e}"))?;
eprintln!("Loaded {} checkpoint from {path}", checkpoint_kind.as_str());
}
engine
} else {
HolaEngine::from_config(study_config)
.map_err(|e| format!("Failed to create engine: {e}"))?
};

let auth_token = configured_token(auth_token);
if !is_local_host(&host) && auth_token.is_none() {
Expand Down
4 changes: 2 additions & 2 deletions hola-py/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "hola-py"
version = "1.0.1-rc8"
version = "1.0.1"
edition = "2024"
description = "Python bindings for the HOLA hyperparameter optimization engine"
documentation = "https://github.com/blackrock/HOLA/blob/main/docs/python-guide.md"
Expand All @@ -18,7 +18,7 @@ name = "hola_opt"
crate-type = ["cdylib"]

[dependencies]
hola_engine = { package = "hola", version = "1.0.1-rc8", path = "../hola", features = ["server"] }
hola_engine = { package = "hola", version = "1.0.1", path = "../hola", features = ["server"] }
pyo3 = { version = "0.29", features = ["extension-module", "abi3-py310"] }
serde = "1"
serde_json = "1"
Expand Down
91 changes: 89 additions & 2 deletions hola-py/benchmarks/adapters/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,10 @@

from __future__ import annotations

from collections.abc import Callable
import json
from collections.abc import Callable, Mapping
from dataclasses import dataclass, field
from typing import Any, Protocol, runtime_checkable
from typing import Any, Protocol, cast, runtime_checkable

import numpy as np

Expand Down Expand Up @@ -70,6 +71,92 @@ def __init__(self, actual: int, expected: int, optimizer: str) -> None:
)


class EmpiricalExploitationError(RuntimeError):
"""A GMM benchmark run did not produce authenticated empirical exploitation."""

def __init__(self, actual: int, observed_diagnostics: Mapping[str, object]) -> None:
self.actual = actual
self.observed_diagnostics = dict(observed_diagnostics)
observed = json.dumps(
self.observed_diagnostics,
allow_nan=False,
ensure_ascii=True,
separators=(",", ":"),
sort_keys=True,
)
super().__init__(
"GMM empirical-exploitation gate failed "
"(requires gmm_fit_epoch>=1, gmm_sampling_ready=true, "
"gmm_origin_suggestions>=5, and "
"issued_suggestions==completed_evaluations); "
f"observed_diagnostics={observed}"
)


def empirical_exploitation_gate_configuration() -> dict[str, object]:
"""Return the manifest-bound practical-benchmark GMM gate."""

return {
"minimum_gmm_fit_epoch": 1,
"minimum_gmm_origin_suggestions": 5,
"on_failure": "preserved_error_outcome",
}


def _stable_observed_value(value: object) -> object:
if value is None or type(value) in {bool, int, str}:
return value
if type(value) is float:
return value if np.isfinite(value) else f"<nonfinite {value!r}>"
return f"<malformed {type(value).__name__}>"


def require_empirical_gmm_exploitation(study: object, completed_evaluations: int) -> None:
"""Fail closed unless a completed GMM run demonstrably used its fitted sampler."""

if type(completed_evaluations) is not int or completed_evaluations < 0:
raise ValueError("completed_evaluations must be a non-negative integer")
required_fields = (
"gmm_fit_epoch",
"gmm_origin_suggestions",
"gmm_sampling_ready",
"issued_suggestions",
)
observed: dict[str, object] = {"completed_evaluations": completed_evaluations}
try:
diagnostics_method = cast(Any, study).strategy_diagnostics
diagnostics = diagnostics_method()
except Exception as error:
observed["diagnostics_error"] = f"{type(error).__name__}: {error}"
raise EmpiricalExploitationError(completed_evaluations, observed) from error
if type(diagnostics) is not dict:
observed["diagnostics"] = f"<malformed {type(diagnostics).__name__}>"
raise EmpiricalExploitationError(completed_evaluations, observed)
diagnostics = cast(dict[object, object], diagnostics)
for field_name in required_fields:
observed[field_name] = (
"<missing>"
if field_name not in diagnostics
else _stable_observed_value(diagnostics[field_name])
)

fit_epoch = diagnostics.get("gmm_fit_epoch")
origin_suggestions = diagnostics.get("gmm_origin_suggestions")
sampling_ready = diagnostics.get("gmm_sampling_ready")
issued_suggestions = diagnostics.get("issued_suggestions")
valid = (
type(fit_epoch) is int
and fit_epoch >= 1
and type(origin_suggestions) is int
and origin_suggestions >= 5
and sampling_ready is True
and type(issued_suggestions) is int
and issued_suggestions == completed_evaluations
)
if not valid:
raise EmpiricalExploitationError(completed_evaluations, observed)


def assert_exact_evaluations(actual: int, expected: int, optimizer: str) -> None:
"""Fail a benchmark run whose objective-call count missed its contract."""
if actual != expected:
Expand Down
Loading