Skip to content

[Code scan] Scatter plot reports latency under the efficiency field #441

Description

@njzjz

This issue was found by a Codex global repository scan of tracked non-test files at commit 8c93925cb10b401b2b83c738bd9263fd74474468.

Relevant code

zeroshot_raw = self.metrics_calculator.calculate_mean_m_bar_domain(model)
if (
efficiency_raw is None
or efficiency_raw["average_time"] is None
or zeroshot_raw is None
):
continue
results.append(
{
"name": model.model_metadata.pretty_name,
"family": model.model_family,
"nparams": model.model_metadata.num_parameters,
"efficiency": np.round(efficiency_raw["average_time"], 2),
"std": np.round(efficiency_raw["standard_deviation"], 2),
"generalizability error": np.round(zeroshot_raw, 2),

def calculate_efficiency_results(self) -> dict[str, float]:
efficiency_results = self.fetcher.fetch_inference_efficiency_results()
# filter out models with missing efficiency results
efficiency_results = {
model: metrics
for model, metrics in efficiency_results.items()
if metrics is not None and metrics["average_time"] is not None
}
if not efficiency_results:
logging.warning("No inference efficiency results found.")
return {}
# extract inference time and calculate efficiency score
efficiency_results = {
model: 100 / metrics["average_time"]
for model, metrics in efficiency_results.items()
}

We define an efficiency score, $M_E^m$, by normalizing the average inference time (with unit $\mathrm{\mu s/atom}$), $\bar \eta^m$, of a given LAM measured over 900 configurations with respect to an artificial reference value, thereby rescaling it to a range between zero and positive infinity. A larger value indicates higher efficiency.
$$M_E^m = \frac{\eta^0 }{\bar \eta^m },\quad \eta^0= 100\ \mathrm{\mu s/atom}, \quad \bar \eta^m = \frac{1}{900}\sum_{i}^{900} \eta_{i}^{m}$$

Impact

The ranking code defines efficiency as 100 / average_time, and the README describes the same inverse-latency score. generate_scatter_plot() stores efficiency_raw["average_time"] directly under the JSON field named efficiency.

This makes the scatter data use the opposite direction from the ranking score: larger values mean slower models in the scatter JSON but better models in the ranking metric.

Suggested fix

Either emit 100 / average_time for the scatter efficiency field, or rename the field to latency and update downstream visualization labels and docs accordingly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions