This issue was found by a Codex global repository scan of tracked non-test files at commit 8c93925cb10b401b2b83c738bd9263fd74474468.
Relevant code
|
zeroshot_raw = self.metrics_calculator.calculate_mean_m_bar_domain(model) |
|
if ( |
|
efficiency_raw is None |
|
or efficiency_raw["average_time"] is None |
|
or zeroshot_raw is None |
|
): |
|
continue |
|
results.append( |
|
{ |
|
"name": model.model_metadata.pretty_name, |
|
"family": model.model_family, |
|
"nparams": model.model_metadata.num_parameters, |
|
"efficiency": np.round(efficiency_raw["average_time"], 2), |
|
"std": np.round(efficiency_raw["standard_deviation"], 2), |
|
"generalizability error": np.round(zeroshot_raw, 2), |
|
def calculate_efficiency_results(self) -> dict[str, float]: |
|
efficiency_results = self.fetcher.fetch_inference_efficiency_results() |
|
# filter out models with missing efficiency results |
|
efficiency_results = { |
|
model: metrics |
|
for model, metrics in efficiency_results.items() |
|
if metrics is not None and metrics["average_time"] is not None |
|
} |
|
if not efficiency_results: |
|
logging.warning("No inference efficiency results found.") |
|
return {} |
|
|
|
# extract inference time and calculate efficiency score |
|
efficiency_results = { |
|
model: 100 / metrics["average_time"] |
|
for model, metrics in efficiency_results.items() |
|
} |
|
We define an efficiency score, $M_E^m$, by normalizing the average inference time (with unit $\mathrm{\mu s/atom}$), $\bar \eta^m$, of a given LAM measured over 900 configurations with respect to an artificial reference value, thereby rescaling it to a range between zero and positive infinity. A larger value indicates higher efficiency. |
|
|
|
$$M_E^m = \frac{\eta^0 }{\bar \eta^m },\quad \eta^0= 100\ \mathrm{\mu s/atom}, \quad \bar \eta^m = \frac{1}{900}\sum_{i}^{900} \eta_{i}^{m}$$ |
Impact
The ranking code defines efficiency as 100 / average_time, and the README describes the same inverse-latency score. generate_scatter_plot() stores efficiency_raw["average_time"] directly under the JSON field named efficiency.
This makes the scatter data use the opposite direction from the ranking score: larger values mean slower models in the scatter JSON but better models in the ranking metric.
Suggested fix
Either emit 100 / average_time for the scatter efficiency field, or rename the field to latency and update downstream visualization labels and docs accordingly.
This issue was found by a Codex global repository scan of tracked non-test files at commit
8c93925cb10b401b2b83c738bd9263fd74474468.Relevant code
LAMBench/lambench/metrics/vishelper/plot_generation.py
Lines 35 to 49 in 8c93925
LAMBench/lambench/metrics/vishelper/metrics_calculations.py
Lines 200 to 216 in 8c93925
LAMBench/lambench/metrics/results/README.md
Lines 109 to 111 in 8c93925
Impact
The ranking code defines efficiency as
100 / average_time, and the README describes the same inverse-latency score.generate_scatter_plot()storesefficiency_raw["average_time"]directly under the JSON field namedefficiency.This makes the scatter data use the opposite direction from the ranking score: larger values mean slower models in the scatter JSON but better models in the ranking metric.
Suggested fix
Either emit
100 / average_timefor the scatterefficiencyfield, or rename the field tolatencyand update downstream visualization labels and docs accordingly.