feat(api): add backward compatibility for Eval API method signatures #4683

saichandrapandraju · 2026-01-22T00:20:14Z

What does this PR do?

Adds a backward compatibility layer for the Eval API to support both old-style (individual parameters) and new-style (request objects introduced in #4425) calling conventions. When using the deprecated old-style, a DeprecationWarning is emitted.

Changes:

Add compat.py module with resolver functions
Update EvalRouter to accept both calling styles
Export compat helpers from llama_stack_api

Example:

New style (preferred, introduced in #4425)

await eval.run_eval(RunEvalRequest(benchmark_id="...", benchmark_config=...))

Old style (deprecated, emits warning - added in this PR)

await eval.run_eval(benchmark_id="...", benchmark_config=...)

Test Plan

Run unit tests for current changes: uv run pytest tests/backward_compat/test_eval_compat.py -v
Make sure feat(api): migrate Eval API to FastAPI router (#4345) #4425 tests pass - uv run pytest tests/unit/test_eval_models.py tests/unit/providers/nvidia/test_eval.py
Run pre-commit hooks: uv run pre-commit run --all-files
Verify deprecation warnings are emitted when using old-style parameters

cc @leseb @r-bit-rry

r-bit-rry · 2026-01-22T14:26:38Z

src/llama_stack_api/eval/compat.py

+    warnings.warn(
+        _DEPRECATION_MESSAGE.format(method_name=method_name, request_class=request_class),
+        DeprecationWarning,
+        stacklevel=2,


The stacklevel=2 in _emit_deprecation_warning points to the resolver function rather than user code. The call chain is:

_emit_deprecation_warning (stacklevel=1)

resolve_* function (stacklevel=2) ← current target

Router method (stacklevel=3)

User code (stacklevel=4) ← desired target

Fix: Change stacklevel=2 to stacklevel=4 or make it configurable.

Oops, yes. Changed stacklevel=2 to stacklevel=4 so warnings point to user code.

r-bit-rry · 2026-01-22T14:28:17Z

src/llama_stack_api/eval/compat.py

+
+    # Old-style parameters
+    if benchmark_id is not None and benchmark_config is not None:
+        _emit_deprecation_warning("run_eval", "RunEvalRequest")


When a request object is passed, it's returned immediately without validation. If someone passes RunEvalRequest(benchmark_id="", benchmark_config=None) (e.g., via Pydantic's construct()), the error surfaces deep in the router with confusing messages.
Add explicit validation when request object is provided, or rely on Pydantic validation being enforced at construction.

Added _validate_not_empty() checks for required fields when request objects are passed, catching empty/None values before they reach the router.

r-bit-rry · 2026-01-22T14:30:09Z

src/llama_stack_api/eval/compat.py

+            benchmark_config=benchmark_config,
+        )
+
+    raise ValueError("Either 'request' (RunEvalRequest) or both 'benchmark_id' and 'benchmark_config' must be provided")


for the valueerrors in compat:
Error messages don't indicate which parameters were actually provided vs. which are missing. Users must guess which parameter they forgot.

You can try:

raise ValueError( f"Either 'request' or all required params must be provided. " f"Missing: {', '.join(missing)}" )

Added _format_missing_params() helper that shows both missing and provided parameters in error messages.

r-bit-rry · 2026-01-22T14:30:53Z

tests/backward_compat/test_eval_compat.py

+            resolve_job_status_request(job_id="job-456")  # missing benchmark_id
+
+
+class TestResolveJobCancelRequest:


TestResolveJobCancelRequest and TestResolveJobResultRequest are missing test_missing_parameters_raises_error tests that exist for other resolvers

Added these tests to both TestResolveJobCancelRequest and TestResolveJobResultRequest.

r-bit-rry · 2026-01-22T14:32:09Z

tests/backward_compat/test_eval_compat.py

Nit: No test documents what happens when BOTH a request object AND individual parameters are provided. The code silently ignores individual parameters, which could confuse users. (could be also resolved with a documentation or ignored-> nit)

Added test_request_object_takes_precedence_over_individual_params tests and documented the behavior in docstrings.

r-bit-rry · 2026-01-22T14:33:08Z

src/llama_stack/core/routers/eval_scoring.py

+        Args:
+            request: The new-style request object (preferred)
+            benchmark_id: (Deprecated) The benchmark ID
+            job_id: (Deprecated) The job ID


All other methods document their return type, but job_cancel does not document that it returns None (which is part of the signature).

Added Returns: None to the job_cancel docstring.

r-bit-rry · 2026-01-22T14:33:42Z

src/llama_stack_api/eval/compat.py

+    RunEvalRequest,
+)
+
+_DEPRECATION_MESSAGE = (


Standard practice is to include a target version for removal (@leseb maybe you can suggest a version).

Got it, I think it'd be 0.6.0 but I'll wait for @leseb

Hi @leseb , is it reasonable to specify 0.6.0 as target version for removal? thanks!

r-bit-rry · 2026-01-22T14:34:30Z

tests/backward_compat/test_eval_compat.py

Nit: Empty strings pass the is not None check but create invalid requests. Consider adding validation or at least a test documenting this behavior.

Old-style params now use truthiness checks (if benchmark_id and ...) which reject empty strings/lists. Request objects are explicitly validated with _validate_not_empty(). Added TestEmptyValueValidation tests.

saichandrapandraju · 2026-01-26T23:30:40Z

Thanks for the review, @r-bit-rry ! I addressed the comments and happy to discuss any remaining concerns!

feat(api): add backward compatibility for Eval API method signatures

e13d0a4

saichandrapandraju requested review from ashwinb, bbrowning, cdoern, ehhuang, franciscojavierarceo, leseb, mattf and raghotham as code owners January 22, 2026 00:20

meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Jan 22, 2026

Merge branch 'main' into eval-api-bkwd-compat

b7f04bd

r-bit-rry suggested changes Jan 22, 2026

View reviewed changes

validate request objects + format missing params; add respective tests

0594b67

saichandrapandraju requested a review from r-bit-rry January 26, 2026 23:30

saichandrapandraju and others added 2 commits January 29, 2026 21:45

update deprecation warning message to specify version target

99fd695

Merge branch 'main' into eval-api-bkwd-compat

7928679

		resolve_job_status_request(job_id="job-456") # missing benchmark_id


		class TestResolveJobCancelRequest:

feat(api): add backward compatibility for Eval API method signatures #4683

Are you sure you want to change the base?

feat(api): add backward compatibility for Eval API method signatures #4683

Uh oh!

Conversation

saichandrapandraju commented Jan 22, 2026

What does this PR do?

New style (preferred, introduced in #4425)

Old style (deprecated, emits warning - added in this PR)

Test Plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saichandrapandraju Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saichandrapandraju commented Jan 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

saichandrapandraju Jan 26, 2026 •

edited

Loading