Skip to content

Count non-OK responses in failed-requests metric#4564

Open
pepone wants to merge 2 commits intoicerpc:mainfrom
pepone:fix/metrics-non-ok-responses
Open

Count non-OK responses in failed-requests metric#4564
pepone wants to merge 2 commits intoicerpc:mainfrom
pepone:fix/metrics-non-ok-responses

Conversation

@pepone
Copy link
Copy Markdown
Member

@pepone pepone commented Apr 28, 2026

Fix #4435

MetricsInterceptor and MetricsMiddleware only counted thrown
exceptions in failed-requests, ignoring returned IncomingResponse /
OutgoingResponse with non-OK StatusCode. The library's own internal
components routinely return non-OK responses rather than throwing —
DeadlineMiddleware returns DeadlineExceeded, Router.NotFoundDispatcher
returns NotFound, generated dispatchers convert DispatchException to
non-OK responses — so a typical production setup silently
under-reported failures and produced metrics that depended on whether
each component chose to throw or return.

Most strikingly, the same logical event produced different metrics on
the two sides of a request: DeadlineInterceptor throws and counts as
a failure on the client, while DeadlineMiddleware returns a non-OK
response and was counted as success on the server.

Inspect StatusCode after awaiting the next invoker/dispatcher and
increment failed-requests when it is not Ok.

Closes icerpc#4435
Copilot AI review requested due to automatic review settings April 28, 2026 13:33
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates IceRpc.Metrics to count non-OK responses as failures in the failed-requests metric, aligning metrics behavior between “returned error responses” and “thrown exceptions” (Fix #4435).

Changes:

  • Increment failed-requests when dispatch/invocation completes with a non-StatusCode.Ok response.
  • Add regression tests covering non-OK OutgoingResponse (dispatch) and non-OK IncomingResponse (invocation).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File Description
src/IceRpc.Metrics/MetricsMiddleware.cs Counts non-OK OutgoingResponse.StatusCode as a failed dispatch.
src/IceRpc.Metrics/MetricsInterceptor.cs Counts non-OK IncomingResponse.StatusCode as a failed invocation.
tests/IceRpc.Metrics.Tests/MetricsMiddlewareTests.cs Adds coverage asserting non-OK dispatch responses increment failed-requests.
tests/IceRpc.Metrics.Tests/MetricsInterceptorTests.cs Adds coverage asserting non-OK invocation responses increment failed-requests.

Copy link
Copy Markdown
Member

@externl externl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@pepone
Copy link
Copy Markdown
Member Author

pepone commented Apr 28, 2026

One case I'm not sure here is iceprc ApplicationError or ice UserException, should this count we count them as failures in the metrics?

Copy link
Copy Markdown
Member

@InsertCreativityHere InsertCreativityHere left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

@InsertCreativityHere
Copy link
Copy Markdown
Member

One case I'm not sure here is iceprc ApplicationError or ice UserException, should this count we count them as failures in the metrics?

Also not clear to me whether these should be counted as failures... I'd lean towards "yes", but it's not a strong "yes" : v)
Maybe worth blogging about.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Audit-Medium] failed-requests ignores normal non-OK responses, so fa…

4 participants