Http Proxy Manager Token Expired Errors

[KarissaBrummell]

What product are you having troubles with?

Console

What Console version are you using?

Suite v14.0, Console v1.118.9

Description

We are having a problem with the Http Proxy Manager. It seems to be making calls to downstream applications with tokens that are milliseconds from expiring, thereby causing those applications to return 401s and the Http Proxy Manager to log the "token expired" error.

From my understanding, this is how the plugin handles tokens:

• The HttpProxyManager requests a token from the authorization server specified in the tokenIssuerUrl configuration.
• It stores this token in an in-memory cache until it expires.
• When a token expires, the HttpProxyManager logs "token expired" and removes it from the cache.
• It then requests a new token for subsequent requests.
• The service also proactively removes tokens from the cache when it receives 401 or 403 responses, even if the token might not actually be expired.

Right now, our token expiration is set to 1 hour (3600 seconds, technically). Currently in our busiest namespaces, we see that every hour on/near the hour mark, microservices return 401s and there is an accompanied "token expired" log by the Proxy Manager. It seems like what's causing this is that the token is milliseconds from expiring, we make one final call, get a 401 back, log the error, then refresh the token and everything is fine for another hour.

This is causing pain for our teams as it's triggering alerts and failed requests in our older workflows that don't have high resiliency.

It's worth noting that we have this problem in projects using various versions of the plugin, from latest (3.5.0) all the way back to 1.x and everything in between.

I believe this is the same root issue that was brought up in this discussion: #386. It's also worth noting that a comment on the thread in that discussion mentioned setting up a meeting to talk about this further, but that meeting never actually happened.

Actual Outcome

On a cadence that matches our configured token expiration (currently set to 1 hour), applications downstream of the Http Proxy Manager return 401s and the Proxy Manager logs a "token expired" error.

Expected Outcome

Http Proxy Manager handles token expiration in a way that ensures that downstream applications are called with a valid token and won't return 401s on an hourly cadence. Perhaps we could have a configurable way to preemptively mark a token as expired.

[marcofilippi]

Hi @KarissaBrummell,
thanks for bringing this to our attention and for the detailed description. We will look into this issue so that we can come up with a solution that can solve this inconvenience as soon as possible.

[Danielecina]

Hello, @KarissaBrummell.
We have improved the microservice to avoid this race condition by modifying the logic.
In addition, in version v3.6.0, we have added the ability to configure TOKEN_PREEMPTIVE_EXPIRY_SECONDS (the default value is 30 seconds).
Please let us know if your issue has been resolved.

[KarissaBrummell]

Hey @Danielecina,

Thank you for getting back with us on this! We have tried out version 3.6.0 and have a couple of thoughts. The first thing we noticed was that the logs don't mention anything about preemptive refreshing taking place, only that tokens had expired. This lack of clarity seems like it would make tracing through the flow more difficult in the future or for someone who has less context about this specific edge case. Also, while the preemptive token expiry does solve this for the majority of use cases, I think there is still potential for unusually long-running requests to encounter the bug. We ended up adding a custom resource that retries all 401s returned to the HttpProxyManager. That, coupled with your implementation, should ensure that no requests fall through the cracks.

All that to say, I think we can consider this problem solved. Thank you again for your cooperation and assistance!

[dallas-fletchall]

@Danielecina, the changelog does not include this version and the latest version in our marketplace is 3.5.0.

We tested the container version 3.6.0, but do not want to detach from the marketplace to allow upgrade to that version. Can you validate these items?

[Danielecina]

Hello @dallas-fletchall, thank you for the tag. I would like to clarify that I no longer work for the company, so I am no longer responsible for its maintenance. For further information or assistance, please contact the current maintenance managers @FedericoOldrini