YARN-11937. Yarn Proxy Behind a Reverse Proxy#8300
Merged
brumi1024 merged 1 commit intoapache:trunkfrom Mar 10, 2026
Merged
Conversation
|
💔 -1 overall
This message was automatically generated. |
p-szucs
approved these changes
Mar 6, 2026
Contributor
|
Thanks for the patch @K0K0V0K, LGTM! |
Description
When the Yarn Proxy is deployed behind a reverse proxy that is also used in application tracking URLs, the Yarn Proxy should redirect requests to that proxy instead of attempting to proxy them internally.
Use Case
Consider the following scenario:
• A user runs a Spark job.
• The Spark UI is hosted in the Spark History Server (SHS).
• Multiple SHS instances are deployed for high availability (HA).
• The tracking URL points to a Knox Gateway, which routes requests to the available SHS instances.
This setup ensures high availability for the tracking UI. If one SHS instance becomes unavailable, another can continue serving the UI.
Problem Statement
When the Knox Gateway forwards a user’s HTTP request to the Yarn Proxy, the Yarn Proxy attempts to proxy the request back to the Knox Gateway. However, this proxied request does not include the JWT token. As a result, Knox initiates authentication instead of forwarding the request to the appropriate SHS instance.
Proposed Solution
For security reasons, the JWT token must not be forwarded to the tracking URL. Therefore, when an application registers a tracking URL that includes a specific flag indicating that it is served behind a reverse proxy, the Yarn Proxy should redirect the user directly to the tracking URL instead of attempting to proxy the request internally.
Config
New config was created: yarn.web-proxy.redirect-flag
slfan1989
approved these changes
Mar 9, 2026
|
🎊 +1 overall
This message was automatically generated. |
|
Hi All, Created a follow up PR for this because the current implementation might not correctly identify this parameter when it is part of a larger query string containing multiple parameters. For example in the following URL: Currently the logic expects the query string to be exactly yarn_knox_proxy=true, it will fail to recognize the flag when other parameters (like doAs, parameter1, or parameter2) are present. The parameter detection should be improved to properly parse the query string and identify the yarn_knox_proxy=true key-value pair regardless of its position or the presence of other parameters. Could you please take a look? Thank you! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
When the Yarn Proxy is deployed behind a reverse proxy that is also used in application tracking URLs, the Yarn Proxy should redirect requests to that proxy instead of attempting to proxy them internally.
Use Case
Consider the following scenario:
• A user runs a Spark job.
• The Spark UI is hosted in the Spark History Server (SHS).
• Multiple SHS instances are deployed for high availability (HA).
• The tracking URL points to a Knox Gateway, which routes requests to the available SHS instances.
This setup ensures high availability for the tracking UI. If one SHS instance becomes unavailable, another can continue serving the UI.
Problem Statement
When the Knox Gateway forwards a user’s HTTP request to the Yarn Proxy, the Yarn Proxy attempts to proxy the request back to the Knox Gateway. However, this proxied request does not include the JWT token. As a result, Knox initiates authentication instead of forwarding the request to the appropriate SHS instance.
Proposed Solution
For security reasons, the JWT token must not be forwarded to the tracking URL. Therefore, when an application registers a tracking URL that includes a specific flag indicating that it is served behind a reverse proxy, the Yarn Proxy should redirect the user directly to the tracking URL instead of attempting to proxy the request internally.
Config
New config was created: yarn.web-proxy.redirect-flag
How was this patch tested?
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?AI Tooling
If an AI tool was used:
where is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html