feat: support schemeless proxied URLs with a configurable default scheme#141
Open
lemuelbarango wants to merge 7 commits into
Open
feat: support schemeless proxied URLs with a configurable default scheme#141lemuelbarango wants to merge 7 commits into
lemuelbarango wants to merge 7 commits into
Conversation
Requesting a URL without an http(s):// prefix (e.g. /example.com/page) previously failed with 'unsupported protocol scheme ""'. The proxy now prepends a configured default scheme to such requests so they Just Work. The default is 'https' and is configurable via the DEFAULT_SCHEME env var or the --default-scheme / -s CLI flag. Only 'http' and 'https' are accepted; any other value causes a startup error. Behavior for URLs that already include a scheme is unchanged, and the existing referer-based relative-path reconstruction still takes precedence when the referer points at a proxied URL.
…d fetchSite check
…nit log.Fatal - ensureScheme now uses a '://' substring check instead of magic-length and dot/colon heuristics, so single-segment hosts (localhost/api) and schemeless host:port URLs (example.com:8443/x) now work. - Strip a stray leading '://' so '://example.com' no longer produces 'https://://example.com'. - Move normalization out of fetchSite and into the handler boundaries (extractUrl, api, raw) — single normalization point per entry. - Drop the init() block that called log.Fatal on invalid DEFAULT_SCHEME. Validation now happens once in cmd/main.go where it's recoverable, so utility commands like --merge-rulesets aren't killed by an unrelated env var.
lemuelbarango
changed the title
The applyRules function was defined but never called from any code path, making the entire ruleset infrastructure (regexRules, injections) dead code. The only ruleset fields that took effect were Headers, URLMods, and the incidental side effects of rewriteHtml (which renames `src="/..."` to `script="..."` on relative-URL script tags). Invoke applyRules in fetchSite, BEFORE rewriteHtml. Order matters: - regexRules patterns like `<script[^>]*src="..."` need to see original src attributes (rewriteHtml renames them and breaks subsequent matches). - injections add content to <head>, position-independent — but easier to follow if all ruleset work happens before URL munging.
Three tests in handlers/apply_rules_test.go: - TestApplyRulesUnit — applyRules itself accepts regexRules + injections and applies both. Always passed; documents the surface area. - TestFetchSiteInvokesApplyRules — regression test for the dead-code bug. Spins up an httptest.NewServer, configures a rule with regexRules and injections, calls fetchSite, asserts both effects landed. Fails without the applyRules invocation in fetchSite. - TestApplyRulesOrderingBeforeRewriteHtml — codifies the ordering: applyRules must run BEFORE rewriteHtml because rewriteHtml renames src= attributes on relative-URL script tags, which would otherwise prevent src-based regex patterns from matching. Fails if the call order is swapped. Verified: with the fix in proxy.go reverted, the second and third tests fail with clear error messages pointing at the ruleset effects that didn't land.
lemuelbarango
force-pushed
the
feat/default-scheme
branch
from
June 25, 2026 17:23
4f1e6e3 to
58ee7e0
Compare
Contributor
|
I'll need a moment to test this. But looks promising. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Requesting a URL through the proxy without an
http://orhttps://prefix (e.g.http://localhost:8080/example.com/page) currently fails with:This PR makes the proxy prepend a configured default scheme to schemeless URLs so they work end-to-end.
Behavior
DEFAULT_SCHEMEenv var and--default-scheme/-sCLI flag. Default ishttps.httpandhttpsare accepted; anything else is rejected at startup incmd/main.gowith a clear error (nolog.Fatalfrominit(), so utility flows like--merge-rulesetsaren't affected by a stale env var).scheme://prefix (includinghttp://,https://,ftp://,chrome-extension://, etc.) are passed through unchanged.Referer-based relative-path reconstruction still takes precedence — only when the referer can't reconstruct a proxied URL does the default-scheme fallback kick in.extractUrlfallback forProxySite,Api, andRaw), not insidefetchSite.Implementation notes
ensureSchemeuses a simple://-substring check rather thanurl.Parse, because Go's parser is RFC-3986 lenient enough to treatexample.com:8443/pageas having schemeexample.com— exactly the input we want to normalize. The trade-off is that non-network URI schemes without://(e.g.mailto:foo@bar.com) also get the default scheme prepended; they still fail, just at DNS rather than at the stdlib protocol check. This is documented in the test cases.Test plan
handlers/default_scheme_test.gocovers:SetDefaultSchemeacceptshttp/https(case-insensitive, trimmed) and rejects everything else.ensureSchemeunit tests across schemed URLs, bare hosts,host/path,host:port/path, single-segment hosts (localhost/api), relative paths (/images/foo.jpg), stray://, non-network schemes, andftp://.extractUrlintegration via Fiber: schemeless URL with no referer gets the default scheme; explicithttp://andhttps://URLs pass through; relative paths reconstruct from a proxied referer; configuredhttpscheme is honored.unsupported protocol scheme ""error so we'd notice if Go changes the wording.Rawhandler against anhttptestupstream, exercising the new boundary normalization.Run:
Both green.