diff --git a/CHANGELOG.md b/CHANGELOG.md index 5c7dad7..4bb8a61 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,157 @@ diagnostic codes in surface: adding a code is a minor change, changing what one means is a breaking one. +## Unreleased + +The start of `v0.7.0`: the bounds §10 of the design policy names and the tree +did not yet enforce, and the per-stage configuration surface that lets a host +state them for one stage rather than for the whole process. + +### Added + +- **Per-stage configuration through `ArResolverContext`.** A context is made + from a string, through OpenUSD's own entry point, with the environment's own + names: + + ```python + ctx = Ar.GetResolver().CreateContextFromString( + "https", "USD_HTTP_RESOLVER_DESTINATIONS=public; USD_HTTP_RESOLVER_MAX_RETRIES=0") + stage = Usd.Stage.Open("https://example.org/scenes/main.usda", ctx) + ``` + + One vocabulary and one parser: a value in a context is refused or adjusted + for exactly the reasons the same value in the environment would be, judged + over the environment it will be layered on. Eight variables may be set per + stage — the three deadlines, retries, redirects, the destination policy, and + the two coalescing limits. The block size, the two budgets, and the persistent + directory stay the environment's, because every stage shares the store they + configure and the store's stripes are sized for one block size. Precedence is + context, then environment, then default. Values are kept as the parser read + them, so `060000` and `60000`, or `private, public` and `public,private`, are + one context rather than two to every table OpenUSD keys on one. Problems are + reported once, when the context is created, and never per bind. + +- **A stage's context cannot be walked past through another stage.** Four ways + it could have been, each closed and each with its case in + `httpResolver_stage`. Every identifier this resolver owns is context-dependent, + because for a path that is not, OpenUSD's layer registry finds a loaded layer + by identifier whatever `Resolve` has just said. The opens `Resolve` retains + are keyed by transport options as well as identifier, because a reader keeps + the options it was opened with. Resolutions inside an `ArResolverScopedCache` + are cached by this resolver, keyed the same way, because OpenUSD caches them + by path alone for a resolver that does not — and a scope routinely spans two + stages. And asset info answers from memory only for a caller whose policy + could have reached the asset, so a refusing stage is not told the size and + token of an asset another stage opened. + +- **Installing the bundle still changes nothing about a local-only process.** + Implementing contexts means OpenUSD constructs this resolver in every process + that opens any stage, so the constructor does nothing: the environment is + read, the process stores configured, the persistent directory created, and + problems reported at the first resolve, open, asset-info query, or context + creation. A child process whose only stage is local, with + `USD_HTTP_RESOLVER_PERSISTENT_CACHE_DIR` set, leaves no directory behind; with + configuration back in the constructor, it did. + +- **A context is readable from Python**, as its canonical string: + `Ar.ResolverContext('USD_HTTP_RESOLVER_DESTINATIONS=public')`. Without a + to-Python conversion, `ctx.Get()` raised and `Usd.Stage.__repr__` printed + `pathResolverContext=`. The conversion is registered once Python + is running, under the GIL and no other lock, so a Python thread and a C++ + thread creating contexts at once cannot wait on each other. The context type + lives in this bundle's namespace, because `ArResolverContext` matches context + objects by type name. + +- **A destination policy**, `USD_HTTP_RESOLVER_DESTINATIONS`: which classes of + address — `public`, `private`, `loopback`, `link-local`, `metadata` — a + connection may reach. §10.2 of the design policy makes reach a declared policy + rather than whatever the host's network allows, because an identifier can + arrive from a layer nobody here authored and a resolver that fetches whatever + it is told is a request-forgery primitive. + + The default is `public,private,loopback`. Loopback and private networks stay + reachable, because local fixture servers and intranet hosts are what `http` is + registered for and a default that broke them would be overridden everywhere; + link-local is refused, and so are the well-known instance-metadata endpoints, + which are a class of their own and classified by value, because no range + contains them: `169.254.169.254` is where most clouds put theirs, but AWS's + IPv6 endpoint is unique-local, Alibaba's is in the shared address space, and + Azure's WireServer is public. Permitting `link-local` does not permit + `metadata`. + + Judged three times, and none is redundant. At connect time, against the + address libcurl is about to connect to — after the name was resolved, before a + socket exists — which is what makes it hold for `localhost` and for a name + whose answer changed between lookups. Before each request, against the host + as libcurl's own URL parser will send it, which is what makes it hold through + a proxy: libcurl normalizes `2852039166`, `0xa9fea9fe`, and + `%31%36%39.254.169.254` to `169.254.169.254` before the proxy sees the + request, and without this check each of them reached a proxy that forwarded it + there. And at every redirect hop against a canonical literal, before any + transport sees it. Removing the connect-time check lets `localhost` through a + policy that refuses loopback; removing the client-side one lets the spellings + above through a proxy; each has the case that says so. + + An IPv6 address carrying an IPv4 one — mapped, compatible, or NAT64 — is the + class of the address it carries, so `[::ffff:169.254.169.254]` is `metadata`. + A refusal is `AccessDenied` (`HTTP002`) naming the class, with no request sent + and no retry: the code a `403` gets, because a caller does the same thing about + both. A list is read with its line breaks trimmed, so a list broken across + lines is the list written rather than a refused value that falls back to the + wider default. Sockets the policy admits are created close-on-exec. + +- **The scheme allowlist in the client as well as the parser.** libcurl is told + `http,https` and nothing else, so a parser that ever widened would widen into a + refusal rather than into a `file:` read. + +- **A bound on the response header block**, 64 KiB per exchange and summed + across interim `1xx` responses, counted in the transport before a line is + stored. With the caller's buffer bounding the body, a response can no longer + choose how much this process allocates for it, which is the whole of §10.1's + "bound the response header block and the total response size". A response + abandoned at the bound is refused whole, as `InvalidResponse` naming the + bound, whatever its status — its status line arrived intact, and an open that + read `Content-Length` and `Accept-Ranges` out of the prefix that fit would be + acting on a response nobody finished receiving. It is not retried, even when + the status line that did arrive was a `503`: asking again does not make the + block smaller. + + The bound was not optional, and the corpus is how that is known rather than + argued. With it removed, libcurl 8.7.1 opens an asset behind a megabyte of + ordinary header fields without complaint; the library's own ceilings are on a + single line, and a block of kilobyte lines never reaches them. + +- **`OversizedHeaders`**, a nineteenth corpus row: a correct response padded + with a megabyte of kilobyte-sized fields, placed after the ones that matter. + Kilobyte fields rather than one enormous one, so that what a client has to + bound is the block and not the line — a row made of one huge line would be + caught by the library's limit and would prove nothing about the client's. The + self-test asserts the size from the bytes on the wire and everything else + about the response against the Normal row, so a client cannot pass by + refusing a response that was also malformed. + +- **The scheme allowlist, asserted at the redirect hop.** It already held, as a + consequence of a `Location` going through the same parser as an identifier; + it is now a case, because a consequence is the kind of property nothing + notices losing. Seven targets are refused and never requested — `file:` in + two spellings, `ftp:`, `gopher:`, `data:`, `s3:`, and an `https:` with no + authority — and the two scheme-less forms that stay inside the allowlist, a + network-path reference and an absolute path, are still followed. + +### Changed + +- **An adjusted configuration value says it was used.** A block size rounded + down to a power of two, or a coalescing gap capped under the request ceiling, + used to be reported with the same ending as a refused value — "using the + default" — which was false: the adjusted value was the one in force. Adjusted + and refused values are now told apart, and a refused context value says that + its stage falls back to the environment rather than to the default. + +- **The environment is read at the resolver's first use, not at its + construction.** Nothing changes for a process that uses the resolver; a + process that only opens local stages no longer has its environment read, its + process stores reconfigured, or configuration warnings posted on its behalf. + ## `v0.5.0` - 2026-08-27 The resolver becomes an independently composable geospatial-runtime component. diff --git a/docs/architecture/DIAGNOSTICS.md b/docs/architecture/DIAGNOSTICS.md index 470ed38..9a22df5 100644 --- a/docs/architecture/DIAGNOSTICS.md +++ b/docs/architecture/DIAGNOSTICS.md @@ -133,7 +133,7 @@ diagnostics. The mapping is one-way and total: | Code | `HTTPxxx` | OpenUSD | Typical cause | | --- | --- | --- | --- | | `NotFound` | `HTTP001` | error | `404`, or a resolved path that does not exist | -| `AccessDenied` | `HTTP002` | error | `401`, `403` | +| `AccessDenied` | `HTTP002` | error | `401`, `403`, or a destination the policy refuses ([CONFIGURATION.md](../reference/CONFIGURATION.md) §2.1) | | `RangeNotSupported` | `HTTP003` | error | no `Accept-Ranges`, or `200` in response to `Range` | | `InvalidResponse` | `HTTP004` | error | wrong `Content-Range`, truncated body, bad framing | | `NetworkError` | `HTTP005` | error | connection reset, DNS, TLS failure | diff --git a/docs/architecture/RESOLVER.md b/docs/architecture/RESOLVER.md index 0ccc455..bfccf9a 100644 --- a/docs/architecture/RESOLVER.md +++ b/docs/architecture/RESOLVER.md @@ -10,8 +10,7 @@ it is described in [ASSET_READER.md](ASSET_READER.md). Sections marked **Planned** are direction, not shipped behavior. Status: implemented in `v0.2.0`, except §3, which is `v0.4.0`, and §6, which is -`v0.6.0` apart from the environment variables named in -[CONFIGURATION.md](../reference/CONFIGURATION.md). +`v0.7.0`. §3 has landed: asset info and identity stability are implemented, and what that surface may and may not publish is stated there rather than left to the code. @@ -24,11 +23,17 @@ The bundle registers a URI-scheme resolver, not the primary resolver: "Types": { "HttpResolver": { "bases": ["ArResolver"], + "implementsContexts": true, + "implementsScopedCaches": true, "uriSchemes": ["http", "https"] } } ``` +The two `implements` flags are §6's, and neither changes what a local asset +does: they are what lets a stage's context configure this resolver, and what +keeps OpenUSD from caching its resolutions by path alone. + Consequences that are contract, not detail: - The host's primary resolver is unchanged. Local paths keep resolving exactly @@ -146,7 +151,9 @@ two revisions, and each is individually consistent, which is exactly the guarantee §2.1 of [ASSET_READER.md](ASSET_READER.md) makes. A failure is **not** retained. Caching one would turn a server that was -restarting into an asset that does not exist for the rest of the process. +restarting into an asset that does not exist for the rest of the process. The +one place a failure is kept is inside an `ArResolverScopedCache`, for the life +of the scope, because that is what a scope is for (§6) and a scope ends. The table of retained opens is **bounded**. A resolve that is never followed by an open is legal and normal — a host probing for existence does it constantly — @@ -370,16 +377,98 @@ explicitly. Assets are immutable; publishing a new revision at a new path is the supported editing model, per §6 of the [design policy](../design/DESIGN_POLICY.md). -## 6. Context and configuration — Planned (`v0.6.0`) +## 6. Context and configuration + +`ArResolverContext` binding is where per-stage configuration belongs, and as of +`v0.7.0` it is where this resolver reads it from. The environment is the +process's configuration and the bootstrap for everything else; a context +overrides it for the stage it is bound to, and for nothing else +([CONFIGURATION.md](../reference/CONFIGURATION.md) §4). + +A context is created from a string, through OpenUSD's own entry point, and in +no other way: -`ArResolverContext` binding is where per-stage configuration belongs: cache -budget, timeouts, retry policy, and — later — a credential provider. It is -resolved at bind time, never read from a global on each request. +```text +ArGetResolver().CreateContextFromString("https", + "USD_HTTP_RESOLVER_DESTINATIONS=public; USD_HTTP_RESOLVER_MAX_RETRIES=0") +``` -Environment variables are the v0.x mechanism and are documented in -[CONFIGURATION.md](../reference/CONFIGURATION.md). They are a bootstrap, not -the final surface: a host that opens two stages against two servers with two -credentials cannot be served by a process-global. +The names are the environment's, so the configuration surface stays one +vocabulary, and the entry point is OpenUSD's, so no host includes a header from +this repository to configure it — the property ADR-0001 holds consumers to, +extended to the hosts that configure them. What the object carries is the +overrides as written, after validation; what they produce is resolved against +the environment when a call is made under it. + +Five consequences are contract rather than detail. + +**A context sets what binds a reader, and not what the process shares.** The +transport bounds, the destination policy, and the coalescing limits may be set +per stage. The block size, the two cache budgets, and the persistent directory +may not: the block store and the persistent tier are shared by every stage in +the process (CACHE.md §7), and the store's stripes are sized for one block size. +A context that names one of those is told so when it is created. + +**Every identifier this resolver owns is context-dependent.** Not because a path +resolves to a different path under two contexts — an identifier resolves to +itself — but because whether it resolves *at all* can, and OpenUSD's layer +registry acts on the answer. For a path that is not context-dependent, +`SdfLayer::FindOrOpen` finds an already-loaded layer by its identifier whatever +`Resolve` has just said; for one that is, it looks the layer up by the path +`Resolve` returned. Answering no would let one stage's destination policy be +walked past by opening the same URL in another stage first, and +`httpResolver_stage` asserts that it cannot be. + +**A retained open is handed only to a caller it fits.** §2.3's table of +retained opens is keyed by the identifier *and* the transport options the +reader was opened with, because a reader keeps those options for its lifetime. +A reader a resolve retained under one policy is never handed to an `OpenAsset` +under a narrower one; that call opens again, under its own. + +**A scoped cache is this resolver's, and keyed the same way.** Inside an +`ArResolverScopedCache`, OpenUSD caches `Resolve` on behalf of any resolver that +does not implement scoped caches — by path alone. A scope routinely spans more +than one stage, and a path resolved under a permissive context would then be +answered under a refusing one without this resolver being asked. So the bundle +declares `implementsScopedCaches` and keeps the scope's resolutions itself, +keyed by identifier and configuration. It keeps what OpenUSD's cache kept, +failures included, for the life of the scope, because composition resolves one +reference once per arc and the scope is what stops that costing one request per +arc. + +**Identity is shared across contexts, but not told across a policy.** A +validator describes the bytes at a URL, not the configuration that fetched +them, and §3.2's record of a republish is kept per identifier. What asset info +will not do is answer from memory for a caller who could not have reached the +asset: an identity is remembered with the destination policies it was reached +under, and answered only for a caller whose own policy covers one of them. A +stage whose context refuses a destination is told what it would have been told +had nobody opened the asset there. + +Implementing contexts has a cost that is paid in the constructor's shape rather +than in behavior. OpenUSD constructs every resolver that implements contexts or +scoped caches in any process that binds a context or opens a scope — which is +every process that opens a stage, local ones included — and may construct two +at once and keep one. So the constructor does nothing: the environment is read, +the process stores are configured, the persistent directory is created, and the +environment's problems are reported at the first resolve, open, asset-info query, +or context creation. §1's promise that installing this bundle never changes how +a local asset opens includes not creating a directory for a host that never +named a remote one, and `httpResolver_stage` asserts it from a child process +whose first contact with the resolver is a local stage. + +What a context reaches is what OpenUSD resolves and opens while it is bound, and +the boundary is worth stating because it is OpenUSD's rather than this +resolver's. The context is read from the calling thread, per call. Composition +binds a layer stack's context on every thread it computes a prim index on, so +the layers and references a stage composes are resolved under the stage's +context however parallel the composition is. A plugin that opens an asset on a +thread of its own, with nothing bound, is configured by the environment — and +the reader it gets keeps that configuration for its lifetime, because a reader +is bound when it is opened and not per read. + +A credential provider is the context's to carry when authentication arrives. +Nothing here carries one yet. ## 7. Thread safety diff --git a/docs/design/DESIGN_POLICY.md b/docs/design/DESIGN_POLICY.md index 05ab657..94da5b6 100644 --- a/docs/design/DESIGN_POLICY.md +++ b/docs/design/DESIGN_POLICY.md @@ -1,6 +1,6 @@ # Development Policy -Last updated: 2026-09-04 +Last updated: 2026-09-11 This document is the standing development policy for `usd-http-resolver`. The roadmap and architecture documents refine it; they do not override it. @@ -73,9 +73,11 @@ The read contract, the local backend, the shared boundary suite, the hostile-server corpus, the HTTP backend, the `ArResolver` bundle, the block cache, identity exposed to consumers, the on-disk cache under it, and the packaged product that composes the bundle as a runtime component are implemented -and passing, and released through `v0.5.0`. What is not is the first consumer -integration, the configuration and authentication seams, adaptive read-ahead, -and every transport after HTTP. The contracts under +and passing, and released through `v0.5.0`. On `main` since then: the bounds +§10 names — the response header block and the destination policy — and +per-stage configuration through `ArResolverContext`. What is not is the first +consumer integration, the authentication seam, adaptive read-ahead, and every +transport after HTTP. The contracts under [architecture/](../architecture/) were written before their implementation, which is deliberate: the boundary is the product, and it is cheaper to fix here than in five consumers — and every one of those implementations has since landed @@ -446,14 +448,19 @@ because a redirect target is parsed by the same parser as an original identifier and that parser accepts two schemes. A `Location` naming `file:` is an unusable location, not a followed one. -Two things are named here as scope rather than as shipped properties. The -destination policy — whether loopback and private-network addresses are -reachable — does not exist, and its difficulty is that the hostile-server corpus -*is* loopback, so the setting has to distinguish a fixture from a deployment -rather than forbid one to protect the other. Nor is the response header block -separately bounded; today the caller's buffer bounds the body and nothing bounds -what precedes it. Both land with the configuration surface, because a policy with -no way to state it is a default nobody can override. +The response header block is bounded too, at 64 KiB per exchange, which with the +caller's buffer bounding the body is the whole of §10.1's "total response size". + +And the destination policy exists: `USD_HTTP_RESOLVER_DESTINATIONS`, a set of +address classes, judged against the address a connection is about to be made to +and against any literal in the URL at every hop +([CONFIGURATION.md](../reference/CONFIGURATION.md) §2.1). Its default is +`public,private,loopback`, which is how the fixture and the deployment are told +apart without forbidding either: the corpus is loopback and runs under the +default, intranet hosts are private and are what `http` is registered for, and +the two classes refused — link-local, and the well-known instance-metadata +endpoints wherever they sit — are ones nothing legitimate serves USD from. A deployment that wants a +narrower reach states it, and a refusal is `AccessDenied` naming the class. ## 11. Testing @@ -666,13 +673,10 @@ one integration that decides whether the abstraction is real. number, which prices a round trip at nearly zero and therefore cannot price the trade this architecture makes. This is not a separate task from 1 so much as the reason 1 is first. -3. **The configuration surface and the network policy that needs it**, per §10.2 - and [CONFIGURATION.md](../reference/CONFIGURATION.md): the transport bounds - resolved from `ArResolverContext` as well as the environment, and the scheme - and destination policy stated somewhere a host can override. -4. **The authentication interception point**, per §4.3 — the seam, and no - provider. -5. **Adaptive read-ahead**, cache level 3 in §5, *after* 2 and not before it. +3. **The authentication interception point**, per §4.3 — the seam, and no + provider. It has somewhere to live now: a credential provider is the + context's to carry, and the context exists. +4. **Adaptive read-ahead**, cache level 3 in §5, *after* 2 and not before it. Deliberately not on this list, and each for a stated reason: freezing the internal API before the consumer integration has argued with it (§3.3); @@ -681,6 +685,19 @@ measured (§15); and any second transport before the first has a consumer (§4.4 Done and no longer pending: +- The configuration surface and the network policy that needed it. The + destination policy of §10.2 is `USD_HTTP_RESOLVER_DESTINATIONS`, judged at + connect time, before each request as the client will send the host, and at + every hop, with a default that refuses link-local and the instance-metadata + endpoints and nothing else; the response header block of §10.1 is bounded; and the + transport bounds and the policy are a stage's through `ArResolverContext`, + with the environment as the default a context overrides + ([CONFIGURATION.md](../reference/CONFIGURATION.md) §2.1 and §4). Two things + the work had to decide that no document had said: every identifier this + resolver owns is context-dependent, because OpenUSD's layer registry + otherwise hands a loaded layer to a stage whose context would refuse it; and + a reader retained under one configuration is never handed to a caller under + another. - The packaged, composable product. `v0.5.0` publishes the workspace as an aggregate product with a component-owned acceptance probe that runs from the installed artifact rather than from a producer build directory, which is the diff --git a/docs/reference/CAPABILITY_MATRIX.md b/docs/reference/CAPABILITY_MATRIX.md index a3a55a3..3565bac 100644 --- a/docs/reference/CAPABILITY_MATRIX.md +++ b/docs/reference/CAPABILITY_MATRIX.md @@ -4,7 +4,7 @@ This document describes what the current tree implements. It is not a plan. Intent lives in the [roadmap](../roadmap/README.md); contracts live in [architecture/](../architecture/). -Last updated: 2026-09-04, against `main` at `v0.5.0`. +Last updated: 2026-09-12, against `main` after `v0.5.0`. ## Summary @@ -32,7 +32,7 @@ byte-equivalent to the local backend over every fixture size and 10,000 generated cases. `tests/fixture-server` is no longer a passing oracle waiting for a subject. -`tests/corpus` is the subject: every one of the 18 named behaviors is projected +`tests/corpus` is the subject: every one of the 19 named behaviors is projected onto a `StatusCode`, and the coverage is asserted at runtime rather than claimed. Neither side knows the other — nothing in the fixture server has heard of `StatusCode`, and nothing in the backend has heard of `Behavior` — so a @@ -58,7 +58,7 @@ byte-equivalent to the reader underneath. The resolver takes it. Every asset the bundle opens is decorated and bound into the process store, and the four cache variables in -[CONFIGURATION.md](CONFIGURATION.md) are read at construction. +[CONFIGURATION.md](CONFIGURATION.md) are read at the resolver's first use. Identity now leaves the process, and so do the bytes. `GetAssetInfo` publishes the resolved identifier, the size, an opaque validation token, and a stability @@ -122,8 +122,10 @@ not planned explicitly out of scope | Resume of a short transfer | implemented | The remainder is re-requested from where it stopped, bounded by the same budget; past it, `InvalidResponse` | | Range unsupported → hard error | implemented | `RangeNotSupported`, at open when `Accept-Ranges` is absent and at the first read when it was advertised and then ignored. No whole-asset fallback, per [ADR-0002](../adr/0002-range-unsupported-policy.md) | | Response body bounded by the request | implemented | The caller's buffer is the bound. A `200` answering a 64 KiB range request moves 64 KiB and is cut off | -| Scheme allowlist, at the first hop and at every redirect | implemented | `http` and `https` only. A redirect target is parsed by the same parser as an original identifier, so a `Location` naming `file:` or `s3:` is an unusable location rather than a followed one | -| Response header-block or total-response bound | not implemented | The caller's buffer bounds the body; nothing bounds what precedes it. §10.1 of the [design policy](../design/DESIGN_POLICY.md) | +| Scheme allowlist, at the first hop and at every redirect | implemented | `http` and `https` only. A redirect target is parsed by the same parser as an original identifier, so a `Location` naming `file:` or `s3:` is an unusable location rather than a followed one — and never requested. Asserted per scheme in `usdAssetHttp_protocol`, because the property is a consequence of the parser and nothing else would notice it widening | +| Per-stage configuration through `ArResolverContext` | implemented | `CreateContextFromString("http"` or `"https", "NAME=value; ...")`, with the environment's names. Eight variables per stage — the deadlines, retries, redirects, destinations, and the two coalescing limits — and the four the process shares stay the environment's. Values are kept as the parser read them, so contexts that say the same thing are equal. Every identifier is context-dependent, a retained open is handed only to a caller with the same transport options, a scoped cache is keyed by configuration, and asset info is not answered across a policy — so a stage whose context refuses a destination cannot reach it through another stage's layer, reader, scope, or identity. The resolver is constructed in every process that opens a stage and does nothing until first used, so a local-only host is untouched. Printable from Python as its canonical string. [CONFIGURATION.md](CONFIGURATION.md) §4 | +| Destination policy: loopback, private, link-local, and metadata reach | implemented | `USD_HTTP_RESOLVER_DESTINATIONS`, default `public,private,loopback` — link-local and the well-known instance-metadata endpoints refused, the latter by value wherever they sit (`fd00:ec2::254` is unique-local, `100.100.100.200` is shared address space). Judged three times: at connect time against the address a name resolved to; before each request against the host as libcurl will send it, so decimal, octal, hexadecimal, and percent-encoded spellings are judged as the address they normalize to, through a proxy as well as without one; and at every redirect hop against a canonical literal. IPv4-mapped, -compatible, and NAT64 addresses are the class of the IPv4 address they carry. A refusal is `AccessDenied` naming the class, with no request sent. Asserted as a table in `usdAssetHttp_destination`, over a scripted transport in `usdAssetHttp_protocol`, and over a real socket — through `localhost`, `127.1`, and a proxy — in `usdAssetHttp_destination_policy`. [CONFIGURATION.md](CONFIGURATION.md) §2.1 | +| Response header-block and total-response bound | implemented | 64 KiB of header per exchange, summed across interim responses, and the caller's buffer for the body. A response abandoned at the header bound is `InvalidResponse` whatever its status, and is not retried. The corpus's `OversizedHeaders` row pads a correct response with a megabyte of ordinary fields, which libcurl alone accepts. §10.1 of the [design policy](../design/DESIGN_POLICY.md) | | Loopback / private-network destination policy | not implemented | §10.2 of the design policy. It lands with the configuration surface, and it has to distinguish a fixture server from a deployment, since the hostile corpus is itself loopback | | Content encoding refused, not decoded | implemented | `Accept-Encoding: identity` on every request. A compressed range response would make the byte accounting describe the wire rather than the asset, and a decompressing client is a client with an unbounded output for a bounded input | | Bounded whole-asset fallback | deferred | Its own residency model; needs a new ADR and a demonstrated need | @@ -206,7 +208,7 @@ not planned explicitly out of scope | CI: sanitizer cells | implemented | `core-ci.yml`, `sanitizers` job, Linux. A sanitizer is a property of the compiler, and MSVC implements only `address` — unverified at that | | CI: generated OpenStrata support matrix | implemented | `openstrata.ci.yaml`, six `pull_request` cells, and `ost-source-ci.yml` generated from it: the dependency graph on Linux and Windows, `ost build` + `ost test` on Linux and macOS arm64, and the bundle through the pyramid to L1 on the same two. The L1 cap is [report 02](../reports/ost/02-2026-08-18-resolver-bundle-under-the-pyramid.md) §2 | | CI: plugin lane on Windows | implemented | `plugin-windows-ci.yml`, hand-authored: no generated cell can hand CMake the vcpkg prefix libcurl needs. It reads its pins back out of `openstrata.ci.yaml` and asserts `httpResolver_stage` by name from the `ctest` log; see [report 03](../reports/ost/03-2026-08-18-a-support-matrix-with-one-hand-authored-lane.md) | -| Hostile-server fixture corpus | implemented | `tests/fixture-server`; 18 behaviors covering all nine conditions in §11.2 of the design policy. Additional to the boundary suite, not a substitute | +| Hostile-server fixture corpus | implemented | `tests/fixture-server`; 19 behaviors covering all nine conditions in §11.2 of the design policy, and the header-block bound of §10.1. Additional to the boundary suite, not a substitute | | Fixture-server self-test | implemented | Asserts over a raw socket that each behavior puts on the wire what its name claims, with a client that shares no HTTP code with the server | | Corpus projection onto the typed vocabulary | implemented | `tests/corpus`; every behavior maps to a `StatusCode`, and coverage against `AllBehaviors()` is asserted at runtime rather than claimed | | Boundary suite against the HTTP backend | implemented | `tests/boundary/backends/boundary_http_main.cpp`, one row, running the suite unchanged over a real server and a real socket | diff --git a/docs/reference/CONFIGURATION.md b/docs/reference/CONFIGURATION.md index dbf30d7..5616d2b 100644 --- a/docs/reference/CONFIGURATION.md +++ b/docs/reference/CONFIGURATION.md @@ -1,9 +1,11 @@ # Configuration This document defines the configuration surface. The five transport bounds are -implemented as of `v0.2.0` and the four cache variables as of `v0.3.0`; all nine -are read by `plugins/http-resolver`, once, when the resolver is constructed. The -`ArResolverContext` form arrives in `v0.7.0`. +implemented as of `v0.2.0`, the four cache variables as of `v0.3.0`, the two +persistence variables as of `v0.4.0`, and the destination policy as of `v0.7.0`; +all twelve are read by `plugins/http-resolver`, once, when the resolver is first +used. Eight of them can also be set per stage, through an `ArResolverContext`, +as of `v0.7.0` (§4). ## 1. Two mechanisms, in order @@ -15,8 +17,8 @@ ArResolverContext per stage, v0.7.0 Environment variables are a bootstrap, not the destination. A host that opens two stages against two servers with two credentials cannot be served by a process-global, and the moment authentication is real, the context form is the -only correct one. Both will coexist: environment values become the defaults -that a bound context overrides. +only correct one. Both coexist: environment values are the defaults a bound +context overrides. ## 2. Variables @@ -37,6 +39,7 @@ that the defaults are wrong. | `USD_HTTP_RESOLVER_TOTAL_TIMEOUT_MS` | `300000` | Total per-request deadline, headers and body | | `USD_HTTP_RESOLVER_MAX_RETRIES` | `2` | Retry ceiling for retryable failures; `0` disables retry | | `USD_HTTP_RESOLVER_MAX_REDIRECTS` | `5` | Redirect chain ceiling; `0` refuses to follow any | +| `USD_HTTP_RESOLVER_DESTINATIONS` | `public,private,loopback` | Address classes a connection may reach, as a comma-separated set of `public`, `private`, `loopback`, `link-local`, and `metadata`; see §2.1 | | `USD_HTTP_RESOLVER_METRICS_DUMP` | unset | When set, dumps the metrics aggregate at process exit | An unparseable value is a diagnostic at first use, not a silent fallback to the @@ -45,8 +48,8 @@ fails. The three deadlines are the ones the backend separates so that `Timeout` (`HTTP006`) can name which one elapsed, which DIAGNOSTICS.md requires of it. The -value read is the one the resolver was constructed with: these are process-wide, -and per-stage values are what `ArResolverContext` is for in `v0.7.0`. +value read is the one the resolver read when it was first used, unless the +stage's context sets another (§4). The four cache defaults are measured constants and the measurement that chose them is [BLOCK_POLICY.md](BLOCK_POLICY.md). Two of them are labelled there as @@ -61,7 +64,7 @@ discovering: §10 of the [design policy](../design/DESIGN_POLICY.md) exists to forbid. For the counters it means "do not retry" and "do not follow", which are both legitimate things to ask for. -- **One bad value does not discard the other eight.** Each variable is parsed +- **One bad value does not discard the others.** Each variable is parsed independently, so a configuration that is mostly right stays mostly in force, and the warning names the variable, its value, and what was wrong with it. - **An adjustment is reported, not only a rejection.** A block size that is not @@ -73,6 +76,91 @@ discovering: rather than clamped: a budget below one block means the caller wanted no cache, and there is no variable for that. +### 2.1 The destination policy + +§10.2 of the [design policy](../design/DESIGN_POLICY.md): an identifier can +arrive from a layer the user did not author, so the reach it has is bounded by +declared policy rather than by whatever the host's network happens to allow. +`USD_HTTP_RESOLVER_DESTINATIONS` is the declaration. + +| Class | Addresses | +| --- | --- | +| `metadata` | the well-known instance-metadata and credential endpoints, by value: `169.254.169.254`, `169.254.170.2`, `169.254.170.23`, `169.254.0.23`, `100.100.100.200`, `168.63.129.16`, `fd00:ec2::254`, and `fd00:ec2::23` | +| `loopback` | `127.0.0.0/8` and `::1`; and `0.0.0.0/8` and `::`, because a connect to them reaches this host | +| `link-local` | `169.254.0.0/16` and `fe80::/10`, apart from the metadata addresses in them | +| `private` | `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, the shared address space `100.64.0.0/10`, `fc00::/7`, and the deprecated `fec0::/10`, apart from the metadata addresses in them | +| `public` | everything else | + +`metadata` is checked first and is a class of its own, because no range +contains it. An address range is a statement about routing, and the property a +request-forgery policy cares about — that answering this request hands a +stranger the instance's credentials — follows the provider rather than the +range. `169.254.169.254` is where most clouds put it; AWS's IPv6 endpoint is +unique-local, Alibaba's is in the shared address space, and Azure's WireServer +is public. A policy that refused link-local and called the metadata endpoint +refused would be true of one address out of eight. + +An IPv6 address that carries an IPv4 one — mapped, compatible, or behind the +NAT64 well-known prefix — is the class of the IPv4 address, because that is +where the connection ends up. `[::ffff:169.254.169.254]` is `metadata`. + +**The default is `public,private,loopback`, and it is a middle rather than +either end.** Loopback and private networks stay reachable because `http` is +registered for local fixture servers and intranet hosts +([RESOLVER.md](../architecture/RESOLVER.md) §1), and a default that broke the +uses the scheme exists for would be a default every deployment overrode. +Link-local and metadata are refused because nothing legitimate serves USD from +either, and a metadata address is where a cloud instance hands out its +credentials. Permitting `link-local` does not permit `metadata`; that is never +a side effect. A deployment that wants less reach says so: `public` alone for a +render farm that must never reach its own network from a layer it did not +author, `private` alone for one that must never leave it — for the whole process +in the environment, or for one stage in its context (§4). + +The policy is judged three times, and none of the three is redundant: + +- **At connect time**, against the numeric address the transport is about to + connect to, after the name was resolved and before a socket exists. This is + the check that makes the policy hold for a name at all: for a name that + resolves to a refused address, and for a name whose answer changed between + lookups. A name with several addresses is refused only when every one of them + is; a refused IPv6 address followed by a permitted IPv4 one that did not + answer is a network failure, not a refusal. +- **Before each request, as the client will send the host.** The host is taken + from libcurl's own URL parser — the one the transfer will use on the same + string — so that `2852039166`, `0xa9fea9fe`, `169.254.43518`, and + `%31%36%39.254.169.254` are each judged as the `169.254.169.254` libcurl + normalizes them to. This is the check that holds through a proxy, where the + address this process connects to is the proxy's and the destination goes out + as text for the proxy to resolve. A non-ASCII host is judged in the ASCII form + the client would put on the wire, where the libcurl in use can produce one. +- **At every redirect hop, before any transport sees it**, against a literal in + the URL in its canonical spelling. This is the one that does not depend on + which client is underneath, and it is what makes a redirect to + `http://169.254.169.254/` a refusal of a string rather than of a connection. + +A refusal is `AccessDenied` (`HTTP002`), naming the class, and no request is +sent. The code is the one a `403` gets because a caller does the same thing +about both — nothing, and tell whoever owns the configuration — and the class is +named so that the person told goes to their own policy rather than to the +origin's permissions. + +A value is a set, not a level, because the classes are not an order. Whitespace +around a name is tolerated — spaces, tabs, and line breaks, so that a list +broken across lines, or an environment file saved with CRLF endings, is the list +that was written. An unknown name refuses the whole value and keeps the default, +because a policy that silently dropped the word it did not recognize is a +different policy from the one written. + +Two interactions are named rather than solved. Through a proxy, a *name* is +resolved by the proxy, so the policy cannot see where it leads, and the +connect-time check judges the proxy's own address instead: a proxy on loopback +needs `loopback` in the set. And a libcurl built without IDN support sends a +non-ASCII host as written; if a proxy then maps it — look-alike digits folded to +ASCII ones — into an address, that address is the proxy's to police. A libcurl +with IDN support converts the host first, and the converted form is what is +judged. + ## 3. What is not configurable Some things are deliberately absent, because making them configurable would @@ -112,11 +200,86 @@ turn a correctness property into a deployment mistake: environment or the context, and no credential is ever named in a variable this resolver defines, printed, or persisted. -## 4. Precedence +## 4. Precedence, and the context form ```text ArResolverContext > environment variable > built-in default ``` -Resolved at bind time, not per request. A per-request read of a global is both -slow and unpredictable when a host mutates the environment mid-session. +The environment is read once, at the resolver's first use, and kept. First use +rather than construction, and the difference is a promise kept: because this +resolver implements contexts, OpenUSD constructs it in every process that opens +any stage, local ones included, and a constructor that read the environment +would configure the process stores, create the persistent cache directory, and +warn about settings for a host that never names an `http` URL. So construction +does nothing, and the first resolve, open, asset-info query, or context creation +does the rest. A context is resolved against that snapshot rather than against +`getenv`, so a host that mutates its environment mid-session does not change +what a stage it opened earlier is configured by — and a reader keeps the options +it was opened with, so a stage's configuration is bound when its assets are +opened, not per request. + +A context is created from a string, through OpenUSD, with either scheme name — +one resolver type serves both: + +```python +ctx = Ar.GetResolver().CreateContextFromString( + "https", + "USD_HTTP_RESOLVER_DESTINATIONS=public; USD_HTTP_RESOLVER_TOTAL_TIMEOUT_MS=60000") +stage = Usd.Stage.Open("https://example.org/scenes/main.usda", ctx) +``` + +The string is `NAME=value` entries separated by `;`. The names are the +environment's, spelled exactly as §2 spells them, and each value goes through +the parser the environment's would — over the environment it will be layered +on, so that a coalescing gap is judged against the request ceiling that will +actually apply — and is refused or adjusted for the same reasons, reported the +same way. Whitespace around an entry, a name, or a value is tolerated, and so is +an empty entry — a trailing `;` is what concatenation leaves behind. A name set +twice considers only the last value, as an environment assignment would, and +says so; if that last value is refused, that is said too, and the stage takes +the environment's value. + +Eight variables may be set in a context, and four may not: + +| May be set per stage | Environment only | +| --- | --- | +| `CONNECT_TIMEOUT_MS`, `READ_TIMEOUT_MS`, `TOTAL_TIMEOUT_MS` | `BLOCK_SIZE` | +| `MAX_RETRIES`, `MAX_REDIRECTS` | `CACHE_BUDGET` | +| `DESTINATIONS` | `PERSISTENT_CACHE_DIR` | +| `COALESCE_GAP`, `MAX_REQUEST_BYTES` | `PERSISTENT_CACHE_BUDGET` | + +The left column is what binds a reader or its cache wrap, and is therefore a +property of whoever opened the asset. The right column is what the process +shares: one block store with one budget (CACHE.md §7), one persistent +directory, and a store whose stripes are sized for one block size — eight blocks +to a stripe, so a stage that asked for blocks larger than a stripe would fetch +each one and watch it evicted on arrival. A context that names one of those is +told, when it is created, that the value is read from the environment. + +Problems are reported once, when the context is created, and never at bind +time: a context is created once and bound on every thread that composes the +stage, and a warning per bind would be one typo rendered once per prim. What a +context carries is what it admitted, *as the parser read it* — `60000` for +`060000`, `public,private` for `private, public` — so two contexts that say the +same thing compare and hash equal however they were spelled, and are one context +to every table OpenUSD keys on one. A context whose every entry was refused +configures a stage exactly as no context does. + +In Python, a context reads back as its canonical string — +`Ar.ResolverContext('USD_HTTP_RESOLVER_DESTINATIONS=public')` — which is what +`Usd.Stage.__repr__` prints for a stage opened with one. + +Four properties follow for the destination policy in particular, and each is +asserted in `httpResolver_stage` ([RESOLVER.md](../architecture/RESOLVER.md) §6). +A stage whose context refuses a destination cannot reach it through a layer +another stage has already loaded: every identifier this resolver owns is +context-dependent, which is what makes OpenUSD's layer registry look the layer +up by the path `Resolve` returned rather than by its name. It cannot reach it +through a reader another stage's resolve left behind: a retained open is handed +only to a caller that would have opened it under the same transport options. It +cannot reach it through an `ArResolverScopedCache` that spans both stages: this +resolver keeps the scope's cache itself, keyed by configuration, rather than +letting OpenUSD key it by path. And it is not told the asset's size and token by +asset info: an identity is answered from memory only for a caller whose policy +could have reached the asset itself. diff --git a/docs/roadmap/README.md b/docs/roadmap/README.md index ebe3d52..57231c7 100644 --- a/docs/roadmap/README.md +++ b/docs/roadmap/README.md @@ -315,6 +315,12 @@ See [consumer integration](consumer-integration.md). ### `v0.7.0` — configuration, network policy, and the auth seam +Status: in progress on `main`, ahead of `v0.6.0` for the reason `v0.5.0` was — +it is code, and the consumer integration is waiting on a fixture and a host. The +configuration surface, the destination policy, and the header-block bound have +landed; the request interception point and formation composition have not. See +[implementation status](implementation-status.md). + Scope: the configuration surface (block size, budgets, timeouts, retry policy) resolved from `ArResolverContext` as well as the environment, so a bound is a property of a stage rather than of a process; the declared network policy of @@ -358,7 +364,7 @@ content, per §3.5 of the design policy. | 4 | Identity exposure, persistent cache, stability metadata | Complete for `v0.4.0` | Everything that makes identity outlive a reader — and, with the disk tier, outlive the process | | 5 | Packaging: aggregate product and artifact-owned acceptance | Complete for `v0.5.0` | Behavior unchanged; the probe runs from the installed artifact, not from a build tree | | 6 | First consumer integration and amplification baseline | Planned for `v0.6.0` | The abstraction's real test, and the first measurement over distance | -| 7 | Configuration, network policy, auth seam, formation composition | Planned for `v0.7.0` | Seams and policy only, no providers | +| 7 | Configuration, network policy, auth seam, formation composition | In progress for `v0.7.0` | Configuration surface, destination policy, and header-block bound landed; the auth seam and formation composition have not. Seams and policy only, no providers | | 8 | Adaptive read-ahead | Planned for `v0.8.0` | Blocked on phase 6, not on phase 7: it needs a latency number, not a config surface | | 9 | Second consumer (`usd-3dgs-plugins`) | Deferred | Camera-driven streaming; validates generality | | 10 | Package composition: ranges inside a remote package | Deferred | `https://host/model.usdz[texture.png]` without downloading the package; needs the package resolver's cooperation and a new ADR | @@ -381,7 +387,7 @@ content, per §3.5 of the design policy. | W8 | Identity exposure, persistence, cross-stage reuse rules | 4 | Done — `GetAssetInfo`, `DiskBlockStore`, and the one rule that governs both | | W9 | Aggregate product and artifact-owned acceptance probe | 5 | Done — `share/usd-http-resolver/probes/packaged_probe.py`, run against the installed artifact | | W10 | Consumer integration and amplification baselines | 6 | Planned | -| W11 | Configuration, network policy, auth seam, formation composition | 7 | Planned | +| W11 | Configuration, network policy, auth seam, formation composition | 7 | In progress — `ArResolverContext` configuration, the destination policy, and the header-block bound are done | | W12 | Adaptive read-ahead | 8 | Planned — gated on W10's latency numbers | | W13 | Fuzzing the parsers, per §11.6 of the design policy | Parallel | Planned — CI work; no release gate | | W14 | Async, prefetch, Wasm research | Parallel | No release gate | diff --git a/docs/roadmap/implementation-status.md b/docs/roadmap/implementation-status.md index ba9d4fa..e03c678 100644 --- a/docs/roadmap/implementation-status.md +++ b/docs/roadmap/implementation-status.md @@ -4,7 +4,7 @@ Task-level tracking of what is done, in progress, and outstanding. Behavior belongs in [capability matrix](../reference/CAPABILITY_MATRIX.md); this file tracks work. -Last updated: 2026-09-04. +Last updated: 2026-09-11. Phases 0 and 1 are complete and `v0.1.0` is released. The read contract, the local backend, and the shared boundary suite are in the tree and passing; the @@ -69,6 +69,16 @@ The transport, validator, cache, and public C++ behavior are unchanged from version number imply otherwise. The consumer integration moved to `v0.6.0` intact; nothing in its scope was cut. +Phase 7 is in progress, ahead of phase 6 for the reason phase 5 was: it is +code, and the consumer integration is waiting on a fixture and a host. Three of +its rows have landed. The response header block is bounded, which the corpus +proved was not optional; the destination policy of §10.2 exists, with a default +that refuses link-local and the instance-metadata endpoints and nothing else; +and the transport bounds and that +policy are a stage's, through `ArResolverContext`, with the environment as the +default a context overrides. What remains is the request interception point for +authentication and formation composition. + **`v0.2.0` is released.** The gate is walked and [its record](../releases/v0.2.0.md) is written. Gates 4 and 6 bound for the first time and both pass; gate 9 turned out not to bind, because it binds a release @@ -220,10 +230,10 @@ therefore ship unexercised. | Task | Status | | --- | --- | -| Configuration surface resolved from `ArResolverContext`, not only the environment | Outstanding — the environment form ships since `v0.2.0` as a process-wide bootstrap | -| Declared scheme allowlist, re-applied at every redirect hop | Done, as a consequence rather than as a feature — a redirect target goes through the same parser as an original identifier, and that parser accepts `http` and `https` only, so a `Location` naming `file:` is an unusable location. Worth an explicit case, since nothing today would notice if the parser widened | -| Loopback and private-network destination policy, with a documented default | Outstanding — §10.2 of the [design policy](../design/DESIGN_POLICY.md). It has to distinguish a fixture server from a deployment, since the corpus depends on loopback | -| Response header-block and total-response bounds | Outstanding — the caller's buffer bounds the body today; the header block is not separately bounded | +| Configuration surface resolved from `ArResolverContext`, not only the environment | Done — `CreateContextFromString` with the environment's own names, eight variables per stage and the four the process shares refused with a reason. Two things had to be decided that the contract had not said. Every identifier is declared context-dependent, because OpenUSD's layer registry otherwise finds a loaded layer by name whatever `Resolve` said, and one stage's destination policy could be walked past by opening the URL in another stage first. And the retained opens are keyed by transport options as well as identifier, because a reader keeps the options it was opened with. Both were checked by removing them, and each removal fails exactly the case written for it | +| Declared scheme allowlist, re-applied at every redirect hop | Done, as a consequence rather than as a feature — a redirect target goes through the same parser as an original identifier, and that parser accepts `http` and `https` only, so a `Location` naming `file:` is an unusable location. Now with the explicit case it was owed: `usdAssetHttp_protocol` redirects to seven schemes and spellings that must be refused, asserts each is never requested, and asserts the two scheme-less forms that stay inside the allowlist are still followed | +| Loopback and private-network destination policy, with a documented default | Done — `USD_HTTP_RESOLVER_DESTINATIONS`, default `public,private,loopback`: the corpus is loopback and runs under the default, intranet hosts stay reachable, and link-local and the instance-metadata endpoints are refused — the latter a class of its own, by value, because AWS's IPv6 endpoint is unique-local and Alibaba's is in the shared address space. Judged at connect time against the resolved address, before each request against the host as libcurl will send it, and at every hop against a canonical literal. Each check was removed in turn: without the connect-time one `localhost` walks past a policy that refuses loopback, and without the client-side one `2852039166` reaches a proxy that forwards it to 169.254.169.254. [CONFIGURATION.md](../reference/CONFIGURATION.md) §2.1 | +| Response header-block and total-response bounds | Done — 64 KiB of header per exchange, counted in the transport before a line is stored, and the caller's buffer for the body. A response abandoned at the bound is refused whole at any status, and never retried. The corpus gained a row for it, `OversizedHeaders`, and the row found that the bound was not optional: without it, libcurl 8.7.1 opened an asset behind a megabyte of header fields | | Request interception point for authentication | Outstanding — the seam, no provider | | OpenStrata formation composition and pinned artifacts | Outstanding | | Reproducible binary output | Outstanding — measured at the `v0.2.0` gate: two builds agree on 24 of 28 installed files, and the four that differ differ only in embedded build timestamps. Closing it is a link flag and belongs with the packaging work | @@ -309,6 +319,43 @@ dependency, resolved as libcurl in measured exactly and whose denominator is zero. `v0.6.0` is where distance arrives, and it is why read-ahead is scheduled behind it rather than beside the cache work it belongs to. +3. The rest of phase 7: the request interception point for authentication — a + seam with no provider, which now has somewhere to be supplied from, since a + credential provider is the context's to carry — and formation composition. + +Done, and no longer next: the configuration surface and the network policy. The +contract had said what they were for and not two things they turned out to +need. OpenUSD's layer registry finds an already-loaded layer by identifier for +any path that is not context-dependent, whatever `Resolve` has just answered, +so a per-stage destination policy is only a policy if every identifier this +resolver owns is declared context-dependent — without that, opening a URL in +one stage hands the layer to every other stage whatever its context says. And +the retained opens of RESOLVER.md §2.3 had been keyed by identifier alone, +which was correct while one process had one configuration and stopped being +correct the moment it had several: a reader keeps the options it was opened +with. Both were found by writing the case first and then removing the fix, and +each removal fails exactly its own case. + +A review of that work before it merged found more, and all of it was the same +shape: a guarantee that held in the case written for it and not in the one next +to it. The destination policy judged canonical literals before a request and +addresses at connect time, and through a proxy neither sees `2852039166` — the +connect is the proxy's, and libcurl normalizes the spelling to +`169.254.169.254` before the proxy reads it — so the host is now judged as +libcurl's own URL parser will send it. Refusing link-local did not refuse the +metadata endpoints that are not link-local, so those became a class of their +own. OpenUSD caches `Resolve` by path inside a scoped cache for a resolver that +does not implement scoped caches, which walked past a per-stage policy the +same way the layer registry had. And implementing contexts turned out to mean +being constructed in every process that opens any stage, so a constructor that +configured the persistent tier created its directory for hosts that never named +a remote URL. Each fix has a case that fails without it. + +The header-block bound found something of its own. The corpus row written for +it — a correct response padded with a megabyte of ordinary fields — opened an +asset under libcurl 8.7.1 with the bound removed, so the bound was not the +belt-and-braces it might have been taken for: the library's ceilings are on a +single line, and a block of kilobyte lines never reaches them. Done, and no longer next: the packaged product. What it settled is a question that had been answerable only in the affirmative-by-assumption until then — diff --git a/libs/usd-asset-http/CMakeLists.txt b/libs/usd-asset-http/CMakeLists.txt index c9e3f2c..7e6e07c 100644 --- a/libs/usd-asset-http/CMakeLists.txt +++ b/libs/usd-asset-http/CMakeLists.txt @@ -61,6 +61,7 @@ find_package(CURL REQUIRED) add_library(usdAssetHttp STATIC src/CurlTransport.cpp + src/Destination.cpp src/Framing.cpp src/HttpAssetReader.cpp src/Transport.cpp diff --git a/libs/usd-asset-http/README.md b/libs/usd-asset-http/README.md index 8b5ea29..a1f3962 100644 --- a/libs/usd-asset-http/README.md +++ b/libs/usd-asset-http/README.md @@ -62,6 +62,7 @@ struct HttpOptions { // all of it bounded int transferTimeoutMs = 300000; int maxRedirects = 5; int maxAttempts = 3; // requests per logical operation, retries included + DestinationPolicy destinations; // public, private, loopback; not link-local or metadata std::string userAgent; // empty takes the default }; @@ -79,9 +80,11 @@ OpenResult OpenAsset(const std::string& url, const HttpOptions& options); ``` `HttpOptions` is a parameter rather than a set of constants so that a test can -make a deadline elapse in milliseconds. It is not yet resolved from the -environment or from an `ArResolverContext`; that is the configuration surface in -`v0.6.0`. +make a deadline elapse in milliseconds. Nothing here reads the environment: +resolving these from a deployment's settings — the environment, or a stage's +`ArResolverContext` — is the resolver's configuration surface +([CONFIGURATION.md](../../docs/reference/CONFIGURATION.md)), and this module is +the mechanism underneath it. ## Dependencies @@ -102,6 +105,7 @@ installed header. ```text Open(url) -> parse the URL absolute http/https, or InvalidArgument + -> destination policy, every hop a refused literal is AccessDenied, unsent -> HEAD, following bounded redirects one metadata round trip, no content -> size from Content-Length absent is a refusal, not a guess -> range support from Accept-Ranges absent is RangeNotSupported, terminal @@ -131,8 +135,19 @@ HTTP/1.1 over libcurl, with almost every convenience turned off: | Raw response status | ADR-0002 makes a `200` answering a `Range` request `RangeNotSupported`, and a client that normalizes a partial response into "here are your bytes" cannot implement that | | `Accept-Encoding: identity` | A compressed range response would make the byte accounting describe the wire rather than the asset | | Bounded write callback | The caller's buffer is the bound. A server answering a 64 KiB range request with a 10 GB body moves 64 KiB and is then cut off | +| `curl_url` host, before each request | The destination policy judged against the host as libcurl will send it: its own URL parser normalizes `2852039166`, `0xa9fea9fe`, and `%31%36%39.254.169.254` to `169.254.169.254` before a proxy ever sees them, so that is what is judged | +| `CURLOPT_OPENSOCKETFUNCTION` | The destination policy's connect-time half: each address libcurl is about to connect to is classified and refused or admitted before a socket exists, which is the one point where a resolved name's address is known and not yet reached. Sockets it creates are close-on-exec, so a host that forks does not hand them to its children | +| `CURLOPT_PROTOCOLS_STR` `http,https` | The scheme allowlist a second time. The parser enforces it; this makes a parser that ever widened widen into a refusal | +| Bounded header callback | 64 KiB per exchange, interim responses included (`kMaxResponseHeaderBytes`). Without it the header table is a buffer whose size the server chooses: libcurl 8.7.1 on its own accepts a megabyte of ordinary header fields, measured against the corpus's `OversizedHeaders` row | | `CURLOPT_NOSIGNAL` | libcurl's alarm-based DNS timeout is not safe to use from a thread | +The two bounded callbacks together are the whole of §10.1's "bound the response +header block and the total response size": a response can deliver at most 64 KiB +of header and at most what the caller's buffer holds of body, whatever it +declares. A response abandoned at the header bound is refused whole — its status +line arrived intact, and an open that read a `Content-Length` out of the prefix +that fit would be acting on a response nobody finished receiving. + No libcurl error string ever reaches a `Status::message`. They embed the effective URL, and a message built from one would undo the credential elision this repository already ships. @@ -206,6 +221,7 @@ response whose weak validator has changed is still positive evidence of | --- | --- | | `404`, `410` | `NotFound` | | `401`, `403` | `AccessDenied` | +| A destination `DestinationPolicy` refuses, literal or resolved | `AccessDenied`, naming the class; no request sent, not retried | | No `Accept-Ranges` at open; `200` answering a `Range` | `RangeNotSupported` | | `Content-Range` that does not cover the request | `InvalidResponse` | | Missing or unparseable `Content-Length` at open | `InvalidResponse` | @@ -216,6 +232,7 @@ response whose weak validator has changed is still positive evidence of | Redirect chain past `maxRedirects`; no `Location`; unusable `Location` | `InvalidResponse` | | `https` → `http` redirect | `InvalidResponse` | | Response that is not HTTP | `InvalidResponse` | +| Header block past 64 KiB, at any status | `InvalidResponse`, naming the bound; not retried, even behind a `503` | | DNS, refusal, TLS handshake, reset connection | `NetworkError` | | `5xx` or `429` after the retry budget | `NetworkError`, with the status attached | | Connect, response, or transfer deadline | `Timeout`, naming which | @@ -337,12 +354,13 @@ The toolchain activates vcpkg's own wrapper, which supplies the release and debug paths explicitly. Naming the triplet matters too: the default is the dynamic one. -Three suites, and they are not interchangeable: +Four suites, and they are not interchangeable: | Suite | Where | Asserts | | --- | --- | --- | -| Module tests | `tests/` | URI arithmetic, framing as a pure function of headers, and the protocol policies over a scripted transport | +| Module tests | `tests/` | URI arithmetic, address classification, framing as a pure function of headers, and the protocol policies over a scripted transport | | Corpus projection | `tests/corpus/` | Which hostile server behavior produces which `StatusCode`, against a real server | +| Destination policy | `tests/corpus/` | The connect-time half of the policy, through a real resolver: a name that resolves to a refused address is refused before a socket exists | | Boundary suite | `tests/boundary/` | The read contract, byte-equivalent to the local backend over an independent oracle | The scripted transport in the module tests is deliberately never used for diff --git a/libs/usd-asset-http/include/usdAssetHttp/HttpAssetReader.h b/libs/usd-asset-http/include/usdAssetHttp/HttpAssetReader.h index f498661..056a9bb 100644 --- a/libs/usd-asset-http/include/usdAssetHttp/HttpAssetReader.h +++ b/libs/usd-asset-http/include/usdAssetHttp/HttpAssetReader.h @@ -40,6 +40,115 @@ namespace usdasset { namespace http { +/// What kind of network an address belongs to, for the destination policy. +/// +/// Five classes, and only the five a decision about request forgery turns on +/// (§10.2 of the design policy). An address is classified by its numeric value +/// and never by a name, so a hostname that resolves to a loopback address is a +/// loopback destination however it is spelled. +/// +/// Metadata the well-known instance-metadata and credential endpoints of +/// the major clouds, wherever they sit: 169.254.169.254, +/// 169.254.170.2, 169.254.170.23, 169.254.0.23, +/// 100.100.100.200, 168.63.129.16, fd00:ec2::254, and +/// fd00:ec2::23. Checked before the ranges below, because two of +/// them are inside the private ones and one is public +/// Loopback 127.0.0.0/8, 0.0.0.0/8, ::1, and the unspecified address ::, +/// which a connect on the common stacks treats as this host +/// LinkLocal 169.254.0.0/16 and fe80::/10, apart from the metadata +/// addresses in them +/// Private 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, the shared address +/// space 100.64.0.0/10, the unique-local fc00::/7, and the +/// deprecated site-local fec0::/10, apart from the metadata +/// addresses in them +/// Public everything else +/// +/// `Metadata` is a class of its own rather than a part of `LinkLocal` because +/// the ranges do not contain it. An address range is a statement about routing, +/// and the property a request-forgery policy cares about -- that answering +/// this request hands a stranger the instance's credentials -- follows the +/// provider rather than the range: AWS's IPv6 endpoint is unique-local, and +/// Alibaba's sits in the shared address space. +/// +/// An IPv6 address that carries an IPv4 one -- mapped (`::ffff:a.b.c.d`), +/// compatible (`::a.b.c.d`), or behind the NAT64 well-known prefix +/// (`64:ff9b::a.b.c.d`) -- is the class of the address it carries, because +/// that is where the connection ends up. +enum class AddressClass { + Public, + Private, + Loopback, + LinkLocal, + Metadata, +}; + +/// The stable lowercase spelling of a class: `public`, `private`, `loopback`, +/// `link-local`, `metadata`. These are also the words the resolver's +/// configuration takes, so that a message and the setting that would change it +/// use one vocabulary. +const char* AddressClassName(AddressClass addressClass) noexcept; + +/// Which classes of address a reader may connect to. +/// +/// §10.2 of the design policy: an identifier can arrive from a layer the user +/// did not author, which makes a resolver a request-forgery primitive unless +/// its reach is bounded by declared policy rather than by whatever the host's +/// network happens to allow. The default is that declaration, and it is a +/// deliberate middle rather than either end: +/// +/// public, private, loopback permitted -- `http` is registered for local +/// fixture servers and intranet hosts +/// (RESOLVER.md §1), and refusing either would +/// break the uses the scheme exists for +/// link-local, metadata refused -- nothing legitimate serves USD from +/// either, and a metadata address is where a +/// cloud instance hands out its credentials +/// +/// A deployment that wants a narrower reach says so; a render farm that must +/// never reach its own intranet from a layer it did not author sets `public` +/// alone. Permitting `link-local` does not permit `metadata`: the second is +/// never a side effect of the first. +/// +/// Judged three times, because each judgement covers what the others cannot. +/// The address a connection is made to is judged at connect time, which is +/// what makes the policy hold for a name that resolves to a refused address. +/// The host the client will actually send is judged before each request, as +/// the client itself parses it -- decimal, octal, and hexadecimal spellings of +/// an address, and percent-encoded ones, normalized the way it will normalize +/// them -- which is what makes the policy hold through a proxy, where the +/// address this process connects to is the proxy's and the destination is the +/// proxy's to resolve. And a literal in the URL is judged at every redirect hop +/// before any transport sees it, so that the rule does not depend on which +/// client is underneath. +struct DestinationPolicy { + bool publicAddresses = true; + bool privateNetworks = true; + bool loopback = true; + bool linkLocal = false; + bool metadata = false; + + bool Permits(AddressClass addressClass) const noexcept; + + /// True when every class `other` permits, this permits too. A + /// destination reached under `other` is reachable under this. + bool Covers(const DestinationPolicy& other) const noexcept { + return (publicAddresses || !other.publicAddresses) && + (privateNetworks || !other.privateNetworks) && + (loopback || !other.loopback) && (linkLocal || !other.linkLocal) && + (metadata || !other.metadata); + } + + bool operator==(const DestinationPolicy& other) const noexcept { + return publicAddresses == other.publicAddresses && + privateNetworks == other.privateNetworks && + loopback == other.loopback && linkLocal == other.linkLocal && + metadata == other.metadata; + } + bool operator!=(const DestinationPolicy& other) const noexcept { + return !(*this == other); + } +}; + /// Transport policy, all of it bounded. /// /// These are the knobs §10 of the design policy requires to exist -- "bound @@ -49,9 +158,10 @@ namespace http { /// a caller that passes nothing gets, and they are the values the release is /// measured with. /// -/// They are not yet resolved from the environment or from an -/// `ArResolverContext`: that is the configuration surface in `v0.6.0` -/// (RESOLVER.md §6). Until then a caller passes them or takes the defaults. +/// Nothing here reads the environment. Resolving these from a deployment's +/// settings is the resolver's configuration surface (CONFIGURATION.md), which +/// is a policy about where values come from; this module is a mechanism, and a +/// caller of it passes values or takes the defaults. struct HttpOptions { /// Establishing a connection. Its own deadline because a connect that /// never completes and a server that never answers are different faults, @@ -77,6 +187,12 @@ struct HttpOptions { /// that is what the network saw (METRICS.md §3). int maxAttempts = 3; + /// Which classes of address this reader may connect to, at the first hop + /// and at every redirect. A refusal is `AccessDenied` and issues no + /// request: it is a configuration decision, which is what `403` is too, + /// and a caller does the same thing about both. + DestinationPolicy destinations; + /// Sent as `User-Agent`. Empty takes the module's default. std::string userAgent; }; diff --git a/libs/usd-asset-http/src/CurlTransport.cpp b/libs/usd-asset-http/src/CurlTransport.cpp index c5feac5..fc5370a 100644 --- a/libs/usd-asset-http/src/CurlTransport.cpp +++ b/libs/usd-asset-http/src/CurlTransport.cpp @@ -17,13 +17,27 @@ #include +#if defined(_WIN32) +// `curl.h` has already included Winsock on Windows, which is where +// `sockaddr_in6` and `socket` live there. `windows.h` is for +// `SetHandleInformation`. +#include +#else +#include +#include +#include +#endif + +#include #include #include #include +#include #include #include #include +#include "Destination.h" #include "Transport.h" namespace usdasset { @@ -51,6 +65,23 @@ struct Exchange { /// then stalled", which `Timeout` is required to name. bool headersComplete = false; + /// Header bytes delivered so far on this exchange, interim responses + /// included, and whether they ran past `kMaxResponseHeaderBytes`. + std::size_t headerBytes = 0; + bool headersTooLarge = false; + + /// The destination policy, and what it decided about each address libcurl + /// offered. Counted rather than flagged, because a name can resolve to + /// several addresses and libcurl tries them in turn: only an exchange on + /// which *every* address was refused is a refusal. One on which a refused + /// IPv6 address was followed by a permitted IPv4 one that did not answer + /// is a connection failure, and saying otherwise would send an operator to + /// the wrong setting. + DestinationPolicy destinations; + int addressesRefused = 0; + int addressesAdmitted = 0; + std::optional refusedClass; + unsigned char* body = nullptr; std::size_t capacity = 0; std::size_t written = 0; @@ -69,6 +100,18 @@ struct Exchange { std::size_t OnHeader(char* data, std::size_t size, std::size_t count, void* userdata) { Exchange& exchange = *static_cast(userdata); const std::size_t bytes = size * count; + + // Counted before the line is copied or stored, so that the bound holds for + // the allocation it exists to prevent rather than for the one after it. + // Returning anything other than `bytes` aborts the transfer, which libcurl + // reports as a write error; `headersTooLarge` is what tells that apart from + // the body bound's own abort below. + if (bytes > kMaxResponseHeaderBytes - exchange.headerBytes) { + exchange.headersTooLarge = true; + return 0; + } + exchange.headerBytes += bytes; + std::string line(data, bytes); while (!line.empty() && (line.back() == '\r' || line.back() == '\n')) { @@ -124,6 +167,143 @@ std::size_t OnBody(char* data, std::size_t size, std::size_t count, void* userda return bytes; } +/// Reads the numeric address out of what libcurl is about to connect to. False +/// for a family the destination policy cannot classify. +/// +/// Copied out rather than cast in place. `curl_sockaddr::addr` is declared as a +/// plain `sockaddr`, sixteen bytes, and libcurl stores a `sockaddr_in6` there +/// in storage it sized for one; `addrlen` is the length that is actually +/// valid, and it is checked before a byte past the declared member is read. +bool ClassifySocketAddress(const curl_sockaddr& address, AddressClass* out) { + const unsigned char* raw = reinterpret_cast(&address.addr); + if (address.family == AF_INET && address.addrlen >= sizeof(sockaddr_in)) { + sockaddr_in v4; + std::memcpy(&v4, raw, sizeof(v4)); + std::array bytes{}; + std::memcpy(bytes.data(), &v4.sin_addr, bytes.size()); + *out = ClassifyIPv4(bytes); + return true; + } + if (address.family == AF_INET6 && address.addrlen >= sizeof(sockaddr_in6)) { + sockaddr_in6 v6; + std::memcpy(&v6, raw, sizeof(v6)); + std::array bytes{}; + std::memcpy(bytes.data(), &v6.sin6_addr, bytes.size()); + *out = ClassifyIPv6(bytes); + return true; + } + return false; +} + +/// The destination policy's connect-time half. +/// +/// libcurl calls this with each address it is about to connect to -- after the +/// name was resolved, before a socket exists -- which is the one point at which +/// the address is both known and not yet reached. That is what makes the policy +/// hold for a name that resolves to a refused address, for a name whose answer +/// changed between two lookups, and for every legacy spelling of an address a +/// system resolver accepts: none of those is visible in the URL, and all of them +/// are visible here. +/// +/// Refusing is returning `CURL_SOCKET_BAD`, which libcurl treats as a failed +/// connect and moves on to the next address, if there is one. The counts on the +/// exchange are what tell a refusal apart from a network that did not answer. +curl_socket_t OnOpenSocket(void* userdata, curlsocktype purpose, curl_sockaddr* address) { + Exchange& exchange = *static_cast(userdata); + + AddressClass addressClass = AddressClass::Public; + const bool classified = purpose == CURLSOCKTYPE_IPCXN && address != nullptr && + ClassifySocketAddress(*address, &addressClass); + if (!classified || !exchange.destinations.Permits(addressClass)) { + ++exchange.addressesRefused; + exchange.refusedClass = + classified ? std::optional(addressClass) : std::nullopt; + return CURL_SOCKET_BAD; + } + ++exchange.addressesAdmitted; + + // Created here, which means created without whatever libcurl would have + // done itself -- so the one property a host depends on is set here too. A + // DCC that forks render workers or shell tools while a reader holds a + // connection must not hand that socket to every child it starts. +#if defined(_WIN32) + const curl_socket_t created = + socket(address->family, address->socktype, address->protocol); + if (created != CURL_SOCKET_BAD) { + SetHandleInformation(reinterpret_cast(created), HANDLE_FLAG_INHERIT, 0); + } +#elif defined(SOCK_CLOEXEC) + // Atomically, where the platform can: a fork between `socket` and `fcntl` + // would inherit the descriptor anyway. + const curl_socket_t created = + socket(address->family, address->socktype | SOCK_CLOEXEC, address->protocol); +#else + const curl_socket_t created = + socket(address->family, address->socktype, address->protocol); + if (created != CURL_SOCKET_BAD) fcntl(created, F_SETFD, FD_CLOEXEC); +#endif + return created; +} + +bool HasNonAscii(const std::string& text) noexcept { + for (const char c : text) { + if (static_cast(c) >= 0x80) return true; + } + return false; +} + +/// The destination policy's pre-flight half, as the client will see the host. +/// +/// The protocol layer judges a literal it can read, and reads canonical +/// spellings only. That is not enough through a proxy, where the connect-time +/// check sees the proxy's address and the host goes out as text: libcurl reads +/// `2852039166`, `0xa9fea9fe`, `169.254.43518`, and `%31%36%39.254.169.254` as +/// 169.254.169.254, and normalizes each to it before the proxy ever sees the +/// request. So the host is taken from libcurl's own URL parser -- the one +/// `curl_easy_perform` will use on the same string -- and judged as that. +/// +/// A host that is not ASCII is asked for in the ASCII form the client would put +/// on the wire, where this libcurl can produce one; a compatibility mapping can +/// turn look-alike digits into an address. A libcurl without IDN support sends +/// such a host as written, and what a proxy then makes of it is the proxy's. +/// +/// Returns false, with the refused class, when the policy refuses the host. +/// Anything this cannot read -- a URL libcurl will itself refuse, an allocation +/// that failed -- is left to the transfer and the connect-time check. +bool PermittedByClient(const std::string& url, const DestinationPolicy& policy, + std::optional* refusedOut) { + CURLU* parsed = curl_url(); + if (parsed == nullptr) return true; + + std::string host; + if (curl_url_set(parsed, CURLUPART_URL, url.c_str(), 0) == CURLUE_OK) { + char* text = nullptr; + if (curl_url_get(parsed, CURLUPART_HOST, &text, 0) == CURLUE_OK && text != nullptr) { + host = text; + } + curl_free(text); +#if LIBCURL_VERSION_NUM >= 0x075800 + if (HasNonAscii(host)) { + char* ascii = nullptr; + if (curl_url_get(parsed, CURLUPART_HOST, &ascii, CURLU_PUNYCODE) == CURLUE_OK && + ascii != nullptr) { + host = ascii; + } + curl_free(ascii); + } +#endif + } + curl_url_cleanup(parsed); + + AddressClass addressClass = AddressClass::Public; + if (!host.empty() && ClassifyHostLiteral(host, &addressClass) && + !policy.Permits(addressClass)) { + *refusedOut = addressClass; + return false; + } + return true; +} + /// The progress callback needs the handle to read elapsed time from, so the /// two travel together. struct ProgressContext { @@ -215,6 +395,18 @@ TransportError ClassifyCurlError(CURLcode code, const Exchange& exchange, bool c case CURLE_WEIRD_SERVER_REPLY: return TransportError::Malformed; +#if LIBCURL_VERSION_NUM >= 0x080600 + case CURLE_TOO_LARGE: + // libcurl's own ceilings -- one header line past 100 KiB, or a + // block past its total -- fire before `OnHeader` ever sees the + // line, so the refusal is the library's rather than this file's. + // It is the same fact about the server, and it is reported as one. + // Older libcurl names it something vaguer, and that is classified + // below as a transport fault: failed closed either way, and only + // the words differ. + return TransportError::HeadersTooLarge; +#endif + case CURLE_OUT_OF_MEMORY: return TransportError::Internal; @@ -282,6 +474,15 @@ class CurlTransport final : public Transport { TransportResponse Perform(const TransportRequest& request) override { TransportResponse response; + // Before a handle, a connection, or a byte: a host the client would + // send to a refused address is not sent anywhere, proxy or not. + std::optional refused; + if (!PermittedByClient(request.url, request.destinations, &refused)) { + response.error = TransportError::DestinationRefused; + response.refusedClass = refused; + return response; + } + CURL* handle = _pool.Acquire(); if (handle == nullptr) { response.error = TransportError::Internal; @@ -293,6 +494,7 @@ class CurlTransport final : public Transport { exchange.capacity = request.body == nullptr ? 0 : request.bodyCapacity; exchange.responseMs = request.timeouts.responseMs; exchange.transferMs = request.timeouts.transferMs; + exchange.destinations = request.destinations; ProgressContext progress; progress.exchange = &exchange; @@ -342,6 +544,20 @@ class CurlTransport final : public Transport { // cannot test and this repository's counter cannot bound. curl_easy_setopt(handle, CURLOPT_FOLLOWLOCATION, 0L); + // The scheme allowlist, a second time. The parser above this seam is + // what enforces it, for an identifier and for every `Location`; this + // makes the client refuse too, so that a parser that ever widened + // would widen into a refusal rather than into a `file:` read. +#if LIBCURL_VERSION_NUM >= 0x075500 + curl_easy_setopt(handle, CURLOPT_PROTOCOLS_STR, "http,https"); +#else + curl_easy_setopt(handle, CURLOPT_PROTOCOLS, + static_cast(CURLPROTO_HTTP | CURLPROTO_HTTPS)); +#endif + + curl_easy_setopt(handle, CURLOPT_OPENSOCKETFUNCTION, &OnOpenSocket); + curl_easy_setopt(handle, CURLOPT_OPENSOCKETDATA, &exchange); + if (request.method == Method::Head) { curl_easy_setopt(handle, CURLOPT_NOBODY, 1L); } else { @@ -392,7 +608,26 @@ class CurlTransport final : public Transport { response.bodyOverflowed = exchange.overflowed; response.connected = pretransferTime > 0; - if (code == CURLE_WRITE_ERROR && exchange.overflowed) { + if (code != CURLE_OK && exchange.addressesRefused > 0 && + exchange.addressesAdmitted == 0) { + // Every address was the policy's to refuse, and it refused them + // all. Whatever code libcurl chose for "no connection could be + // made" is beside the point; nothing was attempted. + response.error = TransportError::DestinationRefused; + response.refusedClass = exchange.refusedClass; + } else if (code == CURLE_WRITE_ERROR && exchange.headersTooLarge) { + // Abandoned by `OnHeader`, at the bound. The status line may well + // have arrived and been perfectly ordinary; what the response did + // not do was finish describing itself within the space it was + // given, and nothing it said is worth acting on. + // + // So none of it is handed up. The table holds the first 64 KiB of + // a block that did not end, and a `Content-Length` or an + // `Accept-Ranges` found in it would be a fact read out of a + // response nobody finished receiving. + response.error = TransportError::HeadersTooLarge; + response.headers.Clear(); + } else if (code == CURLE_WRITE_ERROR && exchange.overflowed) { // Not a failure. The transfer was cut off deliberately, by this // file, because the server had more to send than the caller was // willing to receive. Whether that is an error depends on what the diff --git a/libs/usd-asset-http/src/Destination.cpp b/libs/usd-asset-http/src/Destination.cpp new file mode 100644 index 0000000..b5a4df0 --- /dev/null +++ b/libs/usd-asset-http/src/Destination.cpp @@ -0,0 +1,282 @@ +// SPDX-License-Identifier: Apache-2.0 + +#include "Destination.h" + +#include + +namespace usdasset { +namespace http { +namespace { + +constexpr std::size_t kNpos = std::string_view::npos; + +int HexValue(char c) noexcept { + if (c >= '0' && c <= '9') return c - '0'; + if (c >= 'a' && c <= 'f') return c - 'a' + 10; + if (c >= 'A' && c <= 'F') return c - 'A' + 10; + return -1; +} + +/// Parses colon-separated hexadecimal groups into `groups`, returning how many +/// sixteen-bit groups were written, or -1. +/// +/// A dotted quad is admitted as the final part when `allowTrailingIPv4` is set, +/// and counts as two groups. It is admitted nowhere else: `1.2.3.4::` is not an +/// address, and a parser that read it as one would be classifying a string no +/// resolver connects to. +int ParseGroups(std::string_view text, std::uint16_t* groups, int capacity, + bool allowTrailingIPv4) noexcept { + if (text.empty()) return 0; + + int count = 0; + std::size_t at = 0; + for (;;) { + const std::size_t colon = text.find(':', at); + const std::string_view part = + colon == kNpos ? text.substr(at) : text.substr(at, colon - at); + + if (part.find('.') != kNpos) { + if (!allowTrailingIPv4 || colon != kNpos) return -1; + std::array v4{}; + if (!ParseIPv4(part, &v4) || count + 2 > capacity) return -1; + groups[count++] = static_cast((v4[0] << 8) | v4[1]); + groups[count++] = static_cast((v4[2] << 8) | v4[3]); + return count; + } + + // An empty part is a stray colon -- `:1::2`, `1:2:` -- since the one + // legal pair of adjacent colons was split off by the caller. + if (part.empty() || part.size() > 4 || count + 1 > capacity) return -1; + unsigned value = 0; + for (const char c : part) { + const int digit = HexValue(c); + if (digit < 0) return -1; + value = value * 16 + static_cast(digit); + } + groups[count++] = static_cast(value); + + if (colon == kNpos) return count; + at = colon + 1; + } +} + +std::array TrailingIPv4(const std::array& address) noexcept { + return {address[12], address[13], address[14], address[15]}; +} + +bool AllZero(const std::array& address, std::size_t first, + std::size_t last) noexcept { + for (std::size_t i = first; i < last; ++i) { + if (address[i] != 0) return false; + } + return true; +} + +/// The well-known instance-metadata and credential endpoints, by value. +/// +/// A list and not a range, because no range contains them: they are where +/// each provider chose to put them. 169.254.169.254 is the common one -- AWS, +/// Azure, Google, Oracle, OpenStack, and most of the rest -- and the others +/// are the ones that are not there: AWS's container and pod credential agents +/// (169.254.170.2, 169.254.170.23), Tencent's metadata service (169.254.0.23), +/// Alibaba's (100.100.100.200, inside the shared address space), and Azure's +/// WireServer (168.63.129.16, inside public space). +bool IsMetadataIPv4(const std::array& address) noexcept { + static constexpr std::uint8_t kKnown[][4] = { + {169, 254, 169, 254}, + {169, 254, 170, 2}, + {169, 254, 170, 23}, + {169, 254, 0, 23}, + {100, 100, 100, 200}, + {168, 63, 129, 16}, + }; + for (const auto& known : kKnown) { + if (address[0] == known[0] && address[1] == known[1] && + address[2] == known[2] && address[3] == known[3]) { + return true; + } + } + return false; +} + +/// AWS's IPv6 metadata endpoint and pod credential agent, fd00:ec2::254 and +/// fd00:ec2::23 -- both inside unique-local space, which is why they cannot +/// be left to the range they happen to sit in. +bool IsMetadataIPv6(const std::array& address) noexcept { + if (address[0] != 0xfd || address[1] != 0x00 || address[2] != 0x0e || + address[3] != 0xc2 || !AllZero(address, 4, 14)) { + return false; + } + // The last group is hexadecimal as written: `::254` is 0x0254, and `::23` + // is 0x0023. + return (address[14] == 0x02 && address[15] == 0x54) || + (address[14] == 0x00 && address[15] == 0x23); +} + +} // namespace + +const char* AddressClassName(AddressClass addressClass) noexcept { + switch (addressClass) { + case AddressClass::Public: return "public"; + case AddressClass::Private: return "private"; + case AddressClass::Loopback: return "loopback"; + case AddressClass::LinkLocal: return "link-local"; + case AddressClass::Metadata: return "metadata"; + } + return "unknown"; +} + +bool DestinationPolicy::Permits(AddressClass addressClass) const noexcept { + switch (addressClass) { + case AddressClass::Public: return publicAddresses; + case AddressClass::Private: return privateNetworks; + case AddressClass::Loopback: return loopback; + case AddressClass::LinkLocal: return linkLocal; + case AddressClass::Metadata: return metadata; + } + return false; +} + +AddressClass ClassifyIPv4(const std::array& address) noexcept { + // First, and by value: two of these sit inside ranges that are permitted + // by default, and the range they sit in says nothing about what answers. + if (IsMetadataIPv4(address)) return AddressClass::Metadata; + // 0.0.0.0/8 is "this host on this network". A connect to 0.0.0.0 reaches + // the local machine on Linux and macOS alike, so it is loopback for the + // purpose this classification serves, whatever the registry calls it. + if (address[0] == 127 || address[0] == 0) return AddressClass::Loopback; + if (address[0] == 169 && address[1] == 254) return AddressClass::LinkLocal; + if (address[0] == 10) return AddressClass::Private; + if (address[0] == 172 && (address[1] & 0xf0) == 0x10) return AddressClass::Private; + if (address[0] == 192 && address[1] == 168) return AddressClass::Private; + // RFC 6598's shared address space. Carrier-grade NAT, and in practice the + // inside of a good many corporate and cloud networks: private for every + // purpose a request-forgery policy has. + if (address[0] == 100 && (address[1] & 0xc0) == 0x40) return AddressClass::Private; + return AddressClass::Public; +} + +AddressClass ClassifyIPv6(const std::array& address) noexcept { + if (AllZero(address, 0, 12)) { + // `::` and `::1`. The unspecified address reaches this host for the + // same reason 0.0.0.0 does. + if (address[12] == 0 && address[13] == 0 && address[14] == 0 && + (address[15] == 0 || address[15] == 1)) { + return AddressClass::Loopback; + } + // IPv4-compatible, `::a.b.c.d`. Deprecated, and still an address a + // stack may route to the IPv4 one it spells. + return ClassifyIPv4(TrailingIPv4(address)); + } + // IPv4-mapped, `::ffff:a.b.c.d`: a connection to it *is* a connection to + // the IPv4 address, on every dual-stack socket. Classifying it by its IPv6 + // prefix would let `[::ffff:169.254.169.254]` walk past a policy that + // refuses 169.254.169.254. + if (AllZero(address, 0, 10) && address[10] == 0xff && address[11] == 0xff) { + return ClassifyIPv4(TrailingIPv4(address)); + } + // The NAT64 well-known prefix, 64:ff9b::/96, where a translator forwards + // to the IPv4 address in the low bits. + if (address[0] == 0x00 && address[1] == 0x64 && address[2] == 0xff && + address[3] == 0x9b && AllZero(address, 4, 12)) { + return ClassifyIPv4(TrailingIPv4(address)); + } + if (IsMetadataIPv6(address)) return AddressClass::Metadata; + if (address[0] == 0xfe && (address[1] & 0xc0) == 0x80) return AddressClass::LinkLocal; + if (address[0] == 0xfe && (address[1] & 0xc0) == 0xc0) return AddressClass::Private; + if ((address[0] & 0xfe) == 0xfc) return AddressClass::Private; + return AddressClass::Public; +} + +bool ParseIPv4(std::string_view text, std::array* out) noexcept { + std::array bytes{}; + std::size_t at = 0; + for (std::size_t part = 0; part < 4; ++part) { + if (part > 0) { + if (at >= text.size() || text[at] != '.') return false; + ++at; + } + const std::size_t start = at; + unsigned value = 0; + while (at < text.size() && text[at] >= '0' && text[at] <= '9') { + value = value * 10 + static_cast(text[at] - '0'); + // Checked per digit, so that a long run of digits cannot overflow + // `value` before the length check below sees it. + if (value > 255) return false; + ++at; + } + const std::size_t digits = at - start; + if (digits == 0 || digits > 3) return false; + if (digits > 1 && text[start] == '0') return false; + bytes[part] = static_cast(value); + } + if (at != text.size()) return false; + *out = bytes; + return true; +} + +bool ParseIPv6(std::string_view text, std::array* out) noexcept { + std::uint16_t head[8] = {}; + std::uint16_t tail[8] = {}; + int headCount = 0; + int tailCount = 0; + + const std::size_t gap = text.find("::"); + if (gap == kNpos) { + headCount = ParseGroups(text, head, 8, true); + if (headCount != 8) return false; + } else { + // One `::` at most. A second -- including the overlapping one in + // `:::` -- would make the number of zero groups ambiguous. + if (text.find("::", gap + 1) != kNpos) return false; + headCount = ParseGroups(text.substr(0, gap), head, 8, false); + tailCount = ParseGroups(text.substr(gap + 2), tail, 8, true); + if (headCount < 0 || tailCount < 0) return false; + // The gap stands for at least one group of zeros. + if (headCount + tailCount > 7) return false; + } + + std::uint16_t groups[8] = {}; + for (int i = 0; i < headCount; ++i) groups[i] = head[i]; + for (int i = 0; i < tailCount; ++i) groups[8 - tailCount + i] = tail[i]; + + std::array bytes{}; + for (int i = 0; i < 8; ++i) { + bytes[2 * i] = static_cast(groups[i] >> 8); + bytes[2 * i + 1] = static_cast(groups[i] & 0xff); + } + *out = bytes; + return true; +} + +bool ClassifyHostLiteral(std::string_view host, AddressClass* out) noexcept { + if (!host.empty() && host.front() == '[') { + if (host.size() < 2 || host.back() != ']') return false; + std::string_view inner = host.substr(1, host.size() - 2); + // RFC 6874 writes a zone as `%25` followed by its name. The zone + // chooses an interface and not an address, so it plays no part in the + // class; a bare `%` is taken the same way, since that is how a system + // resolver reads it. + const std::size_t zone = inner.find('%'); + if (zone != kNpos) inner = inner.substr(0, zone); + + std::array v6{}; + if (!ParseIPv6(inner, &v6)) return false; + *out = ClassifyIPv6(v6); + return true; + } + + // One trailing dot is the fully qualified spelling of the same address -- + // libcurl keeps it, and a proxy or a system resolver that reads the host + // as an address reads it without the dot. Two are not an address at all. + std::string_view quad = host; + if (!quad.empty() && quad.back() == '.') quad.remove_suffix(1); + + std::array v4{}; + if (!ParseIPv4(quad, &v4)) return false; + *out = ClassifyIPv4(v4); + return true; +} + +} // namespace http +} // namespace usdasset diff --git a/libs/usd-asset-http/src/Destination.h b/libs/usd-asset-http/src/Destination.h new file mode 100644 index 0000000..cf30934 --- /dev/null +++ b/libs/usd-asset-http/src/Destination.h @@ -0,0 +1,65 @@ +// SPDX-License-Identifier: Apache-2.0 +// +// Address classification for the destination policy: §10.2 of the design +// policy, and `DestinationPolicy` in the public header. +// +// Pure functions over bytes and text. No socket header, no resolver, and no +// libcurl: the transport hands this file the address it is about to connect to +// as sixteen or four bytes, and the protocol layer hands it a URI host. Keeping +// the arithmetic here is what makes it testable as a table and fuzzable as a +// parser (§11.6), and what lets a second transport reuse it without inheriting +// the first one's client. +// +// Internal to usdAssetHttp. No header outside src/ includes this. + +#ifndef USDASSETHTTP_DESTINATION_H +#define USDASSETHTTP_DESTINATION_H + +#include +#include +#include + +#include "usdAssetHttp/HttpAssetReader.h" + +namespace usdasset { +namespace http { + +/// The class of an IPv4 address, in network byte order. +AddressClass ClassifyIPv4(const std::array& address) noexcept; + +/// The class of an IPv6 address, in network byte order. An address that +/// carries an IPv4 one is the class of the address it carries. +AddressClass ClassifyIPv6(const std::array& address) noexcept; + +/// Parses canonical dotted-quad IPv4 text: four decimal parts, each 0 to 255, +/// with no leading zeros and nothing else. +/// +/// Strict on purpose, and the strictness is safe rather than merely tidy. The +/// legacy forms a client or a system resolver also accepts -- `127.1`, +/// `0x7f.0.0.1`, `017700000001` -- are not literals here; the transport judges +/// them after its client has normalized them, and the connect-time check sees +/// the address they became. Reading `010.0.0.1` as decimal would be worse than +/// not reading it: a client that honours the leading zero connects to 8.0.0.1, +/// and a pre-flight that judged 10.0.0.1 would be judging an address nobody +/// connects to. +bool ParseIPv4(std::string_view text, std::array* out) noexcept; + +/// Parses IPv6 text in RFC 4291 §2.2 form: hexadecimal groups, at most one +/// `::`, and an optional trailing dotted quad. No brackets and no zone. +bool ParseIPv6(std::string_view text, std::array* out) noexcept; + +/// Classifies a URI host when it is a literal address: a dotted quad, with or +/// without the one trailing dot of a fully qualified name, or a bracketed IPv6 +/// literal with or without an RFC 6874 zone (`[fe80::1%25en0]`). +/// +/// Returns false for a name, which is not a failure: a name is judged at +/// connect time, by the address it resolves to. The legacy spellings a client +/// or a resolver also reads as an address are left to the caller to normalize +/// first -- the transport does, with the client's own parser, so that what is +/// judged is what will be sent. +bool ClassifyHostLiteral(std::string_view host, AddressClass* out) noexcept; + +} // namespace http +} // namespace usdasset + +#endif // USDASSETHTTP_DESTINATION_H diff --git a/libs/usd-asset-http/src/HttpAssetReader.cpp b/libs/usd-asset-http/src/HttpAssetReader.cpp index ac678dd..3c0024b 100644 --- a/libs/usd-asset-http/src/HttpAssetReader.cpp +++ b/libs/usd-asset-http/src/HttpAssetReader.cpp @@ -8,6 +8,7 @@ #include #include "usdAssetIo/RangeMath.h" +#include "Destination.h" #include "Framing.h" #include "TestSupport.h" #include "Transport.h" @@ -115,6 +116,24 @@ std::string Where(const Uri& uri) { return " (" + ElideSecrets(uri.ToIdentity()) + ")"; } +/// The destination policy's refusal, from whichever of its checks refused. +/// +/// `AccessDenied`, and not a code of its own, on the test DIAGNOSTICS.md §1 +/// sets for a code: what a caller does about it is what it does about a `403` +/// -- nothing, because a retry asks the same rule the same question, and tell +/// whoever owns the configuration. The class is named, because "access denied" +/// alone would send that person to the origin's permissions rather than to +/// their own policy. +Status DestinationRefusedStatus(const std::optional& refused, + const Uri& uri) { + const std::string what = + refused ? std::string(AddressClassName(*refused)) + " addresses" + : std::string("an address it cannot classify"); + return Status::Error(StatusCode::AccessDenied, + "the destination policy does not permit connecting to " + + what + Where(uri)); +} + Status ProjectTransportError(TransportError error, const Uri& uri) { switch (error) { case TransportError::ConnectFailed: @@ -147,6 +166,19 @@ Status ProjectTransportError(TransportError error, const Uri& uri) { case TransportError::Malformed: return Status::Error(StatusCode::InvalidResponse, "the response could not be parsed as HTTP" + Where(uri)); + case TransportError::HeadersTooLarge: + // `InvalidResponse`, because what a caller does about it is what it + // does about any other response it cannot use: nothing, and tell a + // human. The bound is named so that the human can tell an origin + // that misbehaved from a limit that was too tight. + return Status::Error(StatusCode::InvalidResponse, + "the response header block exceeded " + + std::to_string(kMaxResponseHeaderBytes) + + " bytes" + Where(uri)); + case TransportError::DestinationRefused: + // Reached only without the refused class, which the exchange + // passes to `DestinationRefusedStatus` itself when it has one. + return DestinationRefusedStatus(std::nullopt, uri); case TransportError::Internal: return Status::Error(StatusCode::NetworkError, "the HTTP client could not issue the request" + @@ -295,6 +327,22 @@ ExchangeResult PerformExchange(Transport& transport, int redirects = 0; for (;;) { + // The destination policy's pre-flight, at every hop and before any + // transport sees the request. A canonical literal in the URL is judged + // here by what it spells, so that the rule holds whichever client is + // underneath and a redirect to `http://169.254.169.254/` is refused as + // a string rather than as a connection. The transport judges twice + // more: the host as its client will actually send it, which is what + // holds through a proxy for every other spelling, and every address it + // connects to. + AddressClass literal = AddressClass::Public; + if (ClassifyHostLiteral(current.host, &literal) && + !options.destinations.Permits(literal)) { + result.finalUri = current; + result.status = DestinationRefusedStatus(literal, current); + return result; + } + TransportResponse response; for (;;) { @@ -308,6 +356,7 @@ ExchangeResult PerformExchange(Transport& transport, request.timeouts.connectMs = options.connectTimeoutMs; request.timeouts.responseMs = options.responseTimeoutMs; request.timeouts.transferMs = options.transferTimeoutMs; + request.destinations = options.destinations; request.body = body; request.bodyCapacity = capacity; @@ -321,9 +370,16 @@ ExchangeResult PerformExchange(Transport& transport, // Retried only when nothing usable came back. A response whose // headers arrived is the caller's to judge, and a body that stopped // early is resumed by the read loop rather than re-fetched whole. + // + // A block abandoned at the header bound is never retried, whatever + // its status line said. Its `503` is the one part of it that + // arrived, and it is not evidence of anything a second attempt + // could change -- it is an invitation to buffer the same 64 KiB + // again, as many times as the budget allows. const bool retryable = - response.status == 0 ? IsRetryableTransportError(response.error) - : IsRetryableStatus(response.status); + response.error == TransportError::HeadersTooLarge ? false + : response.status == 0 ? IsRetryableTransportError(response.error) + : IsRetryableStatus(response.status); if (!retryable) break; --*retriesRemaining; sink.Retry(); @@ -331,6 +387,31 @@ ExchangeResult PerformExchange(Transport& transport, result.finalUri = current; + if (response.error == TransportError::DestinationRefused) { + // The connect-time half: every address the name resolved to was + // one the policy refuses. Not retried -- `IsRetryableTransportError` + // does not admit it -- because asking again asks the same rule. + result.response = std::move(response); + result.status = + DestinationRefusedStatus(result.response.refusedClass, current); + return result; + } + + if (response.error == TransportError::HeadersTooLarge) { + // Before the status is looked at, because the status is the one + // part of this response that did arrive intact. A `200` whose + // header block never ended is not a `200` with some headers: it is + // a response that was abandoned, and letting it through to the + // branches below would have an open judge `Content-Length` and + // `Accept-Ranges` from whichever prefix of the block fit. + result.response = std::move(response); + result.status = ProjectTransportError(result.response.error, current); + if (result.response.status != 0) { + result.status.WithTransportStatus(result.response.status); + } + return result; + } + if (response.status == 0) { result.response = std::move(response); result.status = ProjectTransportError(result.response.error, current); diff --git a/libs/usd-asset-http/src/Transport.cpp b/libs/usd-asset-http/src/Transport.cpp index 6c09f1f..29a9894 100644 --- a/libs/usd-asset-http/src/Transport.cpp +++ b/libs/usd-asset-http/src/Transport.cpp @@ -66,6 +66,8 @@ const char* TransportErrorName(TransportError error) noexcept { case TransportError::ConnectionLost: return "ConnectionLost"; case TransportError::IncompleteBody: return "IncompleteBody"; case TransportError::Malformed: return "Malformed"; + case TransportError::HeadersTooLarge: return "HeadersTooLarge"; + case TransportError::DestinationRefused: return "DestinationRefused"; case TransportError::Internal: return "Internal"; } return "Unknown"; diff --git a/libs/usd-asset-http/src/Transport.h b/libs/usd-asset-http/src/Transport.h index c73c0cb..ba8b9b0 100644 --- a/libs/usd-asset-http/src/Transport.h +++ b/libs/usd-asset-http/src/Transport.h @@ -32,11 +32,14 @@ #include #include #include +#include #include #include #include #include +#include "usdAssetHttp/HttpAssetReader.h" + namespace usdasset { namespace http { @@ -109,11 +112,51 @@ enum class TransportError { IncompleteBody, /// The response could not be parsed as HTTP at all. Malformed, + /// The header block ran past `kMaxResponseHeaderBytes`, and the exchange + /// was abandoned before the block ended. Separate from `Malformed` because + /// every line that arrived may have been perfectly well formed -- what was + /// wrong was how many of them there were. + HeadersTooLarge, + /// The request's `DestinationPolicy` refused it before a connection was + /// attempted: either the host as the client would send it is a refused + /// address, or every address the name resolved to was one. Separate from + /// `ConnectFailed` because it is not a fact about the network: it is the + /// caller's own declared policy, and retrying it would be asking the same + /// question of the same rule. + DestinationRefused, /// The transport itself failed -- out of memory, a handle that would not /// initialize. Never a property of the server. Internal, }; +/// The most header bytes one exchange may deliver: status lines, fields, and +/// line terminators, counted as they arrive and summed across every interim +/// `1xx` response on the exchange. +/// +/// §10.1 of the design policy requires a bound on "the response header block +/// and the total response size, not only the body the caller asked for". The +/// body already has one -- `TransportRequest::bodyCapacity`, sized from what +/// the caller asked for -- and this is the other half. Without it, the header +/// table is a buffer whose size the server chooses: one response of ten million +/// short fields is ten million allocations in a process that asked for 64 KiB. +/// +/// Summed across interim responses rather than reset at each status line, +/// because a server that sends an unending stream of small `100 Continue`s is +/// the same attack spelled differently, and a bound that restarts at every +/// status line bounds nothing. +/// +/// A bound and not a tuned value, and labelled as one for invariant 11's sake. +/// Real origins send a few hundred bytes to a few kilobytes, and nginx, the +/// commonest thing in front of one, refuses by default to relay a response +/// whose header block does not fit in one memory page. 64 KiB is an order of +/// magnitude of headroom over that, and below libcurl's own 100 KiB ceiling on +/// a single line, so for a block of ordinary lines this bound is the one that +/// decides rather than the library's. It is a constant rather than a variable +/// for the reason CONFIGURATION.md §3 gives about the cache bypass threshold: it +/// is a correctness-of-policy rule, and a deployment that could raise it +/// without limit could remove it. +constexpr std::size_t kMaxResponseHeaderBytes = 64 * 1024; + const char* TransportErrorName(TransportError error) noexcept; enum class Method { @@ -148,6 +191,14 @@ struct TransportRequest { Timeouts timeouts; + /// Which classes of address a connection may be opened to. Judged by the + /// transport against the numeric address it is about to connect to, after + /// name resolution and before the socket exists, because that is the only + /// point at which the address is both known and not yet reached. A reused + /// connection is not judged again: it was admitted under the same reader's + /// policy when it was opened, and a reader's policy does not change. + DestinationPolicy destinations; + /// Where the body goes, and the bound §10 of the design policy requires: /// "never allocate from a server-declared length without a bound". The /// caller sizes this from what it asked for, so a server answering a 64 KiB @@ -181,6 +232,13 @@ struct TransportResponse { /// connect deadline from a response deadline, which `Timeout` (`HTTP006`) /// is required to name. bool connected = false; + + /// With `TransportError::DestinationRefused`: the class of an address the + /// policy refused, for the message. Empty when the refused address was of a + /// family the policy cannot classify, which is refused rather than + /// admitted -- a policy that let through what it could not read would be a + /// policy with a hole the shape of every address family it had not heard of. + std::optional refusedClass; }; /// The seam itself. diff --git a/libs/usd-asset-http/tests/CMakeLists.txt b/libs/usd-asset-http/tests/CMakeLists.txt index 1868043..e273360 100644 --- a/libs/usd-asset-http/tests/CMakeLists.txt +++ b/libs/usd-asset-http/tests/CMakeLists.txt @@ -15,6 +15,7 @@ # replaced without a deprecation. set(_usd_asset_http_tests + destination framing protocol uri) diff --git a/libs/usd-asset-http/tests/ScriptedTransport.h b/libs/usd-asset-http/tests/ScriptedTransport.h index 1ca0133..25d040b 100644 --- a/libs/usd-asset-http/tests/ScriptedTransport.h +++ b/libs/usd-asset-http/tests/ScriptedTransport.h @@ -56,6 +56,11 @@ struct SentRequest { std::string range; std::string ifRange; std::size_t capacity = 0; + /// What the transport was told to admit. The connect-time half of the + /// destination policy is the transport's to enforce, so the policy has to + /// reach it on every request -- redirect hops and reads included -- and a + /// request that arrived without it is a request judged by the default. + usdasset::http::DestinationPolicy destinations; }; /// The script: a function of the request and how many have come before it. @@ -89,6 +94,7 @@ class ScriptedTransport final : public Transport { sent.range = request.range; sent.ifRange = request.ifRange; sent.capacity = request.bodyCapacity; + sent.destinations = request.destinations; _script->sent.push_back(std::move(sent)); index = static_cast(_script->sent.size()) - 1; } diff --git a/libs/usd-asset-http/tests/test_destination.cpp b/libs/usd-asset-http/tests/test_destination.cpp new file mode 100644 index 0000000..6872491 --- /dev/null +++ b/libs/usd-asset-http/tests/test_destination.cpp @@ -0,0 +1,297 @@ +// SPDX-License-Identifier: Apache-2.0 +// +// Address classification for the destination policy, as a table. +// +// The policy is only as good as the arithmetic under it, and the arithmetic is +// where request-forgery bypasses live: an address that means loopback spelled +// in a form the classifier did not expect. So every row below is either a +// boundary of one of the five classes or a spelling of an address that a +// careless classifier would put in the wrong one -- and the whole file needs no +// socket, because the transport hands the classifier bytes and the protocol +// layer hands it text. + +#include +#include +#include +#include + +#include "Check.h" +#include "Destination.h" + +namespace { + +using usdasset::http::AddressClass; +using usdasset::http::AddressClassName; +using usdasset::http::ClassifyHostLiteral; +using usdasset::http::ClassifyIPv4; +using usdasset::http::ClassifyIPv6; +using usdasset::http::DestinationPolicy; +using usdasset::http::ParseIPv4; +using usdasset::http::ParseIPv6; + +void Fail(const char* what, const std::string& input, const char* detail) { + std::fprintf(stderr, "FAIL [%s] %s: %s\n", what, input.c_str(), detail); + ++usdassettest::FailureCount(); +} + +struct LiteralCase { + const char* host; + AddressClass expected; +}; + +void TestHostLiterals() { + // The class a URI host names when it is a literal. Every row is one the + // pre-flight check has to get right, because through a proxy it is the + // only check that sees the destination. + const LiteralCase cases[] = { + // Loopback, and the addresses a connect treats as this host. + {"127.0.0.1", AddressClass::Loopback}, + {"127.255.255.254", AddressClass::Loopback}, + {"0.0.0.0", AddressClass::Loopback}, + {"0.1.2.3", AddressClass::Loopback}, + {"[::1]", AddressClass::Loopback}, + {"[::]", AddressClass::Loopback}, + {"[0:0:0:0:0:0:0:1]", AddressClass::Loopback}, + + // The instance-metadata endpoints, by value, whichever range they sit + // in -- two of them are inside ranges permitted by default. + {"169.254.169.254", AddressClass::Metadata}, + {"169.254.170.2", AddressClass::Metadata}, + {"169.254.170.23", AddressClass::Metadata}, + {"169.254.0.23", AddressClass::Metadata}, + {"100.100.100.200", AddressClass::Metadata}, + {"168.63.129.16", AddressClass::Metadata}, + {"[fd00:ec2::254]", AddressClass::Metadata}, + {"[fd00:ec2::23]", AddressClass::Metadata}, + {"[FD00:0EC2:0:0:0:0:0:254]", AddressClass::Metadata}, + // And their neighbours, which are only what their ranges say. + {"169.254.169.253", AddressClass::LinkLocal}, + {"100.100.100.201", AddressClass::Private}, + {"168.63.129.17", AddressClass::Public}, + {"[fd00:ec2::255]", AddressClass::Private}, + {"[fd00:ec3::254]", AddressClass::Private}, + + // Link-local, the rest of it. + {"169.254.0.0", AddressClass::LinkLocal}, + {"169.254.255.255", AddressClass::LinkLocal}, + {"[fe80::1]", AddressClass::LinkLocal}, + {"[febf:ffff::1]", AddressClass::LinkLocal}, + {"[fe80::1%25en0]", AddressClass::LinkLocal}, + {"[FE80::A9FE:A9FE]", AddressClass::LinkLocal}, + + // Private, at both edges of each range. + {"10.0.0.0", AddressClass::Private}, + {"10.255.255.255", AddressClass::Private}, + {"172.16.0.0", AddressClass::Private}, + {"172.31.255.255", AddressClass::Private}, + {"192.168.0.1", AddressClass::Private}, + {"100.64.0.0", AddressClass::Private}, + {"100.127.255.255", AddressClass::Private}, + {"[fc00::1]", AddressClass::Private}, + {"[fdff:ffff::1]", AddressClass::Private}, + {"[fec0::1]", AddressClass::Private}, + + // Public, just outside each range -- the rows that catch a mask that + // is one bit too wide. + {"172.15.255.255", AddressClass::Public}, + {"172.32.0.0", AddressClass::Public}, + {"169.253.255.255", AddressClass::Public}, + {"169.255.0.0", AddressClass::Public}, + {"100.63.255.255", AddressClass::Public}, + {"100.128.0.0", AddressClass::Public}, + {"126.255.255.255", AddressClass::Public}, + {"128.0.0.0", AddressClass::Public}, + {"8.8.8.8", AddressClass::Public}, + {"[2001:4860:4860::8888]", AddressClass::Public}, + {"[fbff::1]", AddressClass::Public}, + {"[fe00::1]", AddressClass::Public}, + + // An IPv6 address carrying an IPv4 one is the IPv4 one's class. These + // are the classic bypasses: each spells a refused address in a family + // a naive classifier files as public. + {"[::ffff:127.0.0.1]", AddressClass::Loopback}, + {"[::ffff:7f00:1]", AddressClass::Loopback}, + {"[::ffff:169.254.169.254]", AddressClass::Metadata}, + {"[::ffff:a9fe:a9fe]", AddressClass::Metadata}, + {"[::ffff:169.254.1.1]", AddressClass::LinkLocal}, + {"[::ffff:10.1.2.3]", AddressClass::Private}, + {"[::ffff:8.8.8.8]", AddressClass::Public}, + {"[::127.0.0.1]", AddressClass::Loopback}, + {"[::169.254.169.254]", AddressClass::Metadata}, + {"[64:ff9b::169.254.169.254]", AddressClass::Metadata}, + {"[64:ff9b::100.100.100.200]", AddressClass::Metadata}, + {"[64:ff9b::7f00:1]", AddressClass::Loopback}, + {"[64:ff9b::8.8.8.8]", AddressClass::Public}, + + // The one trailing dot of a fully qualified name is the same address. + {"169.254.169.254.", AddressClass::Metadata}, + {"127.0.0.1.", AddressClass::Loopback}, + }; + for (const LiteralCase& row : cases) { + AddressClass actual = AddressClass::Public; + if (!ClassifyHostLiteral(row.host, &actual)) { + Fail("literal", row.host, "not recognized as a literal address"); + continue; + } + if (actual != row.expected) { + std::fprintf(stderr, "FAIL [literal] %s: %s, expected %s\n", row.host, + AddressClassName(actual), AddressClassName(row.expected)); + ++usdassettest::FailureCount(); + } + } +} + +void TestNamesAreNotLiterals() { + // A name is judged at connect time, by what it resolves to. A spelling of + // an address that is not canonical is not read here either: the transport + // judges it as its client normalizes it, and reading `010.0.0.1` as + // 10.0.0.1 here would judge an address a client that honours the leading + // zero never connects to. + const char* const names[] = { + "example.org", + "localhost", + "127.1", + "0x7f.0.0.1", + "017700000001", + "2130706433", + "010.0.0.1", + "127.0.0.01", + "256.0.0.1", + "1.2.3", + "1.2.3.4.5", + "1.2.3.4..", + ".1.2.3.4", + "1..2.3", + "[]", + "[::1", + "::1", + "[1.2.3.4]", + "[v1.fe80::1]", + "", + }; + for (const char* name : names) { + AddressClass ignored = AddressClass::Public; + if (ClassifyHostLiteral(name, &ignored)) { + Fail("name", name, "classified as a literal address"); + } + } +} + +void TestIPv6Grammar() { + // RFC 4291 §2.2, and the malformations adjacent to it. A parser that + // accepted any of the second list would be classifying a string that no + // resolver turns into the address it was classified as. + const char* const valid[] = { + "::", + "::1", + "1::", + "1:2:3:4:5:6:7:8", + "1:2:3:4:5:6:7::", + "::2:3:4:5:6:7:8", + "1:2:3:4:5:6:1.2.3.4", + "::ffff:1.2.3.4", + "fe80::abcd:1234", + "ABCD:ef01::", + }; + for (const char* text : valid) { + std::array out{}; + if (!ParseIPv6(text, &out)) Fail("ipv6", text, "refused a valid address"); + } + + const char* const invalid[] = { + "", + ":", + ":::", + "1:::2", + "1::2::3", + ":1::2", + "1::2:", + "1:2:3:4:5:6:7", + "1:2:3:4:5:6:7:8:9", + "1:2:3:4:5:6:7:8::", + "::1:2:3:4:5:6:7:8", + "12345::", + "g::", + "1.2.3.4", + "1.2.3.4::", + "::1.2.3.4:5", + "1:2:3:4:5:6:7:1.2.3.4", + "::256.0.0.1", + }; + for (const char* text : invalid) { + std::array out{}; + if (ParseIPv6(text, &out)) Fail("ipv6", text, "accepted a malformed address"); + } + + // And the bytes, for the one form with arithmetic in it. + std::array mapped{}; + CHECK(ParseIPv6("::ffff:169.254.1.2", &mapped)); + const std::array expected = {0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0xff, 0xff, 169, 254, 1, 2}; + CHECK(mapped == expected); +} + +void TestIPv4Bytes() { + std::array out{}; + CHECK(ParseIPv4("192.0.2.255", &out)); + CHECK(out[0] == 192 && out[1] == 0 && out[2] == 2 && out[3] == 255); + CHECK(ClassifyIPv4({169, 254, 169, 254}) == AddressClass::Metadata); + CHECK(ClassifyIPv4({169, 254, 1, 1}) == AddressClass::LinkLocal); + + std::array loopback{}; + loopback[15] = 1; + CHECK(ClassifyIPv6(loopback) == AddressClass::Loopback); +} + +void TestDefaultPolicy() { + // The documented default, asserted so that changing it is a change to a + // test and not only to a comment: CONFIGURATION.md states it, and a + // default nobody can see move is a default nobody chose. + const DestinationPolicy policy; + CHECK(policy.Permits(AddressClass::Public)); + CHECK(policy.Permits(AddressClass::Private)); + CHECK(policy.Permits(AddressClass::Loopback)); + CHECK(!policy.Permits(AddressClass::LinkLocal)); + CHECK(!policy.Permits(AddressClass::Metadata)); + + // Permitting link-local is not permitting the metadata endpoints in it. + DestinationPolicy linkLocal; + linkLocal.linkLocal = true; + CHECK(!linkLocal.Permits(AddressClass::Metadata)); + + // Covers: every destination reachable under the narrower policy is + // reachable under the wider one, and not the other way round. + CHECK(linkLocal.Covers(policy)); + CHECK(!policy.Covers(linkLocal)); + CHECK(policy.Covers(policy)); + + DestinationPolicy publicOnly; + publicOnly.privateNetworks = false; + publicOnly.loopback = false; + CHECK(publicOnly.Permits(AddressClass::Public)); + CHECK(!publicOnly.Permits(AddressClass::Private)); + CHECK(!publicOnly.Permits(AddressClass::Loopback)); + CHECK(publicOnly != policy); + + // The spellings the configuration takes, pinned: they are a compatibility + // surface as soon as one deployment has written one down. + CHECK_EQ(std::string(AddressClassName(AddressClass::Public)), std::string("public")); + CHECK_EQ(std::string(AddressClassName(AddressClass::Private)), std::string("private")); + CHECK_EQ(std::string(AddressClassName(AddressClass::Loopback)), + std::string("loopback")); + CHECK_EQ(std::string(AddressClassName(AddressClass::LinkLocal)), + std::string("link-local")); + CHECK_EQ(std::string(AddressClassName(AddressClass::Metadata)), + std::string("metadata")); +} + +} // namespace + +int main() { + TestHostLiterals(); + TestNamesAreNotLiterals(); + TestIPv6Grammar(); + TestIPv4Bytes(); + TestDefaultPolicy(); + return usdassettest::Report("usdAssetHttp/destination"); +} diff --git a/libs/usd-asset-http/tests/test_protocol.cpp b/libs/usd-asset-http/tests/test_protocol.cpp index c6135cc..d4e3071 100644 --- a/libs/usd-asset-http/tests/test_protocol.cpp +++ b/libs/usd-asset-http/tests/test_protocol.cpp @@ -803,6 +803,231 @@ void TestConflictingContentLengthIsRefused() { } } +/// A response the transport abandoned at the header bound, carrying whatever +/// prefix of the block arrived. `CurlTransport` clears that prefix before +/// handing the response up; this script deliberately does not, because the rule +/// under test is the exchange layer's, and it has to hold for any transport. +TransportResponse AbandonedAtHeaderBound(TransportResponse prefix) { + prefix.error = TransportError::HeadersTooLarge; + return prefix; +} + +void TestAbandonedHeaderBlockIsNotAResponse() { + // §10.1 of the design policy: the header block is bounded, and a response + // whose block ran past the bound is refused whole. The status line is the + // one part of it that arrived intact, and the risk is a layer that reads + // it -- a `200`, with a `Content-Length` and an `Accept-Ranges` in the + // prefix -- and opens the asset on a response nobody finished receiving. + { + auto script = MakeScript([](const TransportRequest&, int) { + return AbandonedAtHeaderBound(MetadataResponse(kSize, "\"v1\"")); + }); + const HttpOpenResult opened = OpenWith(script); + CHECK_EQ(opened.status.code, StatusCode::InvalidResponse); + CHECK(opened.reader == nullptr); + CHECK(opened.status.message.find("header block") != std::string::npos); + CHECK(opened.status.transportStatus.has_value()); + // Not retried. Nothing about asking again makes the block smaller. + CHECK_EQ(script->Count(), 1); + } + { + // Nor when its status line is one the retry policy would otherwise + // act on. A `503` followed by a megabyte of fields is a hostile + // response, not a transient one, and retrying it buffers the bound + // again for every attempt the budget allows. + const int statuses[] = {503, 429, 502, 504}; + for (const int status : statuses) { + auto script = MakeScript([status](const TransportRequest&, int) { + TransportResponse response; + response.status = status; + response.connected = true; + response.headers.Add("Retry-After", "0"); + return AbandonedAtHeaderBound(response); + }); + HttpOptions options; + options.maxAttempts = 3; + CHECK_EQ(OpenWith(script, options).status.code, StatusCode::InvalidResponse); + CHECK_EQ(script->Count(), 1); + } + } + { + // A redirect whose block did not end is not followed, whatever its + // `Location` said. + auto script = MakeScript([](const TransportRequest&, int) { + return AbandonedAtHeaderBound(RedirectResponse("/elsewhere")); + }); + CHECK_EQ(OpenWith(script).status.code, StatusCode::InvalidResponse); + CHECK_EQ(script->Count(), 1); + } + { + // And a range response: a correct `Content-Range` in the prefix is + // not a framed body, and the read is neither resumed nor retried. + auto script = MakeScript([](const TransportRequest& request, int index) { + if (index == 0) return MetadataResponse(kSize, "\"v1\""); + return AbandonedAtHeaderBound( + PartialResponse(request, 0, kSize, "\"v1\"", 0xA5)); + }); + HttpOpenResult opened = OpenWith(script); + CHECK(opened.reader != nullptr); + if (!opened.reader) return; + + std::vector buffer(256, 0); + const ReadResult read = opened.reader->Read(0, buffer.data(), 256); + CHECK_EQ(read.status.code, StatusCode::InvalidResponse); + CHECK_EQ(read.bytesRead, std::size_t(0)); + CHECK_EQ(script->Count(), 2); + } +} + +void TestRedirectTargetsAreHeldToTheSchemeAllowlist() { + // §10.2 of the design policy: the scheme set is an allowlist, applied at + // every hop and not only the first. It holds today as a consequence rather + // than as a check -- a `Location` goes through the same parser as an + // original identifier, and that parser accepts two schemes -- which is + // exactly why it is asserted here: nothing else would notice if the parser + // ever widened. + // + // Each target is refused before it is requested, so the one request in + // the log is the one that discovered the hop. + const char* const refused[] = { + "file:///etc/passwd", + "FILE:///etc/passwd", + "ftp://example.org/data/survey.copc", + "gopher://example.org/1", + "data:text/plain,hello", + "s3://bucket/key", + "https:/no-authority", + }; + for (const char* location : refused) { + auto script = MakeScript([location](const TransportRequest&, int) { + return RedirectResponse(location); + }); + const HttpOpenResult opened = OpenWith(script); + CHECK_EQ(opened.status.code, StatusCode::InvalidResponse); + CHECK(opened.status.message.find("unusable location") != std::string::npos); + CHECK_EQ(script->Count(), 1); + } + + // The two forms that stay inside the allowlist without naming a scheme -- + // a network-path reference, which inherits the base's, and an absolute + // path -- are followed. Refusing them would be a different policy, and + // an origin moving an asset to its own CDN is the ordinary case. + const char* const followed[] = { + "//cdn.example.net/data/survey.copc", + "/data/moved.copc", + }; + for (const char* location : followed) { + auto script = MakeScript([location](const TransportRequest&, int index) { + if (index == 0) return RedirectResponse(location); + return MetadataResponse(kSize, "\"v1\""); + }); + const HttpOpenResult opened = OpenWith(script); + CHECK(opened.reader != nullptr); + CHECK_EQ(script->Count(), 2); + if (script->Count() == 2) { + CHECK_EQ(script->sent[1].url.compare(0, 8, "https://"), 0); + } + } +} + +void TestDestinationPolicy() { + // §10.2 of the design policy: reach is bounded by declared policy. These + // are the halves of it that are the protocol layer's -- the pre-flight on a + // literal address, at every hop, and the projection of the transport's own + // refusal. The connect-time half is the transport's, and is exercised over + // a real socket in `tests/corpus`. + { + // The default refuses the metadata endpoints, and a literal is refused + // before any request is issued for it: the instance-metadata address + // never sees a packet from this process. + auto script = MakeScript(Wellbehaved("\"v1\"")); + const HttpOpenResult opened = usdasset::http::testing::OpenWithTransport( + "http://169.254.169.254/latest/meta-data/", HttpOptions(), Factory(script)); + CHECK_EQ(opened.status.code, StatusCode::AccessDenied); + CHECK(opened.reader == nullptr); + CHECK(opened.status.message.find("metadata") != std::string::npos); + CHECK_EQ(script->Count(), 0); + } + { + // And at a redirect hop, which is where a hostile origin would put it. + // Every spelling that carries the refused address in another family is + // the same refusal. + const char* const targets[] = { + "https://169.254.169.254/latest/meta-data/", + "https://[::ffff:169.254.169.254]/latest/meta-data/", + "https://[fe80::1%25en0]/x", + }; + for (const char* target : targets) { + auto script = MakeScript([target](const TransportRequest&, int) { + return RedirectResponse(target); + }); + const HttpOpenResult opened = OpenWith(script); + CHECK_EQ(opened.status.code, StatusCode::AccessDenied); + // One request: the one that discovered the hop. + CHECK_EQ(script->Count(), 1); + } + } + { + // A narrower policy refuses what it says, and nothing else. Loopback + // is permitted by default -- the fixture server is loopback -- and a + // deployment that refuses it gets the refusal it asked for. + HttpOptions options; + options.destinations.loopback = false; + auto script = MakeScript(Wellbehaved("\"v1\"")); + const HttpOpenResult refused = usdasset::http::testing::OpenWithTransport( + "http://127.0.0.1:8080/a.usda", options, Factory(script)); + CHECK_EQ(refused.status.code, StatusCode::AccessDenied); + CHECK(refused.status.message.find("loopback") != std::string::npos); + CHECK_EQ(script->Count(), 0); + + const HttpOpenResult permitted = usdasset::http::testing::OpenWithTransport( + "http://127.0.0.1:8080/a.usda", HttpOptions(), Factory(script)); + CHECK(permitted.reader != nullptr); + } + { + // The policy reaches the transport on every request -- the metadata + // request, each redirect hop, and every read -- because the + // connect-time half is judged there and a request without it would be + // judged by the default instead. + HttpOptions options; + options.destinations.privateNetworks = false; + options.destinations.linkLocal = true; + auto script = MakeScript([](const TransportRequest& request, int index) { + if (index == 0) return RedirectResponse("/moved.copc"); + if (request.method == Method::Head) return MetadataResponse(kSize, "\"v1\""); + return PartialResponse(request, 0, kSize, "\"v1\"", 0xA5); + }); + HttpOpenResult opened = OpenWith(script, options); + CHECK(opened.reader != nullptr); + if (opened.reader) { + std::vector buffer(64, 0); + CHECK_EQ(opened.reader->Read(0, buffer.data(), 64).status.code, + StatusCode::Ok); + } + CHECK_EQ(script->Count(), 3); + for (const usdassethttptest::SentRequest& sent : script->sent) { + CHECK(sent.destinations == options.destinations); + } + } + { + // The transport's refusal -- every address the name resolved to was + // refused -- is `AccessDenied` naming the class, and is not retried: + // asking again asks the same rule the same question. + auto script = MakeScript([](const TransportRequest&, int) { + TransportResponse response = + TransportFailure(TransportError::DestinationRefused); + response.refusedClass = usdasset::http::AddressClass::Private; + return response; + }); + HttpOptions options; + options.maxAttempts = 3; + const HttpOpenResult opened = OpenWith(script, options); + CHECK_EQ(opened.status.code, StatusCode::AccessDenied); + CHECK(opened.status.message.find("private") != std::string::npos); + CHECK_EQ(script->Count(), 1); + } +} + void TestCallerErrors() { auto script = MakeScript(Wellbehaved("\"v1\"")); HttpOpenResult opened = OpenWith(script); @@ -840,6 +1065,9 @@ int main() { TestRetryBudgetIsSharedAcrossOneRead(); TestRefusedRangeThatMeansTheAssetMoved(); TestConflictingContentLengthIsRefused(); + TestAbandonedHeaderBlockIsNotAResponse(); + TestRedirectTargetsAreHeldToTheSchemeAllowlist(); + TestDestinationPolicy(); TestCallerErrors(); return usdassettest::Report("usdAssetHttp/protocol"); } diff --git a/plugins/http-resolver/CMakeLists.txt b/plugins/http-resolver/CMakeLists.txt index a3294e8..d086970 100644 --- a/plugins/http-resolver/CMakeLists.txt +++ b/plugins/http-resolver/CMakeLists.txt @@ -71,6 +71,7 @@ set(PLUGIN_NAME HttpResolver) add_library(${PLUGIN_NAME} SHARED src/Configuration.cpp + src/Context.cpp src/Diagnostics.cpp src/HttpResolver.cpp src/Identifier.cpp diff --git a/plugins/http-resolver/README.md b/plugins/http-resolver/README.md index c595966..cb650c5 100644 --- a/plugins/http-resolver/README.md +++ b/plugins/http-resolver/README.md @@ -23,6 +23,8 @@ When this README and that document disagree, the document wins. ```json "HttpResolver": { "bases": ["ArResolver"], + "implementsContexts": true, + "implementsScopedCaches": true, "uriSchemes": ["http", "https"] } ``` @@ -62,7 +64,8 @@ removed because §4.3 of the resolver API, and an identifier *is* the resolver API — a URL that needs credentials therefore fails at the origin with `HTTP002` rather than succeeding with a secret in every log line. Authentication arrives as the interception -point in `v0.6.0`, not as a URL component. +point in `v0.7.0`, supplied through the resolver context, and not as a URL +component. Relative references anchor to the layer they were authored in, per RFC 3986 §5.2, which is what makes a remote scene work at all: a layer published to a CDN @@ -166,9 +169,11 @@ invalid timestamp costs a reload, never a wrong answer. ## Configuration -The five transport bounds and the four cache values in -[CONFIGURATION.md](../../docs/reference/CONFIGURATION.md), read once when the -resolver is constructed: +The transport bounds, the destination policy, the cache values, and the +persistent tier in [CONFIGURATION.md](../../docs/reference/CONFIGURATION.md), +read once, when the resolver is first used — not when it is constructed, because +OpenUSD constructs it in every process that opens a stage and a host that only +ever opens local ones must not have a cache directory created for it: | Variable | Maps to | Default | | --- | --- | --- | @@ -177,6 +182,7 @@ resolver is constructed: | `USD_HTTP_RESOLVER_TOTAL_TIMEOUT_MS` | whole-transfer deadline | 300000 | | `USD_HTTP_RESOLVER_MAX_RETRIES` | attempts, minus one | 2 | | `USD_HTTP_RESOLVER_MAX_REDIRECTS` | redirect hops | 5 | +| `USD_HTTP_RESOLVER_DESTINATIONS` | address classes a connection may reach | `public,private,loopback` | | `USD_HTTP_RESOLVER_BLOCK_SIZE` | cache block size, in bytes | 65536 | | `USD_HTTP_RESOLVER_CACHE_BUDGET` | process-wide cache budget, in bytes | 134217728 | | `USD_HTTP_RESOLVER_COALESCE_GAP` | blocks of gap merged into one request | 1 | @@ -196,6 +202,15 @@ Only a `Stable` identity is written there directory is reversible to a URL — an entry's identity is a SHA-256 digest, because a resolved identifier can be a signed one. +`USD_HTTP_RESOLVER_DESTINATIONS` is the reach an identifier from a layer nobody +here authored is allowed to have, per §10.2 of the +[design policy](../../docs/design/DESIGN_POLICY.md). The default refuses +link-local addresses and the well-known instance-metadata endpoints — by value, +wherever they sit, because two of them are inside ranges that are otherwise +private — and keeps loopback and private networks reachable, because local +fixture servers and intranet hosts are what `http` is registered for. A refusal +is `HTTP002` naming the class, and no request is sent. + A value that does not parse is a warning at construction and then the default; one bad value does not discard the others, and a value that is adjusted rather than refused — a block size rounded down to a power of two — warns and takes the @@ -203,9 +218,23 @@ adjustment. `0` is legal for the two counters and means "do not", and is rejected for the three deadlines, because to most transports a zero deadline means *no* deadline — the one value §10 of the design policy exists to forbid. -Per-stage configuration through `ArResolverContext` is `v0.6.0`. A host that -opens two stages against two servers cannot be served by a process-global, and -that is the surface the environment variables are a bootstrap for. +Eight of these can also be set per stage, through an `ArResolverContext` made +from a string with the same names — the transport bounds, the destination +policy, and the two coalescing limits: + +```python +ctx = Ar.GetResolver().CreateContextFromString( + "https", "USD_HTTP_RESOLVER_DESTINATIONS=public; USD_HTTP_RESOLVER_MAX_RETRIES=0") +stage = Usd.Stage.Open("https://example.org/scenes/main.usda", ctx) +``` + +The block size, the two budgets, and the persistent directory stay the +environment's, because every stage in the process shares the store they +configure. A context is validated when it is created and warns then; what it +carries afterwards is what it admitted. The rules, and why every identifier this +bundle owns is declared context-dependent, are in +[CONFIGURATION.md](../../docs/reference/CONFIGURATION.md) §4 and +[RESOLVER.md](../../docs/architecture/RESOLVER.md) §6. ## Plugin discovery and installation @@ -285,7 +314,7 @@ arithmetic, and a mistake in any of them is invisible from the outside: | Test | Asserts | | --- | --- | | `httpResolver_identifier` | normalization, anchoring, what is not claimed, idempotence | -| `httpResolver_configuration` | the five variables, and what a bad value does | +| `httpResolver_configuration` | every variable, and what a bad value does | | `httpResolver_diagnostics` | the `HTTPxxx` table, the message form, and that no secret survives | | `httpResolver_identity` | what asset info may publish for a strong, weak, absent, or contradicted validator, and that no credential reaches it | | `httpResolver_stage` | a remote stage over a real socket, against the hostile fixture corpus | @@ -295,8 +324,9 @@ The fifth is the release's claim: it stands up an origin on loopback, opens a a 4 KiB window out of a 1 MiB asset and checks the `Range` header the server actually received, and confirms that a `404` is silent, that a failure is not, that range-unsupported is terminal, that asset info reports the identity of the -open rather than of a new request, and that a local stage still opens exactly as -it did. +open rather than of a new request, that a resolver context configures the stage +it is bound to and no other — including through a layer another stage already +loaded — and that a local stage still opens exactly as it did. One of its cases asserts nothing at all in a `CHECK`: it resolves an asset it never opens, leaving a retained reader to be destroyed during static teardown, @@ -322,9 +352,11 @@ recorded in [NOTICE](../../NOTICE); nothing in this bundle adds one. ## Known limitations -- **Nothing cached outlives the process.** The block cache is in front of every - asset this bundle opens (`v0.3.0`), and it is in memory only: on-disk - persistence is `v0.4.0`, admitted for a strong validator alone. +- **Only a `Stable` identity outlives the process.** The block cache is in front + of every asset this bundle opens (`v0.3.0`), and the on-disk tier under it + (`v0.4.0`) admits a strong validator the origin issued and nothing weaker, so + an asset with a weak or absent validator is re-fetched by every process that + reads it. - **Identity is exposed through `GetAssetInfo` and nowhere else.** There is no side-channel API, and `GetModificationTimestamp` is invalid by design rather than by omission. See *Asset info and identity* above. diff --git a/plugins/http-resolver/plugin/resources/httpResolver/plugInfo.json.in b/plugins/http-resolver/plugin/resources/httpResolver/plugInfo.json.in index 98811e7..097934d 100644 --- a/plugins/http-resolver/plugin/resources/httpResolver/plugInfo.json.in +++ b/plugins/http-resolver/plugin/resources/httpResolver/plugInfo.json.in @@ -11,6 +11,8 @@ "HttpResolver": { "bases": ["ArResolver"], "displayName": "HTTP range-read asset resolver", + "implementsContexts": true, + "implementsScopedCaches": true, "uriSchemes": ["http", "https"] } } diff --git a/plugins/http-resolver/src/Configuration.cpp b/plugins/http-resolver/src/Configuration.cpp index dd900a5..a8ab0dd 100644 --- a/plugins/http-resolver/src/Configuration.cpp +++ b/plugins/http-resolver/src/Configuration.cpp @@ -4,7 +4,9 @@ #include #include +#include #include +#include #include namespace usdhttpresolver { @@ -15,6 +17,7 @@ constexpr const char* kReadTimeout = "USD_HTTP_RESOLVER_READ_TIMEOUT_MS"; constexpr const char* kTotalTimeout = "USD_HTTP_RESOLVER_TOTAL_TIMEOUT_MS"; constexpr const char* kMaxRetries = "USD_HTTP_RESOLVER_MAX_RETRIES"; constexpr const char* kMaxRedirects = "USD_HTTP_RESOLVER_MAX_REDIRECTS"; +constexpr const char* kDestinations = "USD_HTTP_RESOLVER_DESTINATIONS"; constexpr const char* kBlockSize = "USD_HTTP_RESOLVER_BLOCK_SIZE"; constexpr const char* kCacheBudget = "USD_HTTP_RESOLVER_CACHE_BUDGET"; @@ -58,6 +61,35 @@ bool ParseCount(const std::string& text, long long min, long long max, return true; } +/// A problem that ended in the value being used, after an adjustment. +ConfigurationProblem Adjusted(const char* variable, std::string value, + std::string reason) { + ConfigurationProblem problem; + problem.variable = variable; + problem.value = std::move(value); + problem.reason = std::move(reason); + problem.adjusted = true; + return problem; +} + +/// Strips spaces, tabs, and line breaks from both ends. A context string is +/// often written across lines in a host's configuration file, and a newline +/// before a name is not part of the name. +std::string Trim(const std::string& text) { + const char* const blanks = " \t\r\n"; + const std::size_t first = text.find_first_not_of(blanks); + if (first == std::string::npos) return std::string(); + const std::size_t last = text.find_last_not_of(blanks); + return text.substr(first, last - first + 1); +} + +bool Contains(const std::vector& names, const std::string& name) { + for (const char* candidate : names) { + if (name == candidate) return true; + } + return false; +} + /// `getenv`, and the pragma MSVC needs to allow it. /// /// `_dupenv_s` exists on one toolchain and allocates; this runs once, at @@ -89,6 +121,96 @@ void ReadInto(const EnvironmentLookup& lookup, const char* name, long long min, *target = static_cast(value); } +/// Every address class, in the order a canonical destination list names them. +constexpr usdasset::http::AddressClass kAddressClasses[] = { + usdasset::http::AddressClass::Public, usdasset::http::AddressClass::Private, + usdasset::http::AddressClass::Loopback, usdasset::http::AddressClass::LinkLocal, + usdasset::http::AddressClass::Metadata, +}; + +/// The canonical spelling of a destination set: the classes it permits, in +/// `kAddressClasses` order, separated by commas without spaces. +std::string DestinationsString(const usdasset::http::DestinationPolicy& policy) { + std::string text; + for (const usdasset::http::AddressClass addressClass : kAddressClasses) { + if (!policy.Permits(addressClass)) continue; + if (!text.empty()) text += ','; + text += usdasset::http::AddressClassName(addressClass); + } + return text; +} + +/// Parses a destination set: address class names separated by commas, each +/// spelled as `AddressClassName` spells it. +/// +/// A set rather than a level, because the four classes are not an order. An +/// intranet-only deployment permits `private` and refuses `public`; a render +/// farm permits `public` and refuses the rest; neither is a point on a scale +/// that also contains the other. +/// +/// Whitespace around a name is tolerated -- `public, private` is how a person +/// writes a list -- and nothing else is. An unknown name refuses the whole +/// value rather than the one name, because a policy that silently dropped the +/// word it did not recognize is a different policy from the one written, and +/// the difference is always in the direction of a destination nobody meant to +/// permit or refuse. +bool ParseDestinations(const std::string& text, usdasset::http::DestinationPolicy* out, + std::string* reasonOut) { + using usdasset::http::AddressClass; + + if (text.empty()) { + *reasonOut = "empty"; + return false; + } + + usdasset::http::DestinationPolicy policy; + policy.publicAddresses = false; + policy.privateNetworks = false; + policy.loopback = false; + policy.linkLocal = false; + + std::size_t at = 0; + for (;;) { + const std::size_t comma = text.find(',', at); + // `Trim`, and not a narrower one: a list broken across lines in a + // context string, or an environment file saved with CRLF endings, puts + // a line break beside a name, and refusing the whole value for it + // would put the stage on the default -- a wider policy than the one + // written, which is the one direction this parser must not fail in. + const std::string name = Trim(text.substr( + at, comma == std::string::npos ? std::string::npos : comma - at)); + + if (name.empty()) { + *reasonOut = "an empty entry in the list"; + return false; + } + bool known = false; + for (const AddressClass addressClass : kAddressClasses) { + if (name != usdasset::http::AddressClassName(addressClass)) continue; + known = true; + switch (addressClass) { + case AddressClass::Public: policy.publicAddresses = true; break; + case AddressClass::Private: policy.privateNetworks = true; break; + case AddressClass::Loopback: policy.loopback = true; break; + case AddressClass::LinkLocal: policy.linkLocal = true; break; + case AddressClass::Metadata: policy.metadata = true; break; + } + } + if (!known) { + *reasonOut = "unknown address class '" + name + + "'; expected public, private, loopback, link-local, or " + "metadata"; + return false; + } + + if (comma == std::string::npos) break; + at = comma + 1; + } + + *out = policy; + return true; +} + /// The 64-bit form of `ReadInto`, for the variables that are byte counts. /// /// A block size and a budget do not fit in the `int` the transport bounds are, @@ -187,31 +309,31 @@ usdasset::cache::CacheOptions CacheOptionsFrom( // rather than from a byte count. const usdasset::cache::CacheOptions normalized = options.Normalized(); if (problemsOut != nullptr && normalized.blockSize != options.blockSize) { - problemsOut->push_back({kBlockSize, std::to_string(options.blockSize), - "rounded down to the power of two " + - std::to_string(normalized.blockSize)}); + problemsOut->push_back(Adjusted(kBlockSize, std::to_string(options.blockSize), + "rounded down to the power of two " + + std::to_string(normalized.blockSize))); } if (problemsOut != nullptr && normalized.coalesceGapBlocks != options.coalesceGapBlocks) { problemsOut->push_back( - {kCoalesceGap, std::to_string(options.coalesceGapBlocks), - "capped at " + std::to_string(normalized.coalesceGapBlocks) + - ", the widest gap that can fit under " - "USD_HTTP_RESOLVER_MAX_REQUEST_BYTES"}); + Adjusted(kCoalesceGap, std::to_string(options.coalesceGapBlocks), + "capped at " + std::to_string(normalized.coalesceGapBlocks) + + ", the widest gap that can fit under " + "USD_HTTP_RESOLVER_MAX_REQUEST_BYTES")); } if (problemsOut != nullptr && normalized.budgetBytes != options.budgetBytes) { problemsOut->push_back( - {kCacheBudget, std::to_string(options.budgetBytes), - "raised to " + std::to_string(normalized.budgetBytes) + - ", one block: a budget that cannot hold a block does not " - "cache nothing, it fetches a block and drops it"}); + Adjusted(kCacheBudget, std::to_string(options.budgetBytes), + "raised to " + std::to_string(normalized.budgetBytes) + + ", one block: a budget that cannot hold a block does not " + "cache nothing, it fetches a block and drops it")); } if (problemsOut != nullptr && normalized.maxRequestBytes != options.maxRequestBytes) { problemsOut->push_back( - {kMaxRequestBytes, std::to_string(options.maxRequestBytes), - "raised to " + std::to_string(normalized.maxRequestBytes) + - ", one block: a merged request that cannot carry a block " - "cannot carry the block it was merging"}); + Adjusted(kMaxRequestBytes, std::to_string(options.maxRequestBytes), + "raised to " + std::to_string(normalized.maxRequestBytes) + + ", one block: a merged request that cannot carry a block " + "cannot carry the block it was merging")); } return options; @@ -245,20 +367,30 @@ usdasset::http::HttpOptions OptionsFrom( // Zero redirects is legal and means "refuse to follow any". ReadInto(lookup, kMaxRedirects, 0, 100, &options.maxRedirects, problemsOut); + // The destination policy of §10.2. Unset is the documented default -- + // `public,private,loopback`, which is `DestinationPolicy`'s own -- and not + // "everything": a deployment that says nothing about link-local does not + // reach the instance-metadata address by accident. + std::string destinations; + if (lookup(kDestinations, &destinations)) { + std::string reason; + if (!ParseDestinations(destinations, &options.destinations, &reason) && + problemsOut != nullptr) { + problemsOut->push_back({kDestinations, destinations, reason}); + } + } + return options; } -usdasset::http::HttpOptions OptionsFromEnvironment( - std::vector* problemsOut) { - const EnvironmentLookup lookup = [](const char* name, std::string* valueOut) { - const char* value = ReadEnvironment(name); - if (value == nullptr) return false; - valueOut->assign(value); - return true; - }; - return OptionsFrom(lookup, problemsOut); +bool ReadEnvironmentVariable(const char* name, std::string* valueOut) { + const char* value = ReadEnvironment(name); + if (value == nullptr) return false; + valueOut->assign(value); + return true; } + ResolverConfiguration ConfigurationFrom( const EnvironmentLookup& lookup, std::vector* problemsOut) { @@ -269,16 +401,6 @@ ResolverConfiguration ConfigurationFrom( return configuration; } -ResolverConfiguration ConfigurationFromEnvironment( - std::vector* problemsOut) { - const EnvironmentLookup lookup = [](const char* name, std::string* valueOut) { - const char* value = ReadEnvironment(name); - if (value == nullptr) return false; - valueOut->assign(value); - return true; - }; - return ConfigurationFrom(lookup, problemsOut); -} const std::vector& ConfiguredVariables() { static const std::vector variables = { @@ -292,8 +414,191 @@ const std::vector& ConfiguredVariables() { kReadTimeout, kTotalTimeout, kMaxRetries, - kMaxRedirects}; + kMaxRedirects, + kDestinations}; return variables; } +const std::vector& ContextVariables() { + static const std::vector variables = { + kCoalesceGap, + kMaxRequestBytes, + kConnectTimeout, + kReadTimeout, + kTotalTimeout, + kMaxRetries, + kMaxRedirects, + kDestinations}; + return variables; +} + +std::map OverridesFrom( + const std::string& text, + const EnvironmentLookup& base, + std::vector* problemsOut) { + std::map entries; + + const auto refuse = [problemsOut](std::string variable, std::string value, + std::string reason) { + if (problemsOut == nullptr) return; + ConfigurationProblem problem; + problem.variable = std::move(variable); + problem.value = std::move(value); + problem.reason = std::move(reason); + problem.fromContext = true; + problemsOut->push_back(std::move(problem)); + }; + + std::size_t at = 0; + for (;;) { + const std::size_t separator = text.find(';', at); + const std::string entry = Trim(text.substr( + at, separator == std::string::npos ? std::string::npos : separator - at)); + + if (!entry.empty()) { + const std::size_t equals = entry.find('='); + if (equals == std::string::npos) { + // Reported with an empty variable: there is no name to report + // it under, and inventing one would point the reader at a + // setting the entry never named. + refuse(std::string(), entry, "not a NAME=value entry"); + } else { + const std::string name = Trim(entry.substr(0, equals)); + const std::string value = Trim(entry.substr(equals + 1)); + if (!Contains(ConfiguredVariables(), name)) { + refuse(name, value, "not a variable this resolver reads"); + } else if (!Contains(ContextVariables(), name)) { + refuse(name, value, + "shared by every stage in the process, so it is read " + "from the environment and not from a context"); + } else { + // Said before the value is judged, so it says only what + // is true before then: the earlier value is gone. Whether + // the last one is used is the parser's to say, and if it + // is refused that is reported as its own problem below. + if (entries.find(name) != entries.end() && problemsOut != nullptr) { + ConfigurationProblem repeated = + Adjusted(name.c_str(), value, + "set more than once in one context; only the " + "last value is considered"); + repeated.fromContext = true; + problemsOut->push_back(std::move(repeated)); + } + entries[name] = value; + } + } + } + + if (separator == std::string::npos) break; + at = separator + 1; + } + + // The values, through the parsers the environment's go through, all at + // once and over `base`: a coalescing gap is capped against a request + // ceiling and a block size, and judging the context's value against the + // built-in default of the other would warn about an adjustment that is not + // the one applied. Only problems with the context's own variables are the + // context's to report; the environment's were reported when it was read. + std::vector valueProblems; + const ResolverConfiguration applied = + ConfigurationFrom(Layered(entries, base), &valueProblems); + for (ConfigurationProblem& problem : valueProblems) { + if (entries.find(problem.variable) == entries.end()) continue; + // A refused value leaves the context, so that what the context carries + // -- and what it compares and hashes by -- is what is in force. An + // adjusted one stays, and is adjusted again wherever it is applied. + if (!problem.adjusted) entries.erase(problem.variable); + problem.fromContext = true; + if (problemsOut != nullptr) problemsOut->push_back(std::move(problem)); + } + + // And what stays is kept as the parser read it rather than as it was + // written. `060000` and `60000` are one deadline, and `private, public` + // and `public,private` one policy; a context that compared them unequal + // would be two contexts to every table OpenUSD keys on one -- a stage cache + // that missed, a layer stack built twice. + for (auto& entry : entries) { + const std::string& name = entry.first; + const usdasset::http::HttpOptions& transport = applied.transport; + if (name == kConnectTimeout) { + entry.second = std::to_string(transport.connectTimeoutMs); + } else if (name == kReadTimeout) { + entry.second = std::to_string(transport.responseTimeoutMs); + } else if (name == kTotalTimeout) { + entry.second = std::to_string(transport.transferTimeoutMs); + } else if (name == kMaxRetries) { + entry.second = std::to_string(transport.maxAttempts - 1); + } else if (name == kMaxRedirects) { + entry.second = std::to_string(transport.maxRedirects); + } else if (name == kDestinations) { + entry.second = DestinationsString(transport.destinations); + } else if (name == kCoalesceGap) { + entry.second = std::to_string(applied.cache.coalesceGapBlocks); + } else if (name == kMaxRequestBytes) { + entry.second = std::to_string(applied.cache.maxRequestBytes); + } + } + + return entries; +} + +std::string CanonicalContextString(const std::map& overrides) { + std::string text; + for (const auto& entry : overrides) { + if (!text.empty()) text += ';'; + text += entry.first; + text += '='; + text += entry.second; + } + return text; +} + +EnvironmentLookup LookupIn(const std::map& values) { + return [&values](const char* name, std::string* valueOut) { + const auto found = values.find(name); + if (found == values.end()) return false; + valueOut->assign(found->second); + return true; + }; +} + +EnvironmentLookup Layered(const std::map& overrides, + EnvironmentLookup base) { + return [&overrides, base = std::move(base)](const char* name, std::string* valueOut) { + const auto found = overrides.find(name); + if (found != overrides.end()) { + valueOut->assign(found->second); + return true; + } + return base ? base(name, valueOut) : false; + }; +} + +std::map Snapshot(const EnvironmentLookup& lookup) { + std::map values; + for (const char* name : ConfiguredVariables()) { + std::string value; + if (lookup(name, &value)) values.emplace(name, std::move(value)); + } + return values; +} + +std::string TransportFingerprint(const usdasset::http::HttpOptions& options) { + const usdasset::http::DestinationPolicy& reach = options.destinations; + std::string text; + text += "connect=" + std::to_string(options.connectTimeoutMs); + text += ";response=" + std::to_string(options.responseTimeoutMs); + text += ";transfer=" + std::to_string(options.transferTimeoutMs); + text += ";attempts=" + std::to_string(options.maxAttempts); + text += ";redirects=" + std::to_string(options.maxRedirects); + text += ";reach="; + text += reach.publicAddresses ? 'P' : '-'; + text += reach.privateNetworks ? 'R' : '-'; + text += reach.loopback ? 'L' : '-'; + text += reach.linkLocal ? 'K' : '-'; + text += reach.metadata ? 'M' : '-'; + text += ";agent=" + options.userAgent; + return text; +} + } // namespace usdhttpresolver diff --git a/plugins/http-resolver/src/Configuration.h b/plugins/http-resolver/src/Configuration.h index 17b6b49..e5faaf5 100644 --- a/plugins/http-resolver/src/Configuration.h +++ b/plugins/http-resolver/src/Configuration.h @@ -1,16 +1,19 @@ // SPDX-License-Identifier: Apache-2.0 // -// The environment-variable configuration surface of CONFIGURATION.md §2: the -// five transport bounds, which arrived in `v0.2.0`, the four cache variables, -// which arrived in `v0.3.0` with the cache they configure, and the two -// persistence variables, which arrive in `v0.4.0` with the tier they turn on. -// The `ArResolverContext` form arrives in `v0.6.0`. +// The configuration surface of CONFIGURATION.md: the five transport bounds, +// which arrived in `v0.2.0`, the four cache variables, which arrived in +// `v0.3.0` with the cache they configure, the two persistence variables, which +// arrived in `v0.4.0` with the tier they turn on, and the destination policy, +// which arrived in `v0.7.0` -- read from the environment for the process, and +// from an `ArResolverContext` for a stage. // // Parsing is separated from reading the environment, and from reporting, on // purpose. An `ArResolver` is constructed once per process by `Plug`, so the // interesting cases -- a value that does not parse, a value out of range, a // value that is zero -- are otherwise reachable only by a test that mutates the // process environment and then hopes about ordering. Here they are a table. +// The context form is the same table read through a different lookup, which is +// what keeps it one vocabulary rather than two. // // No OpenUSD header. `httpResolver_test_configuration` links this translation // unit alone. @@ -19,6 +22,7 @@ #define USDHTTPRESOLVER_CONFIGURATION_H #include +#include #include #include @@ -38,6 +42,17 @@ struct ConfigurationProblem { std::string variable; std::string value; ///< As set. These variables never carry a secret. std::string reason; + + /// The value was used after an adjustment -- rounded, capped, raised -- + /// rather than refused. The two are reported differently because they end + /// differently: an operator told "using the default" about a value that + /// was in fact used, rounded, has been told something false. + bool adjusted = false; + + /// Written in a resolver context rather than in the environment. A refused + /// context value leaves that stage on the environment's value, which is not + /// the same fallback as the environment's own, and the message says so. + bool fromContext = false; }; /// Reads one variable. Returns false when it is unset; an empty string is a @@ -46,18 +61,19 @@ struct ConfigurationProblem { using EnvironmentLookup = std::function; +/// Reads one variable from the process environment, and the only place this +/// bundle calls `getenv`. The resolver takes a `Snapshot` through it once, and +/// everything after that reads the snapshot. +bool ReadEnvironmentVariable(const char* name, std::string* valueOut); + /// The transport options `lookup` describes, starting from the defaults. /// -/// Every variable is independent: one bad value leaves the other four in force +/// Every variable is independent: one bad value leaves the others in force /// rather than discarding the whole configuration. usdasset::http::HttpOptions OptionsFrom( const EnvironmentLookup& lookup, std::vector* problemsOut); -/// The same, against the process environment. -usdasset::http::HttpOptions OptionsFromEnvironment( - std::vector* problemsOut); - /// The cache policy `lookup` describes, starting from the shipped defaults. /// /// The values the defaults are is a measured question and its answer is @@ -95,13 +111,81 @@ ResolverConfiguration ConfigurationFrom( const EnvironmentLookup& lookup, std::vector* problemsOut); -ResolverConfiguration ConfigurationFromEnvironment( - std::vector* problemsOut); - /// The variables this version reads, in the order CONFIGURATION.md lists them. /// Exposed so a test asserts the set rather than restating it. const std::vector& ConfiguredVariables(); +// --- the context form ---------------------------------------------------------- + +/// The variables a resolver context may set: the ones that bind a reader or a +/// wrap, and are therefore a property of whoever opened the asset. +/// +/// Not the others, and the reason is the same for all four. The block store and +/// the persistent tier are shared by every stage in the process -- one budget, +/// CACHE.md §7, and one directory -- and the store's stripes are sized for one +/// block size, eight blocks to a stripe. A stage that asked for blocks larger +/// than a stripe would fetch each one and watch it evicted on arrival. So the +/// block size, the two budgets, and the directory are the environment's alone. +const std::vector& ContextVariables(); + +/// Parses a context string: `NAME=value` entries separated by `;`, each `NAME` +/// one of `ContextVariables()` spelled exactly as the environment spells it. +/// +/// Returns the entries admitted, keyed by name, each value in its canonical +/// spelling -- the number the parser read, or the destination classes in a +/// fixed order -- so that two contexts that say the same thing are equal. One +/// vocabulary rather than two: a value is checked by the parser the +/// environment's value would go through, over `base` (the environment the +/// context will be layered on), and refused or adjusted for the same reasons. +/// Whitespace around an entry, a name, or a value is tolerated, and so is an +/// empty entry -- a trailing `;` is what concatenation leaves behind. A name +/// set twice considers only the last value, the way an environment assignment +/// would, and says so. +/// +/// Everything not admitted is a problem marked `fromContext`, and the stage +/// that binds the context takes the environment's value for it instead. +std::map OverridesFrom( + const std::string& text, + const EnvironmentLookup& base, + std::vector* problemsOut); + +/// The canonical spelling of a set of overrides: `NAME=value` entries, sorted +/// by name, separated by `;`. Two contexts that set the same values print the +/// same, which is what a debug string and a `repr` are for. +std::string CanonicalContextString(const std::map& overrides); + +/// A lookup over a fixed set of values. +/// +/// It refers to `values` rather than copying it -- a lookup is built per call +/// under a context and a copy of the environment per call is a cost with no +/// purpose -- so `values` must outlive it. +EnvironmentLookup LookupIn(const std::map& values); + +/// A lookup that answers from `overrides` first and from `base` after: +/// CONFIGURATION.md §4's precedence, context over environment over default, +/// as a function. Refers to `overrides`, which must outlive it. +EnvironmentLookup Layered(const std::map& overrides, + EnvironmentLookup base); + +/// The value of every variable this version reads that `lookup` has. +/// +/// The resolver takes one of these of its environment at construction, and +/// resolves a context against it rather than against `getenv`. CONFIGURATION.md +/// §4: resolved at bind time, not per request -- a host that mutates its +/// environment mid-session must not change what a stage opened an hour ago is +/// configured by. +std::map Snapshot(const EnvironmentLookup& lookup); + +/// Equal for two option sets exactly when a reader opened under one may be +/// handed to a caller configured with the other. +/// +/// Every field a reader carries for its lifetime is in it -- the deadlines, the +/// retry and redirect bounds, the destination policy -- because a reader keeps +/// the options it was opened with. Handing one opened under a permissive policy +/// to a stage whose context refuses that destination would let the stage read +/// from somewhere its own policy forbids. +std::string TransportFingerprint(const usdasset::http::HttpOptions& options); + } // namespace usdhttpresolver #endif // USDHTTPRESOLVER_CONFIGURATION_H diff --git a/plugins/http-resolver/src/Context.cpp b/plugins/http-resolver/src/Context.cpp new file mode 100644 index 0000000..3f34e64 --- /dev/null +++ b/plugins/http-resolver/src/Context.cpp @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: Apache-2.0 + +#include "Context.h" + +#include +#include +#include + +#include "Configuration.h" + +#ifdef PXR_PYTHON_SUPPORT_ENABLED +#include "pxr/base/tf/pyLock.h" +#include "pxr/base/tf/pyUtils.h" +#include "pxr/external/boost/python/object.hpp" +#include "pxr/external/boost/python/str.hpp" +#include "pxr/external/boost/python/to_python_converter.hpp" +#endif + +PXR_NAMESPACE_USING_DIRECTIVE + +namespace usdhttpresolver { + +HttpResolverContext::HttpResolverContext(std::map overrides) + : _overrides(std::move(overrides)) {} + +std::string HttpResolverContext::GetAsString() const { + return CanonicalContextString(_overrides); +} + +size_t hash_value(const HttpResolverContext& context) { + return std::hash()(context.GetAsString()); +} + +std::string ArGetDebugString(const HttpResolverContext& context) { + return "HttpResolverContext(" + context.GetAsString() + ")"; +} + +#ifdef PXR_PYTHON_SUPPORT_ENABLED +namespace { + +struct HttpResolverContextToPython { + static PyObject* convert(const HttpResolverContext& context) { + return pxr_boost::python::incref( + pxr_boost::python::str(context.GetAsString()).ptr()); + } +}; + +} // namespace +#endif + +void HttpResolverContextEnsurePythonConversion() { +#ifdef PXR_PYTHON_SUPPORT_ENABLED + // Not `std::call_once`, because the question is not "has this been + // attempted" but "has this been done": a first call made before the host + // started Python must not use up the only chance. + // + // And no lock of its own. The flag is written only while the GIL is held, + // so the GIL is what serializes registration; the atomic load before it is + // only the fast path, so that a context created after registration does + // not queue for the interpreter at all. + static std::atomic registered{false}; + if (registered.load(std::memory_order_acquire) || !TfPyIsInitialized()) return; + + TfPyLock python; + if (registered.load(std::memory_order_relaxed)) return; + pxr_boost::python::to_python_converter(); + registered.store(true, std::memory_order_release); +#endif +} + +} // namespace usdhttpresolver diff --git a/plugins/http-resolver/src/Context.h b/plugins/http-resolver/src/Context.h new file mode 100644 index 0000000..c8b9241 --- /dev/null +++ b/plugins/http-resolver/src/Context.h @@ -0,0 +1,112 @@ +// SPDX-License-Identifier: Apache-2.0 +// +// The resolver context: per-stage configuration, CONFIGURATION.md §1 and §4. +// +// The environment is a bootstrap. A host that opens two stages against two +// servers under two policies cannot be served by a process-global, and the +// `ArResolverContext` a stage is opened with is where OpenUSD already keeps +// what differs between stages. This is the object that goes in it. +// +// It is created from a string and from nothing else: +// +// ArGetResolver().CreateContextFromString( +// "https", "USD_HTTP_RESOLVER_DESTINATIONS=public; " +// "USD_HTTP_RESOLVER_TOTAL_TIMEOUT_MS=60000") +// +// No header of this repository reaches a host that way, which is the same +// property ADR-0001 holds consumers to, extended to the hosts that configure +// them: the names are the environment's, and the entry point is OpenUSD's. +// +// What it carries is the overrides as written, after validation. It does not +// carry the configuration they produce, because that depends on the +// environment the resolver snapshot at construction, and a context is a value +// that two resolvers -- or two runs -- must compare equal on the strength of +// what it says rather than what it happened to combine with. + +#ifndef USDHTTPRESOLVER_CONTEXT_H +#define USDHTTPRESOLVER_CONTEXT_H + +#include +#include +#include + +#include "pxr/pxr.h" +#include "pxr/usd/ar/defineResolverContext.h" +#include "pxr/usd/ar/resolverContext.h" + +// In this bundle's namespace and not OpenUSD's. `ArResolverContext` finds a +// context object by comparing type names, so a class called +// `HttpResolverContext` in OpenUSD's namespace would be the same type, to it, +// as any other plugin's class of that name -- and the object one resolver +// found would be cast to the other's layout. The namespace is part of the name +// it compares. +namespace usdhttpresolver { + +class HttpResolverContext { +public: + /// No overrides: a stage bound to this is configured by the environment, + /// exactly as one bound to no context at all. + HttpResolverContext() = default; + + /// `overrides` as `OverridesFrom` admitted them: names from + /// `ContextVariables()`, values the environment's parser accepts. + explicit HttpResolverContext(std::map overrides); + + const std::map& GetOverrides() const noexcept { + return _overrides; + } + + /// The canonical spelling: sorted, `;`-separated `NAME=value` entries. Two + /// contexts that say the same thing print the same, whatever order and + /// whitespace they were written in. + std::string GetAsString() const; + + bool operator<(const HttpResolverContext& other) const { + return _overrides < other._overrides; + } + bool operator==(const HttpResolverContext& other) const { + return _overrides == other._overrides; + } + bool operator!=(const HttpResolverContext& other) const { + return !(*this == other); + } + +private: + std::map _overrides; +}; + +size_t hash_value(const HttpResolverContext& context); + +/// Found by argument-dependent lookup from `ArResolverContext`'s own debug +/// string, which would otherwise print a type name and an address. +std::string ArGetDebugString(const HttpResolverContext& context); + +/// Makes a context readable from Python. +/// +/// `Ar.ResolverContext` hands its objects to Python one at a time, through a +/// to-Python conversion, and an object with none cannot be handed. Measured +/// without this: `ctx.Get()` raises `TypeError: No to_python (by-value) +/// converter found`, and `Usd.Stage.__repr__`, which includes its resolver +/// context, prints `pathResolverContext=` for any stage opened +/// with an `http` context -- in usdview's interpreter, in a pipeline script, +/// wherever Python looks at the stage. The conversion is to the canonical +/// string, which is what the object *is*; a Python class for it would be a +/// binding module this bundle does not otherwise need. +/// +/// Registered once, and only once Python is running: a C++ host that never +/// starts an interpreter never pays for it, and one that does gets it at the +/// first context created afterwards. A no-op in a build without Python. +/// +/// The GIL is the lock. A Python caller of `CreateContextFromString` already +/// holds it when it arrives here, so any other lock taken first would be +/// taken in the opposite order by a C++ thread doing the same thing, and the +/// two would wait on each other forever. +void HttpResolverContextEnsurePythonConversion(); + +} // namespace usdhttpresolver + +PXR_NAMESPACE_OPEN_SCOPE +AR_DECLARE_RESOLVER_CONTEXT(usdhttpresolver::HttpResolverContext); +PXR_NAMESPACE_CLOSE_SCOPE + +#endif // USDHTTPRESOLVER_CONTEXT_H diff --git a/plugins/http-resolver/src/HttpResolver.cpp b/plugins/http-resolver/src/HttpResolver.cpp index 4900736..35f2b66 100644 --- a/plugins/http-resolver/src/HttpResolver.cpp +++ b/plugins/http-resolver/src/HttpResolver.cpp @@ -2,6 +2,7 @@ #include "HttpResolver.h" +#include #include #include #include @@ -13,6 +14,7 @@ #include "pxr/usd/ar/writableAsset.h" #include "Configuration.h" +#include "Context.h" #include "Diagnostics.h" #include "Identifier.h" #include "Identity.h" @@ -43,12 +45,27 @@ usdasset::Status UnsupportedWrite() { } // namespace -HttpResolver::HttpResolver() { +HttpResolver::HttpResolver() = default; + +HttpResolver::~HttpResolver() = default; + +void HttpResolver::_EnsureConfigured() const { + std::call_once(_configureOnce, [this] { _Configure(); }); +} + +void HttpResolver::_Configure() const { + // The environment, once. Everything this resolver is configured by -- the + // base configuration here, and every context resolved later -- reads this + // snapshot rather than `getenv`, so one process has one environment. + _environment = usdhttpresolver::Snapshot(&usdhttpresolver::ReadEnvironmentVariable); + std::vector problems; const usdhttpresolver::ResolverConfiguration configuration = - usdhttpresolver::ConfigurationFromEnvironment(&problems); - _options = configuration.transport; - _cacheOptions = configuration.cache.Normalized(); + usdhttpresolver::ConfigurationFrom(usdhttpresolver::LookupIn(_environment), + &problems); + _base.transport = configuration.transport; + _base.cache = configuration.cache.Normalized(); + _base.fingerprint = usdhttpresolver::TransportFingerprint(_base.transport); // The budget belongs to the process store rather than to this resolver, so // it is applied where it lives. Refused only when something is already bound @@ -56,10 +73,10 @@ HttpResolver::HttpResolver() { // stage opens cannot happen -- and if it somehow does, the store keeps the // budget it has and says so rather than being rebuilt underneath a live // reader. - if (!usdasset::cache::BlockCache::ConfigureProcess(_cacheOptions)) { + if (!usdasset::cache::BlockCache::ConfigureProcess(_base.cache)) { problems.push_back( {"USD_HTTP_RESOLVER_CACHE_BUDGET", - std::to_string(_cacheOptions.budgetBytes), + std::to_string(_base.cache.budgetBytes), "the process block store was already in use; its budget and block " "size were left as they were"}); } @@ -79,15 +96,14 @@ HttpResolver::HttpResolver() { } for (const usdhttpresolver::ConfigurationProblem& problem : problems) { - // At first use, per CONFIGURATION.md §2, which for a process-global - // surface is when the resolver is constructed. A typo that silently - // does nothing is worse than one that is reported. + // At first use, per CONFIGURATION.md §2 -- which is exactly when this + // runs. A typo that silently does nothing is worse than one that is + // reported, and a report in a process that never used the resolver is + // noise about a setting nothing read. usdhttpresolver::ReportConfigurationProblem(problem); } } -HttpResolver::~HttpResolver() = default; - std::string HttpResolver::_CreateIdentifier( const std::string& assetPath, const ArResolvedPath& anchorAssetPath) const { @@ -111,11 +127,43 @@ ArResolvedPath HttpResolver::_Resolve(const std::string& assetPath) const { usdhttpresolver::CreateIdentifier(assetPath, std::string()); if (identifier.empty()) return ArResolvedPath(); - const std::shared_ptr<_Opened> entry = _GetOrCreate(identifier); + // Under the stage's own configuration, when one is bound. A resolve under a + // context that refuses a destination fails here, and -- because the path is + // context-dependent -- that failure is what OpenUSD's layer registry acts + // on, even for a layer another stage has already loaded. + const _Effective effective = _EffectiveConfiguration(); + const std::string key = _OpenKey(identifier, effective); + + // Inside a scope, the scope's answer for this identifier under this + // configuration, if it has one. Looked up under the scope's lock and + // resolved outside it: a round trip under a lock every thread of the scope + // shares would serialize the composition the scope exists to speed up. + // Two threads that miss together are single-flighted below, by the + // retained entry, and the second insertion is a no-op. + const std::shared_ptr<_ResolveCache> scope = _resolveCache.GetCurrentCache(); + if (scope) { + std::lock_guard lock(scope->mutex); + const auto found = scope->resolved.find(key); + if (found != scope->resolved.end()) return found->second; + } + + const ArResolvedPath resolved = _ResolveOnce(identifier, key, effective); + + if (scope) { + std::lock_guard lock(scope->mutex); + scope->resolved.emplace(key, resolved); + } + return resolved; +} + +ArResolvedPath HttpResolver::_ResolveOnce(const std::string& identifier, + const std::string& key, + const _Effective& effective) const { + const std::shared_ptr<_Opened> entry = _GetOrCreate(key); std::lock_guard lock(entry->mutex); if (!entry->opened) { - entry->result = usdasset::http::Open(identifier, _options); + entry->result = usdasset::http::Open(identifier, effective.transport); entry->opened = true; if (entry->result.reader) { // Copied off the reader while it is still here. The reader leaves @@ -135,7 +183,7 @@ ArResolvedPath HttpResolver::_Resolve(const std::string& assetPath) const { // Remembered here rather than at the open in `_OpenAsset`, because that // reader is handed out once and the consumer that asks for its identity // asks after it is gone. RESOLVER.md §3. - _RememberIdentity(identifier, entry->metadata); + _RememberIdentity(identifier, entry->metadata, effective.transport.destinations); return ArResolvedPath(identifier); } @@ -147,7 +195,7 @@ ArResolvedPath HttpResolver::_Resolve(const std::string& assetPath) const { // it and a third has opened the identifier successfully, and erasing by key // would throw away that third thread's reader. const usdasset::Status status = entry->result.status; - _Forget(identifier, entry); + _Forget(key, entry); if (status.code != usdasset::StatusCode::NotFound) { usdhttpresolver::Report(status, identifier); @@ -167,9 +215,15 @@ std::shared_ptr HttpResolver::_OpenAsset( resolvedPath.GetPathString(), std::string()); if (identifier.empty()) return nullptr; + // The reader `_Resolve` retained is taken only when it was opened the way + // this call would open it. Under a different context -- a narrower + // destination policy, a shorter deadline -- it is left for a caller it + // fits, and this call opens its own. + const _Effective effective = _EffectiveConfiguration(); + std::unique_ptr reader; - if (const std::shared_ptr<_Opened> entry = _Take(identifier)) { + if (const std::shared_ptr<_Opened> entry = _Take(_OpenKey(identifier, effective))) { std::lock_guard lock(entry->mutex); reader = std::move(entry->result.reader); } @@ -178,7 +232,7 @@ std::shared_ptr HttpResolver::_OpenAsset( // Either nothing resolved this identifier in this process, or the // reader `_Resolve` captured has already been handed to somebody. usdasset::http::HttpOpenResult result = - usdasset::http::Open(identifier, _options); + usdasset::http::Open(identifier, effective.transport); if (!result.reader) { usdhttpresolver::Report(result.status, identifier); return nullptr; @@ -191,7 +245,7 @@ std::shared_ptr HttpResolver::_OpenAsset( // free; a reader opened here may never have been resolved through this // process at all, and this is the only point at which its identity is // known. - _RememberIdentity(identifier, reader->Metadata()); + _RememberIdentity(identifier, reader->Metadata(), effective.transport.destinations); // Captured before the reader is moved from, and valid for as long as the // reader is: it is a member of the reader's own implementation, and the @@ -221,7 +275,7 @@ std::shared_ptr HttpResolver::_OpenAsset( usdasset::OpenResult opened; opened.reader = std::unique_ptr(reader.release()); usdasset::OpenResult cached = usdasset::cache::WrapAsset( - std::move(opened), metrics, _cacheOptions, nullptr); + std::move(opened), metrics, effective.cache, nullptr); if (!cached.reader) { usdhttpresolver::Report(cached.status, identifier); return nullptr; @@ -267,7 +321,8 @@ ArAssetInfo HttpResolver::_GetAssetInfo( // happened, and asset info is not the call that should discover a dead // origin -- for a layer being reloaded against one, that is a second // identical round trip behind the one `_Resolve` has just paid for. - if (!_IdentityFor(identifier, resolved, &metadata, &contradicted)) { + if (!_IdentityFor(identifier, resolved, _EffectiveConfiguration(), &metadata, + &contradicted)) { return info; } @@ -307,9 +362,88 @@ std::string HttpResolver::_GetExtension(const std::string& assetPath) const { return usdhttpresolver::ExtensionOf(assetPath); } +ArResolverContext HttpResolver::_CreateContextFromString( + const std::string& contextStr) const { + // Validated over the environment it will be layered on, which has to have + // been read for that. + _EnsureConfigured(); + + std::vector problems; + std::map overrides = usdhttpresolver::OverridesFrom( + contextStr, usdhttpresolver::LookupIn(_environment), &problems); + + // Reported here and nowhere else. A context is created once and bound many + // times, often from worker threads, and a warning per bind would be one + // typo rendered once per composed prim. + for (const usdhttpresolver::ConfigurationProblem& problem : problems) { + usdhttpresolver::ReportConfigurationProblem(problem); + } + + // Before the context exists, so that the first `repr` of a stage opened + // with it can already print it (Context.h). + usdhttpresolver::HttpResolverContextEnsurePythonConversion(); + + // A context even when nothing was admitted. An empty one configures a + // stage exactly as the environment does, and returning no context at all + // would be indistinguishable, to the host, from a resolver that does not + // implement contexts -- when what happened is that it read the string and + // said what was wrong with it. + return ArResolverContext(usdhttpresolver::HttpResolverContext(std::move(overrides))); +} + +void HttpResolver::_BeginCacheScope(VtValue* cacheScopeData) { + _resolveCache.BeginCacheScope(cacheScopeData); +} + +void HttpResolver::_EndCacheScope(VtValue* cacheScopeData) { + _resolveCache.EndCacheScope(cacheScopeData); +} + +bool HttpResolver::_IsContextDependentPath(const std::string& assetPath) const { + // Every path that reaches this resolver is one of its own: the dispatching + // resolver routes by scheme. See the header for why the answer is yes. + (void)assetPath; + return true; +} + +HttpResolver::_Effective HttpResolver::_EffectiveConfiguration() const { + _EnsureConfigured(); + + const usdhttpresolver::HttpResolverContext* context = + _GetCurrentContextObject(); + if (context == nullptr || context->GetOverrides().empty()) return _base; + + // Resolved per call rather than cached per context. It is a dozen short + // string parses against a request that crosses a network, and a cache + // keyed by context would be a second table to bound, lock, and get wrong + // for no measurable return. No problems are collected: the context's were + // reported when it was created, and the environment's when this resolver + // was. + const usdhttpresolver::ResolverConfiguration configuration = + usdhttpresolver::ConfigurationFrom( + usdhttpresolver::Layered(context->GetOverrides(), + usdhttpresolver::LookupIn(_environment)), + nullptr); + + _Effective effective; + effective.transport = configuration.transport; + effective.cache = configuration.cache.Normalized(); + effective.fingerprint = usdhttpresolver::TransportFingerprint(effective.transport); + return effective; +} + +std::string HttpResolver::_OpenKey(const std::string& identifier, + const _Effective& effective) { + // A newline cannot appear in a normalized identifier -- it is a control + // byte, and normalization encodes those -- so the two halves cannot run + // into each other. + return identifier + '\n' + effective.fingerprint; +} + bool HttpResolver::_RememberIdentity( const std::string& identifier, - const usdasset::AssetMetadata& metadata) const { + const usdasset::AssetMetadata& metadata, + const usdasset::http::DestinationPolicy& reachedUnder) const { std::lock_guard lock(_identityMutex); // The fingerprint first, because it is the half that decides an answer's @@ -333,8 +467,16 @@ bool HttpResolver::_RememberIdentity( const auto found = _identities.find(identifier); if (found != _identities.end()) { found->second.metadata = metadata; + std::vector& policies = + found->second.reachedUnder; + // A handful at most -- one per distinct policy that reached it -- so a + // linear scan is the whole data structure. + if (std::find(policies.begin(), policies.end(), reachedUnder) == + policies.end()) { + policies.push_back(reachedUnder); + } } else { - _identities.emplace(identifier, _Identity{metadata}); + _identities.emplace(identifier, _Identity{metadata, {reachedUnder}}); _identityOrder.push_back(identifier); while (_identityOrder.size() > kMaxRememberedIdentities) { _identities.erase(_identityOrder.front()); @@ -346,6 +488,7 @@ bool HttpResolver::_RememberIdentity( } bool HttpResolver::_KnownIdentity(const std::string& identifier, + const usdasset::http::DestinationPolicy& policy, usdasset::AssetMetadata* metadata, bool* contradicted) const { std::lock_guard lock(_identityMutex); @@ -353,6 +496,15 @@ bool HttpResolver::_KnownIdentity(const std::string& identifier, const auto found = _identities.find(identifier); if (found == _identities.end()) return false; + const std::vector& policies = + found->second.reachedUnder; + const bool reachable = + std::any_of(policies.begin(), policies.end(), + [&policy](const usdasset::http::DestinationPolicy& reached) { + return policy.Covers(reached); + }); + if (!reachable) return false; + const auto fingerprint = _fingerprints.find(identifier); *metadata = found->second.metadata; @@ -363,21 +515,30 @@ bool HttpResolver::_KnownIdentity(const std::string& identifier, bool HttpResolver::_IdentityFor(const std::string& identifier, bool mayOpen, + const _Effective& effective, usdasset::AssetMetadata* metadata, bool* contradicted) const { - if (_KnownIdentity(identifier, metadata, contradicted)) return true; + // Only an identity this caller could have reached itself. A stage whose + // context refuses a destination is not told the size and token of an + // asset another stage opened there -- it is told what it would be told had + // nobody opened it, which is, with an empty resolved path, nothing. + if (_KnownIdentity(identifier, effective.transport.destinations, metadata, + contradicted)) { + return true; + } if (!mayOpen) return false; // Nothing in this process has opened it, or the answer has aged out of the // bounded table. Opening it here goes through the same retained table // `_Resolve` fills, so the metadata request this costs is the one an // `_OpenAsset` that follows would have made rather than an extra one. - const std::shared_ptr<_Opened> entry = _GetOrCreate(identifier); + const std::string key = _OpenKey(identifier, effective); + const std::shared_ptr<_Opened> entry = _GetOrCreate(key); { std::lock_guard lock(entry->mutex); if (!entry->opened) { - entry->result = usdasset::http::Open(identifier, _options); + entry->result = usdasset::http::Open(identifier, effective.transport); entry->opened = true; if (entry->result.reader) { entry->metadata = entry->result.reader->Metadata(); @@ -391,7 +552,8 @@ bool HttpResolver::_IdentityFor(const std::string& identifier, // no identity, and the caller would hand a consumer an empty // `ArAssetInfo` for an asset it is about to read. *metadata = entry->metadata; - *contradicted = _RememberIdentity(identifier, *metadata); + *contradicted = _RememberIdentity(identifier, *metadata, + effective.transport.destinations); return true; } } @@ -400,30 +562,30 @@ bool HttpResolver::_IdentityFor(const std::string& identifier, // nothing is posted: this is a question about identity rather than an // operation on the asset, and the operation that follows reports the same // failure with the same code. One fault rendered twice is noise. - _Forget(identifier, entry); + _Forget(key, entry); return false; } std::shared_ptr HttpResolver::_GetOrCreate( - const std::string& identifier) const { + const std::string& key) const { // Declared before the lock, and therefore destroyed after it is released. // // That ordering is the whole point of this vector. Dropping the last // reference to an evicted entry runs `~HttpAssetReader`, which tears down a // connection -- a socket close, and a TLS shutdown that can put bytes on the // wire. Doing that while holding the table lock would block every unrelated - // identifier's resolution behind one eviction, which is exactly the "no lock + // key's resolution behind one eviction, which is exactly the "no lock // across a request" property RESOLVER.md §7 requires. std::vector> evicted; std::lock_guard lock(_tableMutex); - const auto found = _table.find(identifier); + const auto found = _table.find(key); if (found != _table.end()) return found->second; std::shared_ptr<_Opened> entry = std::make_shared<_Opened>(); - _table.emplace(identifier, entry); - _order.push_back(identifier); + _table.emplace(key, entry); + _order.push_back(key); while (_order.size() > kMaxRetainedOpens) { // The evicted entry may still be held by a thread that is opening it; @@ -440,16 +602,16 @@ std::shared_ptr HttpResolver::_GetOrCreate( } std::shared_ptr HttpResolver::_Take( - const std::string& identifier) const { + const std::string& key) const { std::lock_guard lock(_tableMutex); - const auto found = _table.find(identifier); + const auto found = _table.find(key); if (found == _table.end()) return nullptr; std::shared_ptr<_Opened> entry = found->second; _table.erase(found); for (auto it = _order.begin(); it != _order.end(); ++it) { - if (*it == identifier) { + if (*it == key) { _order.erase(it); break; } @@ -457,7 +619,7 @@ std::shared_ptr HttpResolver::_Take( return entry; } -void HttpResolver::_Forget(const std::string& identifier, +void HttpResolver::_Forget(const std::string& key, const std::shared_ptr<_Opened>& entry) const { // Same ordering argument as `_GetOrCreate`: whatever this drops is dropped // after the lock is released. A forgotten entry is a failed open and so @@ -467,7 +629,7 @@ void HttpResolver::_Forget(const std::string& identifier, std::lock_guard lock(_tableMutex); - const auto found = _table.find(identifier); + const auto found = _table.find(key); if (found == _table.end() || found->second != entry) { // Somebody else has already replaced this entry. Theirs is newer than // the failure being forgotten, and is not ours to discard. @@ -477,7 +639,7 @@ void HttpResolver::_Forget(const std::string& identifier, _table.erase(found); for (auto it = _order.begin(); it != _order.end(); ++it) { - if (*it == identifier) { + if (*it == key) { _order.erase(it); break; } diff --git a/plugins/http-resolver/src/HttpResolver.h b/plugins/http-resolver/src/HttpResolver.h index cd56896..010ff8a 100644 --- a/plugins/http-resolver/src/HttpResolver.h +++ b/plugins/http-resolver/src/HttpResolver.h @@ -17,6 +17,7 @@ #include #include +#include #include #include #include @@ -27,8 +28,11 @@ #include "pxr/usd/ar/assetInfo.h" #include "pxr/usd/ar/resolvedPath.h" #include "pxr/usd/ar/resolver.h" +#include "pxr/usd/ar/resolverContext.h" +#include "pxr/usd/ar/threadLocalScopedCache.h" #include "pxr/usd/ar/timestamp.h" +#include "Configuration.h" #include "usdAssetCache/CacheOptions.h" #include "usdAssetHttp/HttpAssetReader.h" #include "usdAssetIo/AssetReader.h" @@ -40,6 +44,16 @@ class ArWritableAsset; class HttpResolver final : public ArResolver { public: + /// Does nothing, deliberately. Configuration happens at first use. + /// + /// Because this resolver implements contexts, OpenUSD constructs it in + /// every process that binds a context -- which is every process that opens + /// a stage, local ones included -- and may construct two at once and keep + /// one. A constructor that read the environment, rebuilt the process block + /// store, created the persistent cache directory, and posted warnings would + /// do all of that for a host that never names an `http` URL, twice on a + /// race. RESOLVER.md §1 says installing this bundle never changes how a + /// local asset opens, and a directory appearing on disk is a change. HttpResolver(); ~HttpResolver() override; @@ -143,7 +157,79 @@ class HttpResolver final : public ArResolver { /// and is in fact a resolver bug. std::string _GetExtension(const std::string& assetPath) const override; + /// A context holding an `HttpResolverContext`, from `NAME=value` entries + /// separated by `;` (CONFIGURATION.md §4). Every entry that could not be + /// admitted is a warning here, once, at creation -- which is the first use + /// CONFIGURATION.md §2 means for a value written in a context. + /// + /// Reached through `ArGetResolver().CreateContextFromString("http", ...)` + /// or `"https"`: one type serves both schemes, so either name reaches it. + ArResolverContext _CreateContextFromString( + const std::string& contextStr) const override; + + /// True, for every identifier this resolver owns. + /// + /// Not because the *path* resolves differently under two contexts -- an + /// identifier resolves to itself -- but because whether it resolves at all + /// can: a stage whose context refuses private networks must not resolve an + /// intranet URL that another stage's context permits. And the answer here + /// is what OpenUSD's layer registry acts on. For a path that is not + /// context-dependent, `SdfLayer::FindOrOpen` finds an already-loaded layer + /// by identifier alone, whatever `Resolve` just said; for one that is, it + /// looks the layer up by the path `Resolve` returned, and a refusal finds + /// nothing. Answering false would let one stage's policy be walked past by + /// opening the same URL in another stage first. + bool _IsContextDependentPath(const std::string& assetPath) const override; + + /// Resolve caching within an `ArResolverScopedCache`, done here rather than + /// by OpenUSD. + /// + /// For a resolver that does not implement scoped caches, OpenUSD caches + /// `Resolve` on its behalf -- keyed by the path alone. That key is wrong + /// here: under a scope that spans two stages, a path resolved under a + /// permissive context would be answered from the cache under a refusing + /// one, without this resolver being asked, and the layer registry would + /// then find the layer by the path it returned. So the cache is this + /// resolver's, keyed as the retained opens are, by identifier and + /// configuration. It keeps what OpenUSD's kept -- a failure included, for + /// the life of the scope -- because composition resolves one reference + /// once per arc, and a scope is what stops that being one request per arc. + void _BeginCacheScope(VtValue* cacheScopeData) override; + void _EndCacheScope(VtValue* cacheScopeData) override; + private: + /// Reads the environment, configures the process stores, and reports what + /// was wrong -- once, at the first call that needs any of it. See the + /// constructor for why not before. + void _EnsureConfigured() const; + void _Configure() const; + + /// What one call is configured by: the bound context's overrides over the + /// environment, or the environment alone when nothing is bound. + struct _Effective { + usdasset::http::HttpOptions transport; + usdasset::cache::CacheOptions cache; ///< Normalized. + /// `TransportFingerprint(transport)`, which keys the retained opens: + /// a reader keeps the options it was opened with, so it may only be + /// handed to a caller that would have opened it the same way. + std::string fingerprint; + }; + + /// Resolved from the context bound on the calling thread, if it holds an + /// `HttpResolverContext`. Read per call and never stored: the same + /// resolver serves every stage in the process, each on its own threads. + _Effective _EffectiveConfiguration() const; + + /// The key the retained-open table and the scoped resolve cache are + /// indexed by. + static std::string _OpenKey(const std::string& identifier, + const _Effective& effective); + + /// `_Resolve` without the scope: the round trip, retained per §2.3. + ArResolvedPath _ResolveOnce(const std::string& identifier, + const std::string& key, + const _Effective& effective) const; + /// One identifier's in-flight or completed open. /// /// The reader is the point. `_Resolve` has to open the asset in order to @@ -167,24 +253,25 @@ class HttpResolver final : public ArResolver { usdasset::AssetMetadata metadata; }; - /// Finds or creates the entry for `identifier`. The table lock is held for - /// the lookup and never across a request. - std::shared_ptr<_Opened> _GetOrCreate(const std::string& identifier) const; + /// Finds or creates the entry for `key` -- an `_OpenKey`, the identifier + /// and the options its reader would be opened with. The table lock is held + /// for the lookup and never across a request. + std::shared_ptr<_Opened> _GetOrCreate(const std::string& key) const; /// Removes an entry and returns it, so that a reader is handed out exactly /// once. A second `_OpenAsset` for one identifier opens again rather than /// sharing a reader that is already bound to a revision somebody else is /// mid-composition on. - std::shared_ptr<_Opened> _Take(const std::string& identifier) const; + std::shared_ptr<_Opened> _Take(const std::string& key) const; - /// Removes `identifier`'s entry, but only if it is still `entry`. + /// Removes `key`'s entry, but only if it is still `entry`. /// /// By identity rather than by name, because a failed resolve is forgotten /// and two threads can be holding one failed entry: the second arrives after /// the first has removed it and a third has opened the identifier /// successfully, and erasing by key would discard that third thread's /// reader. - void _Forget(const std::string& identifier, + void _Forget(const std::string& key, const std::shared_ptr<_Opened>& entry) const; /// What one identifier's most recent successful open discovered. @@ -196,6 +283,13 @@ class HttpResolver final : public ArResolver { /// consumer actually holds. struct _Identity { usdasset::AssetMetadata metadata; + + /// The destination policies this identity was reached under. Asset + /// info answers from memory only for a caller whose own policy covers + /// one of them -- one that could have reached the asset itself. A stage + /// whose context refuses a destination is not told the size and token + /// of an asset another stage opened there. + std::vector reachedUnder; }; /// The validator one identifier has been seen with, and whether it has ever @@ -225,10 +319,13 @@ class HttpResolver final : public ArResolver { /// permanent -- see `PublishIdentity` -- and because forgetting it is /// indistinguishable, from the inside, from the asset never having moved. bool _RememberIdentity(const std::string& identifier, - const usdasset::AssetMetadata& metadata) const; + const usdasset::AssetMetadata& metadata, + const usdasset::http::DestinationPolicy& reachedUnder) const; - /// The remembered identity for `identifier`, if there is one. + /// The remembered identity for `identifier`, if there is one that a caller + /// under `policy` could have reached. bool _KnownIdentity(const std::string& identifier, + const usdasset::http::DestinationPolicy& policy, usdasset::AssetMetadata* metadata, bool* contradicted) const; @@ -244,6 +341,7 @@ class HttpResolver final : public ArResolver { /// see `_GetAssetInfo`. bool _IdentityFor(const std::string& identifier, bool mayOpen, + const _Effective& effective, usdasset::AssetMetadata* metadata, bool* contradicted) const; @@ -256,15 +354,33 @@ class HttpResolver final : public ArResolver { /// never costs correctness. static constexpr std::size_t kMaxRetainedOpens = 64; - usdasset::http::HttpOptions _options; + mutable std::once_flag _configureOnce; - /// The block policy every asset this resolver opens is decorated with. + /// The environment, as it was when this resolver was first used. + /// + /// Kept rather than re-read, per CONFIGURATION.md §4: a context is resolved + /// against this, so a host that mutates its environment mid-session does + /// not change what a stage it opened earlier is configured by -- and the + /// environment's own problems were reported once, when it was read, rather + /// than again for every context resolved over them. Written once, under + /// `_configureOnce`, and only read after it. + mutable std::map _environment; + + /// What a call with no context bound is configured by. /// - /// Resolved once, at construction, from the environment. The blocks - /// themselves live in the process-wide store rather than here, because the - /// budget is process-wide and shared across assets (CACHE.md section 7) and - /// a store per resolver would not be one budget. - usdasset::cache::CacheOptions _cacheOptions; + /// The block policy in it is also the one the process store was built + /// for. The blocks live in the process-wide store rather than here, + /// because the budget is process-wide and shared across assets (CACHE.md + /// section 7) and a store per resolver would not be one budget. + mutable _Effective _base; + + /// One scope's resolutions, shared by every thread the scope was handed + /// to. Keyed by `_OpenKey`, never by path. + struct _ResolveCache { + std::mutex mutex; + std::unordered_map resolved; + }; + mutable ArThreadLocalScopedCache<_ResolveCache> _resolveCache; /// Remembered identities the process will hold before dropping the oldest. /// diff --git a/plugins/http-resolver/src/Identifier.h b/plugins/http-resolver/src/Identifier.h index d9ab82a..6edb898 100644 --- a/plugins/http-resolver/src/Identifier.h +++ b/plugins/http-resolver/src/Identifier.h @@ -62,7 +62,7 @@ bool IsClaimedScheme(std::string_view assetPath) noexcept; /// hides one in the part a query-string rule keeps. A URL that needs /// credentials therefore fails at the server with `AccessDenied` rather than /// succeeding with a secret in every log line; authentication is the -/// interception point in `v0.6.0`, not a URL component. +/// interception point in `v0.7.0`, not a URL component. /// /// Anchoring is RFC 3986 §5.2 reference resolution against `anchorAssetPath`, /// which is what makes a remote scene work at all: a layer published to a CDN diff --git a/plugins/http-resolver/src/Report.cpp b/plugins/http-resolver/src/Report.cpp index 1f06f36..966b861 100644 --- a/plugins/http-resolver/src/Report.cpp +++ b/plugins/http-resolver/src/Report.cpp @@ -40,6 +40,33 @@ void ReportRetries(std::uint64_t retryCount, std::string_view identifier) { } void ReportConfigurationProblem(const ConfigurationProblem& problem) { + // Four sentences rather than one with blanks in it, because the four end + // differently and the ending is the part an operator acts on. An adjusted + // value was used; a refused one was not; and a refused *context* value + // leaves its stage on the environment's value, which is a different + // fallback from the environment's own. + if (problem.fromContext) { + if (problem.variable.empty()) { + TF_WARN("a resolver context has an entry '%s', which is %s; it is " + "ignored", + problem.value.c_str(), problem.reason.c_str()); + } else if (problem.adjusted) { + TF_WARN("a resolver context sets %s to '%s': %s", + problem.variable.c_str(), problem.value.c_str(), + problem.reason.c_str()); + } else { + TF_WARN("a resolver context sets %s to '%s', which is %s; stages " + "bound to it use the environment's value or the default", + problem.variable.c_str(), problem.value.c_str(), + problem.reason.c_str()); + } + return; + } + if (problem.adjusted) { + TF_WARN("%s is set to '%s': %s", problem.variable.c_str(), + problem.value.c_str(), problem.reason.c_str()); + return; + } TF_WARN("%s is set to '%s', which is %s; using the default", problem.variable.c_str(), problem.value.c_str(), problem.reason.c_str()); diff --git a/plugins/http-resolver/src/Report.h b/plugins/http-resolver/src/Report.h index a5c6ab6..d733d19 100644 --- a/plugins/http-resolver/src/Report.h +++ b/plugins/http-resolver/src/Report.h @@ -31,9 +31,10 @@ void Report(const usdasset::Status& status, std::string_view identifier); /// the caller can hand it a counter unconditionally. void ReportRetries(std::uint64_t retryCount, std::string_view identifier); -/// Posts a warning for an environment variable that was set and could not be -/// used, per CONFIGURATION.md §2. The variable's value is included because -/// these five carry no secret and because a typo is unfindable otherwise. +/// Posts a warning for a configuration value that was set and could not be +/// used as written -- in the environment or in a resolver context -- per +/// CONFIGURATION.md §2. The value is included because no variable here carries +/// a secret and because a typo is unfindable otherwise. void ReportConfigurationProblem(const ConfigurationProblem& problem); } // namespace usdhttpresolver diff --git a/plugins/http-resolver/tests/test_configuration.cpp b/plugins/http-resolver/tests/test_configuration.cpp index a7bbf0a..27ac871 100644 --- a/plugins/http-resolver/tests/test_configuration.cpp +++ b/plugins/http-resolver/tests/test_configuration.cpp @@ -43,6 +43,77 @@ void TestDefaults() { CHECK_EQ(options.transferTimeoutMs, defaults.transferTimeoutMs); CHECK_EQ(options.maxAttempts, defaults.maxAttempts); CHECK_EQ(options.maxRedirects, defaults.maxRedirects); + CHECK(options.destinations == defaults.destinations); +} + +/// The destination policy of §10.2, as a set of address class names. +void TestDestinations() { + using usdasset::http::DestinationPolicy; + + struct Row { + const char* value; + bool publicAddresses; + bool privateNetworks; + bool loopback; + bool linkLocal; + }; + const Row accepted[] = { + {"public", true, false, false, false}, + {"private,loopback", false, true, true, false}, + // How a person writes a list. + {"public, private , loopback", true, true, true, false}, + {"link-local", false, false, false, true}, + {"public,private,loopback,link-local", true, true, true, true}, + {"public,private,loopback,link-local,metadata", true, true, true, true}, + // Repetition is redundant, not contradictory. + {"public,public", true, false, false, false}, + }; + for (const Row& row : accepted) { + std::vector problems; + const usdasset::http::HttpOptions options = OptionsFrom( + From({{"USD_HTTP_RESOLVER_DESTINATIONS", row.value}}), &problems); + if (!problems.empty()) { + std::fprintf(stderr, "FAIL %s:%d: '%s' was refused: %s\n", __FILE__, + __LINE__, row.value, problems[0].reason.c_str()); + ++::usdassettest::FailureCount(); + continue; + } + CHECK_EQ(options.destinations.publicAddresses, row.publicAddresses); + CHECK_EQ(options.destinations.privateNetworks, row.privateNetworks); + CHECK_EQ(options.destinations.loopback, row.loopback); + CHECK_EQ(options.destinations.linkLocal, row.linkLocal); + // Only a list that names it permits the metadata endpoints: permitting + // link-local is not permitting what lives there. + CHECK_EQ(options.destinations.metadata, + std::string(row.value).find("metadata") != std::string::npos); + } + + // Refused whole, and the default kept, loudly. An unknown name is not + // skipped: a policy that dropped the word it did not recognize is a + // different policy from the one that was written. + const char* const refused[] = { + "", + "public,,private", + "public,", + "Public", + "everything", + "public,internet", + "none", + "127.0.0.1", + }; + for (const char* value : refused) { + std::vector problems; + const usdasset::http::HttpOptions options = OptionsFrom( + From({{"USD_HTTP_RESOLVER_DESTINATIONS", value}}), &problems); + if (problems.size() != 1) { + std::fprintf(stderr, "FAIL %s:%d: '%s' produced %zu problem(s)\n", + __FILE__, __LINE__, value, problems.size()); + ++::usdassettest::FailureCount(); + continue; + } + CHECK(problems[0].variable == "USD_HTTP_RESOLVER_DESTINATIONS"); + CHECK(options.destinations == DestinationPolicy()); + } } void TestEachVariable() { @@ -128,10 +199,10 @@ void TestVariableSet() { const std::vector& variables = usdhttpresolver::ConfiguredVariables(); // Five transport bounds from `v0.2.0`, four cache variables from `v0.3.0`, - // and two persistence variables from `v0.4.0`, which is the whole of - // CONFIGURATION.md §2 except the metrics dump -- that one is read by - // usdAssetIo and not by this resolver. - CHECK_EQ(variables.size(), std::size_t{11}); + // two persistence variables from `v0.4.0`, and the destination policy from + // `v0.7.0`, which is the whole of CONFIGURATION.md §2 except the metrics + // dump -- that one is read by usdAssetIo and not by this resolver. + CHECK_EQ(variables.size(), std::size_t{12}); for (const char* name : variables) { CHECK(std::string(name).rfind("USD_HTTP_RESOLVER_", 0) == 0); // Every variable is a byte count or a bound except the cache directory, @@ -257,10 +328,330 @@ void TestPersistenceVariables() { CHECK_EQ(budgetOnly.persistence.budgetBytes, std::uint64_t{8388608}); } +// --- the context form ---------------------------------------------------------- + +/// A context string read over an environment with nothing set -- which is +/// what every case below means unless it says otherwise. +std::map OverridesFrom( + const std::string& text, std::vector* problemsOut) { + return usdhttpresolver::OverridesFrom(text, From({}), problemsOut); +} + +/// What a context may set, and what it may not. The four it may not are the +/// ones the process shares -- the block store, its block size, and the +/// persistent tier -- and a stage that set them would be setting them for +/// every other stage too. +void TestContextVariableSet() { + const std::vector& configured = usdhttpresolver::ConfiguredVariables(); + const std::vector& context = usdhttpresolver::ContextVariables(); + CHECK_EQ(context.size(), std::size_t{8}); + + for (const char* name : context) { + bool known = false; + for (const char* candidate : configured) { + if (std::string(name) == candidate) known = true; + } + CHECK(known); + } + const char* const processWide[] = { + "USD_HTTP_RESOLVER_BLOCK_SIZE", + "USD_HTTP_RESOLVER_CACHE_BUDGET", + "USD_HTTP_RESOLVER_PERSISTENT_CACHE_DIR", + "USD_HTTP_RESOLVER_PERSISTENT_CACHE_BUDGET", + }; + for (const char* name : processWide) { + for (const char* candidate : context) { + CHECK(std::string(name) != candidate); + } + } +} + +void TestContextStrings() { + { + // How a host writes one: across lines, with spaces, and with the + // trailing separator concatenation leaves behind. + std::vector problems; + const std::map overrides = OverridesFrom( + " USD_HTTP_RESOLVER_MAX_RETRIES = 0 ;\n" + " USD_HTTP_RESOLVER_DESTINATIONS = public, private ;\n", + &problems); + CHECK(problems.empty()); + CHECK_EQ(overrides.size(), std::size_t{2}); + CHECK_EQ(overrides.at("USD_HTTP_RESOLVER_MAX_RETRIES"), std::string("0")); + // Kept as the parser read it: the classes in their fixed order, with + // no spaces, whatever spacing they were written with. + CHECK_EQ(overrides.at("USD_HTTP_RESOLVER_DESTINATIONS"), + std::string("public,private")); + // Canonical: sorted by name, whatever order it was written in. + CHECK_EQ(usdhttpresolver::CanonicalContextString(overrides), + std::string("USD_HTTP_RESOLVER_DESTINATIONS=public,private;" + "USD_HTTP_RESOLVER_MAX_RETRIES=0")); + } + { + // Nothing is not a problem. + std::vector problems; + CHECK(OverridesFrom("", &problems).empty()); + CHECK(OverridesFrom(" ; ;", &problems).empty()); + CHECK(problems.empty()); + } + + // Each of these is refused, reported as the context's, and absent from + // what the context carries -- so that what it compares and hashes by is + // what is in force. + struct Refused { + const char* text; + const char* variable; + }; + const Refused refused[] = { + {"USD_HTTP_RESOLVER_BLOCK_SIZE=16384", "USD_HTTP_RESOLVER_BLOCK_SIZE"}, + {"USD_HTTP_RESOLVER_CACHE_BUDGET=1048576", "USD_HTTP_RESOLVER_CACHE_BUDGET"}, + {"USD_HTTP_RESOLVER_PERSISTENT_CACHE_DIR=/var/tmp/x", + "USD_HTTP_RESOLVER_PERSISTENT_CACHE_DIR"}, + {"USD_HTTP_RESOLVER_PERSISTENT_CACHE_BUDGET=8388608", + "USD_HTTP_RESOLVER_PERSISTENT_CACHE_BUDGET"}, + {"USD_HTTP_RESOLVER_METRICS_DUMP=1", "USD_HTTP_RESOLVER_METRICS_DUMP"}, + {"USD_HTTP_RESOLVER_NO_SUCH_THING=1", "USD_HTTP_RESOLVER_NO_SUCH_THING"}, + {"usd_http_resolver_max_retries=1", "usd_http_resolver_max_retries"}, + {"MAX_RETRIES=1", "MAX_RETRIES"}, + {"USD_HTTP_RESOLVER_MAX_RETRIES=abc", "USD_HTTP_RESOLVER_MAX_RETRIES"}, + {"USD_HTTP_RESOLVER_CONNECT_TIMEOUT_MS=0", "USD_HTTP_RESOLVER_CONNECT_TIMEOUT_MS"}, + {"USD_HTTP_RESOLVER_DESTINATIONS=everything", "USD_HTTP_RESOLVER_DESTINATIONS"}, + {"USD_HTTP_RESOLVER_DESTINATIONS=", "USD_HTTP_RESOLVER_DESTINATIONS"}, + {"=1", ""}, + {"USD_HTTP_RESOLVER_MAX_RETRIES", ""}, + }; + for (const Refused& row : refused) { + std::vector problems; + const std::map overrides = + OverridesFrom(row.text, &problems); + if (problems.size() != 1 || !overrides.empty()) { + std::fprintf(stderr, + "FAIL %s:%d: '%s' gave %zu problem(s) and %zu override(s)\n", + __FILE__, __LINE__, row.text, problems.size(), + overrides.size()); + ++::usdassettest::FailureCount(); + continue; + } + CHECK(problems[0].fromContext); + CHECK(!problems[0].adjusted); + CHECK_EQ(problems[0].variable, std::string(row.variable)); + } + + { + // One bad entry does not discard its neighbours. + std::vector problems; + const std::map overrides = OverridesFrom( + "USD_HTTP_RESOLVER_MAX_REDIRECTS=nonsense;USD_HTTP_RESOLVER_MAX_RETRIES=1", + &problems); + CHECK_EQ(problems.size(), std::size_t{1}); + CHECK_EQ(overrides.size(), std::size_t{1}); + CHECK(overrides.count("USD_HTTP_RESOLVER_MAX_RETRIES") == 1); + } + { + // Set twice: the last wins, as an environment assignment would, and + // the repetition is reported as an adjustment rather than a refusal. + std::vector problems; + const std::map overrides = OverridesFrom( + "USD_HTTP_RESOLVER_MAX_RETRIES=1; USD_HTTP_RESOLVER_MAX_RETRIES=2", + &problems); + CHECK_EQ(problems.size(), std::size_t{1}); + if (!problems.empty()) CHECK(problems[0].adjusted); + CHECK_EQ(overrides.at("USD_HTTP_RESOLVER_MAX_RETRIES"), std::string("2")); + } + { + // And when the last value is then refused, both are said, and neither + // claims the refused value is in force. The earlier value is gone + // either way: it was replaced before the replacement was judged. + std::vector problems; + const std::map overrides = OverridesFrom( + "USD_HTTP_RESOLVER_MAX_RETRIES=1; USD_HTTP_RESOLVER_MAX_RETRIES=abc", + &problems); + CHECK_EQ(problems.size(), std::size_t{2}); + CHECK(overrides.empty()); + for (const ConfigurationProblem& problem : problems) { + CHECK(problem.reason.find("is used") == std::string::npos); + } + } + { + // An adjusted value is kept and reported: a gap wider than a merged + // request can carry is capped wherever it is applied. + std::vector problems; + const std::map overrides = + OverridesFrom("USD_HTTP_RESOLVER_COALESCE_GAP=1024", &problems); + CHECK_EQ(problems.size(), std::size_t{1}); + if (!problems.empty()) { + CHECK(problems[0].adjusted); + CHECK(problems[0].fromContext); + } + CHECK_EQ(overrides.size(), std::size_t{1}); + } +} + +/// What a context carries is what the parser read, so two contexts that say +/// the same thing are one context -- to `ArResolverContext`'s equality, to its +/// hash, and so to every table OpenUSD keys on one. +void TestCanonicalValues() { + const char* const equivalent[][2] = { + {"USD_HTTP_RESOLVER_TOTAL_TIMEOUT_MS=060000", + "USD_HTTP_RESOLVER_TOTAL_TIMEOUT_MS=60000"}, + {"USD_HTTP_RESOLVER_DESTINATIONS=private, public", + "USD_HTTP_RESOLVER_DESTINATIONS=public,private"}, + {"USD_HTTP_RESOLVER_DESTINATIONS=public,public", + "USD_HTTP_RESOLVER_DESTINATIONS=public"}, + {"USD_HTTP_RESOLVER_MAX_RETRIES=00", + " USD_HTTP_RESOLVER_MAX_RETRIES = 0 ;"}, + }; + for (const auto& pair : equivalent) { + std::vector problems; + const std::string first = + usdhttpresolver::CanonicalContextString(OverridesFrom(pair[0], &problems)); + const std::string second = + usdhttpresolver::CanonicalContextString(OverridesFrom(pair[1], &problems)); + if (first != second) { + std::fprintf(stderr, "FAIL %s:%d: '%s' and '%s' canonicalized to '%s' and '%s'\n", + __FILE__, __LINE__, pair[0], pair[1], first.c_str(), + second.c_str()); + ++::usdassettest::FailureCount(); + } + } + + // The destination classes are named in one fixed order. + std::vector problems; + const std::map overrides = OverridesFrom( + "USD_HTTP_RESOLVER_DESTINATIONS=metadata,loopback,public", &problems); + CHECK(problems.empty()); + CHECK_EQ(overrides.at("USD_HTTP_RESOLVER_DESTINATIONS"), + std::string("public,loopback,metadata")); +} + +/// A destination list broken across lines is the list that was written, not a +/// refused value. Refusing it would put the stage on the default -- a wider +/// policy than the one written -- which is the one direction this parser must +/// not fail in. +void TestDestinationsAcrossLines() { + const char* const values[] = { + "public,\n private", + "public\r", + "\tpublic ,\r\n private\n", + }; + for (const char* value : values) { + std::vector problems; + const usdasset::http::HttpOptions options = OptionsFrom( + From({{"USD_HTTP_RESOLVER_DESTINATIONS", value}}), &problems); + CHECK(problems.empty()); + CHECK(options.destinations.publicAddresses); + CHECK(!options.destinations.loopback); + } + + std::vector problems; + const std::map overrides = OverridesFrom( + "USD_HTTP_RESOLVER_DESTINATIONS = public,\n private", &problems); + CHECK(problems.empty()); + CHECK_EQ(overrides.count("USD_HTTP_RESOLVER_DESTINATIONS"), std::size_t{1}); +} + +/// A context's adjustments are judged against the environment it will be +/// layered on, not against the built-in defaults, so that what it warns about +/// is what will happen. +void TestContextJudgedOverTheEnvironment() { + // A request ceiling of 16 KiB is one block of a 4 KiB environment block + // size, and raises nothing; over the default 64 KiB block size it would + // have been "raised to 65536". + std::vector problems; + usdhttpresolver::OverridesFrom( + "USD_HTTP_RESOLVER_MAX_REQUEST_BYTES=16384", + From({{"USD_HTTP_RESOLVER_BLOCK_SIZE", "4096"}}), &problems); + CHECK(problems.empty()); + + // And the reverse: a gap the environment's request ceiling cannot carry is + // capped, and the context is told. + problems.clear(); + const std::map capped = usdhttpresolver::OverridesFrom( + "USD_HTTP_RESOLVER_COALESCE_GAP=16", + From({{"USD_HTTP_RESOLVER_MAX_REQUEST_BYTES", "65536"}}), &problems); + CHECK_EQ(problems.size(), std::size_t{1}); + if (!problems.empty()) { + CHECK(problems[0].adjusted); + CHECK(problems[0].fromContext); + } + CHECK_EQ(capped.size(), std::size_t{1}); + + // The environment's own problems are not the context's to report. + problems.clear(); + usdhttpresolver::OverridesFrom( + "USD_HTTP_RESOLVER_MAX_RETRIES=1", + From({{"USD_HTTP_RESOLVER_BLOCK_SIZE", "100000"}}), &problems); + CHECK(problems.empty()); +} + +/// CONFIGURATION.md §4, as a function: context over environment over default, +/// one variable at a time. +void TestPrecedence() { + const std::map environment = { + {"USD_HTTP_RESOLVER_MAX_RETRIES", "5"}, + {"USD_HTTP_RESOLVER_CONNECT_TIMEOUT_MS", "1500"}, + }; + const std::map overrides = { + {"USD_HTTP_RESOLVER_MAX_RETRIES", "0"}, + {"USD_HTTP_RESOLVER_DESTINATIONS", "public"}, + }; + + std::vector problems; + const usdasset::http::HttpOptions layered = OptionsFrom( + usdhttpresolver::Layered(overrides, usdhttpresolver::LookupIn(environment)), + &problems); + CHECK(problems.empty()); + CHECK_EQ(layered.maxAttempts, 1); // the context's + CHECK_EQ(layered.connectTimeoutMs, 1500); // the environment's + CHECK_EQ(layered.maxRedirects, // the default + usdasset::http::HttpOptions().maxRedirects); + CHECK(layered.destinations.publicAddresses); + CHECK(!layered.destinations.loopback); + + // A snapshot takes what the lookup has of the variables this version + // reads, and nothing else. + const std::map snapshot = usdhttpresolver::Snapshot( + From({{"USD_HTTP_RESOLVER_MAX_RETRIES", "3"}, {"PATH", "/usr/bin"}})); + CHECK_EQ(snapshot.size(), std::size_t{1}); + CHECK_EQ(snapshot.at("USD_HTTP_RESOLVER_MAX_RETRIES"), std::string("3")); +} + +/// The key a retained reader is handed out by. Equal exactly when a reader +/// opened under one configuration may serve a caller under the other. +void TestTransportFingerprint() { + using usdhttpresolver::TransportFingerprint; + const usdasset::http::HttpOptions defaults; + CHECK_EQ(TransportFingerprint(defaults), TransportFingerprint(defaults)); + + usdasset::http::HttpOptions narrower; + narrower.destinations.loopback = false; + CHECK(TransportFingerprint(narrower) != TransportFingerprint(defaults)); + + usdasset::http::HttpOptions impatient; + impatient.transferTimeoutMs = 1000; + CHECK(TransportFingerprint(impatient) != TransportFingerprint(defaults)); + + usdasset::http::HttpOptions persistent; + persistent.maxAttempts = 1; + CHECK(TransportFingerprint(persistent) != TransportFingerprint(defaults)); + + usdasset::http::HttpOptions metadata; + metadata.destinations.metadata = true; + CHECK(TransportFingerprint(metadata) != TransportFingerprint(defaults)); +} + } // namespace int main() { TestDefaults(); + TestDestinations(); + TestContextVariableSet(); + TestContextStrings(); + TestCanonicalValues(); + TestDestinationsAcrossLines(); + TestContextJudgedOverTheEnvironment(); + TestPrecedence(); + TestTransportFingerprint(); TestEachVariable(); TestRejectedValues(); TestIndependence(); diff --git a/plugins/http-resolver/tests/test_stage.cpp b/plugins/http-resolver/tests/test_stage.cpp index 7a4dec5..4364f1c 100644 --- a/plugins/http-resolver/tests/test_stage.cpp +++ b/plugins/http-resolver/tests/test_stage.cpp @@ -37,6 +37,9 @@ #include "pxr/usd/ar/assetInfo.h" #include "pxr/usd/ar/resolvedPath.h" #include "pxr/usd/ar/resolver.h" +#include "pxr/usd/ar/resolverContext.h" +#include "pxr/usd/ar/resolverContextBinder.h" +#include "pxr/usd/ar/resolverScopedCache.h" #include "pxr/usd/ar/timestamp.h" #include "pxr/usd/sdf/layer.h" #include "pxr/usd/usd/attribute.h" @@ -367,6 +370,59 @@ int RunChildMode(const std::string& url, const std::string& reportPath) { return report ? 0 : 1; } +/// The other child: open one *local* stage, and nothing else. +/// +/// It exists to be the host RESOLVER.md §1 makes a promise to -- one with this +/// bundle installed that never names an `http` URL. Opening a local stage binds +/// a resolver context, and binding one constructs every resolver that +/// implements contexts, this one included. +int RunLocalOnlyChildMode(const std::string& layerPath) { + const UsdStageRefPtr stage = UsdStage::Open(layerPath); + if (!stage) { + std::fprintf(stderr, "child: local stage did not open: %s\n", layerPath.c_str()); + return 1; + } + return 0; +} + +/// Installing this bundle changes nothing about a process that opens only +/// local assets -- not even a directory on disk. +/// +/// The case the constructor's emptiness is for. Because this resolver +/// implements contexts, OpenUSD constructs it in any process that opens any +/// stage; a constructor that configured the persistent tier would create the +/// directory `USD_HTTP_RESOLVER_PERSISTENT_CACHE_DIR` names for a host that +/// never asked for a remote asset. Run as a child, because the claim is about +/// a process whose first contact with the resolver is a local stage, and this +/// one has long since configured it. +void TestLocalOnlyProcessIsUntouched(const char* executable, + const std::string& localLayer) { + namespace fs = std::filesystem; + if (executable == nullptr || *executable == '\0') return; + + const auto now = std::chrono::system_clock::now().time_since_epoch().count(); + const fs::path untouched = + fs::temp_directory_path() / ("usd-http-resolver-untouched-" + std::to_string(now)); + std::error_code error; + fs::remove_all(untouched, error); + + const fs::path layer = fs::absolute(localLayer); + std::string command = + "\"" + std::string(executable) + "\" --open-local \"" + layer.string() + "\""; +#if defined(_WIN32) + command = "\"" + command + "\""; +#endif + + SetEnvironment("USD_HTTP_RESOLVER_PERSISTENT_CACHE_DIR", untouched.string()); + const int status = std::system(command.c_str()); + SetEnvironment("USD_HTTP_RESOLVER_PERSISTENT_CACHE_DIR", std::string()); + + CHECK_EQ(status, 0); + // The stage opened, the resolver was constructed, and nothing was made. + CHECK(!fs::exists(untouched)); + fs::remove_all(untouched, error); +} + /// The persistent tier, end to end and across a real process boundary. /// /// This is `v0.4.0`'s claim in the only place it can be made against a real @@ -913,6 +969,254 @@ void TestLocalResolutionIsUnchanged(const std::string& localLayer) { mark.Clear(); } +/// True when `mark` holds an error naming `code`. +bool SawCode(const TfErrorMark& mark, const char* code) { + for (const TfError& error : mark) { + if (error.GetCommentary().find(code) != std::string::npos) return true; + } + return false; +} + +/// CONFIGURATION.md §4: a context configures the stage it is bound to, and +/// nothing else. +/// +/// The context is made the way a host makes one -- from a string, through +/// OpenUSD's own entry point -- because that is the whole interface: no header +/// of this repository reaches a host, and a test that constructed the object +/// directly would be testing a path nobody takes. +void TestContextConfiguresOneStage() { + const std::string path = "/context/scene.usda"; + Serve(path, Bytes("#usda 1.0\n\ndef Xform \"Scoped\"\n{\n}\n"), "\"ctx-1\""); + const std::string url = g_server->Url(path); + + const ArResolverContext refusing = ArGetResolver().CreateContextFromString( + "http", "USD_HTTP_RESOLVER_DESTINATIONS=public"); + CHECK(!refusing.IsEmpty()); + // Printed canonically, which is what a log line or a `repr` shows. + CHECK(refusing.GetDebugString().find( + "HttpResolverContext(USD_HTTP_RESOLVER_DESTINATIONS=public)") != + std::string::npos); + + // Two spellings of one context are one context. Either scheme reaches the + // resolver, because one type serves both. + const ArResolverContext respelled = ArGetResolver().CreateContextFromString( + "https", " USD_HTTP_RESOLVER_DESTINATIONS = public ;"); + CHECK(respelled == refusing); + CHECK(hash_value(respelled) == hash_value(refusing)); + + // And not only in whitespace: values are kept as the parser read them, so + // the order of a list and a leading zero do not make a second context -- + // which, to a stage cache, would be a second stage. + CHECK(ArGetResolver().CreateContextFromString( + "http", "USD_HTTP_RESOLVER_DESTINATIONS=private, public;" + "USD_HTTP_RESOLVER_TOTAL_TIMEOUT_MS=060000") == + ArGetResolver().CreateContextFromString( + "http", "USD_HTTP_RESOLVER_TOTAL_TIMEOUT_MS=60000;" + "USD_HTTP_RESOLVER_DESTINATIONS=public,private")); + + // Under the refusing context the stage does not open, and the origin never + // hears about it: the literal loopback address is refused before a + // connection exists. + { + TfErrorMark mark; + const UsdStageRefPtr refused = UsdStage::Open(url, refusing); + CHECK(!refused); + CHECK(SawCode(mark, "HTTP002")); + CHECK_EQ(RequestsFor(path), std::size_t(0)); + mark.Clear(); + } + + // The same URL, with no context bound, opens -- the environment's default + // permits loopback. And the stage it opens is held, so that its root layer + // stays in OpenUSD's layer registry for the next case. + TfErrorMark mark; + const UsdStageRefPtr permitted = UsdStage::Open(url); + CHECK(permitted != nullptr); + if (permitted) CHECK(permitted->GetPrimAtPath(SdfPath("/Scoped")).IsValid()); + CHECK(mark.IsClean()); + mark.Clear(); + + // The case the whole context-dependence answer is for. The layer is loaded + // and registered; a registry that found it by identifier would hand it to + // the refusing stage without asking this resolver anything, and the policy + // would be walked past by having opened the URL somewhere else first. + { + TfErrorMark refusedMark; + const std::size_t before = RequestsFor(path); + const UsdStageRefPtr stillRefused = UsdStage::Open(url, refusing); + CHECK(!stillRefused); + CHECK_EQ(RequestsFor(path), before); + refusedMark.Clear(); + } + + // And a context that says nothing configures a stage exactly as no + // context does: it opens, whatever the string got wrong. What it got wrong + // was reported when it was created, and is not in what it carries. + TfErrorMark lenientMark; + const ArResolverContext lenient = ArGetResolver().CreateContextFromString( + "http", "USD_HTTP_RESOLVER_BLOCK_SIZE=16384; NOT_A_VARIABLE=1"); + CHECK(lenient.GetDebugString().find("HttpResolverContext()") != std::string::npos); + CHECK(UsdStage::Open(url, lenient) != nullptr); + lenientMark.Clear(); +} + +/// A reader `Resolve` retained under one configuration is handed to an +/// `OpenAsset` under another only when the two would have opened it the same +/// way. Otherwise a stage whose context refuses a destination could read from it +/// through a reader somebody else's resolve left behind. +void TestRetainedOpenIsNotHandedAcrossContexts() { + const std::string path = "/context/retained.bin"; + Serve(path, Pattern(4096), "\"retained-1\""); + const std::string url = g_server->Url(path); + + // Resolved with no context: one metadata request, and the reader is kept + // for the `OpenAsset` that usually follows. + TfErrorMark mark; + CHECK(!ArGetResolver().Resolve(url).empty()); + CHECK_EQ(RequestsFor(path), std::size_t(1)); + + // Opened under a context that refuses loopback. The retained reader was + // opened under a policy this caller does not have, so it is not this + // caller's to take: the open is performed again, under the caller's own + // policy, and refused. + const ArResolverContext refusing = ArGetResolver().CreateContextFromString( + "http", "USD_HTTP_RESOLVER_DESTINATIONS=public,private"); + { + ArResolverContextBinder binder(refusing); + const std::shared_ptr asset = + ArGetResolver().OpenAsset(ArResolvedPath(url)); + CHECK(asset == nullptr); + CHECK(SawCode(mark, "HTTP002")); + } + mark.Clear(); + + // And the reader is still there for a caller it fits: opened with no + // context, it is taken rather than re-opened, so no second metadata + // request reaches the origin. + const std::shared_ptr asset = ArGetResolver().OpenAsset(ArResolvedPath(url)); + CHECK(asset != nullptr); + CHECK_EQ(RequestsFor(path), std::size_t(1)); + CHECK(mark.IsClean()); + mark.Clear(); +} + +/// An `ArResolverScopedCache` keeps what it resolved -- per configuration, and +/// not per path. +/// +/// OpenUSD caches `Resolve` by path alone on behalf of a resolver that does not +/// implement scoped caches, and a scope routinely spans more than one stage. So +/// this resolver keeps the scope's cache itself, keyed as the retained opens +/// are, and this is the case that says why: a refusing context inside a scope +/// in which the path already resolved is still refused. +void TestScopedCacheKeepsContextsApart() { + const std::string path = "/context/scoped.usda"; + Serve(path, Bytes("#usda 1.0\n"), "\"scoped-1\""); + const std::string url = g_server->Url(path); + + const ArResolverContext refusing = ArGetResolver().CreateContextFromString( + "http", "USD_HTTP_RESOLVER_DESTINATIONS=public"); + + TfErrorMark mark; + ArResolverScopedCache scope; + + CHECK(!ArGetResolver().Resolve(url).empty()); + CHECK_EQ(RequestsFor(path), std::size_t(1)); + + { + ArResolverContextBinder binder(refusing); + CHECK(ArGetResolver().Resolve(url).empty()); + CHECK(SawCode(mark, "HTTP002")); + } + mark.Clear(); + + // And what the scope is for still holds. The retained reader is taken by + // an open, so a resolve outside a scope would cost a second metadata + // request; inside it, the scope answers. + CHECK(ArGetResolver().OpenAsset(ArResolvedPath(url)) != nullptr); + CHECK(!ArGetResolver().Resolve(url).empty()); + CHECK_EQ(RequestsFor(path), std::size_t(1)); + CHECK(mark.IsClean()); + mark.Clear(); +} + +/// Asset info answers from what this process remembers only for a caller that +/// could have reached the asset itself. +/// +/// Identity is shared across contexts -- a validator describes the bytes at a +/// URL, not the configuration that fetched them -- but a stage whose context +/// refuses the destination is not told the size and token of an asset another +/// stage opened there. It is told what it would have been told had nobody +/// opened it. +void TestAssetInfoIsNotToldAcrossAPolicy() { + const std::string path = "/context/identity.bin"; + Serve(path, Pattern(4096), "\"identity-ctx-1\""); + const std::string url = g_server->Url(path); + + // Opened with no context bound: the identity is known to this process. + CHECK(ArGetResolver().OpenAsset(ArResolvedPath(url)) != nullptr); + CHECK(!InfoField(ArGetResolver().GetAssetInfo(url, ArResolvedPath(url)), + "validationToken") + .empty()); + + const ArResolverContext refusing = ArGetResolver().CreateContextFromString( + "http", "USD_HTTP_RESOLVER_DESTINATIONS=public"); + { + TfErrorMark mark; + ArResolverContextBinder binder(refusing); + // With the empty resolved path a refused resolve leaves behind, and + // with the path itself: nothing either way, and no diagnostic. + const ArAssetInfo unresolved = ArGetResolver().GetAssetInfo(url, ArResolvedPath()); + CHECK(unresolved.version.empty()); + CHECK(unresolved.resolverInfo.IsEmpty()); + const ArAssetInfo resolved = ArGetResolver().GetAssetInfo(url, ArResolvedPath(url)); + CHECK(resolved.version.empty()); + CHECK(resolved.resolverInfo.IsEmpty()); + CHECK(mark.IsClean()); + mark.Clear(); + } + + // A context that could have reached it -- one that only bounds retries -- + // is told, from memory, without a request. + const ArResolverContext retries = ArGetResolver().CreateContextFromString( + "http", "USD_HTTP_RESOLVER_MAX_RETRIES=0"); + { + ArResolverContextBinder binder(retries); + const std::size_t before = RequestsFor(path); + const ArAssetInfo info = ArGetResolver().GetAssetInfo(url, ArResolvedPath()); + CHECK(!info.version.empty()); + CHECK_EQ(RequestsFor(path), before); + } +} + +/// The transport bounds are a stage's too, not only the destination policy: a +/// context that follows no redirects opens nothing behind one, while the same +/// URL with no context bound follows it. +void TestContextBoundsTheTransport() { + usdassetfixture::AssetSpec spec; + spec.path = "/context/moved.usda"; + spec.content = Bytes("#usda 1.0\n\ndef Xform \"Moved\"\n{\n}\n"); + spec.etag = "\"moved-1\""; + spec.behavior = usdassetfixture::Behavior::RedirectChain; + spec.redirectHops = 1; + g_server->Serve(spec); + const std::string url = g_server->Url(spec.path); + + const ArResolverContext noRedirects = ArGetResolver().CreateContextFromString( + "http", "USD_HTTP_RESOLVER_MAX_REDIRECTS=0"); + { + TfErrorMark mark; + ArResolverContextBinder binder(noRedirects); + CHECK(ArGetResolver().Resolve(url).empty()); + CHECK(SawCode(mark, "HTTP004")); + mark.Clear(); + } + + TfErrorMark mark; + CHECK(!ArGetResolver().Resolve(url).empty()); + CHECK(mark.IsClean()); + mark.Clear(); +} + /// Writes the local fixture beside the test executable's working directory, so /// that the local-resolution case does not depend on an installed fixture path. std::string WriteLocalLayer() { @@ -932,6 +1236,9 @@ int main(int argc, char** argv) { if (argc >= 4 && std::string(argv[1]) == "--read-window") { return RunChildMode(argv[2], argv[3]); } + if (argc >= 3 && std::string(argv[1]) == "--open-local") { + return RunLocalOnlyChildMode(argv[2]); + } std::string error; const std::unique_ptr server = @@ -967,9 +1274,16 @@ int main(int argc, char** argv) { TestAssetInfoUnderConcurrentOpen(); TestAssetInfoDoesNotRediscoverAFailure(); TestAgedOutIdentityStillDetectsARepublish(); + TestContextConfiguresOneStage(); + TestRetainedOpenIsNotHandedAcrossContexts(); + TestScopedCacheKeepsContextsApart(); + TestAssetInfoIsNotToldAcrossAPolicy(); + TestContextBoundsTheTransport(); TestRetainedOpenSurvivesProcessExit(); TestWritingIsRefused(); - TestLocalResolutionIsUnchanged(WriteLocalLayer()); + const std::string localLayer = WriteLocalLayer(); + TestLocalResolutionIsUnchanged(localLayer); + TestLocalOnlyProcessIsUntouched(argv[0], localLayer); } else { std::fprintf(stderr, "FAIL: no resolver claimed %s -- is the bundle's " diff --git a/tests/corpus/CMakeLists.txt b/tests/corpus/CMakeLists.txt index 1455630..8aacb94 100644 --- a/tests/corpus/CMakeLists.txt +++ b/tests/corpus/CMakeLists.txt @@ -37,3 +37,27 @@ add_test(NAME usdAssetHttp_corpus_projection COMMAND usdAssetHttp_test_projectio # *hung* exchange fails the lane rather than holding it, which is the failure # mode a suite full of deadlines has. set_tests_properties(usdAssetHttp_corpus_projection PROPERTIES TIMEOUT 120) + +# The destination policy's connect-time half, which is the one part of §10.2 +# that needs a name, a system resolver, and a listening socket rather than a +# scripted `Location`. Same reverse edge, for the same reason: the fixture server +# is the only listening socket this repository has. +add_executable(usdAssetHttp_test_destinations test_destinations.cpp) + +target_link_libraries(usdAssetHttp_test_destinations + PRIVATE usdasset::http usdasset::fixtureserver) +target_include_directories(usdAssetHttp_test_destinations + PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}") + +if(MSVC) + target_compile_options(usdAssetHttp_test_destinations PRIVATE /utf-8 /W4) +else() + target_compile_options(usdAssetHttp_test_destinations PRIVATE -Wall -Wextra -Wpedantic) +endif() + +if(COMMAND usd_http_resolver_stage_runtime_dependencies) + usd_http_resolver_stage_runtime_dependencies(usdAssetHttp_test_destinations) +endif() + +add_test(NAME usdAssetHttp_destination_policy COMMAND usdAssetHttp_test_destinations) +set_tests_properties(usdAssetHttp_destination_policy PROPERTIES TIMEOUT 60) diff --git a/tests/corpus/test_destinations.cpp b/tests/corpus/test_destinations.cpp new file mode 100644 index 0000000..2b276a8 --- /dev/null +++ b/tests/corpus/test_destinations.cpp @@ -0,0 +1,266 @@ +// SPDX-License-Identifier: Apache-2.0 +// +// The destination policy, over a real socket. +// +// `usdAssetHttp_protocol` asserts the halves of the policy the protocol layer +// owns -- the pre-flight on a literal address, and the projection of a +// refusal -- against a scripted transport. What it cannot assert is the two +// halves that live in the client: that the host is judged as libcurl will send +// it, and that the address libcurl is about to connect to is judged after the +// name was resolved and before the socket exists. Those need a client, a +// resolver, and a listening socket, and the fixture server is the one listening +// socket this repository has. +// +// Loopback is the only destination a CI runner can offer without a network, +// so most cases here are about loopback: permitted by default, and refused when +// a policy says so -- whether the URL spells the address or names a host that +// resolves to it. The second of those is the case that matters. A policy that +// only read the URL would pass the first and let `localhost` through. The last +// case uses the fixture as a *proxy*, which is how a refused address that no +// socket here can reach is still asserted to be refused. + +#include +#include +#include +#include +#include + +#include "usdAssetHttp/HttpAssetReader.h" +#include "usdAssetIo/Diagnostics.h" +#include "usdassetfixture/Corpus.h" +#include "usdassetfixture/Server.h" + +#include "Check.h" + +namespace { + +using usdasset::ReadResult; +using usdasset::StatusCode; +using usdasset::http::HttpOpenResult; +using usdasset::http::HttpOptions; +using usdassetfixture::AssetSpec; +using usdassetfixture::Behavior; +using usdassetfixture::RequestRecord; +using usdassetfixture::Server; + +constexpr std::size_t kSize = 4096; + +HttpOptions FastOptions() { + HttpOptions options; + options.connectTimeoutMs = 2000; + options.responseTimeoutMs = 2000; + options.transferTimeoutMs = 4000; + return options; +} + +HttpOptions RefusingLoopback() { + HttpOptions options = FastOptions(); + options.destinations.loopback = false; + return options; +} + +std::string NamedUrl(const Server& server, const std::string& path) { + return "http://localhost:" + std::to_string(server.Port()) + path; +} + +void ExpectRefused(const HttpOpenResult& opened, const Server& server, + const char* what) { + if (opened.status.code != StatusCode::AccessDenied) { + std::fprintf(stderr, "FAIL [%s] expected AccessDenied, got %s\n", what, + usdasset::ToString(opened.status).c_str()); + ++usdassettest::FailureCount(); + return; + } + CHECK(opened.reader == nullptr); + // Named, so that whoever reads it goes to their own policy rather than to + // the origin's permissions. + CHECK(opened.status.message.find("loopback") != std::string::npos); + // And nothing reached the server. Not a request refused after it was + // made: a connection never opened. + CHECK_EQ(server.RequestCount(), std::size_t(0)); +} + +void TestLoopbackIsPermittedByDefault(Server& server) { + // The default policy is what the whole corpus runs under, and this is the + // one place that says so out loud: loopback is permitted, because `http` + // is registered for local fixture servers and intranet hosts and the + // default is not allowed to break the uses the scheme exists for. + server.ClearLog(); + HttpOpenResult opened = usdasset::http::Open(server.Url("/normal"), FastOptions()); + CHECK(opened.reader != nullptr); + if (!opened.reader) return; + std::vector buffer(128, 0); + const ReadResult read = opened.reader->Read(0, buffer.data(), buffer.size()); + CHECK_EQ(read.status.code, StatusCode::Ok); + + // And through a name, which exercises the connect-time callback on the + // admitting side: `localhost` commonly resolves to `::1` first, which this + // fixture does not listen on, and the callback must admit that attempt and + // the IPv4 one after it rather than stopping at the first. + HttpOpenResult named = + usdasset::http::Open(NamedUrl(server, "/normal"), FastOptions()); + if (!named.reader) { + std::fprintf(stderr, "FAIL [named, permitted] %s\n", + usdasset::ToString(named.status).c_str()); + ++usdassettest::FailureCount(); + } +} + +void TestLiteralIsRefusedBeforeConnecting(Server& server) { + // The pre-flight half: `127.0.0.1` in the URL is judged as text. + server.ClearLog(); + const HttpOpenResult opened = + usdasset::http::Open(server.Url("/normal"), RefusingLoopback()); + ExpectRefused(opened, server, "literal"); +} + +void TestNameIsRefusedAtConnect(Server& server) { + // The connect-time half, and the case the policy is worthless without. The + // URL names no address at all; the pre-flight has nothing to judge, and + // passes it. What refuses it is the transport, looking at the address the + // name resolved to, before a socket for it exists. + server.ClearLog(); + const HttpOpenResult opened = + usdasset::http::Open(NamedUrl(server, "/normal"), RefusingLoopback()); + ExpectRefused(opened, server, "name"); +} + +void TestLegacySpellingIsRefusedAtConnect(Server& server) { + // `127.1` is 127.0.0.1 to every resolver descended from `inet_aton`, and + // to libcurl's own URL parser, and it is not a literal to the protocol + // layer's pre-flight, which reads canonical dotted quads only. That is + // deliberate -- a second parser for a notorious grammar would be a second + // opinion about it -- and it is safe because the transport judges the + // host as libcurl will send it, and the connect-time check sees what it + // became regardless. + const std::string url = + "http://127.1:" + std::to_string(server.Port()) + "/normal"; + + server.ClearLog(); + ExpectRefused(usdasset::http::Open(url, RefusingLoopback()), server, "127.1"); + + // And under the default it is simply loopback, and opens. + HttpOpenResult permitted = usdasset::http::Open(url, FastOptions()); + if (!permitted.reader) { + std::fprintf(stderr, "FAIL [127.1, permitted] %s\n", + usdasset::ToString(permitted.status).c_str()); + ++usdassettest::FailureCount(); + } +} + +/// Sets one environment variable for the life of a scope and puts back what was +/// there. An empty value removes the variable. +class ScopedEnvironment { +public: + ScopedEnvironment(const char* name, const std::string& value) : _name(name) { +#if defined(_MSC_VER) +#pragma warning(suppress : 4996) +#endif + if (const char* previous = std::getenv(name)) { + _had = true; + _previous = previous; + } + Set(value); + } + ~ScopedEnvironment() { Set(_had ? _previous : std::string()); } + + ScopedEnvironment(const ScopedEnvironment&) = delete; + ScopedEnvironment& operator=(const ScopedEnvironment&) = delete; + +private: + void Set(const std::string& value) { +#if defined(_WIN32) + _putenv_s(_name, value.c_str()); +#else + if (value.empty()) { + unsetenv(_name); + } else { + setenv(_name, value.c_str(), 1); + } +#endif + } + + const char* _name; + bool _had = false; + std::string _previous; +}; + +void TestLegacySpellingIsRefusedThroughAProxy(Server& server) { + // Through a proxy the connect-time check sees the proxy's address, and the + // destination goes out as text for the proxy to resolve. So a spelling the + // client will normalize to a refused address has to be refused as the + // client will send it -- and libcurl reads every one of these as + // 169.254.169.254 before the proxy sees the request. The fixture server is + // the proxy here: it logs whatever arrives, absolute-form targets included, + // so "nothing reached the proxy" is a count of its log. + ScopedEnvironment proxy("http_proxy", server.BaseUrl()); + ScopedEnvironment noProxy("no_proxy", std::string()); + ScopedEnvironment noProxyUpper("NO_PROXY", std::string()); + + const char* const spellings[] = { + "http://2852039166/latest/meta-data/", + "http://0xa9fea9fe/latest/meta-data/", + "http://169.254.43518/latest/meta-data/", + "http://%31%36%39.254.169.254/latest/meta-data/", + "http://169.254.169.254./latest/meta-data/", + }; + for (const char* url : spellings) { + server.ClearLog(); + const HttpOpenResult opened = usdasset::http::Open(url, FastOptions()); + if (opened.status.code != StatusCode::AccessDenied || + opened.status.message.find("metadata") == std::string::npos) { + std::fprintf(stderr, "FAIL [proxy] %s: %s\n", url, + usdasset::ToString(opened.status).c_str()); + ++usdassettest::FailureCount(); + } + CHECK_EQ(server.RequestCount(), std::size_t(0)); + } + + // The control, and the reason the case above is not vacuous: under a policy + // that permits the metadata class, the same spelling does go to the proxy, + // and the proxy is handed the address libcurl normalized it to. Without + // the pre-flight, this is what every spelling above would have done. + HttpOptions permissive = FastOptions(); + permissive.destinations.metadata = true; + server.ClearLog(); + usdasset::http::Open("http://2852039166/latest/meta-data/", permissive); + const std::vector log = server.Log(); + CHECK(!log.empty()); + if (!log.empty()) { + CHECK(log.front().target.find("169.254.169.254") != std::string::npos); + } +} + +// Not here: a redirect that crosses from a permitted destination into a +// refused one. Staging it over a socket needs two origins in two classes, and a +// runner without a network has one. The rule is the protocol layer's -- each hop +// is judged as a new request, pre-flight and connect alike -- and it is asserted +// in `usdAssetHttp_protocol`, where a scripted `Location` can name any address. + +} // namespace + +int main() { + std::string error; + std::unique_ptr server = Server::Start(&error); + if (!server) { + std::fprintf(stderr, "FAIL: the fixture server could not bind loopback: %s\n", + error.c_str()); + return 1; + } + + AssetSpec normal; + normal.path = "/normal"; + normal.content.assign(kSize, 0x5a); + normal.behavior = Behavior::Normal; + normal.etag = "\"rev-a\""; + server->Serve(normal); + + TestLoopbackIsPermittedByDefault(*server); + TestLiteralIsRefusedBeforeConnecting(*server); + TestNameIsRefusedAtConnect(*server); + TestLegacySpellingIsRefusedAtConnect(*server); + TestLegacySpellingIsRefusedThroughAProxy(*server); + + server->Stop(); + return usdassettest::Report("usdAssetHttp/destination-policy"); +} diff --git a/tests/corpus/test_projection.cpp b/tests/corpus/test_projection.cpp index baad909..55af28a 100644 --- a/tests/corpus/test_projection.cpp +++ b/tests/corpus/test_projection.cpp @@ -16,10 +16,12 @@ // the read contract -- that is `tests/boundary`, over an oracle, and no server // is involved in it. +#include #include #include #include #include +#include #include #include "usdAssetHttp/HttpAssetReader.h" @@ -291,6 +293,49 @@ void TestFraming(Server& server) { StatusCode::InvalidResponse); } +void TestOversizedHeaders(Server& server) { + // A correct response with a header block no client should buffer: a + // megabyte of well-formed fields after a `Content-Length` and an + // `Accept-Ranges` that are both present and both right. §10.1 of the design + // policy requires a bound on the block and not only on the body, and the + // prefix that fits under the bound is exactly what a careless client would + // open the asset on. + g_exercised.insert(Behavior::OversizedHeaders); + const std::string path = "/oversized-headers"; + server.Serve(MakeSpec(path, Behavior::OversizedHeaders)); + server.ClearLog(); + + const HttpOpenResult opened = usdasset::http::Open(server.Url(path), FastOptions()); + if (opened.status.code != StatusCode::InvalidResponse) { + ReportCode(Behavior::OversizedHeaders, "open", opened.status.code, + StatusCode::InvalidResponse); + return; + } + CHECK(opened.reader == nullptr); + // Named as a size, so that a human can tell an origin that misbehaved from + // a bound that was too tight. + CHECK(opened.status.message.find("header block") != std::string::npos); + + // And not retried: nothing about asking again would make the block + // smaller, and a retry would buffer the same 64 KiB a second time. + // + // Waited for rather than read at once. The server logs a request when its + // response head has been written or abandoned, and a client that stops + // reading at the bound leaves that write to fail on the server's clock + // rather than on this thread's. Every request `Open` issued was on the + // wire before it returned, so once the log is quiet it is complete. + const std::chrono::steady_clock::time_point deadline = + std::chrono::steady_clock::now() + std::chrono::seconds(5); + std::size_t logged = server.RequestCount(); + while (std::chrono::steady_clock::now() < deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds(50)); + const std::size_t now = server.RequestCount(); + if (now > 0 && now == logged) break; + logged = now; + } + CHECK_EQ(server.RequestCount(), std::size_t(1)); +} + void TestValidatorChange(Server& server) { g_exercised.insert(Behavior::ValidatorChangeMidRead); const std::string path = "/moving"; @@ -480,6 +525,7 @@ int main() { TestTransientServerError(*server); TestRangeSupport(*server); TestFraming(*server); + TestOversizedHeaders(*server); TestValidatorChange(*server); TestRedirectLoop(*server); TestDeadlines(*server); diff --git a/tests/fixture-server/README.md b/tests/fixture-server/README.md index e0a0423..e0e98d6 100644 --- a/tests/fixture-server/README.md +++ b/tests/fixture-server/README.md @@ -48,7 +48,7 @@ ctest --test-dir build/core -R usdAssetFixture_corpus ## The corpus -§11.2 of the design policy names nine conditions. Each has a row, and three +§11.2 of the design policy names nine conditions. Each has a row, and four more rows come from constraints fixed elsewhere: | Behavior | Condition | Required by | @@ -60,6 +60,7 @@ more rows come from constraints fixed elsewhere: | `ContentRangeTooShort` | `206` accurately describing a range shorter than the request | §11.2; the example in [DIAGNOSTICS.md](../../docs/architecture/DIAGNOSTICS.md) §6 | | `ContentRangeShifted` | `206` describing a range at a different offset | §11.2; a different check from the row above — start, not length | | `UnknownContentLength` | `200` with no `Content-Length`, body delimited by close | ADR-0003: the client must be able to refuse it | +| `OversizedHeaders` | A correct response padded with `headerBytes` (a megabyte) of kilobyte-sized header fields, after the ones that matter | §10.1 of the design policy: the header block is bounded, not only the body | | `ValidatorChangeMidRead` | The `ETag` and content change underneath an open reader | §11.2, and §6 of the design policy | | `RedirectChain` | `redirectHops` `302`s, then the asset | §11.2 | | `RedirectLoop` | A `302` pointing at its own path, unbounded by the server | ADR-0003: bounding a chain is this repository's counter, not the library's | diff --git a/tests/fixture-server/include/usdassetfixture/Corpus.h b/tests/fixture-server/include/usdassetfixture/Corpus.h index c5da18d..45f0571 100644 --- a/tests/fixture-server/include/usdassetfixture/Corpus.h +++ b/tests/fixture-server/include/usdassetfixture/Corpus.h @@ -83,6 +83,20 @@ enum class Behavior { /// this rather than one that helpfully accumulates until EOF. UnknownContentLength, + /// A correct response -- status, framing, validator, and body all right -- + /// whose header block carries `headerBytes` of padding in fields that are + /// each well formed and individually unremarkable. The block does end; what + /// is wrong with it is its size. §10.1 of the design policy requires a + /// client to bound the header block and not only the body, and a client + /// that buffers until the blank line is a client whose allocation the + /// server chooses. + /// + /// Every field is at most a kilobyte, far under any client's ceiling on a + /// single line, so that what a client has to bound here is the block. A row + /// made of one enormous line would be caught by the library's line limit + /// and would prove nothing about the client's own. + OversizedHeaders, + /// The validator and the content change after `changeAfterRequests` /// requests. A later request carrying the old `If-Range` gets `200` and the /// whole *new* body, per RFC 9110 §13.1.5 -- which is `AssetChanged`, and @@ -195,6 +209,12 @@ struct AssetSpec { /// Requests answered with `503` before service resumes, for /// TransientServerError. int transientFailures = 1; + + /// Bytes of padding OversizedHeaders adds to every response's header + /// block, beyond the fields an ordinary response carries. A megabyte is two + /// orders of magnitude past any header block an ordinary origin sends, so a + /// client that accepts it has no bound worth the name. + std::size_t headerBytes = 1024 * 1024; }; } // namespace usdassetfixture diff --git a/tests/fixture-server/src/Corpus.cpp b/tests/fixture-server/src/Corpus.cpp index 6001c7a..a6b522b 100644 --- a/tests/fixture-server/src/Corpus.cpp +++ b/tests/fixture-server/src/Corpus.cpp @@ -17,6 +17,7 @@ const std::vector& AllBehaviors() { Behavior::ContentRangeTooShort, Behavior::ContentRangeShifted, Behavior::UnknownContentLength, + Behavior::OversizedHeaders, Behavior::ValidatorChangeMidRead, Behavior::RedirectChain, Behavior::RedirectLoop, @@ -41,6 +42,7 @@ const char* BehaviorName(Behavior behavior) noexcept { case Behavior::ContentRangeTooShort: return "ContentRangeTooShort"; case Behavior::ContentRangeShifted: return "ContentRangeShifted"; case Behavior::UnknownContentLength: return "UnknownContentLength"; + case Behavior::OversizedHeaders: return "OversizedHeaders"; case Behavior::ValidatorChangeMidRead: return "ValidatorChangeMidRead"; case Behavior::RedirectChain: return "RedirectChain"; case Behavior::RedirectLoop: return "RedirectLoop"; @@ -73,6 +75,8 @@ const char* BehaviorDescription(Behavior behavior) noexcept { return "206 whose Content-Range starts at a different offset"; case Behavior::UnknownContentLength: return "200 with no Content-Length, body delimited by close"; + case Behavior::OversizedHeaders: + return "a correct response padded with headerBytes of header fields"; case Behavior::ValidatorChangeMidRead: return "the ETag and content change underneath an open reader"; case Behavior::RedirectChain: diff --git a/tests/fixture-server/src/Server.cpp b/tests/fixture-server/src/Server.cpp index 8fab6cc..5d14558 100644 --- a/tests/fixture-server/src/Server.cpp +++ b/tests/fixture-server/src/Server.cpp @@ -55,8 +55,13 @@ struct Knobs { int delayMs = 1000; int changeAfterRequests = 1; int transientFailures = 1; + std::size_t headerBytes = 0; }; +/// The longest padding field OversizedHeaders emits. Small enough that no +/// client's ceiling on a single line is what refuses the block; see Corpus.h. +constexpr std::size_t kPaddingFieldBytes = 1024; + struct AssetState { Knobs knobs; std::shared_ptr current; @@ -142,6 +147,7 @@ class Server::Impl { state.knobs.delayMs = spec.delayMs; state.knobs.changeAfterRequests = spec.changeAfterRequests; state.knobs.transientFailures = spec.transientFailures; + state.knobs.headerBytes = spec.headerBytes; state.current = std::move(current); state.pending = std::move(pending); @@ -654,6 +660,26 @@ class Server::Impl { headers.emplace_back("Connection", "close"); } + if (effective == Behavior::OversizedHeaders) { + // After every field that matters, so that a client which stops + // reading at some bound has already been handed a complete-looking + // `Content-Length` and `Accept-Ranges`. That is the harder case: a + // client that acts on the prefix it managed to read opens the asset + // on the strength of a response it never finished receiving. + // + // Distinct names, because a client is entitled to fold repeated + // fields together, and one that did would be bounding something + // other than the block. + const std::string filler(kPaddingFieldBytes, 'x'); + std::size_t remaining = plan.knobs.headerBytes; + for (std::size_t index = 0; remaining > 0; ++index) { + const std::size_t take = std::min(remaining, filler.size()); + headers.emplace_back("X-Padding-" + std::to_string(index), + filler.substr(0, take)); + remaining -= take; + } + } + // Logged once the status line is on the wire, and logged as `0` when it // never got there. The body may still fail after this -- half the // corpus is built on that -- but the status this record names is one a diff --git a/tests/fixture-server/tests/RawClient.cpp b/tests/fixture-server/tests/RawClient.cpp index 1f3cdf5..d0e2fcc 100644 --- a/tests/fixture-server/tests/RawClient.cpp +++ b/tests/fixture-server/tests/RawClient.cpp @@ -195,6 +195,7 @@ RawResponse RawClient::ReadResponse(int timeoutMs, bool expectBody) { const std::string head = _buffered.substr(0, headEnd + 4); _buffered.erase(0, headEnd + 4); response.headElapsedMs = ElapsedMs(started); + response.headBytes = head.size(); if (!ParseHead(head, &response)) { response.end = ResponseEnd::Error; diff --git a/tests/fixture-server/tests/RawClient.h b/tests/fixture-server/tests/RawClient.h index e3bb2fd..5ebb784 100644 --- a/tests/fixture-server/tests/RawClient.h +++ b/tests/fixture-server/tests/RawClient.h @@ -49,6 +49,12 @@ struct RawResponse { std::vector body; ResponseEnd end = ResponseEnd::NoResponse; + /// Bytes from the first byte of the status line through the blank line + /// that ends the head. Recorded rather than reconstructed from `headers`, + /// because the OversizedHeaders row is a claim about what was on the wire, + /// and a sum over parsed fields would be a claim about this parser. + std::size_t headBytes = 0; + /// Milliseconds from the request being sent to the blank line arriving, /// and to the body finishing. The two slow behaviors differ only in which /// of these grows, which is why `Timeout` is required to name the deadline. diff --git a/tests/fixture-server/tests/test_corpus.cpp b/tests/fixture-server/tests/test_corpus.cpp index eb8995f..647ef30 100644 --- a/tests/fixture-server/tests/test_corpus.cpp +++ b/tests/fixture-server/tests/test_corpus.cpp @@ -422,6 +422,45 @@ void TestUnknownContentLength(Server& server, CHECK(BodyEquals(response, content, 0, kSize)); } +void TestOversizedHeaders(Server& server, const std::vector& content) { + CaseScope scope(BehaviorName(Behavior::OversizedHeaders)); + Exercised(Behavior::OversizedHeaders); + const unsigned short port = server.Port(); + + // The row's claim is that the block is large and nothing else is wrong. So + // the size is asserted from the bytes on the wire, and everything else is + // asserted to be exactly what the Normal row puts there -- a response that + // was also malformed would let a client pass for refusing the wrong thing. + const RawResponse head = + FetchOnce(port, HeadRequest("/oversized-headers"), 5000, false); + CHECK_STATUS(head, 200); + CHECK(head.headBytes > 256 * 1024); + CHECK_EQ(head.Header("Accept-Ranges"), std::string("bytes")); + CHECK_EQ(head.Header("Content-Length"), std::to_string(kSize)); + CHECK_EQ(head.Header("ETag"), std::string("\"v1\"")); + + // Many ordinary lines rather than one enormous one, so that what a client + // has to bound is the block. Checked field by field: a single line past a + // client's per-line ceiling would be refused by that ceiling and would say + // nothing about the client's own bound. + std::size_t padding = 0; + for (const auto& field : head.headers) { + if (field.first.compare(0, 10, "X-Padding-") != 0) continue; + ++padding; + CHECK(field.second.size() <= 1024); + } + CHECK(padding > 256); + + // And the ranged GET is the Normal row's, padded the same way. + const RawResponse ranged = + FetchOnce(port, GetRequest("/oversized-headers", {{"Range", "bytes=16-47"}})); + CHECK_STATUS(ranged, 206); + CHECK(ranged.headBytes > 256 * 1024); + CHECK_EQ(ranged.Header("Content-Range"), + "bytes 16-47/" + std::to_string(kSize)); + CHECK(BodyEquals(ranged, content, 16, 32)); +} + // --- revision binding -------------------------------------------------------- void TestValidatorChangeMidRead(Server& server, @@ -940,6 +979,7 @@ void RegisterFixtures(Server& server, const std::vector& content) server.Serve(base("/short-range", Behavior::ContentRangeTooShort)); server.Serve(base("/shifted-range", Behavior::ContentRangeShifted)); server.Serve(base("/unknown-length", Behavior::UnknownContentLength)); + server.Serve(base("/oversized-headers", Behavior::OversizedHeaders)); server.Serve(base("/loop", Behavior::RedirectLoop)); server.Serve(base("/always-416", Behavior::RangeNotSatisfiable)); server.Serve(base("/missing", Behavior::NotFound)); @@ -1050,6 +1090,7 @@ int main() { TestContentRangeTooShort(*server, content); TestContentRangeShifted(*server, content); TestUnknownContentLength(*server, content); + TestOversizedHeaders(*server, content); TestValidatorChangeMidRead(*server, content); TestRepublish(*server); TestRedirectChain(*server, content);