fix(guardrails): refuse an unscannable /v1/files upload only under a text purpose

[jarvis9443]

Problem

#1113 made POST /v1/files refuse an upload whose bytes are not valid UTF-8 whenever an input guardrail chain resolves. The refusal reads only the file part — it is purpose-blind — so a deployment running any input guardrail started getting 422 for a PDF, an image, or any other binary uploaded under purpose=assistants / vision / user_data. That is well beyond the malformed-JSONL case the refusal was built for, and it breaks legitimate uploads.

What changed

The refusal is now keyed on the declared multipart purpose. An undecodable blob is refused only under a purpose whose payload is contractually UTF-8 text:

declared purpose	undecodable blob, chain attached
batch, fine-tune, evals	refused — 422, unchanged envelope from #1113
assistants, vision, user_data	forwarded
unrecognised value (a case variant, a provider extension, a server-minted batch_output)	forwarded
absent	forwarded

A body declaring purpose more than once is malformed, and the provider picks whichever part it picks — so the classification folds rather than overwrites: any declared text purpose refuses, whatever order the parts arrive in. Taking the last one would let the gateway and the provider disagree about what the payload is.

batch and fine-tune take JSONL; an eval data set referenced by file id is a JSONL source too. The rest legitimately carry binary — an assistants document, a vision image, or "any purpose" — so an undecodable blob there is a normal upload, not an evasion. An upload we cannot classify is not one we can claim should have been text, so an absent or unrecognised purpose is never refused.

The scan itself is unchanged. A non-text-purpose upload keeps the best-effort lossy scan it has always had, so a term that survives from_utf8_lossy still blocks. Narrowing the refusal must not turn into skipping the chain, and one unit leg plus one e2e leg exist specifically to fail if it ever does. The residual asymmetry — a binary upload is scanned on a lossy decode and forwarded verbatim — is this surface's deliberate posture and stays; scan_output_blob's note on the download side is updated to say which signal the input side actually keys on.

This route never validates purpose; it forwards whatever the caller declared, so the classification is an exact match against the text set and nothing else. The other three scan_input_blob call sites all declare the text posture, and the refusal arm is unreachable at every one of them: /v1/batches and /v1/fine_tuning/jobs pass serde_json::to_vec output, which is valid UTF-8 by construction, and no caller sets FwdSpec::body, so the shared forwarder never scans anything today.

Behaviour change

This narrows the 422 introduced by #1113. An upload whose file part is not valid UTF-8 is refused only when the declared purpose is batch, fine-tune or evals, and an input guardrail chain resolves. Under any other purpose, an unrecognised one, or none at all, such an upload is scanned lossily and forwarded to the provider exactly as it was before #1113. Uploads under a text purpose keep #1113's envelope unchanged: 422 with error.type: content_filter, error.code: guardrail_unavailable, and unscannable_body in the message. Deployments with no guardrail attached remain unaffected on every purpose.

Baseline comparison

The established gateways gate file-upload guardrail treatment on purpose rather than applying it blindly — the closest equivalent scans batch input records only when purpose == "batch", and rejects an unlisted purpose outright at 400. This change lands on the same axis (purpose decides), with two deliberate differences: the refusal set also covers fine-tune and evals, which are JSONL by the same contract as batch; and this gateway keeps forwarding an unrecognised purpose rather than validating it, which is a pre-existing passthrough property this PR does not touch. The upstream file-purpose set used above is the one the official SDK declares for the upload parameter (assistants, batch, fine-tune, vision, user_data, evals); assistants_output, batch_output and fine-tune-results are server-minted and appear only on a stored file object.

Tests

Unit (crates/aisix-proxy/src/jobs.rs) — each verified to fail under a mutation before passing:

• undecodable_posture_classifies_every_declared_purpose — the full table in one place: three text purposes refuse; the three binary upload purposes, the three server-minted values, a case variant, a leading-space variant, an empty string, an invented value and None do not.
• non_utf8_upload_under_a_binary_purpose_is_forwarded — the fix(guardrails): refuse an unscannable /v1/files upload instead of forwarding it #1113 regression: 200, bytes forwarded byte-for-byte. Fails on c2e89aa9 with 422.
• non_utf8_upload_without_a_purpose_is_forwarded — also fails on c2e89aa9 with 422.
• binary_purpose_upload_is_still_scanned_lossily — an ASCII term that survives the lossy decode still blocks with a policy hit (no unscannable_body tag). Fails if the chain is skipped for non-text purposes rather than the refusal narrowed. The term sits after the invalid bytes, so replacing the lossy decode with a scan of the valid prefix alone also fails it — which is the shape a real evasion takes.
• duplicate_purpose_fields_take_the_stricter_reading — two purpose parts ordered text-then-binary still refuse. Fails against a last-wins read.
• non_utf8_upload_is_refused_and_never_forwarded and non_utf8_upload_without_a_guardrail_still_forwards from fix(guardrails): refuse an unscannable /v1/files upload instead of forwarding it #1113 are unchanged and still green.

E2E (tests/e2e/src/cases/files-unscannable-upload-e2e.test.ts) — the same boundary against a real binary + etcd + upstream recorder: assistants / vision / user_data and a purpose-less upload forward byte-for-byte; fine-tune refuses on the same terms as batch; and a binary-purpose upload carrying a matchable term is still blocked by the policy rather than refused as unreadable.

Mutations checked: purpose-blind refusal (4 tests red), chain skipped for non-text purposes (1 test red), wrong purpose set (3 tests red), last-wins on a duplicated purpose (1 test red), lossy decode replaced by a valid-prefix-only scan (1 test red).

Ref #1022, #1113

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
  • File uploads now handle encoding according to their declared purpose.
  • Batch, fine-tuning, and evaluation uploads with invalid UTF-8 are rejected when guardrails are enabled.
  • Binary, unknown, or unspecified-purpose uploads continue to support non-text content without altering file bytes.
  • Guardrails scan binary uploads for policy violations instead of allowing them to bypass protection.
  • Multipart uploads preserve and process specified purposes, including repeated purpose fields.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Essentials

Run ID: cd604cc8-f1de-4ed0-9be1-eea3a1483411

📥 Commits

Reviewing files that changed from the base of the PR and between 02297d2 and 80c28f8.

📒 Files selected for processing (1)

• crates/aisix-proxy/src/jobs.rs

Included review availability: Your plan provides up to 5 included reviews per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

File uploads now classify purpose before input scanning. Text purposes reject undecodable content, while binary, unknown, and absent purposes retain lossy scanning and original-byte forwarding. Multipart handling forwards purpose, and JSON request scanning uses strict UTF-8.

Changes

Purpose-aware upload scanning

Layer / File(s)	Summary
Purpose classification and scanning crates/aisix-proxy/src/jobs.rs	Upload scanning distinguishes UTF-8-required purposes from binary, unknown, and absent purposes. Non-text uploads use lossy guardrail scanning and preserve original bytes.
Multipart and JSON scanning wiring crates/aisix-proxy/src/jobs.rs	Multipart uploads accumulate purpose classifications and pass the resulting scanning posture to guardrails. Batch, fine-tuning, and shared JSON request bodies use strict UTF-8 scanning.
Purpose classification regression coverage crates/aisix-proxy/src/jobs.rs, tests/e2e/src/cases/files-unscannable-upload-e2e.test.ts	Tests cover purpose classification, duplicate-purpose handling, text-purpose rejection, binary and absent-purpose forwarding, and policy blocking during lossy scanning.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: ⚪ Minimal · up to 80c28

This change limits unscannable-upload rejection to declared text purposes while preserving binary uploads and lossy policy scanning elsewhere; no actionable merge-blocking risk remains beyond normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant AisixProxy
  participant GuardrailScanner
  participant Upstream
  Client->>AisixProxy: Send multipart upload with optional purpose
  AisixProxy->>AisixProxy: Accumulate purpose classification
  AisixProxy->>GuardrailScanner: Scan strict or lossy content
  GuardrailScanner-->>AisixProxy: Allow, policy block, or unscannable result
  AisixProxy->>Upstream: Forward original multipart bytes when allowed

Loading

Suggested reviewers: nic-6443, membphis, moonming

🚥 Pre-merge checks | ✅ 5 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
E2e Test Quality Review	⚠️ Warning	The E2E harness is relevant and uses a spawned gateway, real etcd configuration, and a real HTTP upstream recorder. The main batch, fine-tune, binary-purpose, no-purpose, and policy-hit flows are read…	Add E2E requests through the existing spawned-app/etcd/upstream-recorder flow for purpose=evals and an unrecognized purpose. Add duplicate-purpose multipart cases in both text-then-binary and binary-then-text order, and assert the 422 `co…

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Security Check	✅ Passed	No explicit security-check failure was introduced by the assessed patch. The patch changes only crates/aisix-proxy/src/jobs.rs and the related E2E test. Category 1 — No issues found: the new log rec…
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main change: /v1/files uploads are refused only when an unscannable upload has a text purpose. It is specific and related to the full changeset.

Full details: E2e Test Quality Review

Explanation

The E2E harness is relevant and uses a spawned gateway, real etcd configuration, and a real HTTP upstream recorder. The main batch, fine-tune, binary-purpose, no-purpose, and policy-hit flows are readable and assert provider contact. However, the PR changes duplicate-purpose handling and explicitly requires that any text purpose refuses the upload regardless of field order, but only a unit test covers this. The E2E case has no duplicate-purpose request. It also does not exercise purpose=evals or an unrecognized purpose end to end. The unit classification table does not replace E2E coverage under this check. The no-purpose E2E case checks only the upload count, unlike the other forwarding cases, so it does not verify byte-for-byte forwarding for that boundary.

Resolution

Add E2E requests through the existing spawned-app/etcd/upstream-recorder flow for purpose=evals and an unrecognized purpose. Add duplicate-purpose multipart cases in both text-then-binary and binary-then-text order, and assert the 422 content_filter response contains unscannable_body and that the upstream received no request. Update the no-purpose forwarding case to assert that the recorded multipart body contains the exact NON_UTF8_LINE bytes.

Full details: Security Check

Explanation

No explicit security-check failure was introduced by the assessed patch. The patch changes only crates/aisix-proxy/src/jobs.rs and the related E2E test. Category 1 — No issues found: the new log records only the model and Utf8Error; it does not log request bytes, credentials, or headers, and the client error uses the fixed unscannable_body tag. Category 2 — No issues found: no database persistence changed. Category 3 — No issues found: no authorization or permission logic changed; create_file still requires AuthenticatedKey. Category 4 — No issues found: no resource lookup or ownership logic changed. Category 5 — No issues found: no TLS configuration changed. Category 6 — No issues found: no shared-resource or cascade behavior changed. Category 7 — No issues found: no environment or secret-reference resolution path changed. The binary-purpose path retains lossy guardrail scanning, and the text-purpose path refuses undecodable bytes; neither behavior matches the seven stated failure categories.

• Fix all pre-merge checks with AI

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/files-purpose-aware-utf8-refusal

---

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

🟡 Changes recommended

The new /v1/files purpose capture overwrites on repeated purpose fields, which can allow bypassing the intended non‑UTF‑8 refusal by sending multiple purpose parts in a crafted order.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR narrows the guardrail “unscannable upload” refusal on POST /v1/files so that invalid UTF-8 uploads are only rejected when the multipart purpose indicates a contractually UTF-8 text payload (e.g., JSONL for batch / fine-tune / evals), while binary-allowed purposes continue to be scanned lossily and forwarded verbatim.

Changes:

• Add purpose-aware classification (Undecodable) to decide whether non‑UTF‑8 uploads are refused or scanned lossily in the /v1/files guardrail path.
• Preserve fail-closed refusal behavior for text-contract purposes while allowing binary purposes (and unknown/absent purpose) to forward.
• Extend unit + e2e coverage to lock in the new purpose matrix and ensure non-text uploads are still scanned (not skipped).

File summaries

File	Description
crates/aisix-proxy/src/jobs.rs	Introduces purpose-based undecodable posture and threads it into scan_input_blob + /v1/files multipart handling; adds unit tests for the posture table and forwarding/refusal behavior.
tests/e2e/src/cases/files-unscannable-upload-e2e.test.ts	Expands e2e coverage to assert forwarding for binary purposes / missing purpose, refusal for fine-tune, and continued lossy scanning under binary purposes.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.