Dataflow reset blocked when materialization requires fields defined only in inferred schema

[jwhartley]

Problem

Dataflow reset on a collection discards the inferred schema. Fields defined only in the inferred schema disappear from the collection schema. Any downstream materialization that lists such a field in require then fails spec validation with:

Capture Save Failed
flow://materialization/<tenant>/<mat-name> / bindings / N / fields / require / <field>
required projection <field> does not exist in collection <tenant>/<collection>

The reset is aborted before it can begin, even though the reset's purpose is usually to re-establish the inferred schema from a known-good state.

Why it matters

• Common workflow: customers do dataflow resets for schema drift, source-side restructuring, and recovery scenarios. Downstream require constraints are also common — used to guarantee key fields exist.
• The error surfaces the wrong subject: the customer is editing the capture, but the error text references a materialization binding they weren't touching.
• The current workaround is a multi-step manual dance:
  1. Edit materialization → advanced spec editor → cut and save the require section → publish
  2. Edit capture → trigger dataflow reset → wait for inference to re-populate the fields
  3. Edit materialization → paste the require section back in → publish
     This is multi-system, multiple publications, and requires the customer to know about the advanced spec editor.
• Alternative workaround (disable materialization binding → reset → re-enable) is similarly cumbersome and loses in-flight materialization state.

Known internally

Multiple internal Slack threads document this as a recurring rough spot. Engineering has sketched at least two resolution directions without shipping either. Customers hitting it include MongoDB-sourced pipelines, Klaviyo, Salesforce, and others — the problem spans many connector categories and is not source-specific.

Prior internal discussion

Internal Slack threads (estuary-only):

• MongoDB → Redshift, _meta/uuid missing after dataflow reset — team member describes this as "a known rough spot"; includes the proposed resolution to "relax the scope of the validation check to not include downstream materializations."
• DDLs-on-materialization blocks reset, multi-step workaround documented — source of the proposed resolution direction (1) below: "relax the check, to handle it in the same way we handle exclude fields today."
• Customer hit same error a second time, forced to recreate capture — current support-pattern workaround using the advanced spec editor.
• Klaviyo → Snowflake, relationships/* fields missing after reset — same error pattern on an unrelated connector.
• MongoDB → Aurora Postgres, transaction/paymentDetails/expYear — same error pattern, customer had onIncompatibleSchemaChange: abort set.
• Design intent of reset wiping inferred schema — explains why the current behaviour exists ("otherwise impossible to ever drop stuff from the inferred schema") and constrains the solution space.

Proposed resolution directions

1. Relax require validation during reset. Mirror the current exclude handling: only error if a new require entry is introduced in the same publication; tolerate pre-existing require entries that rely on inferred fields, on the assumption that inference will re-populate them. If they don't reappear after re-inference, fall back to onIncompatibleSchemaChange semantics. Proposed here.

2. Scope validation differently during reset publications. Skip downstream-materialization validation entirely during reset publications; let onIncompatibleSchemaChange handle any post-reset drift. Proposed here.

3. Promote inferred-schema fields into the write schema at reset time. Preserves the fields across reset so materializations validate. This contradicts the intentional "reset wipes inferred schema" design goal (the whole point of reset is to let the system forget stale inferred fields), so probably the wrong direction.

Direction (1) is likely the lowest-risk path: the weakening of "required" semantics is bounded (only for pre-existing requires, only during reset publications) and aligns with existing precedent for how exclude is handled.

Cross-links

• Umbrella truncations work: End-to-end Captured -> Materialized Truncations #2821
• Dataflow reset drops journal splitting, painful for large captures #2881 — sibling dataflow-reset pain: reset drops journal splitting, painful for large captures. Same root workflow, different failure mode.

Labels

control-plane, enhance