Prototype per-run DoNotCache task results

[wolever-gl]

Overview

This is an exploratory prototype only, opened as a draft for design evidence. It is not an approval to merge or deploy, and it does not close any ticket.

Reference: RD-276 (Linear). This PR intentionally does not say "closes RD-276" — the design is under evaluation, not accepted.

Summary

Adds a public DoNotCache(value) return wrapper that a task can use to opt a single run out of result caching and persistence at return time:

• When a task returns DoNotCache(value), the task still completes in the Completed state and the direct caller still receives the raw value — but no result record is persisted and no cache entry is written under the transaction key.
• Because nothing is written, a later execution with the same cache key runs the task again instead of replaying a cached result.
• Ordinary native Cached behavior for tasks that do not opt out is unchanged.
• Implemented in src/prefect/utilities/annotations.py (public DoNotCache annotation), unwrapped in both the sync and async handle_success paths in src/prefect/task_engine.py, re-exported through src/prefect/main.py/src/prefect/__init__.py, with type overloads in src/prefect/tasks.py so task functions returning DoNotCache[R] are typed as Task[..., R].

Design notes

• Wrapper, not exception. A return wrapper was chosen over raising an exception so the success path (state, events, hooks) stays ordinary and the caller's return value is preserved.
• Existing transaction gate. Suppression reuses the existing write_on_commit transaction mechanism rather than introducing a new persistence switch.
• Sync and async. Both handle_success variants unwrap the wrapper; this duplication is a known maintenance cost (see risks).
• Local state vs API hydration. The in-process Completed state carries the raw value, but a state rehydrated through the API has no persisted record behind it and will surface MissingResult — acceptable for the prototype, a key design question for any real implementation.

Tests

• tests/test_task_engine.py: 183 passed, 2 xfailed (41.21s full run).
• New TestDoNotCache class: 13 passed, covering sync/async opt-out, repeated execution on the same key, no persisted record on disk, commit hooks still firing, no advertised result metadata on Completed events, upstream-state wrapping, and explicit ResultRecord handling.
• 12-case pyright probe against the new overloads: 0 errors.

Review / design risks

• Pre-existing cache hits bypass the return-time opt-out entirely: if a cache record already exists, the task never runs, so DoNotCache never gets a chance to apply. Opt-out is therefore not a guarantee against earlier-cached results.
• The name suppresses both caching and persistence; if callers want one and not the other, the single wrapper may be the wrong granularity.
• API/distributed result hydration yields MissingResult for runs that opted out, since no record exists remotely.
• An outer task receiving an opted-out value may itself persist it, defeating the opt-out for the inner value.
• Generator tasks: yielded DoNotCache values are not unwrapped at runtime; only the terminal return DoNotCache(...) is. The type overloads encode this asymmetry.
• The unwrap logic is duplicated across the sync and async success paths and must be kept in sync.
• Upstream Prefect has since added cache_policy.should_store(context, parameters, result), which is a policy-level alternative to a return-value annotation; any production design should be compared against it.

Documentation

No documentation was added, intentionally: this is a prototype for design evidence. If the design proceeds toward a merge, the public API needs coverage in docs/v3/concepts/caching.mdx and docs/v3/concepts/results.mdx (both currently describe caching/persistence with no return-value opt-out mechanism) before it ships. This missing coverage is accepted only because this PR is a draft prototype.

Documentation preflight (concrete evidence)

• git diff --name-only c69ade89e0c42c4e98d9f2062d6cc64aeb256ed2...HEAD touches only: src/prefect/__init__.py, src/prefect/main.py, src/prefect/task_engine.py, src/prefect/tasks.py, src/prefect/utilities/annotations.py, tests/test_task_engine.py.
• A repo-wide search for DoNotCache finds references only in those source/test files — no existing documentation references it, as expected.
• Relevant docs pages that would need updating before any real merge: docs/v3/concepts/caching.mdx, docs/v3/concepts/results.mdx, docs/v3/how-to-guides/workflows/cache-workflow-steps.mdx (none mention a return-value opt-out today).

AGENTS.md

Checked: no AGENTS.md update is needed — the change introduces no new commands, module structure, or architectural drift.

Checklist

• This pull request references the related work item: RD-276 (Linear). It deliberately does not include "closes" — the prototype does not close the ticket.
• If this pull request adds new functionality, it includes unit tests that cover the changes (new TestDoNotCache suite; 183 passed, 2 xfailed).
• If this pull request removes docs files, it includes redirect settings in mint.json. (N/A — no docs files were removed.)
• If this pull request adds functions or classes, it includes helpful docstrings (DoNotCache carries a docstring with a usage example).

---

Note

Medium Risk
Touches the task success path and result/cache transaction commit behavior in both sync and async engines; opt-out does not apply when a cache hit occurs before the task runs, and API-rehydrated states lack persisted results.

Overview
Introduces a public DoNotCache(value) annotation so a task can complete normally and return the unwrapped value while skipping cache writes and result persistence for that run.

The sync and async handle_success paths unwrap the wrapper, set transaction.write_on_commit = False, and replace terminal state data so events/API payloads do not advertise storage keys for records that were never written; the state stays Completed (not Cached) on repeat runs with the same cache key.

DoNotCache is re-exported from prefect; @task overloads (including ConfiguredTaskDecorator) type tasks that return DoNotCache[R] as Task[..., R], with generator terminal returns handled separately from yielded wrappers.

Adds TestDoNotCache covering repeated execution, disk persistence, hooks, events, and API state behavior. Draft prototype for RD-276—not documented for production yet.

^{Reviewed by Cursor Bugbot for commit 495cd61. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 495cd61. Configure here.}

[cursor]

None results break local state retrieval

Medium Severity

When the unwrapped value is None, terminal_state.data is set to None. Local state.result() treats data is None as a missing persisted record and raises MissingResult, so return_state=True fails for an in-process DoNotCache(None) even though the direct caller still receives None.

Additional Locations (1)

• src/prefect/task_engine.py#L1314-L1326

 

^{Reviewed by Cursor Bugbot for commit 495cd61. Configure here.}