Let the master workflow ask for a rebuild, and stop the docs promising one - #7
Merged
Merged
Conversation
…g one `olaf_master_workflow` called `generate` with no arguments, so the notebook a pipeline actually runs had no way to request a rebuild. The framework has had the parameter since 1.0; only the workflow was missing it. The default stays `False`, and that is an access decision rather than a performance one. `generate`'s idempotency skip is keyed on `config_hash`, which fingerprints the config rows and cannot see the Delta catalog, so a table created since the last generate that matches an existing wildcard is absent from the lock-file and nothing is granted on it. A table created by last night's load does not become readable because a star matched it; opening it up is a decision somebody makes by passing `rebuild = True` — which re-resolves every wildcard, so it takes in every table added since the last generate, not only the one being asked about. Two documented claims said otherwise and are corrected. `config-examples.md` promised a new `sales.*` table "shows up in the mapping on the next generate"; the next generate is usually the one that skips. `architecture.md`'s second key invariant had the same gap. The CLS blacklist example described a newly discovered column the same way and is fixed alongside them. CHANGELOG records this under Unreleased. No version or tag is touched.
`docs/modes.md` is the mode manual, and its `generate` section never mentioned the idempotency skip — while the result-envelope table three sections above it already lists `status=skipped` as an outcome. So the one behaviour an operator meets on nearly every run had no home, and the previous commit's pointer from `config-examples.md` aimed at a section that did not exist. It now covers what `config_hash` fingerprints, the two things it therefore cannot see (the Delta catalog and the member table), the five exceptions that defeat the skip without a parameter, what `rebuild=True` re-resolves and drops, and the reminder that not granting a table is not the same as denying it. `docs/api/Deployment.md` was headed `generate(rebuild=False)` and never said what the parameter did; it does now, and points at the same section.
kengio
added a commit
that referenced
this pull request
Aug 31, 2026
Follow-up to #7. The reference docs described the idempotency skip; the runbook — where an operator goes when something looks wrong — did not. New 3i covers the case an estate using a table glob meets first: a table created since the last generate is not granted, and every run since has reported success. It states why the skip holds (`config_hash` fingerprints config rows and cannot see the catalog), that this is intended rather than a fault, that `rebuild=True` re-resolves every pattern and so takes in every table added since the last real generate, holding one back with `exclude_tables` (and that a zero-match exclude is an error), the stale converse when a table is deleted, and the unrelated conditions that force a re-resolution and widen scope as a side effect. The sharper problem was the word itself. `rebuild` appeared in the runbook in exactly one place — 3e, `setup(rebuild=True)`, which DROPS a control table and loses its data. An operator searching for it found the destructive one and no sign that a harmless one exists. 3e now says the two share a parameter name and nothing else, and points next door. A second sweep, matching on the shape of the claim rather than on the word wildcard, closed two more: the cookbook said re-running an unchanged config is a no-op without saying that a changed catalog is also a no-op, and `config-examples.md`'s CLS whitelist takeaway kept the ambiguity 7a had just lost. Both new sections say plainly that not granting a table is not denying it — readability still depends on workspace and item permissions, so a missing grant must not be read as containment. Docs-only. CHANGELOG under Unreleased; no version bump and no tag.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
olaf_master_workflowcalledgeneratewith no arguments, so the notebook a pipeline actually runs had no way to request a rebuild. The framework has had therebuildparameter since 1.0 — only the workflow was missing it.rebuildadded to the parameters cell (defaultFalse), the parameter table, the run-constants block, and passed through asrun_stage("generate", {"rebuild": REBUILD})Why
Falseis the defaultIt is an access decision, not a performance one.
generate's idempotency skip is keyed onconfig_hash, which fingerprints the config rows and cannot see the Delta catalog. So a table created since the last generate that matches an existing wildcard is absent from the lock-file and nothing is granted on it.A table created by last night's load should not become readable because a star matched it. Opening it up is a decision somebody makes by passing
rebuild = True— which is all-or-nothing: every wildcard is re-resolved, so every table added since the last generate arrives with it, not only the one being asked about.Verified against the runtime on the mock harness: with
include_tables: sales.*already generated, addingsales.dim_newto the catalog leaves the secondgeneratereportingskippedwith the table absent from the mapping;rebuild=Truetakes it in.Docs corrected
Two claims said the opposite, and both are load-bearing for how an operator plans a deployment:
docs/config-examples.mdpromised a newsales.*table "shows up in the mapping on the next generate with no config change needed". The first half is right; the second half is what makes it wrong, because the next generate is usually the one that skips. Now says the table needs a generate that actually runs, and namesrebuild=True.docs/architecture.mdkey invariant 2 read "New tables are absent from the saved mapping until the next generate" — true only of a generate that rebuilds. It now says so, and whyconfig_hashcannot tell the difference.config-examples.mddescribed a newly discovered column the same way, and is fixed alongside them.api/Log.md's similar wording was checked and left alone — it describes a memberglob:pattern, which is genuinely exempt from the skip.Verification
pytest tests/— all passscripts/check_public_release.py tree .— PASS, 0 findingsNot included
CHANGELOG records this under
[Unreleased]. No version bump and no tag — that is a release decision, not this PR's.