Let the master workflow ask for a rebuild, and stop the docs promising one

CHANGELOG.md

@@ -6,7 +6,35 @@ Notable changes to OLAF — OneLake Access Framework — are recorded here using
 
 ## [Unreleased]
 
-No changes yet.
+### Added
+
+- **`olaf_master_workflow` takes a `rebuild` parameter, defaulting to `False`**, and passes it to
+  `generate`. The runner previously called `generate` with no arguments, so there was no way to
+  ask for a rebuild from the notebook a pipeline actually runs — the framework had the parameter,
+  the workflow did not expose it. The default is an access decision rather than a performance one:
+  `generate`'s idempotency skip is keyed on `config_hash`, which fingerprints the config rows and
+  cannot see the catalog, so a table created since the last generate that matches an existing
+  wildcard stays out of the mapping and ungranted until somebody deliberately passes
+  `rebuild = True`. Taking it in is all-or-nothing — every wildcard is re-resolved, so every table
+  added since the last generate arrives with it.
+
+### Fixed
+
+- `docs/config-examples.md` said a new table matching `sales.*` "shows up in the mapping on the
+  next generate with no config change needed". The first half is right and the second half is
+  what makes it wrong: an unchanged config takes the idempotency skip, so the *next* generate is
+  usually the one that rebuilds nothing. The example now says the table needs a generate that
+  actually runs, and names `rebuild=True` as the way to get one. The CLS blacklist example, which
+  described a newly discovered column the same way, is corrected alongside it.
+- `docs/architecture.md`'s second key invariant read "New tables are absent from the saved mapping
+  until the next generate", which is true only of a generate that rebuilds. It now says so, and
+  says why `config_hash` cannot see the difference.
+- `docs/modes.md` documented every mode except the one behaviour an operator meets most often: its
+  `generate` section never mentioned the idempotency skip, even though the result-envelope table
+  above it already advertises `status=skipped`. It now carries the skip, what `config_hash` can and
+  cannot see, the five exceptions that defeat it, and what `rebuild=True` costs.
+- `docs/api/Deployment.md` was headed `generate(rebuild=False)` and never said what the parameter
+  did. It does now.
 
 ## [1.1.0] - 2026-08-27
 

docs/api/Deployment.md

@@ -42,6 +42,12 @@ Validates the authored config, resolves synthetic or approved member-cache entri
 the mapping and review artifact. OLAF deliberately does not call Microsoft Graph; this is a design
 choice, not a claim that Graph access is universally impossible in Fabric notebooks.
 
+`rebuild=False` (the default) is idempotent: an unchanged config rebuilds nothing and returns
+`status=skipped`. The key is `config_hash`, which fingerprints the config rows only — so a table
+that has appeared since the last generate and matches an existing glob is not picked up, and is
+not granted, until `rebuild=True` forces a full re-resolution. See
+[modes.md](../modes.md) for the skip's five exceptions.
+
 Generated mapping/control artifacts are sensitive. Do not commit a real mapping, output, or
 workbook to the public repository.
 

docs/architecture.md

@@ -788,7 +788,7 @@ records when generate ran. Full detail: data-model.md.
 ## Key invariants
 
 1. The plan/apply role build reads **only** the mapping lock-file — never the short config (TOCTOU closed).
-2. New tables are absent from the saved mapping until the next generate. The resulting platform access still depends on workspace/item permissions and engine/access mode; OLAF does not infer that nobody can read them.
+2. New tables are absent from the saved mapping until the next generate that actually rebuilds it. `config_hash` fingerprints the config rows and cannot see the catalog, so an unchanged config takes the idempotency skip and a table matching an existing glob stays out of the mapping — `rebuild=True` is the deliberate way in. The resulting platform access still depends on workspace/item permissions and engine/access mode; OLAF does not infer that nobody can read them.
 3. Glob (A2): 0-match on an include OR an exclude = error; table schema part is literal-only. Case resolves through the catalog. Role and predicate limits are OLAF compatibility guards tied to the cited current platform pages, not permanent no-workaround guarantees.
 4. `NOT IN` against a nullable column drops NULL rows silently — three-valued logic makes the term UNKNOWN for a NULL and `WHERE` keeps only TRUE. **This never widens access; it can only over-reject** — a row vanishes from a deny-list's result, so the failure is missing data, not exposure. Guard it with `OR <col> IS NULL`, or assert the column is never NULL; do one of the two deliberately. OLAF warned about this per row (guardrail G3) until the release noted in the CHANGELOG and no longer does: the check was a substring test on the raw condition, so it fired on `status = 'CANNOT INVOICE'` (no `NOT IN` operator at all) and stayed silent on `region NOT IN ('a') AND type IS NULL` (a different column's `IS NULL` muted it). It could not be made sound without the literal-stripping lexer the other RLS rules share, and it named only one of the two mitigations above.
 5. Cross-row rules (see the Rule catalog above) warn/block ambiguous multi-role policy shapes. Effective access is engine-explicit because SQL endpoint CLS differs from non-SQL CLS. C5 is a conservative guard around Microsoft's documented unsupported RLS/CLS combinations; it does not extrapolate untested behavior across tables or membership paths.

docs/config-examples.md

@@ -100,8 +100,12 @@ name lookups against the catalog.
 | SalesRead | /Tables/sales/returns | Table | sg-analysts |
 
 **Takeaway:** the wildcard is resolved against the live catalog at generate time — a new
-`sales.*` table shows up in the mapping on the next generate with no config change needed. The
-schema part (`sales`) stays literal; only the table part accepts `*`/`?` (rule A2).
+`sales.*` table needs no config change to be covered, but it does need a generate that actually
+*runs*. `generate` is idempotent by default and its skip is keyed on `config_hash`, which
+fingerprints the config rows and cannot see the catalog: an unchanged config skips, and the new
+table stays out of the mapping — and ungranted — until `rebuild=True` (or an edit that changes the
+hash) forces a real generate. The schema part (`sales`) stays literal; only the table part accepts
+`*`/`?` (rule A2). See [modes.md](modes.md) for the skip and its exceptions.
 
 ### 3. All-except (`sales.*` minus `sales.returns`)
 
@@ -218,7 +222,8 @@ declares (include and exclude alike), not only the survivors that reach the mapp
 
 **Takeaway:** `visible_columns` holds the allow-list OLAF places in the DAR request,
 not the authored exclusions. In blacklist mode, a newly discovered column enters the
-allow-list on the next generate unless explicitly excluded; until regeneration, it is
+allow-list on the next generate that actually runs, unless explicitly excluded — an
+unchanged config takes the idempotency skip, so until a real regeneration the column is
 absent from OLAF's saved payload. Actual engine enforcement is outside this mapping
 example and differs by access path. See Microsoft's
 [CLS semantics](https://learn.microsoft.com/en-us/fabric/onelake/security/data-access-control-model#column-level-security).

docs/modes.md

@@ -73,6 +73,42 @@ A changed or unknown post-write boundary stops the chain, leaves the sentinel, a
 reports `possible_exposure` with the affected artifact/table version. Deleting the
 artifact would be containment only and is not described as erasing exposure.
 
+### The idempotency skip, and `rebuild`
+
+`generate(rebuild=False)` — the default — is idempotent. When the config is unchanged
+since the last generate it rebuilds nothing, logs one `no_change` row, and returns
+`status=skipped`, `changed=false`, so a pipeline can call generate every run and
+re-plan only when something moved.
+
+The comparison key is `config_hash`, a content fingerprint of the **config rows**.
+That is the whole of what it sees, and two consequences follow:
+
+- **It cannot see the Delta catalog.** A table created since the last generate that
+  matches an existing table glob leaves `config_hash` byte-identical, so the skip
+  holds and the new table is neither resolved into the mapping nor granted. This is
+  deliberate — a table appearing in a lakehouse is not consent to share it — and
+  `rebuild=True` is the deliberate way in. It re-resolves *every* pattern, so it also
+  takes in every other table added since the last generate, and drops from the mapping
+  any table that has since been deleted.
+- **It cannot see `onelake_security_member`**, which is a live input to every member
+  list. The exceptions below close that gap.
+
+Five things defeat the skip with no parameter, and a `rebuild=True` run never takes it:
+
+1. a config carrying a member `glob:` pattern is never eligible — the member table is
+   then a live grant list, and a principal added there must not be silently skipped;
+2. stamped member objectIds that no longer match what the member table resolves the
+   same names to today;
+3. a mapping stamped for a different workspace/lakehouse than the attached one;
+4. a member table carrying resolution errors — the fast path must not certify a state
+   the full gate calls a hard error;
+5. a mapping stamped by a different `framework_version` — content validated under
+   another version's rules has not been validated under these.
+
+Neither the skip nor a rebuild denies anything on its own. A table OLAF has not granted
+is simply one OLAF has not granted; whether anyone can read it still depends on
+workspace/item permissions and the access path.
+
 OLAF's RLS parser is a conservative guard, not an exhaustive service grammar. The
 current platform source is Microsoft's
 [RLS syntax](https://learn.microsoft.com/en-us/fabric/onelake/security/row-level-security-syntax).

notebooks/olaf_master_workflow.ipynb

@@ -165,6 +165,7 @@
     "| `env`             | `\"dev\"`                                  | Tags every log row, so one estate's dev and prod audit trails stay apart.  |\n",
     "| `auto_approve`    | `False`                                  | The gate. `True` lets `apply` run unattended — nobody reviews the omissions. |\n",
     "| `keep_unmanaged`  | `False`                                  | `False` = config is the whole truth: a live role it omits is left out of the payload. |\n",
+    "| `rebuild`         | `False`                                  | `True` re-resolves wildcards against the live catalog even when the config has not changed. |\n",
     "| `run_setup`       | `False`                                  | `True` after a framework upgrade — `setup` migrates control-table columns. |\n",
     "| `load_config`     | `False`                                  | `True` re-reads the authored workbook into the config + member tables.     |\n",
     "| `config_workbook` | `\"Files/security/onelake_security.xlsx\"` | The authored workbook, resolved on the attached lakehouse.                 |\n",
@@ -186,6 +187,7 @@
     "lakehouse_name = \"YourLakehouse\"  # the attached lakehouse this run secures (setup asserts it)\n",
     "auto_approve = False\n",
     "keep_unmanaged = False\n",
+    "rebuild = False\n",
     "run_setup = False\n",
     "load_config = False\n",
     "config_workbook = \"Files/security/onelake_security.xlsx\"\n",
@@ -213,6 +215,7 @@
     "#                   constant here through 1.0.x, invisible to Base parameters)\n",
     "#   AUTO_APPROVE    True = run apply          ·  False = stop after plan\n",
     "#   KEEP_UNMANAGED  False = config is whole truth ·  omits roles absent from the config\n",
+    "#   REBUILD         True = re-resolve wildcards against the live catalog · see cell 13\n",
     "#   LOAD_CONFIG     True = reload the workbook into the config + member tables\n",
     "#   WORKBOOK        the authored workbook, resolved on the attached lakehouse\n",
     "#   BATCH_ID        the passed-in batch_id, or a fresh one when none was supplied\n",
@@ -223,6 +226,7 @@
     "LAKEHOUSE = lakehouse_name\n",
     "AUTO_APPROVE = auto_approve\n",
     "KEEP_UNMANAGED = keep_unmanaged\n",
+    "REBUILD = rebuild\n",
     "LOAD_CONFIG = load_config\n",
     "WORKBOOK = config_workbook\n",
     "BATCH_ID = batch_id.strip() or str(uuid.uuid4())\n",
@@ -498,7 +502,16 @@
     "`apply` read, and export the versioned CSV to the mapping-history folder.\n",
     "\n",
     "`skipped` means the config hash has not changed and the existing lock-file is still current — that\n",
-    "is a pass, not a problem.\n"
+    "is a pass, not a problem.\n",
+    "\n",
+    "**`rebuild` defaults to `False`, and that is an access decision, not a performance one.** The skip\n",
+    "is keyed on `config_hash`, which fingerprints the config rows only — it cannot see the live\n",
+    "catalog. So a table that appeared since the last generate and matches a wildcard (`sales.*`) is\n",
+    "**not** in the lock-file, and nothing is granted on it: a table created by last night's load does\n",
+    "not become readable because a star matched it. Opening it up is a decision somebody makes by\n",
+    "passing `rebuild = True` — which is all-or-nothing, re-resolving *every* wildcard, so it takes in\n",
+    "**every** table that has appeared since the last generate. Note the converse too: a table that was\n",
+    "**dropped** stays in the lock-file until the next real generate.\n"
    ]
   },
   {
@@ -509,7 +522,7 @@
    "outputs": [],
    "source": [
     "try:\n",
-    "    generate = run_stage(\"generate\")\n",
+    "    generate = run_stage(\"generate\", {\"rebuild\": REBUILD})\n",
     "\n",
     "    if generate[\"status\"] == \"skipped\":\n",
     "        print(\"    → config unchanged, reusing the current lock-file\")\n",