diff --git a/CHANGELOG.md b/CHANGELOG.md index b94b84f..39f4eab 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,35 @@ Notable changes to OLAF — OneLake Access Framework — are recorded here using ## [Unreleased] -No changes yet. +### Added + +- **`olaf_master_workflow` takes a `rebuild` parameter, defaulting to `False`**, and passes it to + `generate`. The runner previously called `generate` with no arguments, so there was no way to + ask for a rebuild from the notebook a pipeline actually runs — the framework had the parameter, + the workflow did not expose it. The default is an access decision rather than a performance one: + `generate`'s idempotency skip is keyed on `config_hash`, which fingerprints the config rows and + cannot see the catalog, so a table created since the last generate that matches an existing + wildcard stays out of the mapping and ungranted until somebody deliberately passes + `rebuild = True`. Taking it in is all-or-nothing — every wildcard is re-resolved, so every table + added since the last generate arrives with it. + +### Fixed + +- `docs/config-examples.md` said a new table matching `sales.*` "shows up in the mapping on the + next generate with no config change needed". The first half is right and the second half is + what makes it wrong: an unchanged config takes the idempotency skip, so the *next* generate is + usually the one that rebuilds nothing. The example now says the table needs a generate that + actually runs, and names `rebuild=True` as the way to get one. The CLS blacklist example, which + described a newly discovered column the same way, is corrected alongside it. +- `docs/architecture.md`'s second key invariant read "New tables are absent from the saved mapping + until the next generate", which is true only of a generate that rebuilds. It now says so, and + says why `config_hash` cannot see the difference. +- `docs/modes.md` documented every mode except the one behaviour an operator meets most often: its + `generate` section never mentioned the idempotency skip, even though the result-envelope table + above it already advertises `status=skipped`. It now carries the skip, what `config_hash` can and + cannot see, the five exceptions that defeat it, and what `rebuild=True` costs. +- `docs/api/Deployment.md` was headed `generate(rebuild=False)` and never said what the parameter + did. It does now. ## [1.1.0] - 2026-08-27 diff --git a/docs/api/Deployment.md b/docs/api/Deployment.md index 32e36c3..82e9a0a 100644 --- a/docs/api/Deployment.md +++ b/docs/api/Deployment.md @@ -42,6 +42,12 @@ Validates the authored config, resolves synthetic or approved member-cache entri the mapping and review artifact. OLAF deliberately does not call Microsoft Graph; this is a design choice, not a claim that Graph access is universally impossible in Fabric notebooks. +`rebuild=False` (the default) is idempotent: an unchanged config rebuilds nothing and returns +`status=skipped`. The key is `config_hash`, which fingerprints the config rows only — so a table +that has appeared since the last generate and matches an existing glob is not picked up, and is +not granted, until `rebuild=True` forces a full re-resolution. See +[modes.md](../modes.md) for the skip's five exceptions. + Generated mapping/control artifacts are sensitive. Do not commit a real mapping, output, or workbook to the public repository. diff --git a/docs/architecture.md b/docs/architecture.md index 9ba6fe7..0a9cfdb 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -788,7 +788,7 @@ records when generate ran. Full detail: data-model.md. ## Key invariants 1. The plan/apply role build reads **only** the mapping lock-file — never the short config (TOCTOU closed). -2. New tables are absent from the saved mapping until the next generate. The resulting platform access still depends on workspace/item permissions and engine/access mode; OLAF does not infer that nobody can read them. +2. New tables are absent from the saved mapping until the next generate that actually rebuilds it. `config_hash` fingerprints the config rows and cannot see the catalog, so an unchanged config takes the idempotency skip and a table matching an existing glob stays out of the mapping — `rebuild=True` is the deliberate way in. The resulting platform access still depends on workspace/item permissions and engine/access mode; OLAF does not infer that nobody can read them. 3. Glob (A2): 0-match on an include OR an exclude = error; table schema part is literal-only. Case resolves through the catalog. Role and predicate limits are OLAF compatibility guards tied to the cited current platform pages, not permanent no-workaround guarantees. 4. `NOT IN` against a nullable column drops NULL rows silently — three-valued logic makes the term UNKNOWN for a NULL and `WHERE` keeps only TRUE. **This never widens access; it can only over-reject** — a row vanishes from a deny-list's result, so the failure is missing data, not exposure. Guard it with `OR IS NULL`, or assert the column is never NULL; do one of the two deliberately. OLAF warned about this per row (guardrail G3) until the release noted in the CHANGELOG and no longer does: the check was a substring test on the raw condition, so it fired on `status = 'CANNOT INVOICE'` (no `NOT IN` operator at all) and stayed silent on `region NOT IN ('a') AND type IS NULL` (a different column's `IS NULL` muted it). It could not be made sound without the literal-stripping lexer the other RLS rules share, and it named only one of the two mitigations above. 5. Cross-row rules (see the Rule catalog above) warn/block ambiguous multi-role policy shapes. Effective access is engine-explicit because SQL endpoint CLS differs from non-SQL CLS. C5 is a conservative guard around Microsoft's documented unsupported RLS/CLS combinations; it does not extrapolate untested behavior across tables or membership paths. diff --git a/docs/config-examples.md b/docs/config-examples.md index 6c390b0..a5ac6b1 100644 --- a/docs/config-examples.md +++ b/docs/config-examples.md @@ -100,8 +100,12 @@ name lookups against the catalog. | SalesRead | /Tables/sales/returns | Table | sg-analysts | **Takeaway:** the wildcard is resolved against the live catalog at generate time — a new -`sales.*` table shows up in the mapping on the next generate with no config change needed. The -schema part (`sales`) stays literal; only the table part accepts `*`/`?` (rule A2). +`sales.*` table needs no config change to be covered, but it does need a generate that actually +*runs*. `generate` is idempotent by default and its skip is keyed on `config_hash`, which +fingerprints the config rows and cannot see the catalog: an unchanged config skips, and the new +table stays out of the mapping — and ungranted — until `rebuild=True` (or an edit that changes the +hash) forces a real generate. The schema part (`sales`) stays literal; only the table part accepts +`*`/`?` (rule A2). See [modes.md](modes.md) for the skip and its exceptions. ### 3. All-except (`sales.*` minus `sales.returns`) @@ -218,7 +222,8 @@ declares (include and exclude alike), not only the survivors that reach the mapp **Takeaway:** `visible_columns` holds the allow-list OLAF places in the DAR request, not the authored exclusions. In blacklist mode, a newly discovered column enters the -allow-list on the next generate unless explicitly excluded; until regeneration, it is +allow-list on the next generate that actually runs, unless explicitly excluded — an +unchanged config takes the idempotency skip, so until a real regeneration the column is absent from OLAF's saved payload. Actual engine enforcement is outside this mapping example and differs by access path. See Microsoft's [CLS semantics](https://learn.microsoft.com/en-us/fabric/onelake/security/data-access-control-model#column-level-security). diff --git a/docs/modes.md b/docs/modes.md index 0c2c15e..b14039a 100644 --- a/docs/modes.md +++ b/docs/modes.md @@ -73,6 +73,42 @@ A changed or unknown post-write boundary stops the chain, leaves the sentinel, a reports `possible_exposure` with the affected artifact/table version. Deleting the artifact would be containment only and is not described as erasing exposure. +### The idempotency skip, and `rebuild` + +`generate(rebuild=False)` — the default — is idempotent. When the config is unchanged +since the last generate it rebuilds nothing, logs one `no_change` row, and returns +`status=skipped`, `changed=false`, so a pipeline can call generate every run and +re-plan only when something moved. + +The comparison key is `config_hash`, a content fingerprint of the **config rows**. +That is the whole of what it sees, and two consequences follow: + +- **It cannot see the Delta catalog.** A table created since the last generate that + matches an existing table glob leaves `config_hash` byte-identical, so the skip + holds and the new table is neither resolved into the mapping nor granted. This is + deliberate — a table appearing in a lakehouse is not consent to share it — and + `rebuild=True` is the deliberate way in. It re-resolves *every* pattern, so it also + takes in every other table added since the last generate, and drops from the mapping + any table that has since been deleted. +- **It cannot see `onelake_security_member`**, which is a live input to every member + list. The exceptions below close that gap. + +Five things defeat the skip with no parameter, and a `rebuild=True` run never takes it: + +1. a config carrying a member `glob:` pattern is never eligible — the member table is + then a live grant list, and a principal added there must not be silently skipped; +2. stamped member objectIds that no longer match what the member table resolves the + same names to today; +3. a mapping stamped for a different workspace/lakehouse than the attached one; +4. a member table carrying resolution errors — the fast path must not certify a state + the full gate calls a hard error; +5. a mapping stamped by a different `framework_version` — content validated under + another version's rules has not been validated under these. + +Neither the skip nor a rebuild denies anything on its own. A table OLAF has not granted +is simply one OLAF has not granted; whether anyone can read it still depends on +workspace/item permissions and the access path. + OLAF's RLS parser is a conservative guard, not an exhaustive service grammar. The current platform source is Microsoft's [RLS syntax](https://learn.microsoft.com/en-us/fabric/onelake/security/row-level-security-syntax). diff --git a/notebooks/olaf_master_workflow.ipynb b/notebooks/olaf_master_workflow.ipynb index 4e34ef7..f79d33a 100644 --- a/notebooks/olaf_master_workflow.ipynb +++ b/notebooks/olaf_master_workflow.ipynb @@ -165,6 +165,7 @@ "| `env` | `\"dev\"` | Tags every log row, so one estate's dev and prod audit trails stay apart. |\n", "| `auto_approve` | `False` | The gate. `True` lets `apply` run unattended — nobody reviews the omissions. |\n", "| `keep_unmanaged` | `False` | `False` = config is the whole truth: a live role it omits is left out of the payload. |\n", + "| `rebuild` | `False` | `True` re-resolves wildcards against the live catalog even when the config has not changed. |\n", "| `run_setup` | `False` | `True` after a framework upgrade — `setup` migrates control-table columns. |\n", "| `load_config` | `False` | `True` re-reads the authored workbook into the config + member tables. |\n", "| `config_workbook` | `\"Files/security/onelake_security.xlsx\"` | The authored workbook, resolved on the attached lakehouse. |\n", @@ -186,6 +187,7 @@ "lakehouse_name = \"YourLakehouse\" # the attached lakehouse this run secures (setup asserts it)\n", "auto_approve = False\n", "keep_unmanaged = False\n", + "rebuild = False\n", "run_setup = False\n", "load_config = False\n", "config_workbook = \"Files/security/onelake_security.xlsx\"\n", @@ -213,6 +215,7 @@ "# constant here through 1.0.x, invisible to Base parameters)\n", "# AUTO_APPROVE True = run apply · False = stop after plan\n", "# KEEP_UNMANAGED False = config is whole truth · omits roles absent from the config\n", + "# REBUILD True = re-resolve wildcards against the live catalog · see cell 13\n", "# LOAD_CONFIG True = reload the workbook into the config + member tables\n", "# WORKBOOK the authored workbook, resolved on the attached lakehouse\n", "# BATCH_ID the passed-in batch_id, or a fresh one when none was supplied\n", @@ -223,6 +226,7 @@ "LAKEHOUSE = lakehouse_name\n", "AUTO_APPROVE = auto_approve\n", "KEEP_UNMANAGED = keep_unmanaged\n", + "REBUILD = rebuild\n", "LOAD_CONFIG = load_config\n", "WORKBOOK = config_workbook\n", "BATCH_ID = batch_id.strip() or str(uuid.uuid4())\n", @@ -498,7 +502,16 @@ "`apply` read, and export the versioned CSV to the mapping-history folder.\n", "\n", "`skipped` means the config hash has not changed and the existing lock-file is still current — that\n", - "is a pass, not a problem.\n" + "is a pass, not a problem.\n", + "\n", + "**`rebuild` defaults to `False`, and that is an access decision, not a performance one.** The skip\n", + "is keyed on `config_hash`, which fingerprints the config rows only — it cannot see the live\n", + "catalog. So a table that appeared since the last generate and matches a wildcard (`sales.*`) is\n", + "**not** in the lock-file, and nothing is granted on it: a table created by last night's load does\n", + "not become readable because a star matched it. Opening it up is a decision somebody makes by\n", + "passing `rebuild = True` — which is all-or-nothing, re-resolving *every* wildcard, so it takes in\n", + "**every** table that has appeared since the last generate. Note the converse too: a table that was\n", + "**dropped** stays in the lock-file until the next real generate.\n" ] }, { @@ -509,7 +522,7 @@ "outputs": [], "source": [ "try:\n", - " generate = run_stage(\"generate\")\n", + " generate = run_stage(\"generate\", {\"rebuild\": REBUILD})\n", "\n", " if generate[\"status\"] == \"skipped\":\n", " print(\" → config unchanged, reusing the current lock-file\")\n",