diff --git a/README.md b/README.md index 9a4147c..84a6656 100644 --- a/README.md +++ b/README.md @@ -11,24 +11,23 @@

tests + Coverage: 100% against in-memory fixtures + Latest release tag License: MIT Status: community Preview

-> [!IMPORTANT] -> **OLAF v1.0.0 is an independent community Preview for evaluation and -> development, not a production-ready security product.** Its mutating path -> depends on Microsoft's bulk Data Access Roles `PUT`, which is officially -> documented as **Preview** and not recommended for production use. -> [Official endpoint status](https://learn.microsoft.com/en-us/rest/api/fabric/core/onelake-data-access-security/create-or-update-data-access-roles) +

+ Python 3.11, 3.12 and 3.13 + Microsoft Fabric Runtime 1.3 / Spark 3.5 or newer + Dependencies installed with --require-hashes + Linted and formatted with ruff +

-> [!CAUTION] -> OLAF stores principal identifiers and authorization/recovery state in control -> tables and under `Files/security`. Sensitive modes are disabled by default and -> require a clean DAR snapshot with an ETag. The per-run workspace isolation -> attestation is optional — OLAF records it as `attested` or `unknown` and -> never gates on it. Complete the external access review **before uploading a -> real workbook**. Start with [Protecting OLAF control data](docs/control-data-security.md). +> [!IMPORTANT] +> **An independent community Preview for evaluation and development, not a production-ready +> security product.** Its mutating path uses Microsoft's bulk Data Access Roles `PUT`, which +> Microsoft documents as [Preview and not recommended for production](https://learn.microsoft.com/en-us/rest/api/fabric/core/onelake-data-access-security/create-or-update-data-access-roles). **OLAF — OneLake Access Framework** is a plan → review → apply workflow for Microsoft Fabric OneLake data access roles. It is one self-contained Fabric @@ -40,6 +39,46 @@ The product names above are used only to describe interoperability. See [Microsoft's Trademark and Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks) and the project's [platform contract](docs/platform-contract.md). +## What is OneLake security? + +[OneLake security](https://learn.microsoft.com/en-us/fabric/onelake/security/data-access-control-model) +is Microsoft Fabric's native, data-plane access control: you define **security roles on the data +itself** (a lakehouse's tables, folders, rows, columns) and OneLake enforces them everywhere — instead +of re-implementing security per engine with T-SQL views, workspace boundaries, or report-side filters. + +- **Role-based access** — grant Read per table (`/Tables/schema/table`) or folder (`/Files/...`); + users with no role see nothing (**deny by default**). + [Create and manage roles →](https://learn.microsoft.com/en-us/fabric/onelake/security/create-manage-roles) +- **Row-level security (RLS)** — SQL-like predicates per table per role (static values; the + OneLake-supported operator subset — `= <> > >= < <= IN NOT AND OR IS BLANK NULL TRUE FALSE`, + rule C9). A member of several roles gets the **union** of what they allow. + [Row-level security →](https://learn.microsoft.com/en-us/fabric/onelake/security/row-level-security) +- **Column-level security (CLS)** — hide specific columns per role. + [Column-level security →](https://learn.microsoft.com/en-us/fabric/onelake/security/column-level-security) +- **Entra-native membership** — role members are Entra users, **security groups**, or service + principals; membership changes happen in Entra, no sync tables to maintain. +- **One definition, every engine** — enforced across the SQL analytics endpoint, Spark, and Direct + Lake semantic models (Power BI), so reports and notebooks see the same filtered data. +- **Granular write** — separate ReadWrite roles for scoped write access (RLS/CLS don't apply there). + +Worth knowing: roles carrying RLS/CLS are read-only; workspace Admins/Members/Contributors bypass +OneLake security (it governs consumers); role definitions are per lakehouse. + +## Why OLAF? + +OneLake security provides the enforcement layer. OLAF makes operating that layer **reviewable, +repeatable, and easier to govern** when deployments grow beyond a handful of roles and tables. + +| What you need | What OLAF does | +|---|---| +| **Know what will change** | `plan` shows every role and grant that will be created, updated, or omitted from the submitted payload. | +| **Prevent an unreviewed write** | `apply` requires a saved plan and refuses stale or drifted state. | +| **Catch risky configuration early** | Validation blocks common RLS/CLS, predicate, casing, and multi-role exposure mistakes before deployment. | +| **Keep the runtime simple** | One unchanged notebook serves every project; configuration and runtime parameters carry project intent. | +| **Explain what happened** | Plan and apply are linked in the audit log with who, what, when, and the relevant configuration state. | + +*Full rule-by-rule detail → [docs/architecture.md](docs/architecture.md).* + ## What OLAF does | Need | OLAF behavior | @@ -58,6 +97,14 @@ and [SQL endpoint enforcement guidance](https://learn.microsoft.com/en-us/fabric ## Before you start +> [!CAUTION] +> OLAF stores principal identifiers and authorization/recovery state in control +> tables and under `Files/security`. Sensitive modes are disabled by default and +> require a clean DAR snapshot with an ETag. The per-run workspace isolation +> attestation is optional — OLAF records it as `attested` or `unknown` and +> never gates on it. Complete the external access review **before uploading a +> real workbook**. Start with [Protecting OLAF control data](docs/control-data-security.md). + - Use Microsoft Fabric Runtime 1.3 / Spark 3.5 or newer, then verify the selected runtime and bundled package versions in your Fabric environment. Microsoft publishes the [runtime lifecycle](https://learn.microsoft.com/en-us/fabric/data-engineering/lifecycle) @@ -68,7 +115,7 @@ and [SQL endpoint enforcement guidance](https://learn.microsoft.com/en-us/fabric and the [bulk DAR authorization contract](https://learn.microsoft.com/en-us/rest/api/fabric/core/onelake-data-access-security/create-or-update-data-access-roles). - Treat the same-lakehouse layout as a trusted-administrator boundary, not cryptographic or transactional isolation. If that threat model is unacceptable, - do not import real principal data or run sensitive modes in v1.0.0. + do not import real principal data or run sensitive modes in this Preview. ## Quick start @@ -211,6 +258,34 @@ partial or ambiguous operation. Follow - [Testing guide](docs/testing.md) - [Roadmap](docs/roadmap.md) +## Roadmap + +Options, not promises or release dates. Each is blocked on a specific, checkable condition, and none +may weaken the control-data boundary or turn an observed service behavior into a platform contract. +**[docs/roadmap.md](docs/roadmap.md)** carries the detail and the official sources behind each one. + +- [ ] **Differential apply** — per-role writes instead of one bulk `PUT`, so an apply touches only + what changed. *Blocked on a stable, concurrency-safe official contract: the single-role + endpoints are themselves Preview, and the design needs documented precondition behavior plus a + recovery model for a partially completed role set.* +- [ ] **RLS + CLS cross-grant detection at `generate`** — closing rule C5's blind spot, where a user + arrives through a group on one side and directly on the other. *The conservative rule stays + until evidence justifies otherwise; OLAF deliberately does not call Microsoft Graph.* +- [ ] **Scheduled drift detection** — a gate an adopter schedules rather than wires themselves. + *Must stay read-only, separate identity drift from policy drift, and keep real principal values + out of CI logs.* +- [ ] **Local authoring tool** — validate and build a config from a developer machine against a + real workspace, so table names, member resolution and rule violations surface while the + config is still being written instead of at `generate`. *Blocked on where real identifiers + and a Fabric token are allowed to live once they leave the Fabric boundary; validation only, + never a second write path.* +- [ ] **Separate control store** — a separately secured store or independently managed keys, for + organizations that cannot accept v1's trusted-administrator boundary. *Not a minimal v1 change.* + +**Not planned:** replacing Fabric enforcement with an OLAF query layer · workspace and item RBAC · +calling Microsoft Graph in v1 · claiming production readiness while the required endpoint is Preview +— [the full list and why](docs/roadmap.md#not-planned). + ## Repository layout ```text @@ -233,6 +308,15 @@ General questions and sanitized bug reports belong in [GitHub issues](https://github.com/kengio/olaf/issues). See [SUPPORT.md](SUPPORT.md) and [CONTRIBUTING.md](CONTRIBUTING.md). +

+ The OLAF Arctic Owl Sentinel — a crystalline owl holding a notebook that carries the OLAF shield crest and its padlock +

+ +

+ The Arctic Owl keeps watch over your lake — every change is seen, reviewed,
+ and approved before it is submitted. +

+ ## License and identity Code, documentation, and project artwork are licensed under [MIT](LICENSE), with diff --git a/assets/brand/PROVENANCE.md b/assets/brand/PROVENANCE.md index 4f75741..4c0feda 100644 --- a/assets/brand/PROVENANCE.md +++ b/assets/brand/PROVENANCE.md @@ -9,9 +9,9 @@ personal attribution or private working files. |---|---|---| | `olaf-logo.png` | `68f46d4054cce80b23f5c8255f5f0c2c306da116adbbe3c094ca0ec0d78efa28` | 512 × 512 px | | `olaf-mascot.png` | `3078c87bd9d9a74030fb73df034f17380d01387a8dde29edf6dca5aa19740f9c` | 512 × 512 px | -| `olaf-lockup-light.png` | `3bb7dc420e58bae403d267277f98ad1807cf859e48c30a01b3d1780899fdb887` | 1200 × 600 px | -| `olaf-lockup-dark.png` | `0a8dcda65273cc73a079cc7b068a046dd9010c65ea50057f1dca42d84a4c0b33` | 1200 × 600 px | -| `olaf-social-preview.png` | `f7cb6da4018700fccc49e59185d329b7491c0c5512bb50e7fc2a68306c07e9db` | 1200 × 630 px | +| `olaf-lockup-light.png` | `f005a1b558ae309a05aacf59097a57022d91f86c45f146469f68db213dc29325` | 1280 × 640 px | +| `olaf-lockup-dark.png` | `7ad09f4cf95564e35a5270050ae8dda75553dae1c8f8aa6b892679527cf6b94e` | 1280 × 640 px | +| `olaf-social-preview.png` | `c5d1fec2a4925a5babad59920c0fd5b475778d288c6285efee45a6ac12ce0e0e` | 1280 × 640 px | ## Origin and transformations @@ -20,9 +20,15 @@ contributors using generative-image tools and contributor-directed cleanup. No customer material, personal likeness, private project asset, stock image, third-party logo, Microsoft logo, or Microsoft product icon was supplied as source material. -The social-preview image is a deterministic composition of `olaf-logo.png` centered -on the project's Frost/Ice background. It contains no text, embedded profile, or -personal metadata and introduces no third-party mark. +The social-preview image places the arctic-owl shield beside the OLAF wordmark on the +project's Frost/Ice background, with the project name and the plan → review → apply +sequence set as text. It carries no embedded profile or personal metadata and +introduces no third-party mark. Its predecessor was a text-free composition of +`olaf-logo.png` alone; the text was added so a shared link says what the project is +without the reader having to open it. + +The project name it sets is the same owner-selected name the lockups carry, so the +trademark note below applies to this file too. The lockups contain the owner-selected project name “OLAF — OneLake Access Framework.” This referential product-name use carries the trademark risk documented diff --git a/assets/brand/olaf-lockup-dark.png b/assets/brand/olaf-lockup-dark.png index a61ad72..1d8c765 100644 Binary files a/assets/brand/olaf-lockup-dark.png and b/assets/brand/olaf-lockup-dark.png differ diff --git a/assets/brand/olaf-lockup-light.png b/assets/brand/olaf-lockup-light.png index d89d2e5..0dc7388 100644 Binary files a/assets/brand/olaf-lockup-light.png and b/assets/brand/olaf-lockup-light.png differ diff --git a/assets/brand/olaf-social-preview.png b/assets/brand/olaf-social-preview.png index 4cc0be1..fdd9273 100644 Binary files a/assets/brand/olaf-social-preview.png and b/assets/brand/olaf-social-preview.png differ diff --git a/docs/roadmap.md b/docs/roadmap.md index f8beea9..34f7770 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -37,6 +37,39 @@ OLAF deliberately does not call Microsoft Graph. That is a project design choice not a claim that Graph tokens are universally unavailable: [NotebookUtils token audiences](https://learn.microsoft.com/en-us/fabric/data-engineering/notebookutils/notebookutils-credentials#get-token). +## Local authoring tool — validate and build config against a real workspace + +**Status: planned; blocked on where real identifiers are allowed to live.** + +Today a config is authored blind. The workbook is filled in against what the author +believes the lakehouse contains, and the first time anything checks it against reality +is `generate` inside Fabric. A tool that runs on a developer machine, authenticates +with Fabric credentials, and reads the live workspace could resolve table and column +names, check member principals resolve, and surface rule violations while the config +is still being written — a `plan` for the config itself, before a notebook is involved. + +What has to be settled first is not the API surface. It is that such a tool moves real +tenant data onto a laptop: + +- **Credentials.** It would hold or broker a token with `OneLake.ReadWrite.All` scope + outside the Fabric boundary. The token, its cache, and its lifetime all become the + tool's responsibility, and a stolen laptop becomes a DAR-write capability. Read-only + scope for authoring is the obvious first constraint; whether that is sufficient to + build a config is not yet established. +- **Real identifiers on disk.** Resolving members means reading real principal object + ids and display names. v1's whole control-data posture says do not put those where + they are not governed — see [Protecting OLAF control data](control-data-security.md). + A local cache, a shell history, an editor's undo file and a crash dump are all places + they would land by default. +- **What it must never become.** An authoring tool reads and validates. It must not + grow a write path to the DAR collection, because that would move the reviewed + plan → approve → apply sequence off the audited runtime and onto a machine with no + audit trail. Validation only, with the deployment path unchanged. + +Until those are answered the honest position is that authoring stays in the workbook and +validation stays in `generate` and `validate`, which already run every rule with zero +writes. + ## Scheduled drift detection **Status: candidate automation after the read-only contract is stable.** diff --git a/notebooks/olaf_cookbook.ipynb b/notebooks/olaf_cookbook.ipynb index f99d1d9..865e03e 100644 --- a/notebooks/olaf_cookbook.ipynb +++ b/notebooks/olaf_cookbook.ipynb @@ -2,132 +2,1120 @@ "cells": [ { "cell_type": "markdown", - "id": "cookbook-intro", + "id": "title", "metadata": {}, "source": [ - "# OLAF cookbook — safe evaluation scaffold\n", + "# olaf_cookbook — copy-paste examples for the `OLAF` facade\n", "\n", - "> **Community Preview:** OLAF — OneLake Access Framework is independent and is not affiliated with, endorsed by, sponsored by, or certified by Microsoft. The bulk DAR mutation endpoint is Preview and documented for evaluation/development, not production use.\n", + "> **Attach a lakehouse (or bind one with `%%configure`) before running any cell** — every\n", + "> action below resolves against the attached lakehouse. Without one, the first call fails\n", + "> with `UnsupportedOperationException — No default context`. See\n", + "> [fabric-import.md](../docs/fabric-import.md) for both binding paths.\n", "\n", - "This notebook contains unexecuted examples only. It has no stored outputs, no live verification evidence, and no permission to access a Fabric environment. Use only synthetic values such as `example.user@example.invalid`. Read [the platform contract](../docs/platform-contract.md), [control-data security](../docs/control-data-security.md), and [the runbook](../docs/runbook.md) before importing any notebook.\n" + "Worked, one-per-cell examples of every interactive `OLAF.(...)` call on the\n", + "self-contained [`olaf`](olaf.ipynb) runtime. Each cell shows the\n", + "exact params the action consumes, a one-line note on what it does, and an **example output** (as a\n", + "trailing comment) so you can see the shape before running. **You don't need this file to run the\n", + "framework** — `olaf.ipynb` alone runs every mode; this cookbook is just runnable examples.\n", + "\n", + "> ⚠️ **Live-Fabric demo — NOT run in CI.** These cells call the live DAR REST API and read real\n", + "> control tables, so they only execute inside a Fabric workspace with `olaf` imported\n", + "> and a lakehouse attached. The CI test + coverage harness targets `olaf.ipynb` only;\n", + "> this cookbook is never executed or measured. Values below (`priya@contoso.com`, `SalesReaders`,\n", + "> `2026-07-15`, …) are illustrative placeholders. **The first cell is a guard** — running this notebook whole (Run All / a pipeline / `notebook.run`) exits immediately without touching anything; run the cells you want **one at a time**.\n", + "\n", + "Every `OLAF.*` method returns a **Spark DataFrame** for an easy `display(...)` — a query method\n", + "returns its result table; an ops method returns a compact DataFrame *view* of the outcome envelope\n", + "(the raw dict stays at `OLAF.last_result`)." ] }, { "cell_type": "code", - "id": "cookbook-guard", "execution_count": null, + "id": "9540642a", "metadata": {}, "outputs": [], "source": [ - "# Safety guard: Run All exits before any example can execute.\n", + "# Safety guard: Run All stops here. Every example below reaches a live lakehouse, so\n", + "# they are meant to be read first and run one at a time, deliberately.\n", "import notebookutils\n", "\n", "notebookutils.notebook.exit(\n", " \"OLAF cookbook is examples-only; review and run individual cells deliberately\"\n", - ")\n" + ")" ] }, { "cell_type": "markdown", - "id": "cookbook-controls", + "id": "toc", "metadata": {}, "source": [ - "## Before an individual evaluation cell\n", + "## Contents\n", "\n", - "Do not upload a real workbook before an external-access review. Sensitive modes, including first setup, are disabled by default. For every sensitive run, keep these independent controls: (1) the reviewed control-data boundary, (2) a per-run workspace-isolation attestation, and (3) a fresh immutable DAR snapshot with its ETag. OLAF creates/reads the sentinel before the first sensitive write and revalidates it with the snapshot before every sensitive write. Cleanup is containment, not proof of erasure or isolation; a same-lakehouse layout is not cryptographic or transactional isolation.\n" + "- **Load & configure** — `%run olaf` + `OLAF.configure(...)`\n", + "- **Deployment** — `explain` (dry preview) · `generate` · `validate` (zero-write dry-run) ·\n", + " `plan` · `apply` (full-replace / incremental) · `rollback`\n", + "- **Audit** — `show` (table / role / member) · `trace` · `grants` · `provenance` · `timeline` · `report` ·\n", + " compliance & live-DAR (`coverage` · `effective_access` · `who_can_access` · `drift`) ·\n", + " config time-travel & lineage (`table_history` · `at` · `config_diff` · `value_history`)\n", + "- **Maintenance** — `setup` · `health` · `status` · `diagnose_member`\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8352e9d69036", + "metadata": {}, + "outputs": [], + "source": [ + "# ─────────────────────────────────────────────────────────────────────────────\n", + "# STOP — this is an EXAMPLES notebook, not one to run whole.\n", + "# The cells below call LIVE operations (generate / apply / rollback / setup) against the\n", + "# attached lakehouse. Running this notebook end-to-end — Run All, a Data Factory pipeline,\n", + "# or notebookutils.notebook.run — would execute them for real. So this guard runs FIRST and\n", + "# exits immediately on any whole-notebook run (the %run below never even loads).\n", + "#\n", + "# To use the cookbook: run the cells you want ONE AT A TIME, deliberately (start at the\n", + "# %run cell), or copy the ones you need into your own notebook.\n", + "# ─────────────────────────────────────────────────────────────────────────────\n", + "import notebookutils\n", + "\n", + "notebookutils.notebook.exit(\n", + " \"olaf_cookbook is examples-only — run cells one at a time, not the whole notebook\"\n", + ")" ] }, { "cell_type": "markdown", - "id": "cookbook-load-doc", + "id": "1e406af3fd46", "metadata": {}, "source": [ - "## Load the runtime\n", + "Load the runtime. `%run` is Fabric's import: it defines every function/class + the `OLAF`\n", + "facade in this notebook's namespace. The ▶️ Run cell is guarded, so `%run` dispatches\n", + "NOTHING — `OLAF` is ready, exactly like `from olaf import OLAF`. (The pipeline path never\n", + "`%run`s; it passes `mode`.)\n", "\n", - "Run the next cell by itself only after reviewing the exact release commit and attaching an authorized non-production lakehouse. The runtime notebook remains the source of truth for the API.\n" + "> A `%run` cell must hold **nothing else** — no comments, no code. Fabric rejects the cell\n", + "> with `MagicUsageError: %run cannot run with other code or magic commands.` That is why this\n", + "> explanation is a markdown cell rather than a comment above the magic." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "load", + "metadata": {}, + "outputs": [], + "source": [ + "%run olaf" ] }, { "cell_type": "code", - "id": "cookbook-load", "execution_count": null, + "id": "configure", "metadata": {}, "outputs": [], "source": [ - "%run olaf\n" + "# Optional: set base params shared by all three OLAF namespaces, so you don't repeat them on\n", + "# every call. Anything the parameters cell accepts works here — env, tenant_id, and the\n", + "# control-table names if they differ from the olaf.* defaults.\n", + "#\n", + "# When:\n", + "# once at the top of a session, only if your env / tenant / table names differ from the dev\n", + "# defaults. Chainable; skip it entirely to run against the defaults.\n", + "# Expect:\n", + "# returns a DataFrame of every parameter the next run will use, each row marked\n", + "# set / default / per-call; every later OLAF.() then uses these base params.\n", + "OLAF.configure(\n", + " env=\"qa\",\n", + " tenant_id=\"00000000-0000-0000-0000-000000000000\", # \"\" = auto-resolve from the runtime context\n", + " config_table=\"ml_gold.security_config\", # override any control-table name —\n", + " log_table=\"ml_gold.security_log\", # they don't have to use the olaf.* defaults\n", + ")\n", + "\n", + "# Just changing the env? One-liner: OLAF.configure(env=\"dev\")" ] }, { "cell_type": "markdown", - "id": "cookbook-read-doc", + "id": "sec-deployment", "metadata": {}, "source": [ - "## Read and validation examples\n", + "## Deployment\n", "\n", - "These calls express OLAF's intended read/validation behavior. They can still perform external reads and therefore require an authorized target. No example output is embedded because this repository does not claim an exact-release live run.\n" + "The `generate → plan → apply → rollback` chain. `generate` freezes the config into the mapping\n", + "lock-file; `plan` diffs it against the live DAR; `apply` pushes it; `rollback` restores a prior\n", + "config version. All return a compact DataFrame view of the outcome (`OLAF.last_result` = raw dict)." ] }, { "cell_type": "code", - "id": "cookbook-read", "execution_count": null, + "id": "dep-explain", "metadata": {}, "outputs": [], "source": [ - "OLAF.show_params()\n", - "OLAF.health()\n", + "# Preview the roles/scopes/predicates a config WOULD produce, BEFORE generate ever runs — a dry\n", + "# projection over generate's own resolution chain. No mapping/log write; it DOES read onelake_security_member —\n", + "# safe to call against a config you haven't committed to yet.\n", + "#\n", + "# When:\n", + "# reviewing a config edit before generate/plan/apply — \"what would this build?\"\n", + "# Expect:\n", + "# one row per role × scope grant (members = RESOLVED display names, ';'-joined — a glob: pattern shows its expansion); an\n", + "# empty/all-inactive config → an empty but typed frame.\n", + "OLAF.explain()\n", + "\n", + "#\n", + "# → Spark DataFrame (one row per role × scope grant; nothing written):\n", + "# +----------------+----------------------+------------+---------------+-----------------+---------------------+\n", + "# | role_name | scope_path | permission | rls_condition | visible_columns | members |\n", + "# +----------------+----------------------+------------+---------------+-----------------+---------------------+\n", + "# | SalesReaders | /Tables/sales/orders | Read | null | null | sg-sales |\n", + "# | FinanceReaders | /Tables/fin/ledger | Read | region='APAC' | amount;region | priya@contoso.com |\n", + "# +----------------+----------------------+------------+---------------+-----------------+---------------------+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dep-generate", + "metadata": {}, + "outputs": [], + "source": [ + "# Build the mapping lock-file from the short config: resolve include/exclude, validate every\n", + "# rule, freeze the grants + write a versioned review CSV. rebuild=False is the normal path.\n", + "#\n", + "# When:\n", + "# after every config edit — always before plan/apply. rebuild=False is idempotent, so\n", + "# re-running an unchanged config is a no-op (status \"skipped\").\n", + "# Expect:\n", + "# status \"success\" with the grants/roles/warnings counts (or \"skipped\" if unchanged),\n", + "# plus a new versioned CSV directly under mapping_history_dir (default Files/security/mapping-history).\n", + "OLAF.generate(rebuild=False)\n", + "\n", + "#\n", + "# → Spark DataFrame (OLAF._view of the generate envelope):\n", + "# +----------+---------+---------+--------------------------+--------+-------+----------+-------------------------------------------------------------------------------------------------+\n", + "# | mode | status | changed | message | grants | roles | warnings | csv |\n", + "# +----------+---------+---------+--------------------------+--------+-------+----------+-------------------------------------------------------------------------------------------------+\n", + "# | generate | success | true | 12 grants across 3 roles | 12 | 3 | 1 | Files/security/mapping-history/onelake_security_mapping_20260715-080000_v7_a1b2c3d4e5f60718.csv |\n", + "# +----------+---------+---------+--------------------------+--------+-------+----------+-------------------------------------------------------------------------------------------------+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dep-plan", + "metadata": {}, + "outputs": [], + "source": [ + "# Diff the desired state (the mapping lock-file) against the live DAR — read-only, and it is\n", + "# what unlocks apply.\n", + "#\n", + "# When:\n", + "# after generate, before apply — required (apply stays blocked until a matching plan exists\n", + "# with no since-plan drift).\n", + "# Expect:\n", + "# one row per change: create / update / omit / no_change. changed=false means live already\n", + "# matches config, so apply is skipped.\n", + "OLAF.plan()\n", + "\n", + "#\n", + "# → Spark DataFrame (one row per changed role):\n", + "# +------+---------+---------+-------------+----------------+--------+\n", + "# | mode | status | changed | message | role | action |\n", + "# +------+---------+---------+-------------+----------------+--------+\n", + "# | plan | success | true | 3 change(s) | FinanceReaders | update |\n", + "# | plan | success | true | 3 change(s) | SalesReaders | create |\n", + "# | plan | success | true | 3 change(s) | TempAuditors | omit |\n", + "# +------+---------+---------+-------------+----------------+--------+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dep-apply-inc", + "metadata": {}, + "outputs": [], + "source": [ + "# Push the planned roles to the live DAR. keep_unmanaged=True = INCREMENTAL upsert: managed roles\n", + "# are created/updated and NOTHING is dropped from the payload — hand-added Default* roles are\n", + "# carried through untouched.\n", + "#\n", + "# When:\n", + "# the deliberate opt-in — you want live roles outside config left alone. NOT the default.\n", + "# Expect:\n", + "# push_status (the bulk PUT's HTTP status, NOT a count) beside roles_written (the real count of\n", + "# roles the PUT body carried); roles absent from config are listed under\n", + "# drift_omission_candidates but are still carried in the submitted payload.\n", + "OLAF.apply(keep_unmanaged=True)\n", + "\n", + "#\n", + "# → Spark DataFrame (OLAF._view of the apply envelope):\n", + "# +-------+---------+---------+---------------------+-------------+---------------+----------------+---------------------+------------------------------------------------------+-------------------------+---------------------------+----------------------------+\n", + "# | mode | status | changed | message | push_status | roles_written | keep_unmanaged | request | backup_path | omitted_role_candidates | drift_omission_candidates | post_state_review_required |\n", + "# +-------+---------+---------+---------------------+-------------+---------------+----------------+---------------------+------------------------------------------------------+-------------------------+---------------------------+----------------------------+\n", + "# | apply | success | true | apply (incremental) | 200 | 3 | true | incremental_payload | Files/security/role-backups/onelake_..._replace.json | [] | [\"TempAuditors\"] | true |\n", + "# +-------+---------+---------+---------------------+-------------+---------------+----------------+---------------------+------------------------------------------------------+-------------------------+---------------------------+----------------------------+\n", + "#\n", + "# push_status is HTTP; roles_written is the count. Incremental PUTs live ∪ desired, so the 3 here is\n", + "# the 2 managed roles PLUS the kept TempAuditors — the whole body the PUT wrote." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dep-apply-full", + "metadata": {}, + "outputs": [], + "source": [ + "# The DEFAULT apply submits config as the whole truth: every live role NOT in the desired set is\n", + "# OMITTED from the payload, including hand-added Default* roles (the omitted list prints first).\n", + "# Omission is a REQUEST shape, not a confirmed outcome — the Preview bulk endpoint does not\n", + "# document deletion-by-omission, so review the post-state before concluding a role is gone.\n", + "#\n", + "# When:\n", + "# routine deployment, right after you have reviewed the plan.\n", + "# Expect:\n", + "# the omission-candidate list printed first (roles_written = the roles the PUT body carried,\n", + "# omitted_role_candidates = the roles dropped by omission from it, and\n", + "# post_state_review_required = true because the request alone settles nothing).\n", + "OLAF.apply()\n", + "\n", + "#\n", + "# → Spark DataFrame:\n", + "# +-------+---------+---------+-----------------+-------------+---------------+----------------+----------------+------------------------------------------------------+-------------------------+---------------------------+----------------------------+\n", + "# | mode | status | changed | message | push_status | roles_written | keep_unmanaged | request | backup_path | omitted_role_candidates | drift_omission_candidates | post_state_review_required |\n", + "# +-------+---------+---------+-----------------+-------------+---------------+----------------+----------------+------------------------------------------------------+-------------------------+---------------------------+----------------------------+\n", + "# | apply | success | true | apply (replace) | 200 | 3 | false | config_payload | Files/security/role-backups/onelake_..._replace.json | [\"TempAuditors\"] | [\"TempAuditors\"] | true |\n", + "# +-------+---------+---------+-----------------+-------------+---------------+----------------+----------------+------------------------------------------------------+-------------------------+---------------------------+----------------------------+\n", + "#\n", + "# If the PUT FAILS, OLAF.apply() still returns a frame — run_mode is the non-raising engine, so the\n", + "# view is the 1-row apply summary carrying status \"error\" (or \"blocked\" when a first-attempt 412\n", + "# refused the conditional PUT); the raw envelope stays at OLAF.last_result. Only the pipeline\n", + "# entrypoint run_and_exit raises on a failed outcome. The record is in\n", + "# onelake_security_log: one `failed` row per planned role (intended action vs PRESENT/ABSENT in a\n", + "# re-read of live) plus a `push` summary row carrying the intended payload and backup_path. Nothing\n", + "# is auto-restored: a PUT that timed out may have succeeded, so restoring is a deliberate act." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dep-rollback", + "metadata": {}, + "outputs": [], + "source": [ + "# Restore the config table to a prior Delta version, then re-run generate → plan → apply. A\n", + "# blank version = the immediately previous one; a value pins an exact one. reason is required.\n", + "#\n", + "# When:\n", + "# after a bad config edit you need to undo.\n", + "# Expect:\n", + "# config restored to the target version, the full pipeline re-run, and two durable audit rows\n", + "# (action \"rollback\", status \"prepared\" then \"restored\") whose message is a JSON record\n", + "# carrying from_version, to_version, reason, source_config_hash and target_config_hash.\n", + "OLAF.rollback(rollback_to_version=\"41\", rollback_reason=\"bad RLS predicate on Orders\")\n", + "\n", + "#\n", + "# → Spark DataFrame (1-row status summary):\n", + "# +----------+---------+---------+-----------------------------------------------------+\n", + "# | mode | status | changed | message |\n", + "# +----------+---------+---------+-----------------------------------------------------+\n", + "# | rollback | success | true | rollback to config v41: bad RLS predicate on Orders |\n", + "# +----------+---------+---------+-----------------------------------------------------+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "dep-validate", + "metadata": {}, + "outputs": [], + "source": [ + "# Dry-run the IDENTICAL validation pipeline generate runs (every rule in the docs/architecture.md\n", + "# rule catalog, the No-Graph member gate, the lakehouse target guard) with ZERO writes -- no\n", + "# mapping, no CSV, no log row (not even a 'rejected' one on a blocked run, unlike generate). Safe\n", + "# to run against a live deployment: it never touches the control tables.\n", + "#\n", + "# When:\n", + "# previewing a config edit's validation outcome before committing to generate (which writes).\n", + "# Expect:\n", + "# status \"success\" with the grant/role/warning summary on a clean config; on an invalid config\n", + "# a blocked view carrying the SAME collect-all error set generate would reject with (never\n", + "# raises — the raw envelope, with counts + every warning, stays at OLAF.last_result).\n", "OLAF.validate()\n", - "OLAF.explain()\n" + "\n", + "#\n", + "# → Spark DataFrame (OLAF._view of the validate envelope — the 1-row status summary; the raw\n", + "# envelope with the grant/role counts + every warning stays at OLAF.last_result):\n", + "# +----------+---------+---------+---------------------------------------------------------------------------+\n", + "# | mode | status | changed | message |\n", + "# +----------+---------+---------+---------------------------------------------------------------------------+\n", + "# | validate | success | false | validate: 12 grant(s) across 3 role(s), 1 warning(s) — dry-run, no writes |\n", + "# +----------+---------+---------+---------------------------------------------------------------------------+" ] }, { "cell_type": "markdown", - "id": "cookbook-sensitive", + "id": "sec-audit", + "metadata": {}, + "source": [ + "## Audit\n", + "\n", + "Read-only queries over the live DAR + the `onelake_security_log` trail — no SQL. `show`/`trace`\n", + "route through `run_mode` (they need the live target); `grants` / `provenance` / `timeline` /\n", + "`report` read the log directly. All return a Spark DataFrame." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aud-show-table", + "metadata": {}, + "outputs": [], + "source": [ + "# Pivot the LIVE DAR by TABLE: who can reach sales.orders, each grant enriched from the log\n", + "# (first_applied/first_granted_by · last_applied/last_granted_by · config_version) and flagged\n", + "# framework vs out-of-band (a Fabric-UI edit).\n", + "#\n", + "# When:\n", + "# answering \"who can access this table?\" — audit / compliance / incident review, any time (read-only).\n", + "# Expect:\n", + "# one row per (role, member) reaching the table, with the enrichment + an out_of_band flag.\n", + "OLAF.show(by=\"table\", subject=\"sales.orders\")\n", + "\n", + "#\n", + "# → Spark DataFrame (one row per role × scope × member; member = objectId, abbreviated here).\n", + "# by=table LEADS with scope_path — every axis returns the same eleven columns, led by its own key:\n", + "# +----------------------+--------------+-----------+-------------------+------------+---------------+------------------+--------------+-----------------+----------------+------------+\n", + "# | scope_path | role_name | member | member_name | permission | first_applied | first_granted_by | last_applied | last_granted_by | config_version | provenance |\n", + "# +----------------------+--------------+-----------+-------------------+------------+---------------+------------------+--------------+-----------------+----------------+------------+\n", + "# | /Tables/sales/orders | SalesReaders | 8f4e2a…c1 | priya@contoso.com | Read | 2026-07-15 | ci@contoso | 2026-08-24 | keng@contoso | 42 | framework |\n", + "# +----------------------+--------------+-----------+-------------------+------------+---------------+------------------+--------------+-----------------+----------------+------------+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aud-show-role", + "metadata": {}, + "outputs": [], + "source": [ + "# Pivot the LIVE DAR by ROLE: every table/folder scope + member that SalesReaders grants.\n", + "#\n", + "# When:\n", + "# reviewing one role — \"what does SalesReaders actually grant?\"\n", + "# Expect:\n", + "# one row per (scope, member) the role covers, with the same log enrichment as by=table.\n", + "OLAF.show(by=\"role\", subject=\"SalesReaders\")\n", + "\n", + "#\n", + "# → Spark DataFrame (same eleven columns, led by role_name; one row per scope × member of the role):\n", + "# +--------------+-----------------------+-----------+-------------------+------------+---------------+------------------+--------------+-----------------+----------------+------------+\n", + "# | role_name | scope_path | member | member_name | permission | first_applied | first_granted_by | last_applied | last_granted_by | config_version | provenance |\n", + "# +--------------+-----------------------+-----------+-------------------+------------+---------------+------------------+--------------+-----------------+----------------+------------+\n", + "# | SalesReaders | /Tables/sales/orders | 8f4e2a…c1 | priya@contoso.com | Read | 2026-07-15 | ci@contoso | 2026-08-24 | keng@contoso | 42 | framework |\n", + "# | SalesReaders | /Tables/sales/returns | 8f4e2a…c1 | priya@contoso.com | Read | 2026-07-15 | ci@contoso | 2026-08-24 | keng@contoso | 42 | framework |\n", + "# +--------------+-----------------------+-----------+-------------------+------------+---------------+------------------+--------------+-----------------+----------------+------------+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aud-show-member", + "metadata": {}, + "outputs": [], + "source": [ + "# Pivot the LIVE DAR by MEMBER: every role + scope priya@contoso.com reaches\n", + "# (accepts an objectId or a name / glob).\n", + "#\n", + "# When:\n", + "# an access review for one person — \"what can they see?\"\n", + "# Expect:\n", + "# one row per (role, scope) the member reaches, enriched + out_of_band-flagged.\n", + "OLAF.show(by=\"member\", subject=\"priya@contoso.com\")\n", + "\n", + "#\n", + "# → Spark DataFrame (same eleven columns; by=member LEADS with member + member_name):\n", + "# +-----------+-------------------+----------------+----------------------+------------+---------------+------------------+--------------+-----------------+----------------+------------+\n", + "# | member | member_name | role_name | scope_path | permission | first_applied | first_granted_by | last_applied | last_granted_by | config_version | provenance |\n", + "# +-----------+-------------------+----------------+----------------------+------------+---------------+------------------+--------------+-----------------+----------------+------------+\n", + "# | 8f4e2a…c1 | priya@contoso.com | SalesReaders | /Tables/sales/orders | Read | 2026-07-15 | ci@contoso | 2026-08-24 | keng@contoso | 42 | framework |\n", + "# | 8f4e2a…c1 | priya@contoso.com | FinanceReaders | /Tables/fin/ledger | Read | 2026-07-16 | ci@contoso | 2026-08-24 | keng@contoso | 42 | framework |\n", + "# +-----------+-------------------+----------------+----------------------+------------+---------------+------------------+--------------+-----------------+----------------+------------+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aud-trace", + "metadata": {}, + "outputs": [], + "source": [ + "# Operational snapshot (the trace mode / Audit.report): the deployed generation, LIVE role +\n", + "# grant counts, staleness, and — when the live client resolves — the out-of-band grant count.\n", + "# Every count describes the CURRENT state except established_ever, which is cumulative across\n", + "# every env and config version and is named so it cannot be mistaken for today.\n", + "#\n", + "# When:\n", + "# a quick \"is the deployment healthy / in sync?\" check, any time (read-only).\n", + "# Expect:\n", + "# a 1-row snapshot: generation, live role/grant counts, is_stale, out_of_band count.\n", + "OLAF.trace()\n", + "\n", + "#\n", + "# → Spark DataFrame (1-row snapshot):\n", + "# +-------+---------+-----------------+------------------+------------------+----------+-------------+\n", + "# | mode | status | live_role_count | live_grant_count | established_ever | is_stale | out_of_band |\n", + "# +-------+---------+-----------------+------------------+------------------+----------+-------------+\n", + "# | trace | success | 3 | 7 | 19 | false | 1 |\n", + "# +-------+---------+-----------------+------------------+------------------+----------+-------------+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aud-grants", "metadata": {}, + "outputs": [], "source": [ - "## Sensitive operations — documentation only\n", + "# Established grants read from the LOG (no live DAR call) — one row per (role, scope, member)\n", + "# with provenance. Optional role / scope / member narrow the listing.\n", + "#\n", + "# When:\n", + "# \"what has the framework granted, and when?\" — from the audit trail, offline.\n", + "# Expect:\n", + "# one row per established (role, scope, member), each with BOTH ends of its deploy range:\n", + "# first_applied/first_granted_by, last_applied/last_granted_by, and config_version.\n", + "OLAF.grants(role=\"SalesReaders\")\n", "\n", - "The following sequence is intentionally non-executable. Enable a sensitive operation only after the control gates above pass:\n", + "#\n", + "# → Spark DataFrame:\n", + "# +--------------+----------------------+-----------+-------------------+---------------+------------------+--------------+-----------------+----------------+\n", + "# | role_name | scope_path | member_id | member_name | first_applied | first_granted_by | last_applied | last_granted_by | config_version |\n", + "# +--------------+----------------------+-----------+-------------------+---------------+------------------+--------------+-----------------+----------------+\n", + "# | SalesReaders | /Tables/sales/orders | 8f4e2a…c1 | priya@contoso.com | 2026-07-15 | ci@contoso | 2026-08-24 | keng@contoso | 42 |\n", + "# +--------------+----------------------+-----------+-------------------+---------------+------------------+--------------+-----------------+----------------+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aud-since", + "metadata": {}, + "outputs": [], + "source": [ + "# One established grant's provenance for a (role, scope[, member]) — the first row of grants(),\n", + "# as a 1-row DataFrame (or an empty frame when no such grant exists).\n", + "#\n", + "# Reports BOTH ends, and deliberately no single `since`. The log records what OLAF did; it\n", + "# cannot see a role deleted straight from the Fabric UI, so an apply, an out-of-band deletion\n", + "# and a re-apply are indistinguishable here from one unbroken grant. Read first_applied as\n", + "# \"first deployed on\" and last_applied as \"last re-asserted on\" — a wide gap between them is a\n", + "# cue to check the live DAR, not evidence either way.\n", + "#\n", + "# When:\n", + "# an access review — \"when did this access start, and has anyone touched it since?\"\n", + "# Expect:\n", + "# a 1-row frame with both ends and the principal who pushed each (or an empty frame if none).\n", + "OLAF.provenance(role=\"SalesReaders\", scope=\"/Tables/sales/orders\")\n", + "\n", + "#\n", + "# → Spark DataFrame (1 row):\n", + "# +--------------+----------------------+-----------+-------------------+---------------+------------------+--------------+-----------------+----------------+\n", + "# | role_name | scope_path | member_id | member_name | first_applied | first_granted_by | last_applied | last_granted_by | config_version |\n", + "# +--------------+----------------------+-----------+-------------------+---------------+------------------+--------------+-----------------+----------------+\n", + "# | SalesReaders | /Tables/sales/orders | 8f4e2a…c1 | priya@contoso.com | 2026-07-15 | ci@contoso | 2026-08-24 | keng@contoso | 42 |\n", + "# +--------------+----------------------+-----------+-------------------+---------------+------------------+--------------+-----------------+----------------+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aud-timeline", + "metadata": {}, + "outputs": [], + "source": [ + "# Every logged config generation as one row: first_seen / last_seen / run count per\n", + "# (config_version, config_hash), ordered by version — the lifetime of each generation.\n", + "#\n", + "# When:\n", + "# reviewing how the config has evolved over time — its change history.\n", + "# Expect:\n", + "# one row per generation: config_version, config_hash, first_seen, last_seen, runs.\n", + "OLAF.timeline()\n", "\n", - "```python\n", - "# OLAF.setup()\n", - "# OLAF.load_config(\"config\", \"\", \"config\")\n", - "# OLAF.generate()\n", - "# OLAF.plan()\n", - "# inspect the exact request, re-attest isolation, capture snapshot + ETag, create/read sentinel\n", - "# runtime revalidates the sentinel and snapshot before every sensitive write\n", - "# OLAF.apply(...)\n", - "# re-read and classify the observed post-state\n", - "```\n", + "#\n", + "# → Spark DataFrame:\n", + "# +----------------+-------------+---------------------+---------------------+------+\n", + "# | config_version | config_hash | first_seen | last_seen | runs |\n", + "# +----------------+-------------+---------------------+---------------------+------+\n", + "# | 41 | 3b9c… | 2026-07-10T09:12:03 | 2026-07-14T18:40:55 | 6 |\n", + "# | 42 | a17f… | 2026-07-15T08:01:22 | 2026-07-16T11:29:07 | 4 |\n", + "# +----------------+-------------+---------------------+---------------------+------+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aud-report", + "metadata": {}, + "outputs": [], + "source": [ + "# The one-call operational snapshot behind trace, as a 1-row DataFrame. Nested values (the\n", + "# generation / last-run dicts) are stringified; the counts and is_stale are scalar.\n", + "#\n", + "# When:\n", + "# same as trace, when you want the full dict fields in one row.\n", + "# Expect:\n", + "# a 1-row wide frame: current_generation, last_generate, last_apply, live counts,\n", + "# established_ever, is_stale, out_of_band.\n", + "OLAF.report()\n", "\n", - "`keep_unmanaged` describes OLAF's request construction. Microsoft's Preview bulk endpoint does not document atomic full-set replacement or deletion-by-omission. A backup is a recovery input, not an exact-restore guarantee. `reset()` and `cleanup()` are sensitive containment operations and are also disabled by default.\n" + "#\n", + "# → Spark DataFrame (1 row; wide — nested dicts shown abbreviated):\n", + "# +---------------------------+------------------+----------------+-----------------+------------------+------------------+----------+-------------+\n", + "# | current_generation | last_generate | last_apply | live_role_count | live_grant_count | established_ever | is_stale | out_of_band |\n", + "# +---------------------------+------------------+----------------+-----------------+------------------+------------------+----------+-------------+\n", + "# | {'config_version': 42, …} | {'mode': 'gene…} | {'mode': 'ap…} | 3 | 7 | 19 | false | 1 |\n", + "# +---------------------------+------------------+----------------+-----------------+------------------+------------------+----------+-------------+" ] }, { "cell_type": "markdown", - "id": "cookbook-sources", + "id": "sec-audit-compliance", "metadata": {}, "source": [ - "## Official references\n", + "### Compliance & live-DAR utilities\n", "\n", - "- [Bulk DAR endpoint](https://learn.microsoft.com/en-us/rest/api/fabric/core/onelake-data-access-security/create-or-update-data-access-roles)\n", - "- [OneLake access model](https://learn.microsoft.com/en-us/fabric/onelake/security/data-access-control-model)\n", - "- [Table, column, and row security](https://learn.microsoft.com/en-us/fabric/onelake/security/table-column-row-security)\n", - "- [Canonical RLS syntax](https://learn.microsoft.com/en-us/fabric/onelake/security/row-level-security-syntax)\n", - "- [NotebookUtils credentials](https://learn.microsoft.com/en-us/fabric/data-engineering/notebookutils/notebookutils-credentials#get-token)\n", + "Read-only queries that pivot the LIVE DAR directly (all but `coverage` need a live client) —\n", + "finding gaps, computing net access, and comparing desired vs. live without touching `plan`/`apply`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aud-coverage", + "metadata": {}, + "outputs": [], + "source": [ + "# Protected vs unprotected table surface — the compliance gap finder. The table universe is every\n", + "# REAL table in the lakehouse catalog, not just the ones named in the mapping, so a table nobody\n", + "# configured at all still gets a row (protected=false) instead of being silently skipped.\n", + "#\n", + "# When:\n", + "# \"what tables have NO security applied at all?\" — a compliance sweep, any time (no live\n", + "# client needed).\n", + "# Expect:\n", + "# one row per catalog table; protected=false + roles_count=0 for anything unconfigured.\n", + "OLAF.coverage()\n", + "\n", + "#\n", + "# → Spark DataFrame (one row per table in the catalog):\n", + "# +-----------------------+-----------+-------------+---------+---------+\n", + "# | table | protected | roles_count | has_rls | has_cls |\n", + "# +-----------------------+-----------+-------------+---------+---------+\n", + "# | sales.orders | true | 1 | true | false |\n", + "# | fin.ledger | true | 1 | true | true |\n", + "# | sales.staging_scratch | false | 0 | false | false |\n", + "# +-----------------------+-----------+-------------+---------+---------+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aud-effective-access", + "metadata": {}, + "outputs": [], + "source": [ + "# Net EFFECTIVE access to one table for one member — most-permissive-wins across every reaching\n", + "# role, plus a synthesized 'effective' union row. engine= is REQUIRED (spark | direct_lake |\n", + "# sql_endpoint) and is echoed as a column: on spark/direct_lake a reaching role with no RLS/CLS on\n", + "# the table nullifies every other role's restriction, mirroring rule C8; on sql_endpoint CLS is the\n", + "# INTERSECTION of the explicit allow-lists instead, so an unrestricted role does NOT erase another\n", + "# role's column restriction (RLS is still nullified on every engine). This models the DAR rules —\n", + "# it is not proof of endpoint identity mode or of enforcement for a given request.\n", + "# Needs a live client (Fabric only).\n", + "# member accepts a name/UPN (resolved case-insensitively against onelake_security_member, the\n", + "# same No-Graph table generate resolves from) or an objectId (GUID) passed straight through; a\n", + "# name absent from the member table raises UsageError naming the No-Graph limitation. The optional\n", + "# member_type (\"Group\"/\"User\"/\"ServicePrincipal\"/\"ManagedIdentity\") scopes resolution to the member\n", + "# table's logical PK (member_type, lower(member_name)) — needed only when a Group and a User share a\n", + "# display name, which is otherwise a hard UsageError naming the ambiguity instead of a silent pick.\n", + "#\n", + "# When:\n", + "# \"what can this person actually see on this table, once every role they're in is combined?\"\n", + "# Expect:\n", + "# one detail row per reaching role, plus one effective=true union row; no reaching role → an\n", + "# empty but typed frame.\n", + "OLAF.effective_access(member=\"priya@contoso.com\", table=\"sales.orders\", engine=\"spark\")\n", + "# member is a name/UPN preloaded in onelake_security_member — a GUID objectId also works unchanged\n", + "\n", + "#\n", + "# → Spark DataFrame (one row per reaching role, plus the union):\n", + "# +----------------+---------------+-----------------+-----------------------------+-----------+--------+\n", + "# | role_name | rls_condition | visible_columns | granting_role | effective | engine |\n", + "# +----------------+---------------+-----------------+-----------------------------+-----------+--------+\n", + "# | SalesReaders | region='APAC' | null | null | false | spark |\n", + "# | FinanceReaders | null | null | null | false | spark |\n", + "# | null | null | null | FinanceReaders;SalesReaders | true | spark |\n", + "# +----------------+---------------+-----------------+-----------------------------+-----------+--------+\n", + "# (FinanceReaders carries no RLS on this table, so the union's rls_condition is null too —\n", + "# \"unrestricted nullifies filter\".)\n", + "\n", + "# Same name, two principal types (a Group AND a User called \"finance-team\") — say which one:\n", + "OLAF.effective_access(\n", + " member=\"finance-team\", table=\"sales.orders\", member_type=\"Group\", engine=\"spark\"\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aud-who-can-access", + "metadata": {}, + "outputs": [], + "source": [ + "# The REVERSE of effective_access — every member who can reach this table, one row per\n", + "# (member, role) pair (a member reachable via two roles gets two rows). Needs a live client.\n", + "#\n", + "# When:\n", + "# \"who can see this table, and through which role?\" — an access review by table.\n", + "# Expect:\n", + "# one row per (member, role); rls_cls_summary is a one-cell digest ('rows: ...' / 'cols: ...' /\n", + "# 'unrestricted').\n", + "OLAF.who_can_access(table=\"sales.orders\")\n", + "\n", + "#\n", + "# → Spark DataFrame:\n", + "# +---------------------+-----------+----------------+------------+---------------------+\n", + "# | member_name | member_id | via_role | permission | rls_cls_summary |\n", + "# +---------------------+-----------+----------------+------------+---------------------+\n", + "# | priya@contoso.com | 8f4e2a…c1 | SalesReaders | Read | rows: region='APAC' |\n", + "# | sg-finance-leads | c9d1e0…7f | FinanceReaders | Read | unrestricted |\n", + "# +---------------------+-----------+----------------+------------+---------------------+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aud-drift", + "metadata": {}, + "outputs": [], + "source": [ + "# Full desired-vs-live DAR comparison, CATEGORIZED and READ-ONLY: framework (matches provenance) /\n", + "# out_of_band (no provenance) / policy (provenanced, but the deployed permission / RLS predicate /\n", + "# CLS allow-list differs from what the mapping declares — out_of_band WINS when a grant qualifies\n", + "# for both) / missing (desired but absent live). member_id + member_name are\n", + "# APPENDED at the end of the frame (out_of_band()'s convention); member_name is resolved id->name\n", + "# from the member cache table (same lookup out_of_band/who_can_access use) and falls back to the id\n", + "# itself when the cache has no row — which is exactly what the id/name PAIR lets you detect. A third\n", + "# rendering exists: an id the cache gives MORE THAN ONE distinct name surfaces as\n", + "# \"\", distinct from a real name AND from the bare-id fallback.\n", + "# A pure\n", + "# comparison view — it never records a plan and never gates apply (plan() stays the one\n", + "# apply-gating recorder). Needs a live client.\n", + "#\n", + "# When:\n", + "# a periodic \"is live DAR exactly what config says?\" check, independent of plan/apply.\n", + "# Expect:\n", + "# one row per grant (or per missing desired grant), tagged with its category + a short detail.\n", + "OLAF.drift()\n", + "\n", + "#\n", + "# → Spark DataFrame:\n", + "# +----------------+----------------------+-------------+-----------------------------------------------+-----------+---------------------+\n", + "# | role_name | scope_path | category | detail | member_id | member_name |\n", + "# +----------------+----------------------+-------------+-----------------------------------------------+-----------+---------------------+\n", + "# | SalesReaders | /Tables/sales/orders | framework | live grant matches framework provenance (...) | 8f4e2a…c1 | priya@contoso.com |\n", + "# | FinanceReaders | /Tables/fin/ledger | out_of_band | live grant has no framework provenance (...) | c9d1e0…7f | sg-finance-leads |\n", + "# | TempAuditors | /Tables/sales/orders | missing | desired grant absent from live DAR (...) | a2b3c4…5d | sg-temp-auditors |\n", + "# +----------------+----------------------+-------------+-----------------------------------------------+-----------+---------------------+" + ] + }, + { + "cell_type": "markdown", + "id": "sec-audit-timetravel", + "metadata": {}, + "source": [ + "### Config time-travel & lineage\n", + "\n", + "Delta version-history utilities over the control tables — `table_history`/`at` work on any of\n", + "config/mapping/log; `config_diff`/`value_history` are config-specific, diffing the authored rows\n", + "across versions." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aud-table-history", + "metadata": {}, + "outputs": [], + "source": [ + "# Delta DESCRIBE HISTORY of a control table, made readable. `table` is one of config/mapping/log\n", + "# (mapped to the actual configured table name).\n", + "#\n", + "# When:\n", + "# \"when did the config table last change, and who/what wrote it?\" — a Delta-native history,\n", + "# independent of onelake_security_log.\n", + "# Expect:\n", + "# one row per Delta commit, newest first (DESCRIBE HISTORY's own order).\n", + "OLAF.table_history(\"config\")\n", + "\n", + "#\n", + "# → Spark DataFrame:\n", + "# +---------+---------------------------+-------------------+-----------+------+\n", + "# | version | timestamp | user | operation | rows |\n", + "# +---------+---------------------------+-------------------+-----------+------+\n", + "# | 42 | 2026-07-15T08:00:00+00:00 | alice@contoso.com | UPDATE | 14 |\n", + "# | 41 | 2026-07-10T09:00:00+00:00 | alice@contoso.com | UPDATE | 13 |\n", + "# +---------+---------------------------+-------------------+-----------+------+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aud-at", + "metadata": {}, + "outputs": [], + "source": [ + "# Snapshot of ANY control table (config/mapping/log) at a Delta version or date — config_at\n", + "# generalized to every control table. Give exactly one of version or date.\n", + "#\n", + "# When:\n", + "# \"what did the mapping lock-file look like right after the v41 rollback?\" — any control\n", + "# table, any point in time.\n", + "# Expect:\n", + "# the resolved table's own schema, as of that version/date.\n", + "OLAF.at(\"mapping\", version=7)\n", "\n", - "OLAF deliberately does not call Microsoft Graph. That is a framework design choice, not a claim that Graph tokens are universally unavailable.\n" + "#\n", + "# → Spark DataFrame (onelake_security_mapping's own schema, as of version 7 — abbreviated here):\n", + "# +--------------+----------------------+------------+-----+\n", + "# | role_name | scope_path | permission | ... |\n", + "# +--------------+----------------------+------------+-----+\n", + "# | SalesReaders | /Tables/sales/orders | Read | ... |\n", + "# +--------------+----------------------+------------+-----+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aud-config-diff", + "metadata": {}, + "outputs": [], + "source": [ + "# Role/scope/member changes between two config Delta versions — added / removed / changed rows,\n", + "# diffed in Python after two time-travel reads (config_at(v1), config_at(v2)).\n", + "#\n", + "# When:\n", + "# \"what actually changed between config v41 and v42?\" — reviewing an edit after the fact.\n", + "# Expect:\n", + "# one row per added/removed role×scope key, or one row per changed field (a role with 3\n", + "# changed fields → 3 'changed' rows).\n", + "OLAF.config_diff(41, 42)\n", + "\n", + "#\n", + "# → Spark DataFrame (scope_key is the config's own include/exclude-column key, '|'-joined):\n", + "# +-------------+--------------+-------------------+---------------+-------+---------------+\n", + "# | change_type | role_name | scope_key | field | old | new |\n", + "# +-------------+--------------+-------------------+---------------+-------+---------------+\n", + "# | changed | SalesReaders | sales.orders||| | rls_condition | null | region='APAC' |\n", + "# | added | TempAuditors | sales.orders||| | null | null | null |\n", + "# +-------------+--------------+-------------------+---------------+-------+---------------+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "aud-value-history", + "metadata": {}, + "outputs": [], + "source": [ + "# How ONE role/scope's config VALUE evolved across every config Delta version — walks every\n", + "# version, keeping only the rows where the subject is present, flagging `changed` on its first\n", + "# appearance and on any later version where a tracked field differs from the last one it appeared in.\n", + "#\n", + "# When:\n", + "# \"when did SalesReaders' RLS predicate actually change, across every version?\" — a full\n", + "# lineage for one role/scope, not just a two-version diff (that's config_diff).\n", + "# Expect:\n", + "# one row per version where the subject is present; changed=true on first appearance.\n", + "OLAF.value_history(subject=\"SalesReaders\")\n", + "\n", + "#\n", + "# → Spark DataFrame (20 columns total — every config_diff field column, abbreviated here):\n", + "# +----------------+--------------+-----------------+------------+---------------+-----+---------+------------------+\n", + "# | config_version | role_name | scope_key | permission | rls_condition | ... | changed | window_truncated |\n", + "# +----------------+--------------+-----------------+------------+---------------+-----+---------+------------------+\n", + "# | 41 | SalesReaders | sales.orders||| | Read | null | ... | true | false |\n", + "# | 42 | SalesReaders | sales.orders||| | Read | region='APAC' | ... | true | false |\n", + "# +----------------+--------------+-----------------+------------+---------------+-----+---------+------------------+" + ] + }, + { + "cell_type": "markdown", + "id": "sec-maintenance", + "metadata": {}, + "source": [ + "## Maintenance\n", + "\n", + "Control-table lifecycle. `setup` is the only maintenance action on `OLAF` today." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "mnt-setup", + "metadata": {}, + "outputs": [], + "source": [ + "# Create (or additively migrate) the four control tables — config / mapping / log / member.\n", + "# Idempotent: safe to re-run; it reconciles schema drift without data loss.\n", + "#\n", + "# setup REQUIRES lakehouse_name, and it is an assertion, not a target: setup always writes to the\n", + "# ATTACHED lakehouse (its DDL uses two-part `olaf.…` names), so naming it here is how you get told\n", + "# you are attached to the wrong one — instead of finding the control tables in another workspace\n", + "# later. Case-insensitive; an attachment from another workspace is refused on the ids too.\n", + "#\n", + "# When:\n", + "# the first thing you run in a new lakehouse, and again after a framework upgrade.\n", + "# Expect:\n", + "# the four tables created/migrated + a 1-row status summary; a `blocked` row naming BOTH the\n", + "# declared and the attached lakehouse if they disagree.\n", + "OLAF.configure(lakehouse_name=\"LH_Gold\") # the lakehouse this notebook is attached to\n", + "OLAF.setup()\n", + "\n", + "#\n", + "# → Spark DataFrame (1-row status summary):\n", + "# +-------+---------+---------+-----------------------------------+\n", + "# | mode | status | changed | message |\n", + "# +-------+---------+---------+-----------------------------------+\n", + "# | setup | success | true | created/migrated 4 control tables |\n", + "# +-------+---------+---------+-----------------------------------+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "mnt-health", + "metadata": {}, + "outputs": [], + "source": [ + "# One-call doctor: 9 independent checks (control tables present, table location, mapping\n", + "# staleness, DAR reachability, control-data exposure, identity preflight, runtime prerequisites,\n", + "# last-apply age, out-of-band grants). One failing check never aborts the rest — health() always\n", + "# returns all 9 rows.\n", + "#\n", + "# When:\n", + "# \"is everything healthy?\" — a quick doctor pass, any time (never raises).\n", + "# Expect:\n", + "# always 9 rows, each pass/warn/fail with a human-readable detail.\n", + "OLAF.health()\n", + "\n", + "#\n", + "# → Spark DataFrame (always 9 rows):\n", + "# +-----------------------+--------+----------------------------------------------------------------------+\n", + "# | check | status | detail |\n", + "# +-----------------------+--------+----------------------------------------------------------------------+\n", + "# | control_tables | pass | all 4 control tables present with the expected schema |\n", + "# | table_location | pass | control tables are in the attached lakehouse (LH_Gold) |\n", + "# | mapping_staleness | pass | mapping matches the active config (version 42) |\n", + "# | dar_reachable | pass | bounded DAR read succeeded with a collection ETag |\n", + "# | control_data_exposure | pass | JSON facts show a safe DAR snapshot and workspace_isolation=attested |\n", + "# | identity_preflight | pass | Fabric token acquired for the ambient identity (alice@...) |\n", + "# | runtime_prerequisites | pass | observed Spark meets the baseline; verify the Fabric Runtime label |\n", + "# | last_apply_age | pass | last apply was 2 day(s) ago |\n", + "# | out_of_band | warn | 1 out-of-band grant(s) with no framework provenance |\n", + "# +-----------------------+--------+----------------------------------------------------------------------+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "mnt-status", + "metadata": {}, + "outputs": [], + "source": [ + "# One-call at-a-glance deployment snapshot, built purely from the log + mapping (no live client\n", + "# needed, unlike health()'s DAR-dependent checks).\n", + "#\n", + "# When:\n", + "# \"what's deployed right now, in one row?\" — a lightweight dashboard check.\n", + "# Expect:\n", + "# a 1-row frame: role/member counts, last generate/apply timestamps, the newest successful\n", + "# deployment and its mode, the live config version, and whether an unapplied change is pending.\n", + "OLAF.status()\n", + "\n", + "#\n", + "# → Spark DataFrame (1 row):\n", + "# +---------+-----------+---------------------------+---------------------------+---------------------------+----------------------+---------------------+----------------+\n", + "# | n_roles | n_members | last_generate | last_apply | last_deployment | last_deployment_mode | live_config_version | pending_change |\n", + "# +---------+-----------+---------------------------+---------------------------+---------------------------+----------------------+---------------------+----------------+\n", + "# | 3 | 5 | 2026-07-15T08:00:00+00:00 | 2026-07-15T08:05:00+00:00 | 2026-07-15T08:05:00+00:00 | apply | 42 | false |\n", + "# +---------+-----------+---------------------------+---------------------------+---------------------------+----------------------+---------------------+----------------+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "mnt-diagnose-member", + "metadata": {}, + "outputs": [], + "source": [ + "# Why can't `member` see data? Walks the same chain a human troubleshooter would, IN ORDER. Once a\n", + "# prerequisite step (member_in_table / id_resolved / in_mapping) fails, every later step is\n", + "# short-circuited with \"skipped — prerequisite failed\" instead of reporting a misleading guess.\n", + "#\n", + "# When:\n", + "# \"why doesn't priya see the data they should?\" — a single, ordered root-cause walk.\n", + "# Expect:\n", + "# always 5 rows, in order: member_in_table → id_resolved → in_mapping → live_in_dar →\n", + "# apply_in_sync.\n", + "OLAF.diagnose_member(\"priya@contoso.com\")\n", + "# An objectId works too — GUID pass-through, the same one effective_access gives (step 1 reports\n", + "# the pass-through instead of a name lookup), so an id copied out of who_can_access() is accepted:\n", + "OLAF.diagnose_member(\"8f4e2a00-0000-0000-0000-0000000000c1\")\n", + "\n", + "#\n", + "# → Spark DataFrame (always 5 rows, in order):\n", + "# +-----------------+------+--------------------------------------------------------------+\n", + "# | step | ok | detail |\n", + "# +-----------------+------+--------------------------------------------------------------+\n", + "# | member_in_table | true | found in olaf.onelake_security_member (type User) |\n", + "# | id_resolved | true | resolved to objectId 8f4e2a…c1 |\n", + "# | in_mapping | true | in role(s): SalesReaders |\n", + "# | live_in_dar | true | live in role(s): SalesReaders |\n", + "# | apply_in_sync | true | mapping is in sync with the last apply (2026-07-15T08:05:00) |\n", + "# +-----------------+------+--------------------------------------------------------------+" + ] + }, + { + "cell_type": "markdown", + "id": "c1d92bb00146", + "metadata": {}, + "source": [ + "## 🔥 Destructive — read before running either cell below\n", + "\n", + "> ### These two are destructive. There is no dry run, no review gate, and no undo prompt.\n", + "> The moment the cell finishes, it is done.\n", + "\n", + "| | `OLAF.reset()` | `OLAF.cleanup()` |\n", + "|---|---|---|\n", + "| **does** | submits an **empty DAR payload**, omitting **every** live role — OLAF's, other people's, and `Default*` | deletes all four control tables (**including the whole audit log**) + **every** mapping-history and role-backup file |\n", + "| **leaves** | the control tables — config, mapping, member, log | the live roles, now with no trail explaining them |\n", + "| **way back** | the pre-request backup it writes first — a recovery *input*, not a guaranteed exact restore, and there is no public replay method: recovery is a break-glass incident (RUNBOOK §3c) | **none. It deletes the backups too.** |\n", + "\n", + "**`reset()` submits a request; it does not report an outcome.** The roles it returns are\n", + "`prior_live_role_candidate`s — observed before submission and omitted from the request. The Preview\n", + "bulk endpoint does not document deletion-by-omission, so OLAF will not tell you the roles are gone,\n", + "that OneLake security is now deny-by-default, or that nobody can read the data. `post_state_review_required`\n", + "comes back **true**: re-read the post-state in the target engine and access mode before drawing any\n", + "access conclusion. Note the platform's `DefaultReader` is among the omitted roles and OLAF does\n", + "**not** recreate platform-managed roles.\n", + "\n", + "⚠️ **Do not assume a uniform outcome across principals** — workspace roles, default roles, engine/access\n", + "mode and shortcut behavior all remain relevant, and OLAF claims neither universal enforcement nor a\n", + "universal privileged-role bypass. Verify the post-state in each engine/access mode you actually care about.\n", + "\n", + "⚠️ **`cleanup()` cannot log what it did** — it drops the log table. The returned frame and the\n", + "printed lines are the only record that run will ever produce.\n", + "\n", + "Neither is a mode: a pipeline passing `mode=\"reset\"` is refused by name. They exist only here." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b65ae578acd3", + "metadata": {}, + "outputs": [], + "source": [ + "# 🔥 DESTRUCTIVE containment request. SUBMITS AN EMPTY DAR PAYLOAD, omitting every data access\n", + "# role on the attached lakehouse — OLAF's own, anyone else's, and Default*. OLAF does not\n", + "# recreate platform-managed/default roles.\n", + "#\n", + "# What comes back are CANDIDATES, not confirmed deletions: roles observed before submission and\n", + "# omitted from the request. The Preview contract does not establish deletion-by-omission,\n", + "# no-OneLake-security, or universal-reader outcomes, so post_state_review_required is true —\n", + "# review the post-state in the target engine/access mode before concluding anything about access.\n", + "#\n", + "# Writes a role backup FIRST; if that capture fails the reset aborts before submitting anything.\n", + "# The backup is a recovery INPUT, not a guarantee of exact platform-state restoration (RUNBOOK 3c).\n", + "# Control tables are untouched, so generate -> plan -> apply can submit the config's payload again.\n", + "#\n", + "# When:\n", + "# handing a lakehouse back, or starting its security over from nothing.\n", + "# Review the post-state afterwards in the engine/access mode you care about — OLAF claims neither\n", + "# universal enforcement nor a universal privileged-role bypass.\n", + "OLAF.reset()\n", + "\n", + "#\n", + "# → Spark DataFrame — one row per prior-live role candidate, with the request label, the backup\n", + "# artifact, and the review flag:\n", + "# +---------------------------+---------------+----------------------------------------------------+----------------------------+\n", + "# | prior_live_role_candidate | request | backup_path | post_state_review_required |\n", + "# +---------------------------+---------------+----------------------------------------------------+----------------------------+\n", + "# | DefaultReader | empty_payload | Files/security/role-backups/onelake_..._reset.json | true |\n", + "# | SalesReaders | empty_payload | Files/security/role-backups/onelake_..._reset.json | true |\n", + "# +---------------------------+---------------+----------------------------------------------------+----------------------------+" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4b54a5d44b98", + "metadata": {}, + "outputs": [], + "source": [ + "# 🔥 THE ONE WITH NO WAY BACK. Drops all four control tables — the authored config, the mapping,\n", + "# the member table and the ENTIRE AUDIT HISTORY — and deletes every mapping-history CSV and every\n", + "# pre-apply role backup. Those backups are the recovery for a bad apply and for reset(); after\n", + "# this there is nothing left to restore from.\n", + "#\n", + "# It does NOT touch the live roles. Any still deployed are listed in the result, now with no audit\n", + "# trail behind them — run reset() FIRST if you want them out of the DAR payload, because\n", + "# afterwards its backup is gone too.\n", + "#\n", + "# The frame also carries what cleanup could NOT do: items it failed to remove, the incident\n", + "# sentinel it deliberately preserves, and an explicit exposure-not-remediated row. Containment\n", + "# is not proof of erasure, and the manifest says so rather than reading as an all-clear.\n", + "#\n", + "# It cannot log what it did (it drops the log). The frame below and the printed lines are the ONLY\n", + "# record. Copy them somewhere before closing the notebook.\n", + "#\n", + "# When:\n", + "# first-time deploy into an environment a trial run left dirty, so setup() starts from nothing.\n", + "OLAF.cleanup()\n", + "\n", + "#\n", + "# → Spark DataFrame — the manifest, and the only record this run produces:\n", + "# +-----------------------------+------------------------------------------------------------------+\n", + "# | kind | name |\n", + "# +-----------------------------+------------------------------------------------------------------+\n", + "# | dropped table | olaf.onelake_security_config |\n", + "# | dropped table | olaf.onelake_security_log |\n", + "# | deleted file | Files/security/role-backups/onelake_..._replace.json |\n", + "# | LIVE ROLE LEFT BEHIND | SalesReaders |\n", + "# | INCIDENT SENTINEL PRESERVED | Files/security/.olaf-sensitive-write.sentinel |\n", + "# | EXPOSURE NOT REMEDIATED | exposure_remediated=false; cleanup cannot retract prior reads, … |\n", + "# +-----------------------------+------------------------------------------------------------------+" ] } ], "metadata": { - "kernelspec": { - "display_name": "Synapse PySpark", + "a365ComputeOptions": null, + "kernel_info": { + "jupyter_kernel_name": null, "name": "synapse_pyspark" }, "language_info": { "name": "python" - } + }, + "sessionKeepAliveTimeout": 0 }, "nbformat": 4, "nbformat_minor": 5 diff --git a/notebooks/olaf_master_workflow.ipynb b/notebooks/olaf_master_workflow.ipynb index 644bfcd..4e34ef7 100644 --- a/notebooks/olaf_master_workflow.ipynb +++ b/notebooks/olaf_master_workflow.ipynb @@ -2,8 +2,8 @@ "cells": [ { "cell_type": "code", - "id": "workflow-binding", "execution_count": null, + "id": "0393df6c34a4", "metadata": {}, "outputs": [], "source": [ @@ -12,38 +12,169 @@ " \"defaultLakehouse\": {\n", " \"name\": {\n", " \"parameterName\": \"lakehouse_name\",\n", - " \"defaultValue\": \"SyntheticEvaluationLakehouse\"\n", + " \"defaultValue\": \"YourLakehouse\"\n", " }\n", " }\n", - "}\n" + "}" ] }, { "cell_type": "markdown", - "id": "workflow-intro", + "id": "0963a30fbcb6", "metadata": {}, "source": [ - "# OLAF master workflow — controlled evaluation scaffold\n", + "> 🔗 **Session binding** — the `%%configure` cell above pins this run's default lakehouse **by\n", + "> name, before the session starts**. On a pipeline run the `lakehouse_name` **base parameter**\n", + "> feeds it — the same parameter the Parameters cell below receives, so one pipeline parameter\n", + "> drives both the session binding and `setup`'s assertion. Interactive users can instead simply\n", + "> attach a lakehouse in the portal; `defaultValue` is then what running the cell would bind, so\n", + "> keep it naming the lakehouse `lakehouse_name` asserts. Fail-visible either way: if no binding\n", + "> lands, the run blocks on OLAF's `no lakehouse attached` guard (and `setup` re-asserts the\n", + "> name) — it never writes anywhere unintended." + ] + }, + { + "cell_type": "markdown", + "id": "0da628dda910", + "metadata": {}, + "source": [ + "# OneLake Security — Master Notebook\n", + "\n", + "Runs `olaf` through a full deployment, **one stage per cell**, with each stage's result deciding the\n", + "next. This is exactly what a Fabric / ADF pipeline does with one `notebook.run` activity per stage —\n", + "same sequence, same branching — just without the pipeline.\n", + "\n", + "> ⚠️ **Before running:** attach the lakehouse you are securing. Every stage resolves its control\n", + "> tables and its target through whatever lakehouse is *attached*, so that attachment is the setting\n", + "> that matters. (The `lakehouse_name` parameter is read only by `setup`, which asserts the attached\n", + "> lakehouse is the one you named — a typo there is caught only when `run_setup = True`.) OLAF also refuses a lakehouse attached from a\n", + "> different workspace — run this from the workspace that owns it.\n", + "\n", + "**`keep_unmanaged = False` is deliberate.** `apply` submits the config as the whole truth, so any\n", + "role the config does not declare is omitted from the payload — including one somebody added by hand in the portal, and including\n", + "`Default*` — a leftover `DefaultReader` reads every path and bypasses RLS/CLS, so leaving it in place\n", + "would quietly undo the whole config. That\n", + "that omission is the point: **the config is the only way to change access.** Editing roles outside the\n", + "framework does not survive the next run, by design — so do not flip this to `True` to make a run\n", + "feel safer.\n", + "\n", + "---\n", + "\n", + "## The flow\n", + "\n", + "```\n", + " setup ─▶ load_config ─▶ validate ─▶ generate ─▶ plan ─┬─▶ no changes ──────────────────▶ done\n", + " ┈┈┈┈┈┈ ┈┈┈┈┈┈┈┈┈┈┈ ┈┈┈┈┈┈┈┈ ┈┈┈┈┈┈┈┈ ┈┈┈┈ │\n", + " opt-in workbook→ no writes lock-file diff └─▶ changes ─▶ gate ─▶ apply ─▶ verify\n", + " control tables ┈┈┈┈ ┈┈┈┈┈\n", + " AUTO_APPROVE the only\n", + " live write\n", + "```\n", "\n", - "> **Community Preview:** OLAF — OneLake Access Framework is independent and is not affiliated with, endorsed by, sponsored by, or certified by Microsoft. Its bulk DAR mutation dependency is Preview and is documented for evaluation/development, not production use.\n", + "**Run it as often as you like.** Every stage is safe to repeat: `setup` is idempotent, `load_config`\n", + "is a full REPLACE from the workbook, `generate` skips when the config hash has not moved, and `plan`\n", + "finds no drift when the live state already matches. Re-running is the intended loop, not an\n", + "exceptional path.\n", "\n", - "This notebook is deliberately inert by default. It contains no live results and grants no permission to access or mutate a Fabric environment. The session binding above is not an isolation or authorization boundary. Read [the platform contract](../docs/platform-contract.md), [control-data security](../docs/control-data-security.md), and [the runbook](../docs/runbook.md) before adapting it.\n" + "> ⚠️ **Edited the workbook? Set `load_config = True` for that run.** It is `False` by default, and a\n", + "> run with it off never reads the workbook: the config table is unchanged, so `generate` skips, `plan`\n", + "> reports no drift, and the whole notebook goes green **without your edit ever reaching the vault**.\n", + "> The failure mode of forgetting is a success verdict, which is why it is called out here and not\n", + "> only at the stage.\n", + "\n", + "| # | Stage | Writes | What it does |\n", + "|---|---|---|---|\n", + "| 1 | `setup` | control tables | Creates them if missing, and migrates columns after a framework upgrade. Off by default — set `run_setup = True` to run it. |\n", + "| 1b | `load_config` | config + member | Loads the authored workbook into the two author-owned control tables. A full REPLACE, so a row deleted in the workbook is deleted here. Writes no log row — this is authoring, not deployment. Off by default. |\n", + "| 2 | `validate` | **nothing** | Every rule `generate` runs, against the config. Collects *all* failures. |\n", + "| 3 | `generate` | mapping + CSV | Commits the config into the lock-file `plan` and `apply` read. |\n", + "| 4 | `plan` | log row | Diffs mapping against live. **No live change.** Sets `changed`. |\n", + "| 5 | `apply` | **live roles** | The only live write. Refuses unless a successful `plan` exists for this same **config_hash**, and the live state still matches it. |\n", + "| 6 | verify | nothing | Reads the live state back through the `OLAF` facade. |\n", + "\n", + "**`validate` runs before `generate`, not after.** It is a dry-run of the very validation\n", + "`generate` performs, with zero writes — so it previews the commit, and previewing *after*\n", + "committing would be the wrong way round.\n", + "\n", + "---\n", + "\n", + "## Two ways to call OLAF, and the choice is load-bearing\n", + "\n", + "This notebook uses **both**, on a rule:\n", + "\n", + "| | used for | on a blocked stage |\n", + "|---|---|---|\n", + "| `notebookutils.notebook.run` | every stage whose refusal must **stop the run** — setup, validate, generate, plan, apply, rollback | **raises**, so the cell fails and the run stops |\n", + "| the `OLAF` facade | `configure` and `load_config`, which have no mode of their own · and the read-only tail (`show`, `trace`) where there is no refusal to gate on | returns a frame, **never raises on outcome** |\n", + "\n", + "> The split is **not** \"is it a mode?\" — `show` and `trace` are modes (all eight are: setup, generate,\n", + "> validate, plan, apply, rollback, show, trace). The split is whether a refusal has to halt the run.\n", + "> A read stage has nothing to halt, so the frame is the more useful shape.\n", + "\n", + "**Do not \"simplify\" this into an all-facade notebook.** The facade is the interactive surface and is\n", + "pinned by tests as never raising on outcome (`OLAF.generate()` on a config the member gate refuses\n", + "hands back a `blocked` frame). Drive the stages that way and a refused config prints a frame and the\n", + "run sails on into plan and apply — a fail-closed refusal turned into a silent continue. You would\n", + "have to re-check `OLAF.last_result[\"status\"]` after every call and raise by hand, which is\n", + "`notebook.run` rewritten badly.\n", + "\n", + "`notebook.run` also gives each stage its own session and its own parameters. For orchestration that\n", + "isolation is the feature, not the overhead — and it is what makes this notebook a faithful stand-in\n", + "for a pipeline with one activity per stage.\n", + "\n", + "## How a stage reports back\n", + "\n", + "`olaf` ends every mode with one envelope:\n", + "\n", + "```\n", + "{ \"mode\" the stage that ran \"data\" per-stage result keys\n", + " \"status\" success | skipped \"batch_id\" ties plan to apply\n", + " blocked | error \"config_hash\" which config version\n", + " \"changed\" is there drift to apply \"message\" the one-line verdict }\n", + "```\n", + "\n", + "| status | What `olaf` does | What this notebook sees |\n", + "|---|---|---|\n", + "| `success` · `skipped` | `notebook.exit(envelope)` | `notebook.run` returns the JSON above |\n", + "| `blocked` · `error` | **raises** | the cell fails and the run stops here |\n", + "\n", + "A failed stage still stops the notebook, so the calling pipeline takes its Failure path — but it is\n", + "not allowed to stop *silently*. Every cell goes through `run_stage` or its own try, which prints the\n", + "stage, the capped raise payload and this run's log rows **before** re-raising. The `if`s below branch\n", + "on *outcomes*; failures are reported and rethrown, never swallowed.\n", + "\n", + "> It also means `blocked` and `error` never arrive as a *value*. Any variable that `json.loads`\n", + "> succeeded on is `success` or `skipped` — nothing else. The one stage where that costs something\n", + "> is `apply`, which has already written to the live role set by the time it can fail, so that cell\n", + "> prints the log row before letting the failure through.\n" ] }, { "cell_type": "markdown", - "id": "workflow-params-doc", + "id": "c98064f94b4c", "metadata": {}, "source": [ "## Parameters\n", "\n", - "The public defaults stop before all external work. A reviewer must deliberately enable an authorized, isolated, non-production evaluation. Never put real tenant, workspace, item, customer, or principal values in this file.\n" + "This cell is tagged `parameters`: Fabric and papermill **replace it wholesale** at run time, so the\n", + "values below are only what a manual run falls back on. Keep it to bare assignments — anything else\n", + "written here is lost on a pipeline run.\n", + "\n", + "| Parameter | Default | What it does |\n", + "|-------------------|------------------------------------------|----------------------------------------------------------------------------|\n", + "| `env` | `\"dev\"` | Tags every log row, so one estate's dev and prod audit trails stay apart. |\n", + "| `auto_approve` | `False` | The gate. `True` lets `apply` run unattended — nobody reviews the omissions. |\n", + "| `keep_unmanaged` | `False` | `False` = config is the whole truth: a live role it omits is left out of the payload. |\n", + "| `run_setup` | `False` | `True` after a framework upgrade — `setup` migrates control-table columns. |\n", + "| `load_config` | `False` | `True` re-reads the authored workbook into the config + member tables. |\n", + "| `config_workbook` | `\"Files/security/onelake_security.xlsx\"` | The authored workbook, resolved on the attached lakehouse. |\n", + "| `batch_id` | `\"\"` | Correlates this run's log rows. Empty = this notebook generates one. |\n" ] }, { "cell_type": "code", - "id": "workflow-params", "execution_count": null, + "id": "01312ee005c9", "metadata": { "tags": [ "parameters" @@ -51,93 +182,588 @@ }, "outputs": [], "source": [ - "lakehouse_name = \"SyntheticEvaluationLakehouse\"\n", - "authorized_evaluation = False\n", - "external_access_reviewed = False\n", - "workspace_isolation_attested_for_run = False\n", - "dar_snapshot_and_etag_captured = False\n", - "sentinel_ready = False\n", - "enable_sensitive_operations = False\n", - "synthetic_workbook_path = \"\"\n" + "env = \"dev\"\n", + "lakehouse_name = \"YourLakehouse\" # the attached lakehouse this run secures (setup asserts it)\n", + "auto_approve = False\n", + "keep_unmanaged = False\n", + "run_setup = False\n", + "load_config = False\n", + "config_workbook = \"Files/security/onelake_security.xlsx\"\n", + "batch_id = \"\"" ] }, { "cell_type": "code", - "id": "workflow-guard", "execution_count": null, + "id": "f6e6474947af", "metadata": {}, "outputs": [], "source": [ + "import json\n", + "import re\n", + "import uuid\n", + "from collections import Counter\n", + "\n", "import notebookutils\n", "\n", + "# ── run constants ─────────────────────────────────────────────────────────────\n", + "# OLAF_NB the OLAF runtime notebook, by name, for notebook.run()\n", + "# LAKEHOUSE the attached lakehouse this run secures — from the lakehouse_name\n", + "# PARAMETER above, so a pipeline can set it per run (it was a hardcoded\n", + "# constant here through 1.0.x, invisible to Base parameters)\n", + "# AUTO_APPROVE True = run apply · False = stop after plan\n", + "# KEEP_UNMANAGED False = config is whole truth · omits roles absent from the config\n", + "# LOAD_CONFIG True = reload the workbook into the config + member tables\n", + "# WORKBOOK the authored workbook, resolved on the attached lakehouse\n", + "# BATCH_ID the passed-in batch_id, or a fresh one when none was supplied\n", + "#\n", + "# Captured by value, on purpose: cell 7's `%run olaf` rebinds the lowercase names, so anything\n", + "# read after it must already live in UPPER_CASE or in PARAMS.\n", + "OLAF_NB = \"olaf\"\n", "LAKEHOUSE = lakehouse_name\n", - "required_controls = {\n", - " \"lakehouse_name\": bool(LAKEHOUSE.strip()),\n", - " \"synthetic_workbook_path\": bool(synthetic_workbook_path.strip()),\n", - " \"authorized_evaluation\": authorized_evaluation,\n", - " \"external_access_reviewed\": external_access_reviewed,\n", - " \"workspace_isolation_attested_for_run\": workspace_isolation_attested_for_run,\n", - " \"dar_snapshot_and_etag_captured\": dar_snapshot_and_etag_captured,\n", - " \"sentinel_ready\": sentinel_ready,\n", - " \"enable_sensitive_operations\": enable_sensitive_operations,\n", + "AUTO_APPROVE = auto_approve\n", + "KEEP_UNMANAGED = keep_unmanaged\n", + "LOAD_CONFIG = load_config\n", + "WORKBOOK = config_workbook\n", + "BATCH_ID = batch_id.strip() or str(uuid.uuid4())\n", + "\n", + "# Setup override parameters\n", + "PARAMS = {\n", + " \"env\": env,\n", + " \"batch_id\": BATCH_ID,\n", + " # Schema-qualified names for the four control tables. These are OLAF's own defaults; point them\n", + " # wherever your estate keeps them — every stage below and the facade all read the same PARAMS,\n", + " # so they cannot drift apart.\n", + " \"config_table\": \"olaf.onelake_security_config\",\n", + " \"mapping_table\": \"olaf.onelake_security_mapping\",\n", + " \"member_table\": \"olaf.onelake_security_member\",\n", + " \"log_table\": \"olaf.onelake_security_log\",\n", "}\n", - "missing = [name for name, value in required_controls.items() if not value]\n", - "if missing:\n", - " notebookutils.notebook.exit(\n", - " \"OLAF Preview workflow stopped; disabled controls: \" + \", \".join(missing)\n", - " )\n" + "\n", + "print(\n", + " \"batch_id\",\n", + " BATCH_ID,\n", + " \"(passed in)\" if batch_id.strip() else \"(generated)\",\n", + " \"· AUTO_APPROVE\",\n", + " AUTO_APPROVE,\n", + ")\n", + "\n", + "\n", + "# ── output helpers ────────────────────────────────────────────────────────────\n", + "STAGE_ICON = {\n", + " \"setup\": \"🧱\",\n", + " \"validate\": \"🔍\",\n", + " \"generate\": \"🧬\",\n", + " \"plan\": \"📋\",\n", + " \"apply\": \"🚀\",\n", + " \"rollback\": \"↩️\",\n", + " \"trace\": \"🔭\",\n", + " \"show\": \"👥\",\n", + " \"log\": \"📜\",\n", + "}\n", + "STATUS_BADGE = {\"success\": \"✅\", \"skipped\": \"⏭️\", \"blocked\": \"🚫\", \"error\": \"❌\"}\n", + "\n", + "\n", + "def _flatten(data, _prefix=\"\"): # envelope data -> (item, value) rows\n", + " rows = []\n", + " for k, v in (data or {}).items():\n", + " if isinstance(v, dict):\n", + " rows += [(f\"{_prefix}{k}.{k2}\", str(v2)) for k2, v2 in v.items()] or [\n", + " (f\"{_prefix}{k}\", \"(none)\")\n", + " ]\n", + " elif isinstance(v, (list, tuple)):\n", + " rows.append((f\"{_prefix}{k}\", \", \".join(map(str, v)) if v else \"(none)\"))\n", + " else:\n", + " rows.append((f\"{_prefix}{k}\", str(v)))\n", + " return rows\n", + "\n", + "\n", + "def report(stage, env, detail=True): # header + message + data table\n", + " icon, status = STAGE_ICON.get(stage, \"•\"), env.get(\"status\", \"?\")\n", + " print(f\"{icon} {stage.upper():9} {STATUS_BADGE.get(status, '•')} {status}\")\n", + " print(f\" {env.get('message', '')}\")\n", + " rows = _flatten(env.get(\"data\")) if detail else []\n", + " if rows:\n", + " display(spark.createDataFrame(rows, \"item STRING, value STRING\"))\n", + " return env\n", + "\n", + "\n", + "def role_actions(env, key=\"plan\"): # per-role verdicts, omissions first\n", + " d = (env.get(\"data\") or {}).get(key) or {}\n", + " if not d:\n", + " return\n", + " # `omit`, not `delete`: a role left out of the payload is an omission candidate. The\n", + " # runtime has never emitted \"delete\" here, so an order keyed on it sorted omissions last.\n", + " order = {\"omit\": 0, \"update\": 1, \"create\": 2}\n", + " rows = sorted(((r, a) for r, a in d.items()), key=lambda x: (order.get(x[1], 9), x[0]))\n", + " print(f\" {key} — {len(rows)} role(s), omissions first\")\n", + " display(spark.createDataFrame(rows, \"role STRING, action STRING\"))\n", + "\n", + "\n", + "def fail_report(stage, exc): # what the pipeline gets to see on a failure\n", + " print(f\"❌ {stage.upper():9} FAILED\")\n", + " print(f\" {str(exc)[:1200]}\")\n", + " try: # validate writes no log row; the rest do\n", + " rows = spark.table(PARAMS[\"log_table\"]).where(f\"batch_id = '{BATCH_ID}'\").orderBy(\"run_at\")\n", + " display(rows) if rows.count() else print(\n", + " \" (no log row — validate never logs; reason is above)\"\n", + " )\n", + " except Exception as e:\n", + " print(\" (log unreadable:\", e, \")\")\n", + "\n", + "\n", + "def run_stage(stage, extra=None, detail=True): # notebook.run + report · the CELL owns the try\n", + " return report(\n", + " stage,\n", + " json.loads(\n", + " notebookutils.notebook.run(OLAF_NB, 3600, {**PARAMS, \"mode\": stage, **(extra or {})})\n", + " ),\n", + " detail=detail,\n", + " )\n", + "\n", + "\n", + "_RULE = re.compile(r\"\\s*\\(rule (\\w+)\\)\\s*$\") # MOST warnings end this way; cross-row\n", + "# and platform-limit warnings carry no \"(rule XX)\" suffix and land together under \"?\"\n", + "_ROW = re.compile(r\"^row (\\d+) \\(([^)]*)\\):\\s*\") # per-row warnings name their role\n", + "\n", + "\n", + "def warnings_table(env): # validate's warnings, one row each\n", + " ws = (env.get(\"data\") or {}).get(\"warnings\") or []\n", + " if not ws:\n", + " print(\" ✅ no warnings\")\n", + " return\n", + " rows = []\n", + " for w in ws:\n", + " text = str(w)\n", + " rule = _RULE.search(text)\n", + " text = _RULE.sub(\"\", text) if rule else text\n", + " row = _ROW.match(text)\n", + " rows.append(\n", + " (\n", + " rule.group(1) if rule else \"?\",\n", + " row.group(1) if row else \"\", # config row number, when it has one\n", + " row.group(2) if row else \"\", # the role it is about\n", + " _ROW.sub(\"\", text).strip(),\n", + " )\n", + " )\n", + " by_rule = Counter(r[0] for r in rows)\n", + " print(\n", + " \" ⚠️ \"\n", + " + str(len(rows))\n", + " + \" warning(s) · \"\n", + " + \" · \".join(f\"{k} × {v}\" for k, v in sorted(by_rule.items()))\n", + " )\n", + " # rarest rule first: one C4 among thirty B4s is the one worth reading\n", + " rows.sort(key=lambda r: (by_rule[r[0]], r[0], int(r[1] or 0)))\n", + " display(spark.createDataFrame(rows, \"rule STRING, row STRING, subject STRING, warning STRING\"))" + ] + }, + { + "cell_type": "markdown", + "id": "c98bd3b3c0fa", + "metadata": {}, + "source": [ + "## 1 · setup\n", + "\n", + "Creates the control tables if they are missing. Idempotent, so it is safe on every run.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "fbf07db53781", + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " if run_setup:\n", + " setup = run_stage(\"setup\", {\"lakehouse_name\": LAKEHOUSE})\n", + " else:\n", + " setup = None\n", + " print(\"⏭️ SETUP skipped — set run_setup = True after a framework upgrade\")\n", + "except Exception as exc:\n", + " fail_report(\"setup\", exc)\n", + " raise" ] }, { "cell_type": "markdown", - "id": "workflow-controls", + "id": "da23b3e60022", "metadata": {}, "source": [ - "## Required control sequence\n", + "## 1b · load the authored workbook → config + member\n", "\n", - "The technical DAR snapshot plus immutable ETag is separate from the per-run workspace-isolation attestation. Recompute both for the current run. OLAF creates/reads the sentinel before the first sensitive write and revalidates it with the snapshot before every sensitive write; an earlier check is not enough. Sensitive modes include first setup, workbook load, generate, apply, rollback, reset, and cleanup.\n", + "`load_config` is a **facade** call, not a mode, so the notebook loads `olaf` as a library first with\n", + "`%run`. That single `%run` serves this stage and the verify stage at the end.\n", "\n", - "Do not upload a real workbook before the external-access review. A same-lakehouse layout is not cryptographic or transactional isolation. Cleanup is containment and does not prove erasure or remove external copies.\n" + "> 🔴 **`OLAF.configure(**PARAMS)` is not optional, and it must come before any facade call.** `%run`\n", + "> brings in olaf's *own* parameter-cell defaults, which are not necessarily the tables this run uses.\n", + "> Without it, `load_config` would write to a different set of control tables than the stages below\n", + "> read. `configure` makes PARAMS sticky for every later facade call and hands back a frame of what is\n", + "> set, so the output is the receipt.\n", + ">\n", + "> `%run` also rebinds the loose lowercase names (`env`, `batch_id`, `config_table`, …). Everything\n", + "> this notebook needs afterwards is either `PARAMS` (a dict olaf never defines) or UPPER_CASE, which\n", + "> is why it survives — do not add lowercase state you expect to outlive this cell.\n", + ">\n", + "> **This is why a re-run starts at the top, not at the stage you want to redo.** After this cell,\n", + "> `env` is back to olaf's own default and `batch_id` is `\"\"` — of the switches in the parameters\n", + "> cell, `env`, `batch_id` and `keep_unmanaged` are the three olaf also defines. Re-running from cell\n", + "> 3 onwards would silently log a `prod` run as `dev`, or turn a deliberate `keep_unmanaged = True`\n", + "> back into a destructive REPLACE, with no error either way. Likewise `LOAD_CONFIG` / `AUTO_APPROVE`\n", + "> are **captured** in cell 3: flipping the lowercase switch takes effect only once cell 3 re-runs.\n", + "\n", + "A full REPLACE from the workbook: a row deleted there is deleted here. The sheet's columns must match\n", + "the table exactly — missing *and* unexpected are both refused, naming them, because a sheet loaded\n", + "with a column missing is a config that silently means something other than what the author edited.\n", + "\n", + "No audit row is written; the Delta commit is the record. This is authoring, not deployment." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "798fde952a6e", + "metadata": {}, + "outputs": [], + "source": [ + "%run olaf" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "71ffbb76a793", + "metadata": {}, + "outputs": [], + "source": [ + "# same PARAMS as every stage · verbosity=\"quiet\" drops olaf's result-key dump\n", + "try:\n", + " print(\"⚙️ CONFIGURE facade pointed at this run's control tables\")\n", + " display(OLAF.configure(**PARAMS, verbosity=\"quiet\"))\n", + "except Exception as exc:\n", + " fail_report(\"configure\", exc)\n", + " raise\n", + "\n", + "try:\n", + " if LOAD_CONFIG:\n", + " print(f\"📥 LOAD {WORKBOOK} → config + member (full REPLACE)\")\n", + " display(OLAF.load_config(\"config\", WORKBOOK, sheet=\"config\"))\n", + " display(OLAF.load_config(\"member\", WORKBOOK, sheet=\"member\"))\n", + " else:\n", + " print(\"⏭️ LOAD skipped — set load_config = True and re-run from the top\")\n", + "except Exception as exc:\n", + " fail_report(\"load_config\", exc)\n", + " raise" ] }, { "cell_type": "markdown", - "id": "workflow-sequence", + "id": "003402f0f3ce", "metadata": {}, "source": [ - "## Adapted sequence — documentation only\n", + "## 2 · validate\n", "\n", - "Review the current runtime parameter contract before uncommenting or moving any step:\n", + "Runs every rule `generate` runs, against the config, and writes **nothing** — no mapping, no CSV,\n", + "not even a log row. So a config that is going to be refused is refused here, before anything has\n", + "been committed, and the existing lock-file is left exactly as it was.\n", "\n", - "```python\n", - "# 1. Recompute the per-run isolation attestation.\n", - "# 2. Capture an immutable DAR snapshot and its current ETag.\n", - "# 3. Run read/validation stages and review the exact request.\n", - "# 4. Create/read the sentinel before the first sensitive write.\n", - "# The runtime revalidates the sentinel and snapshot before every sensitive write.\n", - "# 5. Invoke one sensitive stage with explicit enablement.\n", - "# 6. Re-read the DAR, compare post-state, and contain unexpected state.\n", - "```\n", + "It also collects *all* failures rather than stopping at the first, so one run lists everything\n", + "wrong with the config.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "474dc9cf080f", + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " validate = run_stage(\"validate\")\n", + " warnings_table(validate)\n", + "except Exception as exc:\n", + " fail_report(\"validate\", exc)\n", + " raise" + ] + }, + { + "cell_type": "markdown", + "id": "9f9511176203", + "metadata": {}, + "source": [ + "## 3 · generate\n", + "\n", + "Now that the config is known good, commit it: resolve it into the mapping lock-file that `plan` and\n", + "`apply` read, and export the versioned CSV to the mapping-history folder.\n", "\n", - "`keep_unmanaged` describes OLAF request construction only. The Preview endpoint does not document atomic full-set replacement or deletion-by-omission. Backups support recovery decisions; they do not guarantee exact restoration.\n" + "`skipped` means the config hash has not changed and the existing lock-file is still current — that\n", + "is a pass, not a problem.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6aeaf677a915", + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " generate = run_stage(\"generate\")\n", + "\n", + " if generate[\"status\"] == \"skipped\":\n", + " print(\" → config unchanged, reusing the current lock-file\")\n", + "except Exception as exc:\n", + " fail_report(\"generate\", exc)\n", + " raise" ] }, { "cell_type": "markdown", - "id": "workflow-evidence", + "id": "809c868182f2", "metadata": {}, "source": [ - "## Evidence and reporting\n", + "## 4 · plan\n", + "\n", + "Diffs the mapping against the live roles and writes the plan to the log. **No live change.**\n", "\n", - "If an authorized evaluation is performed, bind evidence to the exact commit SHA and record runtime/API versions, engine/access mode, authorization, restoration, and cleanup. Store only redacted evidence. This repository does not claim that v1.0.0 was live-verified.\n", + "`changed` is the branch: no drift means the deployment is already correct, so apply never runs and\n", + "nobody is asked to approve an empty plan.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8907d629e275", + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " plan = run_stage(\"plan\", detail=False)\n", + " role_actions(plan, \"plan\")\n", + "\n", + " print(\n", + " \"\\n → changes to apply — approval required\"\n", + " if plan[\"changed\"]\n", + " else \"\\n ✅ no drift — nothing to apply, the run ends here\"\n", + " )\n", + "except Exception as exc:\n", + " fail_report(\"plan\", exc)\n", + " raise" + ] + }, + { + "cell_type": "markdown", + "id": "c82f3f39fdcf", + "metadata": {}, + "source": [ + "## 5 · gate → apply\n", + "\n", + "The only human step. In ADF this is an approval check; here it is `AUTO_APPROVE`.\n", + "\n", + "**What actually binds apply to the plan is `config_hash`, not `batch_id`.** `apply` refuses unless\n", + "the log holds a successful `plan` row for this env and this same config hash, *and* the live state\n", + "still matches what that plan diffed against — edit the config between the two and apply refuses\n", + "rather than deploying a plan nobody reviewed. `batch_id` only correlates the run's log rows: it is\n", + "not in the gate's filter, so a shared batch_id neither grants nor withholds anything.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3576a5432b5e", + "metadata": {}, + "outputs": [], + "source": [ + "apply_result = None # bound in every branch, so the exit cell can read it safely\n", + "\n", + "try:\n", + " if not plan[\"changed\"]:\n", + " print(\"⏭️ APPLY skipped — nothing to apply\")\n", + "\n", + " elif not AUTO_APPROVE:\n", + " print(\"⏸️ APPLY held — set auto_approve = True and re-run from the top to apply\")\n", + "\n", + " else:\n", + " apply_result = run_stage(\"apply\", {\"keep_unmanaged\": KEEP_UNMANAGED}, detail=False)\n", + " # apply's envelope carries no per-role map, so there is no role_actions() call here: the\n", + " # plan cell above already printed the verdicts, and what apply adds is what it actually did.\n", + " # (An `applied` key existed once, renamed `push_status` because it holds the bulk PUT's\n", + " # HTTP 200 -- \"applied: 200\" reads as 200 roles to anyone skimming an incident.)\n", + " done = apply_result.get(\"data\") or {}\n", + " wrote, http = done.get(\"roles_written\", \"?\"), done.get(\"push_status\", \"?\")\n", + " print(f\" ✍️ wrote {wrote} role(s) · HTTP {http}\")\n", + " gone = done.get(\"omitted_role_candidates\") or []\n", + " # CANDIDATES, not confirmed deletions: the Preview bulk endpoint does not document\n", + " # deletion-by-omission, so what OLAF can say is which roles it left out of the payload.\n", + " print(f\" ➖ omitted {len(gone)}:\", \", \".join(gone) if gone else \"none\")\n", + " if gone:\n", + " print(\n", + " \" ↳ omission is a REQUEST shape, not a confirmed outcome — check the post-state\"\n", + " )\n", + " print(\" 💾 backup:\", done.get(\"backup_path\", \"—\"))\n", + "except Exception as exc:\n", + " fail_report(\"apply\", exc)\n", + " raise" + ] + }, + { + "cell_type": "markdown", + "id": "7998a857b660", + "metadata": {}, + "source": [ + "## 6 · verify — read the live state back\n", + "\n", + "The `OLAF` facade is already bound and configured — stage 1b did both. The facade is the interactive\n", + "surface: unlike the pipeline path it **never raises on outcome** and hands back a DataFrame, so this\n", + "cell is safe to re-run and safe to read.\n", + "\n", + "This is what turns \"apply returned success\" into \"the live role set looks like this\".\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "bbfb559e8ee2", + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " print(\"🔭 TRACE operational snapshot — the current generation end to end\")\n", + " display(OLAF.trace())\n", + "\n", + " print(\"👥 SHOW the live role set, as it stands after the apply above\")\n", + " display(OLAF.show(by=\"role\", subject=\"*\")) # subject is REQUIRED · \"*\" = every role\n", + "except Exception as exc:\n", + " fail_report(\"verify\", exc)\n", + " raise" + ] + }, + { + "cell_type": "markdown", + "id": "5337b3feac35", + "metadata": {}, + "source": [ + "## What the run left behind\n", + "\n", + "`olaf.onelake_security_log` keeps every row this run wrote, keyed by `batch_id` — the durable audit\n", + "trail, readable long after this session is gone.\n", + "\n", + "The run timestamp column is `run_at` — not `logged_at`. The full column list is in [data-model.md](../docs/data-model.md).\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d454df3922de", + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " print(f\"📜 LOG every row this run wrote · batch_id {BATCH_ID}\")\n", + " display(spark.table(PARAMS[\"log_table\"]).where(f\"batch_id = '{BATCH_ID}'\").orderBy(\"run_at\"))\n", + "except Exception as exc:\n", + " fail_report(\"log\", exc)\n", + " raise" + ] + }, + { + "cell_type": "markdown", + "id": "037d45b94fc9", + "metadata": {}, + "source": [ + "## Recovery — two mechanisms, and they are not interchangeable\n", + "\n", + "`apply` defaults to config-as-whole-truth: it omits every live role the config does not declare from the submitted payload, **including\n", + "`Default*`**. There are two ways back, and picking the wrong one loses roles.\n", + "\n", + "| What went wrong | Use | Why |\n", + "|---|---|---|\n", + "| **apply failed part-way** — the push threw | the **pre-apply backup file** | it restores the live role set *exactly as it was*, including roles the config never declared |\n", + "| **apply succeeded, the config was wrong** | `rollback` | it restores a prior config version and re-runs generate → plan → apply |\n", + "\n", + "> 🔴 **`rollback` is not a substitute for the backup.** It replays *config*, so it cannot rebuild\n", + "> roles config never declared — and that is precisely the set a default `apply` leaves out. After a\n", + "> failed or mistaken run, the backup file is the recovery input for putting `Default*` back; it is a\n", + "> recovery *input*, not a guaranteed exact restoration of platform state.\n", + "\n", + "**The backup is automatic.** `apply` writes the live roles to `Files/security/role-backups/` before\n", + "it pushes, on every run, and **a failed backup aborts the apply** — nothing is pushed without a\n", + "restore point behind it. The path is printed by the apply cell above and carried in the envelope as\n", + "`data.backup_path`. Restoring is `json.load` → `put_roles`, nothing to edit in between\n", + "(RUNBOOK §3c).\n", + "\n", + "**A failed push is not silent either.** Before the exception leaves `apply`, the framework re-reads\n", + "the live roles and writes a forensic record to the log: the per-grant rows re-stamped `failed` (so a\n", + "push that wrote nothing cannot later claim it wrote everything) and one row per planned role saying\n", + "whether it is `PRESENT` or `ABSENT` live. Read it with the log cell above before deciding anything.\n", + "\n", + "Rollback is left commented below on purpose — an escape hatch, not part of the routine flow.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5838b6fe114d", + "metadata": {}, + "outputs": [], + "source": [ + "# Uncomment to roll back. rollback_to_version \"\" = the version immediately before the current one.\n", + "# try:\n", + "# rb = run_stage(\"rollback\", {\"rollback_to_version\": \"\",\n", + "# \"rollback_reason\": \"why this deployment is being withdrawn\"})\n", + "# except Exception as exc:\n", + "# fail_report(\"rollback\", exc)\n", + "# raise" + ] + }, + { + "cell_type": "markdown", + "id": "95182f6445b7", + "metadata": {}, + "source": [ + "## Hand the result back to the caller\n", + "\n", + "`notebook.exit` ends the notebook and returns this payload as the activity's `exitValue`, so the\n", + "pipeline that called this one can branch on whether anything actually changed instead of\n", + "re-querying the log.\n", + "\n", + "Failures never reach here: a failed stage raises, the notebook fails, and the calling activity takes\n", + "its own **Failure** path.\n", + "\n", + "**Keep this cell last.** `notebook.exit` stops execution, so anything below it would never run.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ff454cfd6aca", + "metadata": {}, + "outputs": [], + "source": [ + "try:\n", + " result = {\n", + " \"batch_id\": BATCH_ID,\n", + " \"changed\": plan[\"changed\"], # did plan find drift to apply\n", + " \"applied\": apply_result is not None, # did apply actually run\n", + " \"backup_path\": (apply_result or {}).get(\"data\", {}).get(\"backup_path\"),\n", + " }\n", + " print(\"🏁 DONE returning to the caller\")\n", + " display(\n", + " spark.createDataFrame([(k, str(v)) for k, v in result.items()], \"item STRING, value STRING\")\n", + " )\n", + "except Exception as exc:\n", + " fail_report(\"done\", exc)\n", + " raise\n", "\n", - "Official references: [bulk DAR endpoint](https://learn.microsoft.com/en-us/rest/api/fabric/core/onelake-data-access-security/create-or-update-data-access-roles), [workspace permissions](https://learn.microsoft.com/en-us/fabric/onelake/security/data-access-control-model#onelake-security-and-workspace-permissions), and [engine/user access](https://learn.microsoft.com/en-us/fabric/onelake/security/data-access-control-model#engine-and-user-access-to-data).\n" + "notebookutils.notebook.exit(json.dumps(result))" ] } ], "metadata": { "kernelspec": { "display_name": "Synapse PySpark", + "language": "Python", "name": "synapse_pyspark" }, "language_info": {