Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 98 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,24 +11,23 @@

<p align="center">
<a href="https://github.com/kengio/olaf/actions/workflows/test.yml"><img src="https://github.com/kengio/olaf/actions/workflows/test.yml/badge.svg" alt="tests"></a>
<a href="docs/testing.md"><img src="https://img.shields.io/badge/coverage-100%25%20(fixtures)-1E3A8A" alt="Coverage: 100% against in-memory fixtures"></a>
<a href="https://github.com/kengio/olaf/tags"><img src="https://img.shields.io/github/v/tag/kengio/olaf?label=release&color=1E3A8A" alt="Latest release tag"></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-1E3A8A" alt="License: MIT"></a>
<img src="https://img.shields.io/badge/status-community%20Preview-FB923C" alt="Status: community Preview">
</p>

> [!IMPORTANT]
> **OLAF v1.0.0 is an independent community Preview for evaluation and
> development, not a production-ready security product.** Its mutating path
> depends on Microsoft's bulk Data Access Roles `PUT`, which is officially
> documented as **Preview** and not recommended for production use.
> [Official endpoint status](https://learn.microsoft.com/en-us/rest/api/fabric/core/onelake-data-access-security/create-or-update-data-access-roles)
<p align="center">
<img src="https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13-0EA5E9" alt="Python 3.11, 3.12 and 3.13">
<img src="https://img.shields.io/badge/Fabric-Runtime%201.3%20%2F%20Spark%203.5%2B-0EA5E9" alt="Microsoft Fabric Runtime 1.3 / Spark 3.5 or newer">
<img src="https://img.shields.io/badge/dependencies-hash--pinned-0EA5E9" alt="Dependencies installed with --require-hashes">
<img src="https://img.shields.io/badge/lint-ruff-0EA5E9" alt="Linted and formatted with ruff">
</p>

> [!CAUTION]
> OLAF stores principal identifiers and authorization/recovery state in control
> tables and under `Files/security`. Sensitive modes are disabled by default and
> require a clean DAR snapshot with an ETag. The per-run workspace isolation
> attestation is optional — OLAF records it as `attested` or `unknown` and
> never gates on it. Complete the external access review **before uploading a
> real workbook**. Start with [Protecting OLAF control data](docs/control-data-security.md).
> [!IMPORTANT]
> **An independent community Preview for evaluation and development, not a production-ready
> security product.** Its mutating path uses Microsoft's bulk Data Access Roles `PUT`, which
> Microsoft documents as [Preview and not recommended for production](https://learn.microsoft.com/en-us/rest/api/fabric/core/onelake-data-access-security/create-or-update-data-access-roles).

**OLAF — OneLake Access Framework** is a plan → review → apply workflow for
Microsoft Fabric OneLake data access roles. It is one self-contained Fabric
Expand All @@ -40,6 +39,46 @@ The product names above are used only to describe interoperability. See
[Microsoft's Trademark and Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks)
and the project's [platform contract](docs/platform-contract.md).

## What is OneLake security?

[OneLake security](https://learn.microsoft.com/en-us/fabric/onelake/security/data-access-control-model)
is Microsoft Fabric's native, data-plane access control: you define **security roles on the data
itself** (a lakehouse's tables, folders, rows, columns) and OneLake enforces them everywhere — instead
of re-implementing security per engine with T-SQL views, workspace boundaries, or report-side filters.

- **Role-based access** — grant Read per table (`/Tables/schema/table`) or folder (`/Files/...`);
users with no role see nothing (**deny by default**).
[Create and manage roles →](https://learn.microsoft.com/en-us/fabric/onelake/security/create-manage-roles)
- **Row-level security (RLS)** — SQL-like predicates per table per role (static values; the
OneLake-supported operator subset — `= <> > >= < <= IN NOT AND OR IS BLANK NULL TRUE FALSE`,
rule C9). A member of several roles gets the **union** of what they allow.
[Row-level security →](https://learn.microsoft.com/en-us/fabric/onelake/security/row-level-security)
- **Column-level security (CLS)** — hide specific columns per role.
[Column-level security →](https://learn.microsoft.com/en-us/fabric/onelake/security/column-level-security)
- **Entra-native membership** — role members are Entra users, **security groups**, or service
principals; membership changes happen in Entra, no sync tables to maintain.
- **One definition, every engine** — enforced across the SQL analytics endpoint, Spark, and Direct
Lake semantic models (Power BI), so reports and notebooks see the same filtered data.
- **Granular write** — separate ReadWrite roles for scoped write access (RLS/CLS don't apply there).

Worth knowing: roles carrying RLS/CLS are read-only; workspace Admins/Members/Contributors bypass
OneLake security (it governs consumers); role definitions are per lakehouse.

## Why OLAF?

OneLake security provides the enforcement layer. OLAF makes operating that layer **reviewable,
repeatable, and easier to govern** when deployments grow beyond a handful of roles and tables.

| What you need | What OLAF does |
|---|---|
| **Know what will change** | `plan` shows every role and grant that will be created, updated, or omitted from the submitted payload. |
| **Prevent an unreviewed write** | `apply` requires a saved plan and refuses stale or drifted state. |
| **Catch risky configuration early** | Validation blocks common RLS/CLS, predicate, casing, and multi-role exposure mistakes before deployment. |
| **Keep the runtime simple** | One unchanged notebook serves every project; configuration and runtime parameters carry project intent. |
| **Explain what happened** | Plan and apply are linked in the audit log with who, what, when, and the relevant configuration state. |

*Full rule-by-rule detail → [docs/architecture.md](docs/architecture.md).*

## What OLAF does

| Need | OLAF behavior |
Expand All @@ -58,6 +97,14 @@ and [SQL endpoint enforcement guidance](https://learn.microsoft.com/en-us/fabric

## Before you start

> [!CAUTION]
> OLAF stores principal identifiers and authorization/recovery state in control
> tables and under `Files/security`. Sensitive modes are disabled by default and
> require a clean DAR snapshot with an ETag. The per-run workspace isolation
> attestation is optional — OLAF records it as `attested` or `unknown` and
> never gates on it. Complete the external access review **before uploading a
> real workbook**. Start with [Protecting OLAF control data](docs/control-data-security.md).

- Use Microsoft Fabric Runtime 1.3 / Spark 3.5 or newer, then verify the selected
runtime and bundled package versions in your Fabric environment. Microsoft
publishes the [runtime lifecycle](https://learn.microsoft.com/en-us/fabric/data-engineering/lifecycle)
Expand All @@ -68,7 +115,7 @@ and [SQL endpoint enforcement guidance](https://learn.microsoft.com/en-us/fabric
and the [bulk DAR authorization contract](https://learn.microsoft.com/en-us/rest/api/fabric/core/onelake-data-access-security/create-or-update-data-access-roles).
- Treat the same-lakehouse layout as a trusted-administrator boundary, not
cryptographic or transactional isolation. If that threat model is unacceptable,
do not import real principal data or run sensitive modes in v1.0.0.
do not import real principal data or run sensitive modes in this Preview.

## Quick start

Expand Down Expand Up @@ -211,6 +258,34 @@ partial or ambiguous operation. Follow
- [Testing guide](docs/testing.md)
- [Roadmap](docs/roadmap.md)

## Roadmap

Options, not promises or release dates. Each is blocked on a specific, checkable condition, and none
may weaken the control-data boundary or turn an observed service behavior into a platform contract.
**[docs/roadmap.md](docs/roadmap.md)** carries the detail and the official sources behind each one.

- [ ] **Differential apply** — per-role writes instead of one bulk `PUT`, so an apply touches only
what changed. *Blocked on a stable, concurrency-safe official contract: the single-role
endpoints are themselves Preview, and the design needs documented precondition behavior plus a
recovery model for a partially completed role set.*
- [ ] **RLS + CLS cross-grant detection at `generate`** — closing rule C5's blind spot, where a user
arrives through a group on one side and directly on the other. *The conservative rule stays
until evidence justifies otherwise; OLAF deliberately does not call Microsoft Graph.*
- [ ] **Scheduled drift detection** — a gate an adopter schedules rather than wires themselves.
*Must stay read-only, separate identity drift from policy drift, and keep real principal values
out of CI logs.*
- [ ] **Local authoring tool** — validate and build a config from a developer machine against a
real workspace, so table names, member resolution and rule violations surface while the
config is still being written instead of at `generate`. *Blocked on where real identifiers
and a Fabric token are allowed to live once they leave the Fabric boundary; validation only,
never a second write path.*
- [ ] **Separate control store** — a separately secured store or independently managed keys, for
organizations that cannot accept v1's trusted-administrator boundary. *Not a minimal v1 change.*

**Not planned:** replacing Fabric enforcement with an OLAF query layer · workspace and item RBAC ·
calling Microsoft Graph in v1 · claiming production readiness while the required endpoint is Preview
— [the full list and why](docs/roadmap.md#not-planned).

## Repository layout

```text
Expand All @@ -233,6 +308,15 @@ General questions and sanitized bug reports belong in
[GitHub issues](https://github.com/kengio/olaf/issues). See [SUPPORT.md](SUPPORT.md)
and [CONTRIBUTING.md](CONTRIBUTING.md).

<p align="center">
<img src="assets/brand/olaf-mascot.png" alt="The OLAF Arctic Owl Sentinel — a crystalline owl holding a notebook that carries the OLAF shield crest and its padlock" width="260">
</p>

<p align="center">
<em>The Arctic Owl keeps watch over your lake — every change is seen, reviewed,<br>
and approved before it is submitted.</em>
</p>

## License and identity

Code, documentation, and project artwork are licensed under [MIT](LICENSE), with
Expand Down
18 changes: 12 additions & 6 deletions assets/brand/PROVENANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ personal attribution or private working files.
|---|---|---|
| `olaf-logo.png` | `68f46d4054cce80b23f5c8255f5f0c2c306da116adbbe3c094ca0ec0d78efa28` | 512 × 512 px |
| `olaf-mascot.png` | `3078c87bd9d9a74030fb73df034f17380d01387a8dde29edf6dca5aa19740f9c` | 512 × 512 px |
| `olaf-lockup-light.png` | `3bb7dc420e58bae403d267277f98ad1807cf859e48c30a01b3d1780899fdb887` | 1200 × 600 px |
| `olaf-lockup-dark.png` | `0a8dcda65273cc73a079cc7b068a046dd9010c65ea50057f1dca42d84a4c0b33` | 1200 × 600 px |
| `olaf-social-preview.png` | `f7cb6da4018700fccc49e59185d329b7491c0c5512bb50e7fc2a68306c07e9db` | 1200 × 630 px |
| `olaf-lockup-light.png` | `f005a1b558ae309a05aacf59097a57022d91f86c45f146469f68db213dc29325` | 1280 × 640 px |
| `olaf-lockup-dark.png` | `7ad09f4cf95564e35a5270050ae8dda75553dae1c8f8aa6b892679527cf6b94e` | 1280 × 640 px |
| `olaf-social-preview.png` | `c5d1fec2a4925a5babad59920c0fd5b475778d288c6285efee45a6ac12ce0e0e` | 1280 × 640 px |

## Origin and transformations

Expand All @@ -20,9 +20,15 @@ contributors using generative-image tools and contributor-directed cleanup. No
customer material, personal likeness, private project asset, stock image, third-party
logo, Microsoft logo, or Microsoft product icon was supplied as source material.

The social-preview image is a deterministic composition of `olaf-logo.png` centered
on the project's Frost/Ice background. It contains no text, embedded profile, or
personal metadata and introduces no third-party mark.
The social-preview image places the arctic-owl shield beside the OLAF wordmark on the
project's Frost/Ice background, with the project name and the plan → review → apply
sequence set as text. It carries no embedded profile or personal metadata and
introduces no third-party mark. Its predecessor was a text-free composition of
`olaf-logo.png` alone; the text was added so a shared link says what the project is
without the reader having to open it.

The project name it sets is the same owner-selected name the lockups carry, so the
trademark note below applies to this file too.

The lockups contain the owner-selected project name “OLAF — OneLake Access
Framework.” This referential product-name use carries the trademark risk documented
Expand Down
Binary file modified assets/brand/olaf-lockup-dark.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/brand/olaf-lockup-light.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/brand/olaf-social-preview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
33 changes: 33 additions & 0 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,39 @@ OLAF deliberately does not call Microsoft Graph. That is a project design choice
not a claim that Graph tokens are universally unavailable:
[NotebookUtils token audiences](https://learn.microsoft.com/en-us/fabric/data-engineering/notebookutils/notebookutils-credentials#get-token).

## Local authoring tool — validate and build config against a real workspace

**Status: planned; blocked on where real identifiers are allowed to live.**

Today a config is authored blind. The workbook is filled in against what the author
believes the lakehouse contains, and the first time anything checks it against reality
is `generate` inside Fabric. A tool that runs on a developer machine, authenticates
with Fabric credentials, and reads the live workspace could resolve table and column
names, check member principals resolve, and surface rule violations while the config
is still being written — a `plan` for the config itself, before a notebook is involved.

What has to be settled first is not the API surface. It is that such a tool moves real
tenant data onto a laptop:

- **Credentials.** It would hold or broker a token with `OneLake.ReadWrite.All` scope
outside the Fabric boundary. The token, its cache, and its lifetime all become the
tool's responsibility, and a stolen laptop becomes a DAR-write capability. Read-only
scope for authoring is the obvious first constraint; whether that is sufficient to
build a config is not yet established.
- **Real identifiers on disk.** Resolving members means reading real principal object
ids and display names. v1's whole control-data posture says do not put those where
they are not governed — see [Protecting OLAF control data](control-data-security.md).
A local cache, a shell history, an editor's undo file and a crash dump are all places
they would land by default.
- **What it must never become.** An authoring tool reads and validates. It must not
grow a write path to the DAR collection, because that would move the reviewed
plan → approve → apply sequence off the audited runtime and onto a machine with no
audit trail. Validation only, with the deployment path unchanged.

Until those are answered the honest position is that authoring stays in the workbook and
validation stays in `generate` and `validate`, which already run every rule with zero
writes.

## Scheduled drift detection

**Status: candidate automation after the read-only contract is stable.**
Expand Down
Loading