Plan: LLM-based natural language policy evaluation

[DanverImbue]

Summary

Adds an implementation plan for letting users express approval policies in natural language, with a two-tier evaluation architecture:

• Compile when possible: If the policy is expressible as a Detent JSON Schema rule (e.g. "only allow read operations on Slack"), compile it to permissions.json for fast, deterministic, auditable enforcement.
• Judge model when not: If the policy requires judgment or runtime state (e.g. "don't post anything rude", "no more than 5 calls per minute"), store a refined version and evaluate at runtime via a small model using Simon Willison's llm CLI.
• Series composition: Detent runs first (fast deny), then the judge model provides additional restriction on requests Detent allows.

What's in this PR

plans/llm-based-evaluation.md — the full design document covering architecture, file layout, key design decisions, CLI commands, runtime flow, and implementation order.

No code changes.

Next steps

• Review the plan for feedback on approach, scope, and priorities
• Implement per the order described in the plan

🤖 Generated with Claude Code

[github-actions]

Vet found 0 issues.

[hynek-urban]

@DanverImbue Thanks for the suggestion! I can definitely see use cases for this.

There are actually many scenarios that aren't covered by Detent's current functionality. This is true even for "structural policies" - for example, GraphQL APIs. Another feature that several people requested was client-side rate-limiting.

For this reason, I'm in the process of adding a generic hooks field to Detent's permission format. The idea is that Detent's core will remain JSON Schema based but on top of that, it will also be possible to define arbitrary hooks to run for each request. I believe that all the mentioned additional functionalities, including natural language policy evaluation, can be expressed in that way.

Would that work for you? I think I'll be done with it early next week so maybe we can revisit this once done?