feat: case-insensitive search_by_prefix() via parallel lowercase trie

.planning/PROJECT.md

@@ -0,0 +1,77 @@
+# Case-Insensitive Prefix Search for folio-python
+
+## What This Is
+
+Adding case-insensitive prefix search to the folio-python library's `search_by_prefix()` method. Currently, prefix search is case-sensitive because the underlying `marisa-trie` stores labels in their original case (e.g., "Securities Fraud", "DUI", "M&A Practice Components"), but all realistic callers lowercase their input before searching — causing 100% silent miss rates.
+
+## Core Value
+
+Prefix search must return correct results regardless of input casing — `search_by_prefix("securit")` must match "Securities Fraud" just as `search_by_prefix("Securit")` does today.
+
+## Requirements
+
+### Validated
+
+- Existing case-sensitive `search_by_prefix()` works correctly for exact-case input — existing
+- `search_by_label()` already handles case insensitivity via rapidfuzz — existing
+- Trie is built from both `label_to_index` and `alt_label_to_index` keys — existing
+
+### Active
+
+- [ ] Build parallel lowercase `marisa-trie` alongside existing case-sensitive trie
+- [ ] Add `case_sensitive` parameter to `search_by_prefix()` defaulting to `False`
+- [ ] Fix pure-Python fallback path to also support case-insensitive search
+- [ ] Normalize prefix cache to share entries across case variants when case-insensitive
+- [ ] Map lowercase trie keys back to original `OWLClass` objects correctly
+- [ ] Add tests covering case-insensitive prefix search (lowercase, UPPERCASE, mixed, acronyms like "DUI", symbols like "M&A")
+
+### Out of Scope
+
+- Changing `search_by_label()` behavior — already case-insensitive
+- Unicode normalization beyond `.lower()` — no evidence of non-ASCII labels in FOLIO
+- Removing the existing case-sensitive trie — preserved via `case_sensitive=True` parameter
+
+## Context
+
+- **Upstream issue:** alea-institute/folio-python#15
+- **Maintainer approval:** @mjbommar confirmed Option 1 (parallel lowercase trie) in issue comments
+- **Impact:** folio-mapper's Area of Law branch returns only 1 candidate instead of 5+ due to this bug
+- **Existing workaround:** folio-mapper tries both `.capitalize()` and lowercase, but fails for "DUI", "M&A", "IP" edge cases
+- **Library size:** ~18K FOLIO concepts — memory is not a constraint for a second trie
+- **Trie location:** `graph.py:1004-1012` builds the trie during `_load_folio()`
+- **Search location:** `graph.py:1289-1333` implements `search_by_prefix()`
+
+## Constraints
+
+- **Backward compatibility**: Existing `search_by_prefix("Securit")` must continue to work — the new `case_sensitive=False` default returns a superset of what `case_sensitive=True` would return
+- **No new dependencies**: Only uses existing `marisa_trie` (already a dependency)
+- **Branch strategy**: Feature branch `feature/case-insensitive-prefix` targeting `main`
+
+## Key Decisions
+
+| Decision | Rationale | Outcome |
+|----------|-----------|---------|
+| Option 1: Parallel lowercase trie | Cheap in memory (~18K concepts), O(prefix) lookup, simple mapping back to OWLClass via lowercase→original label dict | -- Pending |
+| `case_sensitive=False` default | Matches user expectation — all realistic callers lowercase input | -- Pending |
+| Shared prefix cache (normalized to lowercase) | Avoids duplicate cache entries for "securit" vs "Securit" when case-insensitive | -- Pending |
+| Fix both trie and pure-Python fallback | Consistent behavior regardless of whether marisa_trie is installed | -- Pending |
+
+## Evolution
+
+This document evolves at phase transitions and milestone boundaries.
+
+**After each phase transition** (via `/gsd-transition`):
+1. Requirements invalidated? -> Move to Out of Scope with reason
+2. Requirements validated? -> Move to Validated with phase reference
+3. New requirements emerged? -> Add to Active
+4. Decisions to log? -> Add to Key Decisions
+5. "What This Is" still accurate? -> Update if drifted
+
+**After each milestone** (via `/gsd-complete-milestone`):
+1. Full review of all sections
+2. Core Value check — still the right priority?
+3. Audit Out of Scope — reasons still valid?
+4. Update Context with current state
+
+---
+*Last updated: 2026-04-07 after initialization*

.planning/REQUIREMENTS.md

@@ -0,0 +1,96 @@
+# Requirements: Case-Insensitive Prefix Search
+
+**Defined:** 2026-04-07
+**Core Value:** Prefix search must return correct results regardless of input casing
+
+## v1 Requirements
+
+### Data Structures
+
+- [ ] **DS-01**: FOLIOGraph declares `_lowercase_label_trie` attribute (parallel `marisa_trie.Trie` with casefolded keys)
+- [ ] **DS-02**: FOLIOGraph declares `_lowercase_to_original` attribute (`Dict[str, List[str]]` mapping casefolded labels to original-case labels)
+- [ ] **DS-03**: FOLIOGraph declares `_ci_prefix_cache` attribute (separate cache for case-insensitive queries)
+
+### Index Building
+
+- [ ] **IDX-01**: `parse_owl()` builds `_lowercase_to_original` dict using `defaultdict(list)` from both `label_to_index` and `alt_label_to_index` keys with `casefold()`
+- [ ] **IDX-02**: `parse_owl()` builds `_lowercase_label_trie` from `_lowercase_to_original.keys()` (deduplicated by construction)
+- [ ] **IDX-03**: `parse_owl()` clears `_prefix_cache` and `_ci_prefix_cache` at start of trie-building block (fixes pre-existing `refresh()` staleness)
+
+### Search API
+
+- [ ] **API-01**: `search_by_prefix()` accepts `case_sensitive: bool = False` parameter
+- [ ] **API-02**: When `case_sensitive=False`, queries the lowercase trie with `prefix.casefold()`
+- [ ] **API-03**: When `case_sensitive=True`, queries the existing trie with original prefix (preserves backward compat)
+- [ ] **API-04**: Case-insensitive results resolve through bridge dict → `label_to_index`/`alt_label_to_index` → `OWLClass`
+- [ ] **API-05**: Results are deduplicated by IRI index using a `seen` set (prevents duplicates from lowercase collisions)
+- [ ] **API-06**: Case-insensitive queries use `_ci_prefix_cache` keyed by `prefix.casefold()`
+
+### Fallback Parity
+
+- [ ] **FB-01**: Pure-Python fallback path (when `marisa_trie` is not installed) supports `case_sensitive` parameter
+- [ ] **FB-02**: Pure-Python fallback applies `casefold()` to both query prefix and labels when `case_sensitive=False`
+
+### Tests
+
+- [ ] **TEST-01**: Test case-insensitive search returns results for lowercase input (e.g., `"securit"` matches "Securities Fraud")
+- [ ] **TEST-02**: Test case-insensitive search handles acronyms (e.g., `"dui"` matches "DUI")
+- [ ] **TEST-03**: Test `case_sensitive=True` preserves original behavior (e.g., `"securit"` returns nothing, `"Securit"` returns results)
+- [ ] **TEST-04**: Test no duplicate OWLClass objects in results
+- [ ] **TEST-05**: Test existing `test_search_prefix` still passes (backward compat — "Mich" result ordering)
+- [ ] **TEST-06**: Test pure-Python fallback produces same results as trie path
+
+## v2 Requirements
+
+### Extended Normalization
+
+- **NORM-01**: NFKD Unicode normalization for non-ASCII labels
+- **NORM-02**: Accent/diacritic stripping for broader matching
+
+### Prefix Length
+
+- **LEN-01**: Lower `MIN_PREFIX_LENGTH` to 2 for short acronyms ("IP", "DK", "EU")
+
+## Out of Scope
+
+| Feature | Reason |
+|---------|--------|
+| Locale-aware case folding | Python 3 `casefold()` is sufficient; FOLIO labels are English |
+| Fuzzy prefix matching | `search_by_label()` already covers fuzzy via rapidfuzz |
+| Changing `search_by_label()` | Already case-insensitive |
+| Removing case-sensitive trie | Preserved via `case_sensitive=True` parameter |
+| `MIN_PREFIX_LENGTH` change | Separate concern, keep at 3 for this PR |
+
+## Traceability
+
+| Requirement | Phase | Status |
+|-------------|-------|--------|
+| DS-01 | Phase 1 | Pending |
+| DS-02 | Phase 1 | Pending |
+| DS-03 | Phase 1 | Pending |
+| IDX-01 | Phase 2 | Pending |
+| IDX-02 | Phase 2 | Pending |
+| IDX-03 | Phase 2 | Pending |
+| API-01 | Phase 3 | Pending |
+| API-02 | Phase 3 | Pending |
+| API-03 | Phase 3 | Pending |
+| API-04 | Phase 3 | Pending |
+| API-05 | Phase 3 | Pending |
+| API-06 | Phase 3 | Pending |
+| FB-01 | Phase 3 | Pending |
+| FB-02 | Phase 3 | Pending |
+| TEST-01 | Phase 4 | Pending |
+| TEST-02 | Phase 4 | Pending |
+| TEST-03 | Phase 4 | Pending |
+| TEST-04 | Phase 4 | Pending |
+| TEST-05 | Phase 4 | Pending |
+| TEST-06 | Phase 4 | Pending |
+
+**Coverage:**
+- v1 requirements: 20 total
+- Mapped to phases: 20
+- Unmapped: 0
+
+---
+*Requirements defined: 2026-04-07*
+*Last updated: 2026-04-07 after initial definition*

.planning/ROADMAP.md

@@ -0,0 +1,79 @@
+# Roadmap: Case-Insensitive Prefix Search
+
+## Overview
+
+This roadmap delivers case-insensitive prefix search for folio-python's `search_by_prefix()` method. The work progresses from declaring new data structures, through building parallel indexes, to modifying search behavior, and finally validating correctness with comprehensive tests. Each phase builds on the previous one, with the first two phases introducing zero behavioral change and Phase 3 delivering the actual feature flip.
+
+## Phases
+
+**Phase Numbering:**
+- Integer phases (1, 2, 3): Planned milestone work
+- Decimal phases (2.1, 2.2): Urgent insertions (marked with INSERTED)
+
+Decimal phases appear between their surrounding integers in numeric order.
+
+- [ ] **Phase 1: Data Structure Declarations** - Declare new attributes on FOLIOGraph for lowercase trie, bridge dict, and CI cache
+- [ ] **Phase 2: Index Building** - Build the parallel lowercase trie and bridge dict during parse_owl()
+- [ ] **Phase 3: Search API and Fallback** - Wire case-insensitive search through both trie and pure-Python paths
+- [ ] **Phase 4: Test Suite** - Validate correctness, backward compatibility, and edge cases
+
+## Phase Details
+
+### Phase 1: Data Structure Declarations
+**Goal**: FOLIOGraph has all new attributes declared so subsequent phases can populate and query them
+**Depends on**: Nothing (first phase)
+**Requirements**: DS-01, DS-02, DS-03
+**Success Criteria** (what must be TRUE):
+  1. FOLIOGraph instances have a `_lowercase_label_trie` attribute initialized to None or empty
+  2. FOLIOGraph instances have a `_lowercase_to_original` dict attribute initialized to empty
+  3. FOLIOGraph instances have a `_ci_prefix_cache` dict attribute initialized to empty
+  4. Existing tests still pass with no behavioral change
+**Plans**: TBD
+
+### Phase 2: Index Building
+**Goal**: parse_owl() populates the lowercase trie and bridge dict so they are ready for queries after loading
+**Depends on**: Phase 1
+**Requirements**: IDX-01, IDX-02, IDX-03
+**Success Criteria** (what must be TRUE):
+  1. After parse_owl() completes, `_lowercase_to_original` maps every casefolded label/altLabel to its original-case variant(s)
+  2. After parse_owl() completes, `_lowercase_label_trie` contains all keys from `_lowercase_to_original`
+  3. Both `_prefix_cache` and `_ci_prefix_cache` are cleared at the start of the trie-building block (fixing pre-existing refresh() staleness)
+  4. Existing tests still pass with no behavioral change
+**Plans**: TBD
+
+### Phase 3: Search API and Fallback
+**Goal**: Users can call search_by_prefix() with any casing and get correct results
+**Depends on**: Phase 2
+**Requirements**: API-01, API-02, API-03, API-04, API-05, API-06, FB-01, FB-02
+**Success Criteria** (what must be TRUE):
+  1. `search_by_prefix("securit")` returns results including "Securities Fraud" (lowercase input matches title-case labels)
+  2. `search_by_prefix("Securit", case_sensitive=True)` returns the same results as before this change (backward compat)
+  3. `search_by_prefix("dui")` returns results including "DUI" (acronym case insensitivity)
+  4. No duplicate OWLClass objects appear in any result set
+  5. When marisa_trie is not installed, the pure-Python fallback produces equivalent case-insensitive results
+**Plans**: TBD
+
+### Phase 4: Test Suite
+**Goal**: Automated tests prove correctness, backward compatibility, and edge case handling
+**Depends on**: Phase 3
+**Requirements**: TEST-01, TEST-02, TEST-03, TEST-04, TEST-05, TEST-06
+**Success Criteria** (what must be TRUE):
+  1. A test asserts lowercase prefix input returns correct OWLClass results (TEST-01)
+  2. A test asserts acronym queries like "dui" match their uppercase labels (TEST-02)
+  3. A test asserts case_sensitive=True preserves original behavior -- "securit" returns nothing, "Securit" returns results (TEST-03)
+  4. A test asserts no duplicate OWLClass objects in results (TEST-04)
+  5. The existing `test_search_prefix` ("Mich" ordering) still passes unchanged (TEST-05)
+  6. A test asserts pure-Python fallback produces same results as the trie path (TEST-06)
+**Plans**: TBD
+
+## Progress
+
+**Execution Order:**
+Phases execute in numeric order: 1 -> 2 -> 3 -> 4
+
+| Phase | Plans Complete | Status | Completed |
+|-------|----------------|--------|-----------|
+| 1. Data Structure Declarations | 0/0 | Not started | - |
+| 2. Index Building | 0/0 | Not started | - |
+| 3. Search API and Fallback | 0/0 | Not started | - |
+| 4. Test Suite | 0/0 | Not started | - |

.planning/STATE.md

@@ -0,0 +1,68 @@
+# Project State
+
+## Project Reference
+
+See: .planning/PROJECT.md (updated 2026-04-07)
+
+**Core value:** Prefix search must return correct results regardless of input casing
+**Current focus:** Phase 1 - Data Structure Declarations
+
+## Current Position
+
+Phase: 1 of 4 (Data Structure Declarations)
+Plan: 0 of 0 in current phase
+Status: Ready to plan
+Last activity: 2026-04-08 - Completed quick task 260408-9yz: Address PR #16 review feedback
+
+Progress: [░░░░░░░░░░] 0%
+
+## Performance Metrics
+
+**Velocity:**
+- Total plans completed: 0
+- Average duration: -
+- Total execution time: 0 hours
+
+**By Phase:**
+
+| Phase | Plans | Total | Avg/Plan |
+|-------|-------|-------|----------|
+| - | - | - | - |
+
+**Recent Trend:**
+- Last 5 plans: -
+- Trend: -
+
+*Updated after each plan completion*
+
+## Accumulated Context
+
+### Decisions
+
+Decisions are logged in PROJECT.md Key Decisions table.
+Recent decisions affecting current work:
+
+- Parallel lowercase trie approach confirmed by maintainer @mjbommar
+- casefold() over lower() for Unicode correctness
+- Separate _ci_prefix_cache to avoid heisenbug with shared cache
+
+### Pending Todos
+
+None yet.
+
+### Blockers/Concerns
+
+- MIN_PREFIX_LENGTH=3 filters out 2-char queries like "IP" -- documented as v2 scope (LEN-01)
+- Pre-existing _prefix_cache staleness in refresh() -- will be fixed in Phase 2 (IDX-03)
+
+### Quick Tasks Completed
+
+| # | Description | Date | Commit | Directory |
+|---|-------------|------|--------|-----------|
+| 260408-9yz | Address PR #16 review feedback: mechanical cleanups, dedup-with-tiebreak on both paths, label-over-alt-label ranking tweak | 2026-04-08 | 4b8262a | [260408-9yz-address-pr-16-review-feedback-mechanical](./quick/260408-9yz-address-pr-16-review-feedback-mechanical/) |
+
+## Session Continuity
+
+Last session: 2026-04-07
+Stopped at: Roadmap creation complete
+Resume file: None