Skip to content

Intent ergonomics overhaul: unified API, explainable outputs, and smarter Groq defaults - #4

Merged
stevekrenzel merged 12 commits into
mainfrom
ergonomics
Dec 22, 2025
Merged

Intent ergonomics overhaul: unified API, explainable outputs, and smarter Groq defaults#4
stevekrenzel merged 12 commits into
mainfrom
ergonomics

Conversation

@stevekrenzel

Copy link
Copy Markdown
Contributor

Summary

This PR renames and consolidates the core reranking API around a single Intent class, adds higher-level helpers (filter() and choice()), and introduces optional
per-item explanations for more transparent results. It also improves Groq provider behavior (reasoning effort + guided retries to repair invalid JSON), expands
configuration/provider selection, and significantly strengthens test coverage (unit + real-wire integration).

Key Changes

  • API ergonomics
    • Rename RerankerIntent and unify constructor configuration into a single options object.
    • Add new high-level APIs: filter() (relevancy subset) and choice() (pick exactly one).
  • Explainability
    • Add optional explain output with per-item reasoning (schema/prompt updates to support explanation alongside scores/decisions).
  • Scoring + schema flexibility
    • Make score range configurable (minScore/maxScore) and relax schema constraints accordingly.
    • Expand schema support for ranking, filtering, and single-choice selection.
  • Config + provider selection
    • Replace INTENT_MODEL-style config with explicit provider selection + clearer defaults.
  • Groq provider improvements
    • Replace temperature usage with reasoningEffort.
    • Add guided retry/repair to handle invalid JSON responses.
  • Tests
    • Add/expand unit tests for new APIs and extractors/config helpers.
    • Expand integration tests (including default constructor examples and improved descriptions).
  • Packaging/docs touch-ups
    • Publish package as @with-logic/intent and align prompts/naming references.

Notes / Breaking Changes

  • Reranker is removed in favor of Intent (constructor shape and method names updated accordingly).
  • LLM call config changes from temperaturereasoningEffort (Groq path).

stevekrenzel and others added 12 commits December 21, 2025 16:51
This refactors the public API from Reranker to Intent, consolidating context, extractors, and config into a single optional constructor options object.

Notable changes:
- Renamed core class/files/tests from reranker -> intent, and rerank() -> rank().
- Added default key/summary extractors (hash-based key + pretty JSON summary) and centralized JSON stringification helpers.
- Adjusted config naming/usage to preserve UPPER_SNAKE_CASE env-backed CONFIG while exposing camelCase options.
- Improved Groq provider testability by injecting an SDK factory, removing module-level mocking and enabling deterministic unit tests.
- Updated schema/message building and tests accordingly; removed obsolete groq-default reranker test and added intent equivalent.

Edge cases/behavior:
- Preserves stable ordering on ties and returns original order on errors.
- Validates relevancyThreshold is within 0–10 and errors early.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Problem
- Groq occasionally returns output that is either invalid JSON or fails server-side json_schema validation (json_validate_failed), which currently bubbles up as a hard failure.

Solution
- Add configurable JSON repair retries to the Groq LlmClient.
- On JSON.parse failure, capture the raw model output and append a repair turn (assistant raw output + user instruction with the parse error) and retry.
- On json_validate_failed, attempt to extract the rejected generation from multiple groq-sdk error shapes (structured payloads and message-embedded payloads), then append a repair turn and retry.

Configuration
- Introduces GROQ_JSON_REPAIR_ATTEMPTS (default: 3, min: 0) and an override option jsonRepairAttempts on createDefaultGroqClient.

Tests
- Expands unit coverage to exercise parse + validation repair paths and error-shape parsing.
- Adds an integration test that demonstrates repairing a schema-validation failure.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- Change LLM response contract from Record<string, number> to Record<string, { explanation: string; score: number }> and update prompt/schema accordingly.
- Add rank(..., { explain: true }) overload returning { item, explanation }[]; default remains T[].
- Preserve existing safety behavior: stable sorting on ties; batch-level and top-level fallbacks return original order (with blank explanations when explain is enabled).
- Add unit + Groq integration test coverage, including schema property ordering (explanation before score) to encourage models to emit explanations first.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- Add min/max score configuration (options + INTENT_MIN_SCORE/INTENT_MAX_SCORE) and validate threshold against the configured range.
- Update prompts and schema generation to use the configured range; clamp returned scores to the range.
- Remove JSON Schema minimum/maximum constraints on score for broader structured-output compatibility.
- Stabilize Groq integration by increasing LLM call timeout and test timeout for the relevance sanity check.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add a broad set of unmocked integration tests for Intent that exercise production wire paths against the configured Groq provider.

Coverage includes:
- obvious-match ranking
- threshold behavior for all-unrelated inputs
- stable ordering for ties/near-ties (when all items are returned)
- explain=true output shape and non-empty explanations
- custom extractors for nested objects
- unicode keys/summaries and punctuation/newline keys
- batching behavior and larger stress case across multiple batches
- per-call userId override

Also refresh the Groq provider relevance sanity test candidates to reduce accidental topical overlap.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- Introduce INTENT_PROVIDER (enum; currently only GROQ) and remove INTENT_MODEL.
- Add enumString() config helper for validated enum env vars.
- Make Intent provider-driven and source the model from GROQ defaults when provider=GROQ.
- Update types and unit tests to cover provider parsing and model resolution.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add real-wire integration coverage for the simplest usage pattern (new Intent()) using primitive string items, validating that default extractors and provider wiring work end-to-end without any configuration.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Rename Intent integration tests to be clearer, more user-facing examples while keeping behavior unchanged.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add two new high-level Intent operations alongside rank():

- filter(query, candidates): boolean relevancy decisions per candidate; returns only relevant items while preserving input order, with optional per-item explanations.
- choice(query, candidates): always returns exactly one winner; uses an enum-based selectedKey schema and a batched tournament strategy for large inputs, with optional explanation.

Implementation includes new schema builders and prompt/message builders for the two operations, plus comprehensive unit coverage (100%) and stricter, deterministic real-wire integration assertions for rank/filter/choice behavior.
Replace the temperature-based tuning surface with Groq's reasoning_effort control to better match reasoning-style models.

- Add GROQ_DEFAULT_REASONING_EFFORT (low|medium|high, default: medium) and remove GROQ_DEFAULT_TEMPERATURE
- Update LlmCallConfig to accept reasoningEffort and plumb it through Intent + selectLlmClient
- Update Groq provider request building to send reasoning_effort (while preserving strict JSON schema response_format)
- Update unit/integration tests for config parsing and request shape changes

Backwards compatibility is intentionally not preserved.
Update README to emphasize choice API, add a concise Quickstart example, and clarify configuration/performance knobs (relevancy threshold behavior, min/max score range, timeout, batching).

Add an integration test that validates the README Quickstart example output.

Refine system prompts for rank and choice to produce end-user-facing explanations (avoid prompt mechanics and first-person language).
Rename the npm package to the @with-logic scope and update README examples to import from the scoped name.

Also set publishConfig.access=public so scoped publishes default to public access.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@claude

claude Bot commented Dec 22, 2025

Copy link
Copy Markdown

Claude finished @stevekrenzel's task —— View job


PR Review: Intent Ergonomics Overhaul ✅

Comprehensive code review completed against Intent library engineering standards.


Executive Summary

This is a high-quality PR that delivers substantial improvements to the Intent library. The architectural changes are sound, documentation is comprehensive, and the code demonstrates strong engineering discipline. Tests pass with 100% coverage, and both linting and type checking succeed.

However, there is one critical code style violation that must be addressed: several instances of single-line if statements without braces, which directly violates the stated coding standards in CLAUDE.md.


Critical Issues (Must Fix)

1. Missing Braces on Single-Line If Statements

Location: src/intent.ts:616, 643, 677, 789 and src/providers/groq.ts:227-228, 301-302, 306-307

The code uses single-line if statements without braces, which directly violates CLAUDE.md's core principle: "Always Use Braces: No single-line if statements, even for simple returns."

// ❌ Current code (src/intent.ts:616)
if (data == null || typeof data !== "object") return null;

// ✅ Required style
if (data == null || typeof data !== "object") {
  return null;
}

Affected lines in intent.ts:

  • Line 616: if (data == null || typeof data !== "object") return null;
  • Line 643: if (data == null || typeof data !== "object") return null;
  • Line 677: if (selectedKey === "") return null;
  • Line 789: if (b.score !== a.score) return b.score - a.score;

Affected lines in groq.ts:

  • Lines 227-228: if (state.remaining <= 1) { return undefined; } (braces present but single line)
  • Lines 301-302: if (!serverError) { return undefined; } (braces present but single line)
  • Lines 306-307: if (code !== "json_validate_failed") { return undefined; } (braces present but single line)

Why this matters: The coding standards explicitly require braces on all if statements for consistency, readability, and to prevent future bugs when someone adds a second line without noticing the missing braces. The examples in CLAUDE.md show each brace on its own line.


Code Quality Strengths

✅ Excellent Function Sizes

All functions are under 30 lines in the core implementation files (intent.ts, groq.ts, extractors.ts). This exceeds the target and demonstrates excellent code organization through small, focused functions.

✅ Comprehensive Documentation

Every function has proper JSDoc with:

  • Clear descriptions
  • @param annotations with detailed explanations
  • @returns documentation
  • @throws documentation where applicable
  • @example blocks for public APIs
  • @private markers for internal functions

The documentation quality is exemplary, especially in src/intent.ts:20-51, 174-221.

✅ Strong TypeScript Type Safety

  • No any types without justification (groq.ts:365, 369 use any for groq-sdk interop with comment explaining why)
  • Proper use of generics (e.g., Intent<T>)
  • Type narrowing with guards (e.g., typeof checks)
  • Overloaded method signatures for explain parameter (intent.ts:265-281)
  • Strict type checking passes with zero errors

✅ Excellent Error Handling

  • Graceful degradation with fallbacks (intent.ts:317-325, preserving original order on errors)
  • Comprehensive error messages with context (intent.ts:139-141, 167-169)
  • Proper error type checking: error instanceof Error (intent.ts:318)
  • Structured retry logic in Groq provider (groq.ts:436-465)

✅ Early Returns and Minimal Nesting

The code consistently uses early returns for fast paths:

  • Zero/one candidate optimizations (intent.ts:283-293, 368-378, 436-446)
  • Null checks with early returns (intent.ts:616, 643, 677, 722-725, 746-748)
  • Maximum nesting depth stays at 2-3 levels

✅ 100% Test Coverage

All unit tests pass with complete statement, branch, function, and line coverage. Integration tests provide additional real-world validation with 165 tests passing.


Architecture Review

API Design: Excellent

The refactoring from Reranker to Intent with unified options object significantly improves ergonomics. The addition of filter() and choice() methods provides clear, purpose-driven APIs.

Particularly well-designed:

  • Overloaded signatures for explain parameter provide type-safe distinction between T[] and Array<{ item: T; explanation: string }> returns
  • Tournament strategy in choice() for handling large candidate sets (intent.ts:451-477)
  • Consistent fallback behavior across all methods

Provider Architecture: Solid

The Groq provider improvements are substantial:

  • JSON repair logic (groq.ts:198-241) is clever and well-structured
  • Breaking repair logic into focused helper functions demonstrates excellent code organization
  • Regex-based extraction of failed generation from error messages (groq.ts:284-285) is pragmatic
  • Guided retry with repair messages (groq.ts:199-214) elegantly handles schema validation failures

Configuration System: Clear

The CONFIG object with environment variable mapping is well-structured. The distinction between UPPER_SNAKE_CASE config keys and camelCase API options is intentional and documented (CLAUDE.md explains this design choice).


Minor Observations (Non-Blocking)

File Sizes

Two files exceed the 300-line guideline:

  • src/intent.ts: 823 lines ⚠️
  • src/providers/groq.ts: 471 lines ⚠️

While these exceed the <300 line guideline, both files remain highly cohesive:

  • intent.ts: Single Intent class with three public methods (rank, filter, choice) and supporting private helpers. Breaking this apart would reduce cohesion.
  • groq.ts: Single provider implementation with retry/repair logic. Could potentially split repair logic into separate file, but current organization is acceptable.

Recommendation: Accept as-is. Both files maintain single, clear purposes despite their length.

Naming Clarity

Generally excellent, but one minor suggestion:

  • buildRepairRetry (groq.ts:226) - name suggests it builds a retry, but it also checks if retries remain. Consider tryBuildRepairRetry or maybeBuildRepairRetry to signal the possibility of undefined return.

This is a nitpick and non-blocking.

Test Organization

Test files are well-structured with:

  • Clear top-level describe blocks
  • Helper functions for creating test fixtures (makeCtx in intent.unit.test.ts:9-27)
  • Proper isolation between tests
  • Descriptive test names following the pattern "method() description"

Summary and Recommendations

Overall Assessment: Strong PR with high code quality, comprehensive tests, and sound architecture. The refactoring from Reranker to Intent is well-executed.

Must address before merge:

  1. Add braces (with each brace on its own line) to single-line if statements in src/intent.ts (lines 616, 643, 677, 789)
  2. Verify brace formatting in src/providers/groq.ts matches the multi-line style shown in CLAUDE.md examples

After addressing the bracing issue, this PR will be ready to merge. The bracing fix is straightforward and mechanical - no design changes needed.


Final Checklist Against Requirements

  • No functions over 50 lines (all under 30!)
  • All if statements use braces ← Must fix
  • All functions have JSDoc documentation
  • No any types without justification
  • No hardcoded values that should be config
  • No commented-out code
  • No console.log statements
  • Type checking passes
  • Linting passes
  • Tests pass with 100% coverage
  • Build succeeds

Score: 11/12 - Excellent work! Just needs the bracing fix to meet 100% of the coding standards.


@stevekrenzel
stevekrenzel merged commit 22d3edc into main Dec 22, 2025
4 checks passed
@stevekrenzel
stevekrenzel deleted the ergonomics branch December 22, 2025 09:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant