Skip to content

Add AI-assisted semantic normalization with provider failover and manual fallbacks#31

Open
Prateek32177 wants to merge 1 commit intomainfrom
codex/update-normalization-framework-with-ai-support
Open

Add AI-assisted semantic normalization with provider failover and manual fallbacks#31
Prateek32177 wants to merge 1 commit intomainfrom
codex/update-normalization-framework-with-ai-support

Conversation

@Prateek32177
Copy link
Contributor

Motivation

  • Normalization across platforms is lossy and brittle because field names vary, so we need a layered approach that preserves raw payloads and lets developers declare the schema they care about.
  • Provide an AI-powered extraction layer to map developer-declared semantic fields to platform payloads while keeping a manual fallback for reliability and migration scenarios.

Description

  • Added semantic types and options in src/types.ts (SemanticNormalizeOptions, SemanticAIOptions, _semantic result shape and per-field meta).
  • Implemented the AI extraction + fallback engine in src/normalization/semantic.ts with provider failover order (groq, cohere, openai, anthropic, google), dynamic module loading, JSON-parsing of AI responses, and confidence threshold handling.
  • Made core normalization async by changing normalizePayload to async in src/normalization/simple.ts, and attached semantic results to normalized payloads under _semantic when normalize.semantic is provided.
  • Updated verification flow in src/index.ts to await normalizePayload(...) so semantic extraction runs inside the verification pipeline.
  • Added a regression test in src/test.ts to verify semantic extraction falls back to manual dot-paths when AI is unavailable/low-confidence, and added documentation and usage examples in README.md (API, provider env vars, example config).

Testing

  • Built the project with npm run build (TypeScript compile) which completed successfully.
  • Ran the test suite with npm test; all tests passed including the new semantic fallback regression test (Semantic fallback: PASSED).

Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Comments