feat: tajweed error detection system (closes #207) - #306
Open
atejuolabello-beep wants to merge 24 commits into
Open
feat: tajweed error detection system (closes #207)#306atejuolabello-beep wants to merge 24 commits into
atejuolabello-beep wants to merge 24 commits into
Conversation
Corrected phrasing in the contributing section regarding participation in the Stellar Drips Wave bounty program.
…idge#271) * Add two-tier cache with scope isolation and token quota tracker * Integrate two-tier cache and rate limiting into chat endpoints * Add default gemini_api_key for test compatibility * Fix datetime UTC import for Python 3.10 compatibility * Add unit tests for scope isolation, exact cache, and token quota tracker * Add integration tests for quota enforcement and oversize rejection * Fix ruff linting errors * Fix Optional type annotation for Python 3.10 compatibility * Fix formatting issues with ruff format * Fix remaining ruff linting errors * Ignore UP042 and UP017 for Python 3.10 compatibility * Fix mypy type annotations and ignore store.py errors
… detection (Deen-Bridge#283) Implements self-consistency sampling to detect hallucinations by generating multiple candidate answers and measuring their agreement. Features: - Self-consistency sampler with configurable sample count - Claim extraction from answers (factual, religious, numerical, citation) - Agreement scoring across sampled answers - Response policy for low-agreement answers (warnings, review flags) - Latency optimization with early exit for clear cases - Session-level caching for results - Integration helper for confidence.py The module produces a `self_consistency` score that flows into build_signals() alongside citation_verification and expressed_certainty. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Sakariyah Abdulhazeem <150973162+zeemscript@users.noreply.github.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
…een-Bridge#285) Add a deterministic, dependency-free sentiment analyzer that reads the emotional and spiritual dimension of a user's question so replies can be tuned in tone without ever altering the underlying religious facts. Approach - Pure stdlib + pydantic/fastapi. Curated keyword/phrase lexicons plus simple weighted, negation-aware rules. Fully deterministic and fast, with no ML dependency, no training data, and no import-time side effects. - Produces three normalized dimensions (emotional, spiritual, informational), boolean flags, a primary-intent classification, a single recommended response tone, and an optional empathetic response prefix. Acceptance-criteria coverage - Emotional distress and spiritual-crisis detection; doubt/uncertainty in faith; comfort-seeking vs information-seeking; urgency in personal fiqh; learning enthusiasm vs obligation; cultural-sensitivity and new-Muslim indicators; seeking validation vs guidance. - Pastoral-care / crisis-referral indicators fire when self-harm or suicidal language crosses a conservative threshold; the referral note points toward qualified human help rather than an automated answer. API - router = APIRouter(prefix="/sentiment"); POST /sentiment/analyze returns the full multi-dimensional analysis + tone + care indicators, and GET /sentiment/taxonomy returns the emotion taxonomy. - Wired into main.py; offline tests in tests/test_sentiment.py and a CI step.
…Deen-Bridge#286) Add a self-contained, dependency-free model routing layer that picks the optimal Gemini tier per query on a multi-criteria basis (accuracy need, latency SLA, cost budget) — deterministic and well under 10ms, no trained weights and no live-service calls at import. model_router.py provides: - Query classification: heuristic feature extraction (token estimate, Arabic-script detection, domain keywords for fiqh/tafsir/hadith/zakat/..., a 0-1 complexity score) into a QueryFeatures model. - Model profile registry (gemini-fast/balanced/pro) with cost, latency, accuracy weight and an availability flag. - Multi-criteria scoring engine that reallocates weight toward accuracy as complexity rises; never routes to an unavailable model and raises when none satisfy the constraints. - Fallback ordering (remaining candidates by score). - Named A/B strategies with deterministic query-hash bucketing. - In-memory decision log + aggregated metrics getter. - record_feedback learning hook adjusting a clamped per-model quality bias that shifts future selection. - APIRouter(prefix="/routing"): POST /route, GET /models, GET /metrics, POST /feedback with pydantic request/response models. Wire the router into main.py and add tests/test_model_router.py (classification buckets, unavailable-skip, fallback order, cost-budget, feedback flip, metrics) to the CI allow-list. ruff/ruff-format/mypy clean on the new module. Closes Deen-Bridge#198
…ge#222) (Deen-Bridge#287) Add a deterministic, dependency-free page-structure analyzer for scanned Islamic book pages. It operates on the structured output of an OCR/layout engine — a list of positioned text blocks — rather than raw pixels, so it needs no CV/ML dependency and runs in the existing FastAPI process. New module page_analysis.py provides: - Pydantic input models PageBlock (bbox + text + optional font_size) and PageInput (page dimensions, blocks, optional RTL flag for Arabic). - Element classification (heading / body-text / footnote / commentary / reference-citation / decorative) via relative font size, vertical position, centering, and text-pattern heuristics. - Reading-order determination with multi-column detection (x-position clustering) and right-to-left column ordering for Arabic script. - Reference extraction for Qur'an surah:ayah and hadith-source citations, linked back to their source block. - Semantic structuring that groups classified blocks into ordered regions by role (heading, main text, commentary, footnotes, references). - APIRouter at /page-analysis with POST /analyze and GET /element-types. Wired into main.py (import + include_router after the tafsir router). Adds tests/test_page_analysis.py (14 offline tests, module-only imports) and a dedicated CI step running them. No new project dependencies. Closes Deen-Bridge#222 Co-authored-by: Sakariyah Abdulhazeem <150973162+zeemscript@users.noreply.github.com>
…scholar eras, fiqh timelines) (Deen-Bridge#288) Add a curated, offline historical-context module that detects verse references, scholar names, and topic keywords in a question or drafted answer and returns the relevant historical records plus a compact block the chat layer can inject. Covers the issue's acceptance criteria functionally: - asbab al-nuzul for relevant Quranic verses - hadith narration circumstances - historical development of fiqh rulings with staged timelines - time-period (hijri/gregorian/century) for classical scholars cited - distinction between time-bound and universal rulings The module is pure and dependency-free (fastapi + pydantic + stdlib) with no import-time side effects. Wired into main.py via a new APIRouter and covered by tests/test_history.py (added to CI). Co-authored-by: Sakariyah Abdulhazeem <150973162+zeemscript@users.noreply.github.com>
…on (Deen-Bridge#290) Add a deterministic, dependency-free module that verifies transcribed audio Hadith narrations against a bundled corpus of authenticated texts. - POST /audio-hadith/verify matches a transcript against known narrations with blended Jaccard/containment scoring and confidence output - Arabic diacritic stripping plus transliteration/synonym folding so ASR output lines up with the corpus - Isnad extraction with narrator identification and short biographies - Authenticity grade (sahih/hasan/daif) surfaced from the matched hadith - Misquotation flagging when wording drifts from the authenticated text - Pluggable Transcriber interface (no heavy ASR dependency bundled) Wire the router into main.py and add a dedicated pytest step to CI. Co-authored-by: Sakariyah Abdulhazeem <150973162+zeemscript@users.noreply.github.com>
…een-Bridge#289) Adds a deterministic, dependency-free contextual interpretation layer on top of the authenticity-grading hadith module. A curated offline knowledge base annotates well-known narrations with the context a classical sharh supplies: occasion of narration (asbab al-wurud), attributed scholarly commentary, complementary/qualifying narrations, practical application, madhab-specific readings, apparent contradictions with reconciliation, key points, and contemporary framing. Exposes an APIRouter at /hadith-context: - GET /hadith-context/list canonical references covered - GET /hadith-context/interpret madhab-aware synthesized interpretation - POST /hadith-context/ask question answering routed to the relevant facet (history, application, contradiction, contemporary, or general interpretation) The madhab lens reorders commentary so the requested school leads without dropping any reading (ikhtilaf preserved). Every interpretive claim is attributed to the named work it is drawn from; no live model is involved. Wires the router into main.py and adds tests/test_hadith_context.py plus a CI step. New root module + own tests keep the change conflict-free. Co-authored-by: Sakariyah Abdulhazeem <150973162+zeemscript@users.noreply.github.com>
…les (Deen-Bridge#282) Co-authored-by: Unclebaffa <alhassannuhu0.com> Co-authored-by: Sakariyah Abdulhazeem <150973162+zeemscript@users.noreply.github.com>
…tion harness (Deen-Bridge#284) Co-authored-by: Unclebaffa <alhassannuhu0.com> Co-authored-by: Sakariyah Abdulhazeem <150973162+zeemscript@users.noreply.github.com>
…rofiler (Deen-Bridge#291) Add a dependency-free, deterministic query optimization module that delivers the functional core of issue Deen-Bridge#217 without a live database: - Static analysis: flags SELECT *, full-table scans, leading-wildcard LIKE, functions on indexed columns, implicit joins, OR chains, negation filters, and unbounded/unsorted results, each with a severity and remediation, plus B-tree index recommendations for WHERE / JOIN / ORDER BY columns. - Runtime profiling: P50/P95/P99 latency, slow-query log against a 500ms threshold, throughput, cache-hit rate, connection-pool efficiency, and N+1 detection via literal-stripped fingerprints. Exposed under /db-optimizer and wired into main.py. Side-effect-free at import so the app still boots in CI. Adds 29 offline tests and a CI step to run them.
Enable users to adjust answer granularity from brief summaries to comprehensive scholarly analyses with full evidence and reasoning. Components: - DepthLevel: Four levels (brief, standard, detailed, scholarly) - DepthConfig: Configuration settings for each level - DepthAdapter: Adapts responses based on depth settings - UserPreferencesStore: Stores user preferences - StructuredAnswer: Hierarchical answer with progressive disclosure Features: - Dynamic terminology density adjustment - Scaled citation frequency by level - Arabic text and transliteration toggles - Scholarly disagreements (ikhtilaf) inclusion - Historical context and madhhab comparisons - Collapsible sections for detailed content - Per-topic level preferences - Answer compression and expansion Fixes Deen-Bridge#221 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Sakariyah Abdulhazeem <150973162+zeemscript@users.noreply.github.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Implement a comprehensive system for organizing and accessing Quranic content by themes with multi-tiered classification based on Islamic scholarly traditions. Components: - ThematicTaxonomy: Multi-tiered theme classification (main, sub, micro) - ThemeVerseStore: Verse-to-theme associations with annotations - ThematicRetriever: API for querying and navigating themes Features: - Theme co-occurrence analysis - Chronological tracking (Meccan vs Medinan) - Comparative theme analysis - Scholarly definitions and explanations - Thematic summaries generation - Includes comprehensive test suite Fixes Deen-Bridge#144 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Sakariyah Abdulhazeem <150973162+zeemscript@users.noreply.github.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Optimize vector database performance for faster semantic search, better retrieval accuracy, and efficient scaling. Components: - VectorStore: Abstract base with InMemory implementation - IndexConfig: Configuration for HNSW, IVF, PQ parameters - QueryCache: Caching layer for frequent queries - HybridRetriever: Combined dense-sparse (BM25) retrieval - VectorStoreBenchmark: Latency, recall, and throughput testing - ABTestFramework: A/B testing for retrieval configurations Features: - Multi-database support architecture (Pinecone, Qdrant, pgvector) - Index tuning with configurable HNSW/IVF parameters - Distance metric optimization (cosine, euclidean, dot product) - Hybrid dense-sparse retrieval with configurable weights - Query caching with TTL and size limits - Benchmark suite for P95 latency, recall@k, NDCG, QPS Fixes Deen-Bridge#220 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Sakariyah Abdulhazeem <150973162+zeemscript@users.noreply.github.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Implement a robust queue system for managing heavy computational operations, batch processing, and long-running tasks. Components: - JobQueue: Main queue manager with worker pool - Job: Job representation with status, priority, progress - JobStore: Persistence layer (InMemory, Redis backends) - JobHandler: Abstract base for job handlers Features: - Priority-based queue management (LOW, NORMAL, HIGH, CRITICAL) - Job scheduling (immediate, delayed, with dependencies) - Real-time progress monitoring with callbacks - Exponential backoff retry with configurable limits - Dead letter queue for permanently failed jobs - Resource constraints per job type - Example handlers for embedding generation and index building - Includes comprehensive test suite Fixes Deen-Bridge#215 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Sakariyah Abdulhazeem <150973162+zeemscript@users.noreply.github.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
…ce (Deen-Bridge#296) Adds a calligraphy.py router that estimates the calligraphic hand (Kufic, Naskh, Thuluth, Diwani, Ruq'ah, Muhaqqaq, Nasta'liq) of a specimen from a vector of measurable style signals (angularity, curvature, stroke contrast, diacritic density, geometric regularity, elongation, slant, letter stacking). This is deliberately a transparent, deterministic RULE-BASED estimate — not a trained vision/OCR model and no heavy ML/image dependency. Each hand carries a hand-encoded ideal trait vector; classification scores an input by RMS distance to those ideals, normalizes to confidences summing to 1, and flags ambiguous specimens where the top two hands sit within a small margin. The catalog is both the documentation and the model — nothing is hidden. Endpoints: GET /calligraphy/styles, GET /calligraphy/styles/{style}, POST /calligraphy/classify, POST /calligraphy/analyze (adds legibility/ embellishment bands and a downstream OCR reconstruction hint). Pure stdlib + the existing fastapi/pydantic; no import-time side effects. Wires the router into main.py and adds a CI pytest step. 19 tests in tests/test_calligraphy.py.
…#152) (Deen-Bridge#302) Introduce a self-contained reasoning-chain engine (reasoning_chains.py) that decomposes a complex Islamic question into ordered, evidence-tagged reasoning steps and lets a user inspect, validate and question the chain. The engine is deterministic and dependency-free (stdlib + pydantic/fastapi only): no live model call and no import-time side effects, so it runs in tests and CI without an API key or network. A model can later fill richer conclusions into the same shapes. Acceptance criteria covered: - Decomposition of compound questions into ordered steps by conjunction and fiqh/tafsir/hadith facet detection. - Each step carries an intermediate conclusion, evidence-source references (Qur'an/hadith/fiqh/tafsir category tags — never a fabricated citation), a logical connector to the next step, a confidence score and a stable id. - Consistency validation flags contradictory conclusions; weak-point detection surfaces the lowest-confidence and unsupported steps. - Branching produces parallel reasoning paths for madhhab differences. - Structured dict plus markdown outline rendering with addressable step ids. Exposes router = APIRouter(prefix="/reasoning") with POST /reasoning/chain, GET /reasoning/templates and POST /reasoning/validate, wired into main.py. Adds tests/test_reasoning_chains.py and a CI step to run them.
…Bridge#303) - Calligraphy style detection (Naskh, Thuluth, Nastaliq, etc.) - Image preprocessing for aged/damaged manuscripts - Arabic-specific text normalization and diacritic handling - Multi-level confidence scoring (character/word/diacritic) - Support for multiple OCR backends (Google Vision, Azure, Tesseract) - Post-processing for common Arabic OCR errors - Comprehensive test suite 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Sakariyah Abdulhazeem <150973162+zeemscript@users.noreply.github.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
…dge#218) (Deen-Bridge#301) Add a self-contained, dependency-free reformulation engine that assesses question quality and proposes better-formed rewrites, entirely offline (no generative model call, no network, no import-time side effects). Criteria coverage: - Quality assessment: flags too-short, subject-less, ambiguous-pronoun, compound, missing-madhhab, and missing-scope questions; scores 0-1 with human-readable reasons. - Reformulation options: multiple ranked rewrites, each with an explanation of why it improves the question -- precise phrasing, added context, splitting compound questions, standard Islamic terminology, specifying a madhhab for fiqh questions, and adding scope/constraints. - Example library of well-formed questions by category (aqidah/fiqh/tafsir/ hadith/history). - Exposes router = APIRouter(prefix="/reformulation") with POST /suggest and GET /examples?category=, plus pydantic request/response models. Wire the router into main.py and add tests plus a CI step. Co-authored-by: Sakariyah Abdulhazeem <150973162+zeemscript@users.noreply.github.com>
…#300) Add an "Authentication & Rate Limiting" section to the README describing the service's actual slowapi configuration and the two additional in-process limits. - Request rate: 60 requests / 60s per client on /chat and /chat/stream (CHAT_RATE_LIMIT_MAX / CHAT_RATE_LIMIT_WINDOW_SECONDS), keyed by X-API-Key when present else X-Forwarded-For / remote address. - Hourly token quota: 100000 tokens/hour on /chat (CHAT_TOKEN_QUOTA_PER_HOUR). - Feedback: 20 requests / 60s per IP on /feedback. The Limiter is constructed without headers_enabled, so X-RateLimit-* headers are NOT emitted; documented that explicitly. Only Retry-After is present, on the /chat and /chat/stream 429s and the token-quota 429 (not on the /feedback 429). Includes the real 429 body shapes and a client requests example doing Retry-After + exponential backoff. This also resolves the pre-existing #authentication--rate-limiting anchor linked from the API table. Docs only; no code changes. Co-authored-by: Sakariyah Abdulhazeem <150973162+zeemscript@users.noreply.github.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
Strict review blocker: this branch conflicts with the base branch. Please rebase and resolve conflicts before requesting merge. |
1 similar comment
Contributor
|
Strict review blocker: this branch conflicts with the base branch. Please rebase and resolve conflicts before requesting merge. |
Contributor
|
Strict review blocker: this branch conflicts with the base branch and/or changes have been requested. Please rebase, resolve conflicts, and address requested changes before requesting merge. |
Contributor
|
@atejuolabello-beep this PR has merge conflicts with the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
12 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rule-based Tajweed error detection: ghunnah, idghaam, ikhfa, iqlab, qalqalah, madd duration, makharij. Educational feedback with corrective exercises. Progressive difficulty levels.
Endpoints
Config
ENABLE_TAJWEED_DETECTION, TAJWEED_MIN_SCORE, TAJWEED_LEVEL
Test plan
Closes #207