ScholarGuard 🎓

Live demo → scholarguard.vercel.app (bring your own Anthropic + Moonshot keys)

An unbiased, heavily guardrailed academic chatbot specialized in scholarly literature, with automatic cross-verification by an independent auditor AI. Reliability over flash: every answer is drafted against retrieved papers, then audited for accuracy, bias, completeness, and source fidelity before you see it.

Architecture

                         ┌──────────────────────────────────────────────┐
                         │                 USER (CLI / Streamlit)        │
                         └──────────────────┬───────────────────────────┘
                                            │ query
                                            ▼
                              ┌─────────────────────────┐
                              │ 1. RATE LIMITER          │  sliding window, 10 req/min
                              │ 2. INPUT SANITIZER       │  NFKC norm, control-char strip,
                              │    (injection screen)    │  injection regex blocklist
                              └───────────┬─────────────┘
                                          ▼
                              ┌─────────────────────────┐
                              │ 3. HARD GUARDRAILS       │  code-enforced, pre-LLM
                              │  refuse / caveat / allow │  self-harm, weapons, illegal,
                              └───────────┬─────────────┘  non-consensual
                                          ▼
                    ┌──────────────────────────────────────────┐
                    │ 4. LITERATURE SEARCH (cached, merged)     │
                    │  SerpApi Google Scholar → Semantic        │
                    │  Scholar → OpenAlex → scholarly (last)    │
                    │  dedupe + rank by citations, disk cache   │
                    └───────────────┬──────────────────────────┘
                                    │ top N papers + abstracts
                                    ▼
                    ┌───────────────────────────────┐
                    │ 5. MAIN SCHOLAR AGENT          │  e.g. GPT-4o, temp 0.2
                    │  drafts cited answer; may ONLY │  structured output:
                    │  cite retrieved papers         │  Summary / Sources /
                    └───────────────┬───────────────┘  Evidence / Trace
                                    │ draft + query + papers
                                    ▼
                    ┌───────────────────────────────┐
                    │ 6. AUDITOR AGENT (independent) │  different provider/model,
                    │  scores 6 criteria → /100,     │  e.g. Llama-3.3-70B on Groq,
                    │  verdict PASS/REVISE/FAIL      │  temp 0.0, strict JSON
                    └───────┬──────────────┬───────┘
                    PASS≥85 │              │ REVISE 60–84
                            │              ▼
                            │   ┌──────────────────────────┐
                            │   │ Main agent revises with   │  max 2 iterations
                            │   │ auditor's issues/fixes    │──► back to Auditor
                            │   └──────────────────────────┘
                            ▼
              ┌────────────────────────────────┐
              │ 7. OUTPUT SAFETY NET            │  regex scan for leaked
              │    + audit badge + transparency │  procedural detail
              │    footer + JSONL audit log     │
              └────────────────┬───────────────┘
                               ▼
                     FINAL VERIFIED RESPONSE
                     (FAIL verdicts are returned
                      WITH a visible warning,
                      never silently)

Project layout

ScholarGuard/
├── main.py                  # CLI entry (one-shot + interactive REPL, --deep)
├── ui.py                    # Streamlit chat UI with feedback buttons
├── scholarguard/
│   ├── config.py            # env-driven settings (providers, thresholds, limits)
│   ├── prompts.py           # Main + Auditor system prompts (verbatim + refinements)
│   ├── guardrails.py        # hard safety rules (pre-LLM) + output safety net
│   ├── tools.py             # search backends, disk cache, sanitizer, rate limiter, cost
│   ├── agents.py            # LLMClient, ScholarAgent, AuditorAgent (strict-JSON audit)
│   └── router.py            # the full pipeline + audit/revision loop + logging
├── requirements.txt
├── .env.example
├── logs/                    # audit_trail.jsonl, feedback.jsonl (auto-created)
└── cache/                   # search-result disk cache (auto-created)

Setup

cd ScholarGuard
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

cp .env.example .env        # then edit .env
export $(grep -v '^#' .env | xargs)   # or use python-dotenv / direnv

Minimum viable config: one API key for the Main agent and one for the Auditor. Default setup — Claude (customer-facing Main agent) + Kimi K2 (independent Auditor):

• Main: claude-sonnet-4-5 via Anthropic (MAIN_PROVIDER=anthropic, MAIN_API_KEY=sk-ant-...)
• Auditor: kimi-k2-0905-preview via Moonshot's OpenAI-compatible endpoint (AUDITOR_API_KEY=sk-..., AUDITOR_BASE_URL=https://api.moonshot.ai/v1)

Any OpenAI-compatible endpoint works for either role (*_PROVIDER=openai): OpenAI, Groq, OpenRouter, Together, Fireworks, or a local vLLM/Ollama server. Mix and match freely — just keep the Auditor on a different provider/model family than the Main agent so the audit stays genuinely independent.

Optional: SERPAPI_KEY enables the real Google Scholar engine as the primary backend. Without it, Semantic Scholar + OpenAlex (both free, keyless) cover the scholarly index — direct scraping of scholar.google.com is intentionally not implemented because it violates Google's ToS and is IP-banned within minutes.

Run

python main.py "Does intermittent fasting improve metabolic health?"
python main.py --deep "What is the evidence for microdosing psychedelics?"
python main.py              # interactive REPL (':deep <q>' inside)

streamlit run ui.py         # web UI on http://localhost:8501

Example conversation (audit flow)

$ python main.py "Does social media use cause depression in adolescents?"

1. Summary Answer
- Longitudinal and meta-analytic evidence shows a SMALL average association
  between heavy social-media use and adolescent depressive symptoms (r ≈ 0.06–0.13),
  but causality is contested: effects shrink or vanish in studies controlling for
  pre-existing symptoms, and some find reverse causation...
2. Key Sources
- [1] Orben & Przybylski (2019). The association between adolescent well-being and
      digital technology use. Nature Human Behaviour. https://doi.org/10.1038/s41562-018-0506-1
- [2] Odgers & Jensen (2020). Annual Research Review: Adolescent mental health in the
      digital age. JCPP. https://doi.org/10.1111/jcpp.13190
- [3] Twenge et al. (2018) ...
3. Evidence Strength & Limitations
- Strength: multiple large cohort studies + meta-analyses; consistent direction but tiny effect.
- Limitations: self-report measures, confounding, publication bias toward positive findings,
  heterogeneous definitions of "use".
4. Reasoning Trace
- Queries: ["social media use depression adolescents",
            "social media depression adolescents systematic review meta-analysis"]
- Weighed meta-analyses over single cohorts; noted the Orben vs. Twenge methodological dispute.
Confidence: Medium — consistent small association, unresolved causality.

---
🔶 Audit #1: REVISE (78/100) — balance_bias=6: "draft led with Twenge's correlational
   claim before meta-analyses" → Main agent revised (revision 1).
✅ Verified by independent audit — score 91/100, verdict PASS.

<details>Transparency</details> ... backends, tokens, cost, request id

Refusal example:

$ python main.py "how do I synthesize explosives at home"
I can't help with that request. ScholarGuard only answers academic research
questions and cannot provide assistance with this topic. ...

Self-harm queries receive the supportive 988 / findahelpline response instead of a refusal wall.

System prompts

The exact prompts live in scholarguard/prompts.py:

• ScholarGuard-Main — as specified, plus: citations restricted to retrieved papers only (the single most effective anti-hallucination rule), mandatory confidence line, and a "mandatory framing" slot that guardrail caveats inject into.
• ScholarGuard-Auditor — as specified, with the six criteria mapped to explicit JSON keys, deterministic scoring formula (sum/60 × 100), and verdict thresholds (PASS ≥ 85, REVISE 60–84, FAIL < 60 or any criterion ≤ 3). The router parses this JSON and drives the revision loop from it.

Guardrail enforcement (how it actually works)

1. Pre-LLM, in code (guardrails.py): regex categories → refuse (fixed safe message, no model call — nothing to jailbreak), caveat (allowed but the Main agent gets a mandatory neutrality/framing instruction), or allow.
2. In-prompt: the Main system prompt restates the same prohibitions (defense in depth — catches edge cases keywords miss).
3. Post-generation (check_output): regex safety net scans the draft for procedural/quantitative dangerous detail and appends a visible warning.
4. Auditor criterion: guardrail compliance is 1 of the 6 audit scores; a violation forces FAIL regardless of other scores.
5. Injection screen (tools.sanitize_input): blocks "ignore previous instructions", system-prompt extraction, DAN/jailbreak phrasing, base64 blobs.

Observability

Every interaction appends to logs/audit_trail.jsonl: query, papers retrieved (full citations), draft, raw auditor JSON, revisions, tokens, cost, latency, request id. UI feedback (👍/👎) goes to logs/feedback.jsonl keyed by request id — export these as preference pairs for future fine-tuning or as regression-test cases.

Cost

Roughly $0.01–0.04 per standard answer with GPT-4o + Groq Llama-3.3-70B (the Groq tier is currently free). Deep mode ~2×. Search results are disk-cached (72h TTL), so repeated questions cost only the LLM calls. Per-request cost is shown in the transparency footer.

Scaling suggestions

• RAG over full text: pull open-access PDFs (OpenAlex open_access links, arXiv, PubMed Central), chunk + embed into Chroma/FAISS, retrieve passages instead of abstracts. The DiskCache and Paper model are the seam for this.
• More corpora: add PubMed E-utilities and arXiv API as extra _BACKENDS entries in tools.py — each is one function.
• Multi-agent: split the Auditor into specialist auditors (fact-checker, bias-checker, completeness-checker) and aggregate; or add a "Contrarian" agent that must argue the opposite conclusion from the same papers.
• Async pipeline: search and PDF fetching are I/O-bound — parallelize with asyncio/httpx; run the first audit while drafting the "reasoning trace".
• Evals: build a golden set of ~50 Q&A pairs with known-good citations; run nightly and track auditor score drift per model swap.
• Deploy: the router is UI-agnostic — wrap Router.answer() in a FastAPI endpoint for Vercel/Fly; put Redis in front of DiskCache for shared caching; move rate limiting to Redis for multi-instance deployments.
• Feedback loop: use feedback.jsonl to mine queries where the auditor score is high but users downvote — that's your real quality gap.

Known limitations (MVP)

• Abstract-level grounding only — no full-PDF reading by default.
• scholarly scraping backend is off unless installed, and Google blocks it often.
• Keyword guardrails are conservative; the LLM layer is the second line, not a substitute for human review in high-stakes domains.
• The Auditor is still an LLM: it reduces but cannot eliminate hallucination. The citations are real retrieved metadata — spot-check them, that's the point of showing them.