feat(backend): thin FastAPI wrapper around the ingen CLI

[maan-iitd2]

What

Adds backend/app/ — a thin, local-dev-only FastAPI service that runs the InGen CLI and returns structured results. It contains no business logic: it shells out to python -m ingen, parses the CLI's output, and hands JSON back to a frontend (the InGen Studio SPA, not part of this PR).

Module	Responsibility
main.py	HTTP routing only — parses requests, delegates, returns JSON
runner.py	Writes posted YAML to a temp file, runs the CLI as a subprocess, parses real log output into a structured RunRecord (per-stage status, timing, log level)
store.py	Persists RunRecords to INGEN_RUNS_DIR for history/lookup
schema_validate.py / validation.py	Validate a config (and surface a run's validation results) without executing it
files.py	Sample-data upload with size/type caps, reads only file headers for column preview (not the whole file)
columns.py	Derives the column list a configured data source would produce

Deliberately excluded from this PR: the inChat assistant (backend/app/chat.py), which pulls in torch/transformers/HuggingFace and is a separable concern with its own dependency weight — follow-up PR. main.py in this PR has no chat import, no /api/chat* routes.

Trust boundary

This wrapper executes whatever config is posted (file reads, SQL, API calls — that's InGen's nature). It's intended for local/trusted development only — no auth, sandboxing, or network egress controls, by design.

Testing

• pytest backend/tests/ — 42 passed.
• Verified backend.app.main imports cleanly and registers all 9 routes with torch/transformers/accelerate/safetensors/tokenizers/huggingface_hub deliberately blocked at import time (via sys.meta_path), proving this PR carries no import-time dependency on the inChat stack even though those packages happened to be present in the test environment.
• End-to-end: booted uvicorn backend.app.main:app --port 8123, then:
  • GET /api/health → {"status": "ok"}
  • POST /api/configs/validate with a real config → validated correctly
  • POST /api/runs with a real XML-source config → the CLI subprocess ran the full read → pre_process → format → validate → write pipeline and the endpoint returned a RunRecord with "status": "success" and per-stage logs.
• backend/README.md had drifted from main.py (missing the files/columns endpoints, wrong default CORS ports/origin). Corrected it to describe exactly what this PR ships.

What it looks like

The full API surface this PR ships, at /docs once uvicorn backend.app.main:app is running:

Nine endpoints, no /api/chat* — confirming the inChat stack is genuinely absent from this PR, not just unimported.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Run and validation results can be inaccurate, while preview and upload paths can perform unbounded work.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a local FastAPI wrapper around the InGen CLI, including execution, validation, uploads, source previews, and persisted run history.

Changes:

• Adds nine backend API endpoints and supporting services.
• Parses CLI output into structured run records.
• Adds backend dependencies, documentation, and unit tests.

File summaries

File	Description
backend/app/__init__.py	Defines the backend package.
backend/app/main.py	Implements HTTP routes and CORS.
backend/app/runner.py	Executes and parses CLI runs.
backend/app/validation.py	Derives validation reports.
backend/app/schema_validate.py	Validates YAML configuration structure.
backend/app/store.py	Persists run records.
backend/app/files.py	Handles uploads and previews.
backend/app/columns.py	Retrieves source columns.
backend/README.md	Documents setup and endpoints.
backend/requirements.txt	Declares backend dependencies.
backend/tests/__init__.py	Defines the test package.
backend/tests/test_runner.py	Tests execution result parsing.
backend/tests/test_validation.py	Tests validation derivation.
backend/tests/test_schema_validate.py	Tests schema checks.
backend/tests/test_store.py	Tests run persistence.
backend/tests/test_columns.py	Tests file-source columns.

Review details

Suppressed comments (5)

backend/app/runner.py:76

• A zero exit code does not prove each requested interface ran: the CLI silently filters unknown --interfaces names and also catches per-interface failures without changing the process exit code. Consequently, requesting a nonexistent interface produces no markers here but is returned as a successful run with all stages ok. Treat every requested interface lacking its success marker as failed; this also handles a crash after an earlier interface succeeded.

    # If the process crashed before any per-interface marker, treat all requested as failed.
    if exit_code != 0 and not succeeded and not failed:
        failed = set(requested)

backend/app/runner.py:140

• The furthest stage is selected by _STAGE_MARKERS list order rather than by where markers occur in the log. Real runs validate raw data before pre-processing, so a later pre-processing failure still contains an earlier validation marker and is incorrectly reported as a validation-stage failure. Compare marker positions and choose the latest occurrence.

    reached = "read"
    for stage, rx in _STAGE_MARKERS:
        if rx.search(slice_):
            reached = stage
    return reached

backend/app/columns.py:37

• A json source can never succeed here: SourceFactory constructs JsonSource from its third dynamic_data argument, but this call omits it, so fetch() always raises JSON string is not provided. Add dynamic JSON data to this endpoint and pass it through, or reject this source type before claiming it can return columns.

    params_map = {"run_date": run_date} if run_date else None
    src = SourceFactory().parse_source(source, params_map)
    df = src.fetch()

backend/app/runner.py:62

• This stage order reports post_process before format, but InGen executes formatting first and then post-processing (ingen/generators/base_interface_generator.py:45-50). The returned stage timeline is therefore out of execution order whenever post-processing is configured.

    stages = ["read", "pre_process"]
    if iface_cfg.get("post_processing"):
        stages.append("post_process")
    stages += ["format", "validate", "write"]
    return stages

backend/app/columns.py:38

• For MySQL and API sources, fetch() materializes the complete query/response solely to inspect df.columns (MYSQLReader uses unbounded pd.read_sql). This makes a column-preview request execute potentially huge remote workloads despite the stated preview cap. Use source-specific schema/zero-row requests, or require a bounded preview query rather than calling the normal full-data reader.

    src = SourceFactory().parse_source(source, params_map)
    df = src.fetch()
    return {"columns": [str(c) for c in df.columns]}

• Files reviewed: 16/16 changed files
• Comments generated: 9
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.