Remove vendored shims and harden deployment pipeline

.env.example

@@ -2,8 +2,10 @@
 ENABLE_PUBLIC_UI=false
 ENABLE_RBAC=false
 ALLOW_CIDR=false
-ADMIN_USER=admin
-ADMIN_PASSWORD=changeme
+# For local demos; set to false in production once you configure real secrets.
+ALLOW_DEV_SECRETS=true
+ADMIN_USER=changeme_admin
+ADMIN_PASSWORD=changeme_password
 CONSENT_PUBLIC_KEY_PATH=keys/dev_ed25519.pub
 REPORT_SIGNING_KEY_PATH=keys/dev_ed25519.priv
 FLASK_SECRET_KEY_FILE=keys/dev_flask_secret.key
@@ -15,3 +17,5 @@ CONNECT_TIMEOUT=5
 READ_TIMEOUT=10
 LOG_MAX_BYTES=10485760
 LOG_BACKUP_COUNT=5
+# Optional bootstrap when running start.py (defaults to disabled for offline use)
+# RECONSCRIPT_BOOTSTRAP=false

.github/workflows/ci-matrix.yml

@@ -23,24 +23,34 @@ jobs:
         with:
           python-version: ${{ matrix.python-version }}
           cache: "pip"
+          cache-dependency-path: |
+            requirements.txt
+            requirements-dev.txt
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install -r requirements.txt
           pip install -r requirements-dev.txt
-      - name: Lint
+          pip install detect-secrets trufflehog
+      - name: Format & Lint
         run: |
-          ruff check .
           black --check .
-      - name: Run unit tests
-        run: pytest -m "not integration" --maxfail=1 --ff
-      - name: Upload coverage data
-        if: always()
-        uses: actions/upload-artifact@v4
+          ruff check .
+      - name: Security checks
+        run: |
+          bandit -r reconscript
+          pip-audit --requirement requirements.txt
+      - name: Run unit tests with coverage
+        run: pytest -m "not integration" --maxfail=1 --ff --cov=reconscript --cov-report=xml --cov-report=term
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v4
         with:
-          name: coverage-${{ matrix.python-version }}
-          path: ./.coverage*
-          if-no-files-found: ignore
+          files: coverage.xml
+          fail_ci_if_error: true
+          verbose: true
+      - name: Secret scanning
+        run: |
+          detect-secrets scan --all-files
+          trufflehog filesystem --no-update --fail .
 
   integration:
     runs-on: ubuntu-latest

.gitignore

@@ -207,8 +207,6 @@ marimo/_lsp/
 __marimo__/
 
 # ReconScript artifacts
-results/
-
 
 # Binary assets (keep placeholders only)
 *.png
@@ -220,3 +218,10 @@ results/
 *.tar
 *.gz
 *.mp4
+# reconscript: generated reports
+/results/*
+!/results/.gitkeep
+
+# Local wheel cache for offline Docker builds
+/wheelhouse/*
+!/wheelhouse/.gitkeep

Dockerfile

@@ -2,13 +2,18 @@
 
 FROM python:3.12-slim AS builder
 
+ARG WHEELHOUSE=wheelhouse
+
 ENV PYTHONDONTWRITEBYTECODE=1 \
-    PYTHONUNBUFFERED=1
+    PYTHONUNBUFFERED=1 \
+    PIP_NO_CACHE_DIR=1 \
+    PIP_FIND_LINKS=/opt/wheelhouse
 
 WORKDIR /app
 
 ARG INCLUDE_DEV_KEYS=false
 
+COPY ${WHEELHOUSE}/ /opt/wheelhouse/
 COPY requirements.txt ./
 
 RUN python -m venv /opt/venv \
@@ -17,6 +22,8 @@ RUN python -m venv /opt/venv \
 
 FROM python:3.12-slim AS runtime
 
+ARG WHEELHOUSE=wheelhouse
+
 ENV PYTHONDONTWRITEBYTECODE=1 \
     PYTHONUNBUFFERED=1 \
     PATH="/opt/venv/bin:${PATH}"
@@ -39,6 +46,7 @@ RUN apt-get update \
     && rm -rf /var/lib/apt/lists/* \
     && adduser --disabled-password --gecos "" reconscript
 
+COPY --from=builder /opt/wheelhouse /opt/wheelhouse
 COPY --from=builder /opt/venv /opt/venv
 COPY . .
 

README.md

@@ -47,9 +47,12 @@ export ADMIN_USER=security-admin
 export ADMIN_PASSWORD='replace-with-strong-passphrase'
 export CONSENT_PUBLIC_KEY_PATH=/secure/path/consent_ed25519.pub
 export REPORT_SIGNING_KEY_PATH=/secure/path/report_ed25519.priv
-python start.py
+# Optional: ask the launcher to install Python requirements before starting
+RECONSCRIPT_BOOTSTRAP=1 python start.py
 ```
-The launcher checks dependencies, loads environment variables from `.env` if present, and starts the Flask server on <http://127.0.0.1:5000>. Use `start.sh`, `start.bat`, or `start.ps1` for platform-specific wrappers. Set `ALLOW_DEV_SECRETS=true` only for local demos that intentionally reuse the sample keys in `keys/`.
+Copy `.env.example` to `.env` for local development; it enables `ALLOW_DEV_SECRETS=true`, sets placeholder credentials (`changeme_admin` / `changeme_password`), and points to the developer keys under `keys/`. **Always replace these values before deploying to shared environments.**
+
+The launcher loads environment variables from `.env` if present and starts the Flask server on <http://127.0.0.1:5000>. Use `start.sh`, `start.bat`, or `start.ps1` for platform-specific wrappers. Leave `RECONSCRIPT_BOOTSTRAP` unset (or `false`) when running in offline environments that already have dependencies installed.
 
 ### Run a CLI Scan
 ```bash
@@ -66,6 +69,21 @@ docker compose up --build
 
 Mount the `results/` directory when running containers so generated artefacts persist outside the container lifecycle. Override the required secrets via environment variables or secrets managers at runtime; the container image omits the developer keys unless built with `--build-arg INCLUDE_DEV_KEYS=true`.
 
+### Offline / Air-gapped builds
+
+Prepare wheels on a connected machine and bake them into the Docker image for offline deployments:
+
+```bash
+# On a connected host
+python -m pip install --upgrade pip
+python -m pip wheel -r requirements.txt -w wheelhouse
+
+# Build using the cached wheels
+docker build --build-arg WHEELHOUSE=wheelhouse -t reconscript-offline .
+```
+
+The Dockerfile automatically points `pip` at `/opt/wheelhouse` via `PIP_FIND_LINKS`. Place custom wheels inside the local `wheelhouse/` directory (a `.gitkeep` placeholder is tracked so the folder is present in the build context).
+
 ### Observability
 ReconScript exposes Prometheus-compatible metrics at `/metrics` and a readiness probe at `/healthz`. Scrape the metrics endpoint to monitor scan durations, completion counts, and open-port histograms.
 

ROADMAP.md

@@ -5,6 +5,9 @@
 - [x] Consolidate GitHub Actions into a cached matrix workflow covering Ruff, Black, and pytest.
 - [x] Repair Markdown exporter fallback logic and add regression coverage for CLI reporting.
 - [ ] Publish an onboarding checklist that walks operators through secret provisioning and CI expectations.
+- [x] Remove vendored protocol shims in favour of upstream `requests` and `PyNaCl` packages.
+- [ ] Enforce automated secret scanning (detect-secrets + trufflehog) in CI pipelines.
+- [ ] Keep end-to-end coverage reports at or above 85% with explicit gating in CI.
 
 ## 60-Day Objectives
 - [ ] Automate signing-key rotation with documentation for Vault/Secrets Manager integrations.

docker-compose.yml

@@ -7,5 +7,29 @@ services:
       - "5000:5000"
     volumes:
       - ./results:/app/results
+    env_file:
+      - .env
     environment:
-      - SERVER_NAME=127.0.0.1:5000
+      SERVER_NAME: ${SERVER_NAME:-127.0.0.1:5000}
+      ADMIN_USER: ${ADMIN_USER:-changeme_admin}
+      ADMIN_PASSWORD: ${ADMIN_PASSWORD:-changeme_password}
+      FLASK_SECRET_KEY_FILE: ${FLASK_SECRET_KEY_FILE:-/run/secrets/flask_secret}
+      CONSENT_PUBLIC_KEY_PATH: ${CONSENT_PUBLIC_KEY_PATH:-/run/secrets/consent_key}
+      REPORT_SIGNING_KEY_PATH: ${REPORT_SIGNING_KEY_PATH:-/run/secrets/report_key}
+    secrets:
+      - flask_secret
+      - consent_key
+      - report_key
+    healthcheck:
+      test: ["CMD", "curl", "-fsS", "http://localhost:5000/healthz"]
+      interval: 15s
+      timeout: 5s
+      retries: 10
+
+secrets:
+  flask_secret:
+    file: ./keys/dev_flask_secret.key
+  consent_key:
+    file: ./keys/dev_ed25519.pub
+  report_key:
+    file: ./keys/dev_ed25519.priv

docs/2025-01-15-audit.md

@@ -0,0 +1,45 @@
+# ReconScript Repository Audit (2025-01-15)
+
+## Overview
+This document captures the findings from a production-readiness, security, and deployment audit of ReconScript. Each section maps to the requested checklist and records status, evidence, and remediation guidance.
+
+## 1. Repository Structure & Hygiene
+- ⚠️ **Tracked artefacts** – Generated scan outputs remain committed under `results/` even though the directory is listed in `.gitignore`, which increases the risk of leaking assessment data and bloats the repo. Remove existing files and keep the directory empty. Apply a placeholder README if needed.  
+  Evidence: `results/juice.json`, `results/127-0-0-1-20251016-032315.html`.
+- ⚠️ **Orphan vendor shims** – Stub packages for `requests` and `nacl` ship in-tree, but production builds should rely on the pinned third-party libraries instead. Their presence causes the source tree to shadow real dependencies when the project is executed from a clone, diverging from packaged behaviour.
+
+## 2. Dependency Integrity
+- ❌ **Package shadowing and duplication** – The source tree vendors its own `requests` and `nacl` modules while `requirements.txt`/`pyproject.toml` pin `requests` and `PyNaCl`. This leads to inconsistent behaviour between editable installs and built wheels, and places cryptography outside of vetted libraries. Remove the stubs or move them behind feature flags/tests.  
+  Evidence: `requests/__init__.py`, `requests/api.py`, `nacl/signing.py`.
+- ⚠️ **Inconsistent locking** – `requirements.lock` is manually curated and pins versions (e.g., `requests==2.31.0`) that no longer match `requirements.txt` (`requests==2.32.3`), risking downgrade attacks or resolver churn during CI/CD. Align the lock file by regenerating it with the current dependency set.
+- ⚠️ **Missing runtime dependency** – `prometheus-client` is required by `reconscript.metrics` but absent from `pyproject.toml`. Add it to the `dependencies` array to keep the packaged project functional.
+- ⚠️ **Over-eager bootstrapper** – `install_dependencies.py` attempts to install optional libraries (tabulate, colorama, WeasyPrint stack) regardless of configuration, which fails in restricted networks and slows startup. Restrict installation to the actual runtime requirements or gate extras behind feature flags.
+
+## 3. Build, Test, & Run Validation
+- ❌ **Formatting and lint debt** – `black --check` flagged 22 files and `ruff` reported 272 issues, including style problems and security warnings (e.g., `S104` for binding to `0.0.0.0`). Adopt the configured formatters and linters to restore signal in CI.
+- ⚠️ **Security tooling unavailable** – `bandit` and `pip-audit` were not runnable in the provided environment due to missing wheels. Ensure these tools are vendored or run inside CI where the network is available, then track their reports.
+- ✅ **Unit tests** – `pytest --maxfail=1 --disable-warnings -q` passed (13 tests, 1 skipped) once dependencies already present in the base image were used. No coverage data was produced; configure `pytest-cov` and enforce ≥85% if required.
+- ⚠️ **Launcher dependency install** – `start.py` re-invokes `pip install` at runtime and blocks when the environment lacks internet access. Provide an offline mode or documented skip flag.
+
+## 4. Environment & Configuration
+- ⚠️ **Default secrets conflict** – `.env.example` ships with `ALLOW_DEV_SECRETS=false` while simultaneously providing placeholder admin credentials. This prevents the UI from starting without manual edits. Either set `ALLOW_DEV_SECRETS=true` for the template or replace the credentials with explicit TODO markers.
+- ✅ **Secret path enforcement** – `reconscript.ui` and `reconscript.consent` refuse to use developer keys unless `ALLOW_DEV_SECRETS=true`, preventing accidental deployment of demo material.
+
+## 5. Integration & Connectivity
+- ⚠️ **Docker Compose omissions** – `docker-compose.yml` exposes the UI but omits required environment variables (ADMIN_USER/PASSWORD, secret paths). Compose spins up an unusable container by default. Add `.env` support or documented overrides.
+- ✅ **HTTP endpoints** – `/healthz` and `/metrics` are implemented in the Flask app; CLI entry points respond to `--help` and dry-run flows succeed in tests.
+
+## 6. Security & Compliance
+- ❌ **Custom cryptography** – The vendored `nacl` package re-implements Ed25519 signing and verification. This should be replaced with the audited `PyNaCl` dependency to avoid logic bugs or timing attacks.
+- ⚠️ **Secret scanning** – `trufflehog`/`detect-secrets` were unavailable locally. CI should run a secret scan to ensure new commits do not leak credentials.
+
+## 7. Deployment Readiness & Observability
+- ⚠️ **CI gaps** – `.github/workflows/ci-matrix.yml` uploads coverage artefacts but never generates coverage data, so uploads are empty. Update pytest invocation to include `--cov` options. Review caching logic for the duplicated `pip install` commands (runtime and dev requirements).
+- ⚠️ **Docker reproducibility** – Offline builds will fail because the Dockerfile installs requirements from PyPI at build time. Mirror wheels or provide instructions for air-gapped deployments. Compose also lacks health/ready checks for dependencies beyond the Flask app.
+
+## 8. Recommended Remediations
+1. Remove the in-repo `requests/` and `nacl/` packages in favour of official dependencies; update tests to rely on real libraries.
+2. Regenerate dependency manifests (`requirements.lock`, `pyproject.toml`) and restrict `install_dependencies.py` to the runtime set.
+3. Run `black`, `ruff --fix`, and re-enable security tooling in CI/CD. Configure coverage reporting.
+4. Clean committed artefacts under `results/` and tighten `.gitignore`/`.dockerignore` to prevent regressions.
+5. Extend Docker Compose/Dockerfile docs to include required environment variables and offline build instructions.