Add read-only JHU dataset access for isolated local development

.github/workflows/ci.yml

@@ -69,7 +69,7 @@ jobs:
       # tests need. numpy is required because constants.py imports it; requests
       # because Authlib's Flask client imports it.
       - name: Install test deps
-        run: python -m pip install --upgrade pip pytest werkzeug SQLAlchemy Flask-SQLAlchemy alembic argon2-cffi Authlib requests flask-cors numpy scipy nibabel openpyxl pillow python-dotenv reportlab websockets scikit-image pandas trimesh
+        run: python -m pip install --upgrade pip pytest SQLAlchemy Flask-SQLAlchemy alembic argon2-cffi Authlib flask-cors numpy scipy nibabel pillow python-dotenv reportlab websockets scikit-image pandas trimesh -r readonly_data/requirements.txt
 
       # Catches syntax errors anywhere in the backend (api_blueprint.py etc.)
       # without installing the heavy runtime deps - nothing else in CI parses
@@ -80,6 +80,9 @@ jobs:
       - name: Path-safety unit tests
         run: python -m pytest tests/unit/test_path_safety.py -v
 
+      - name: Read-only development data boundary and snapshot tests
+        run: python -m pytest tests/unit/test_readonly_data.py -v
+
       # User-dataset admission gatekeeper: quotas, dedup, CT/segmentation validity,
       # promotion into the PanTS-mirroring layout.
       - name: User-dataset gatekeeper unit tests

PanTS-Demo/vite.config.ts

@@ -62,9 +62,9 @@ export default defineConfig({
 		cors: true,
 		proxy: {
 			"/api": {
-				target: env.VITE_API_BASE,
+				target: env.VITE_PROXY_TARGET || env.VITE_API_BASE || "http://127.0.0.1:5001",
 				changeOrigin: true,
-				secure: false,
+				secure: true,
 			},
 			"/ws": {
 				target: env.VITE_WS_BASE || "ws://127.0.0.1:8001",

README.md

@@ -1,5 +1,10 @@
 # Backend
 
+For local frontend/backend development using approved JHU cases, follow
+[read-only development setup](flask-server/deploy/READONLY_DEVELOPMENT.md).
+It uses a restricted data service and a local database; server provisioning is a
+separate reviewed step. Do not point development writes at the production API.
+
 The viewer's AI assistant also needs its model service running. See
 [AI model setup and persistent Ollama service](PanTS-Demo/src/components/AIAssistant/README_AI_MODEL_SETUP.md#hosting-the-bodymaps-ai-models-ollama-on-the-server)
 for installation, restart, and health checks. Restarting Flask alone does not

flask-server/deploy/READONLY_DEVELOPMENT.md

@@ -0,0 +1,194 @@
+# Local development using approved JHU data
+
+This setup copies explicitly approved PanTS cases through a separate read-only
+service. The frontend and normal BodyMaps backend then run on your computer.
+The service does not import the production app, load its `.env`, connect to its
+database, lease inference jobs, generate meshes, or accept uploads.
+
+```text
+LOCAL COMPUTER                             JHU (VPN only)
+frontend :5173 -> local backend :5001
+                      |
+                local snapshot <--- GET --- SSH tunnel -> read service :8766
+                local SQLite DB                         -> existing PanTS files
+                local uploads/results
+```
+
+This first version uses selected-case snapshots, not a live mirror of the whole
+dataset. A fresh snapshot goes into a new directory; existing local work is never
+overwritten. Disconnect the VPN after downloading and continue developing offline.
+Missing source meshes are omitted and may be generated by the **local** backend.
+GPU inference on JHU and production account/subscription testing are not part of
+this connection. Model weights/environments must be installed locally to test
+local inference; a separate approved staging worker is needed for JHU GPU tests.
+
+## What enforces read-only access
+
+- A dedicated service account, without sudo or membership in production groups.
+  The systemd service has a read-only filesystem, hidden home directories, and
+  read-only dataset mounts. Only private temporary files and service logs change.
+- A separate SSH account/key with forwarding restricted to this service's port.
+  Shells, commands, SFTP, other ports and reverse tunnels are denied by sshd.
+- An expiring bearer token per developer, containing a hash on the server and an
+  explicit case allowlist. There is no wildcard/full-dataset grant.
+- Three GET/HEAD endpoints only: `/v1/catalog`, `/v1/cases/<id>`, and
+  `/v1/cases/<id>/assets/<asset>`. Asset names are fixed; arbitrary paths,
+  symlink escapes, request bodies, query parameters and write methods are denied.
+- The local launcher discards inherited application credentials and disables
+  backend `.env` loading. Vite targets only the local backend. Production keys and
+  read-service tokens are never passed to the browser.
+
+Do not use a tunnel to production port 8000 as a substitute. Some production GET
+routes have side effects (job leasing and on-demand mesh generation). Also, an
+unrestricted `visitor` SSH key retains its existing production powers: using that
+key with `ssh -N` does not turn it into read-only access.
+
+## Developer setup (after administrator provisioning)
+
+Use the README's Python 3.11 backend environment and Node installation. Install
+the backend requirements and run `npm ci` in `PanTS-Demo`. Use a private directory
+outside the checkout and OneDrive for tokens, snapshots and test state.
+
+From `flask-server` in PowerShell, with the backend environment activated:
+
+```powershell
+$devRoot = Join-Path $env:LOCALAPPDATA 'BodyMapsDev'
+New-Item -ItemType Directory -Force -Path $devRoot | Out-Null
+python -m readonly_data.credentials --cases 35,8854 --days 7 --token-file "$devRoot\jhu.token" --grant-file "$devRoot\jhu-grant.json"
+```
+
+The cases are examples: the data owner must approve them. Send only
+`jhu-grant.json` and your **dedicated SSH public key** to the administrator over
+the team's approved channel. Keep `jhu.token` private; on Windows verify that its
+file ACL permits only your user and administrators. The grant is inactive until
+an administrator installs it. There is no production login or admin coupon here.
+
+Connect the JHU VPN and open a separate terminal, replacing the SSH username and
+key path with the dedicated account/key the administrator provisioned:
+
+```powershell
+ssh -N -T -o ExitOnForwardFailure=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -i "$env:USERPROFILE\.ssh\bodymaps-data-ro" -L 127.0.0.1:8766:127.0.0.1:8766 YOUR_READONLY_USER@bdmap1.wse.jhu.edu
+```
+
+Keep that terminal open during import. From `flask-server`:
+
+```powershell
+python -m readonly_data.client --token-file "$devRoot\jhu.token" --cases 35,8854 --output "$devRoot\snapshot-01" --max-gb 10
+```
+
+Only approved cases are fetched. The size limit includes all scan/mesh bytes.
+There is no upload operation and HTTP redirects are refused. Source changes or
+interrupted downloads leave an `INCOMPLETE` marker; the launcher refuses that
+snapshot. Retry into a different new directory. Snapshots are copies of existing
+files, not atomic database backups. The shared read limit is 120 requests/minute;
+the client retries HTTP 429 twice at 60-second intervals.
+
+Open two terminals in `flask-server`, with the Python environment activated, and
+set `$devRoot` in each as above:
+
+```powershell
+# Terminal 1: local backend and a new local database (Alembic runs automatically)
+python -m readonly_data.local backend --snapshot "$devRoot\snapshot-01" --state "$devRoot\state-01"
+
+# Terminal 2: local frontend
+python -m readonly_data.local frontend --snapshot "$devRoot\snapshot-01" --state "$devRoot\state-01"
+```
+
+Open **http://127.0.0.1:5173/dashboard**. API requests go through Vite to
+**http://127.0.0.1:5001**. Local email registration uses the new SQLite database;
+OAuth and SMTP credentials are absent. Ports already in use cause startup to
+fail; stop the previous local instance first. Live-room WebSockets need the
+separate local service described in `LIVE_ROOMS.md`; no JHU WebSocket proxy is set.
+
+## Administrator: reviewed one-time installation
+
+**Not an automatic deploy script.** Perform this only after its PR is approved,
+merged and explicitly approved for this host. None of these settings are active
+just because these files exist in Git. Provisioning needs a JHU administrator
+with sudo; a production application account is not a substitute.
+
+1. Record the approved full commit SHA and PR. Export only
+   `flask-server/readonly_data` and `flask-server/deploy/readonly` from that exact
+   merged commit into a **new** root-owned release directory under
+   `/opt/bodymaps-readonly/releases/<sha>/`. Do not copy `.env`, databases,
+   credentials, uploaded scans, or the main app. Make
+   `/opt/bodymaps-readonly/current` point to that reviewed release. The service
+   account must not be able to change any release, symlink or virtual environment.
+2. Create the dedicated system account/group `bodymaps-data-ro` with no login or
+   sudo, a root-owned `/opt/bodymaps-readonly/venv` using Python 3.11, and install
+   that release's `readonly_data/requirements.txt` in it. Create mount targets
+   `/srv/bodymaps-readonly/PanTS` and optionally `meshes`. Verify the real dataset
+   path and existing metadata layout before starting anything. Grant only the
+   required read/traverse ACLs if necessary; do not make production data
+   world-readable or grant access to the production application directory.
+3. Create `/etc/bodymaps-readonly` owned by root, group `bodymaps-data-ro`, mode
+   0750. Create `tokens.json` with `[]`, owner root, group `bodymaps-data-ro`, mode
+   0640. After the data owner approves a grant, append that grant object to the
+   JSON array using an atomic replacement that preserves owner/group/mode.
+   Confirm its case list and expiry. Plaintext tokens never belong on JHU.
+4. Review `readonly/bodymaps-readonly.service`. Confirm its dataset mount; enable
+   the two commented mesh lines only if that verified mesh directory exists.
+   Install as `/etc/systemd/system/bodymaps-readonly.service`. Run
+   `systemd-analyze verify /etc/systemd/system/bodymaps-readonly.service`, then
+   `systemctl daemon-reload` and `systemctl enable --now bodymaps-readonly`.
+   Check `journalctl -u bodymaps-readonly` and `ss -ltnp`: **127.0.0.1:8766 only**.
+   Do not weaken namespace restrictions if startup fails; check ACLs and mounts.
+   Keep one worker because the request limit is process-local. No nginx/public
+   route or production restart is needed.
+5. Create a separate `bodymaps-data-tunnel` group and one nonprivileged SSH
+   account per developer. Those accounts get no dataset/app groups or sudo.
+   Install their dedicated public keys in root-controlled authorized-key files.
+   Review and install the Match block in `readonly/sshd-readonly.conf` under the
+   host's sshd configuration. Never put `visitor` or deployment accounts in this
+   group. Keep an existing admin SSH session open; run `sshd -t` and inspect
+   `sshd -T -C user=THE_USER,host=bdmap1.wse.jhu.edu,addr=CLIENT_VPN_IP` before
+   reloading SSH. Verify all effective restrictions, not just the template.
+6. With the dedicated account, verify a GET succeeds through the tunnel, an
+   unapproved case gives 404, POST gives 405, and an expired token gives 401.
+   Verify shell/SFTP and forwarding to ports 8000/11434 are denied. Inside the
+   service's mount namespace, inspect the mount flags to confirm the dataset is
+   read-only; confirm the service UID cannot read production secrets/database.
+   Check the production site's health and unchanged application revision.
+
+The systemd/SSH restrictions require validation on the actual Linux host. Local
+Windows tests cover the application boundary, not the server's OS configuration.
+Reads consume some JHU disk/network bandwidth and create access logs; they do not
+edit the source scans or app. The service limits CPU, memory and concurrency.
+
+To revoke access, remove the hash from `tokens.json` atomically and remove that
+developer's SSH key. Token changes apply on the next request; a download already
+in progress may finish. To disable the entire facility, stop/disable only
+`bodymaps-readonly`; production gunicorn/nginx/Ollama remain untouched.
+
+## Approval and deployment workflow
+
+1. Develop on a feature branch with the isolated local state above.
+2. Push a PR. Run CI, review the diff, and obtain the team's approval before merge.
+3. Obtain explicit deployment approval for the **merged commit SHA**. A push or
+   PR merge alone must not restart or copy files to JHU.
+4. An authorized deployer installs that exact revision during the approved
+   window, verifies health, and retains the previous release for rollback.
+
+This PR introduces **no automatic JHU deployment**. Repository administrators
+should configure protected `main` rules (required PR approval and CI; no direct
+push/force push) and, before adding automation, a `jhu-production` environment
+with required reviewers. Those GitHub settings require repository-admin access
+and are not enforced by adding a YAML file or this document. Existing privileged
+server credentials can still change production; keep them out of developer
+machines/CI and give routine testers only the restricted credentials above.
+
+## Validation
+
+```powershell
+cd flask-server
+python -m pytest tests/unit/test_readonly_data.py -v
+```
+
+Tests use synthetic bytes and a temporary XLSX, not JHU scans. They cover scoped
+reads, expiry/revocation, malformed grants, traversal/symlink protection, method
+restrictions, existing-file-only behavior, real HTTP snapshot import, size limits,
+source hashes/mtime preservation and local environment separation.
+
+Implementation references: [OpenSSH forwarding/session restrictions](https://man.openbsd.org/sshd_config),
+[systemd read-only mounts and sandboxing](https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html),
+and [python-dotenv disable switch](https://bbc2.github.io/python-dotenv/#disable-load_dotenv).

flask-server/deploy/readonly/bodymaps-readonly.service

@@ -0,0 +1,42 @@
+[Unit]
+Description=BodyMaps approved dataset reads for local development
+After=network.target
+
+[Service]
+Type=simple
+User=bodymaps-data-ro
+Group=bodymaps-data-ro
+WorkingDirectory=/opt/bodymaps-readonly/current/flask-server
+Environment=PYTHONDONTWRITEBYTECODE=1
+Environment=PYTHON_DOTENV_DISABLED=1
+Environment=BODYMAPS_RO_DATA_ROOT=/srv/bodymaps-readonly/PanTS
+Environment=BODYMAPS_RO_TOKENS_FILE=/etc/bodymaps-readonly/tokens.json
+# Existing meshes are optional. Verify the actual production path before enabling.
+# Environment=BODYMAPS_RO_MESH_ROOT=/srv/bodymaps-readonly/meshes
+# BindReadOnlyPaths=/home/visitor/data/render_only:/srv/bodymaps-readonly/meshes
+BindReadOnlyPaths=/mnt/bodymaps/zzhou82/data/PanTS:/srv/bodymaps-readonly/PanTS
+ExecStart=/opt/bodymaps-readonly/venv/bin/gunicorn --workers 1 --threads 2 --bind 127.0.0.1:8766 --timeout 120 --worker-tmp-dir /tmp --access-logfile - readonly_data.service:create_app()
+Restart=on-failure
+RestartSec=5
+UMask=0077
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=tmpfs
+PrivateTmp=true
+PrivateDevices=true
+ProtectKernelTunables=true
+ProtectKernelModules=true
+ProtectControlGroups=true
+ProtectKernelLogs=true
+RestrictSUIDSGID=true
+LockPersonality=true
+CapabilityBoundingSet=
+RestrictAddressFamilies=AF_INET AF_UNIX
+IPAddressDeny=any
+IPAddressAllow=localhost
+CPUQuota=50%
+MemoryMax=512M
+TasksMax=32
+
+[Install]
+WantedBy=multi-user.target

flask-server/deploy/readonly/sshd-readonly.conf

@@ -0,0 +1,18 @@
+# Admin-reviewed template. Dedicated accounts in this group have NO shell/SFTP.
+# Never add visitor, a deploy user, or a privileged account to this group.
+Match Group bodymaps-data-tunnel
+    AuthenticationMethods publickey
+    PasswordAuthentication no
+    KbdInteractiveAuthentication no
+    AllowTcpForwarding local
+    PermitOpen 127.0.0.1:8766
+    PermitListen none
+    AllowStreamLocalForwarding no
+    AllowAgentForwarding no
+    X11Forwarding no
+    PermitTTY no
+    PermitTunnel no
+    PermitUserRC no
+    MaxSessions 0
+    ForceCommand /usr/sbin/nologin
+Match all

flask-server/readonly_data/__init__.py

@@ -0,0 +1 @@
+"""Standalone dataset export service. Never imports the production application."""