Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
node_modules/
dist/
src-tauri/target/
.DS_Store
*.local
215 changes: 214 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1 +1,214 @@
# InsightsTool
# Maine DOC Insight

A desktop analytics tool for Maine DOC education program data.
Built with [Tauri](https://tauri.app/) (Rust backend) + React + Chart.js.

## What it does

**Dashboard views** — six data views for exploring program and resident outcomes:

| View | What it covers |
|------|----------------|
| Home | System-wide KPIs, top programs by completion, near-release engagement snapshot |
| Programs | Completion rates, waitlists vs. throughput, second-enrollment rates, breakdown by LSI/custody/education level |
| Residents | Never-engaged roster, near-release residents by engagement status, no-diploma residents not in education |
| Facilities | Facility completion rates (raw and mix-adjusted), operational focus areas |
| Comparison | Side-by-side facility comparisons |
| Data Quality | Field-level missing data report, metric quality status, integrity checks |

**AI chat panel** — ask natural-language questions about the loaded data. The planner routes your question to the right metrics; the writer synthesizes a cited response. Supports multiple AI providers (see below). No resident names or DOC numbers are sent to any external API — only aggregated metrics.

**Data quality assessment** — automatic checks on every loaded dataset. Missing files, low coverage, and suspicious values surface as per-view badges and a dedicated Data Quality view with field-level detail.

**Demo mode** — explore sample AI responses without loading real data or entering an API key. Toggle the switch in the chat panel header.

## Prerequisites

- [Node.js](https://nodejs.org/) ≥ 18
- [Rust](https://www.rust-lang.org/tools/install) + Cargo (for Tauri)
- [Tauri prerequisites](https://tauri.app/v1/guides/getting-started/prerequisites) for your OS

## Quick start

```bash
npm install

# Run in dev mode (opens a hot-reload window)
npm run tauri dev

# Or run just the web frontend (no Rust required)
npm run dev
# then open http://localhost:5173
```

## Build for distribution

```bash
npm run tauri build
# Output: src-tauri/target/release/bundle/
```

## AI providers

Open the chat panel (✦ Chat tab on the right) and select a provider.

| Provider | Setup | Privacy |
|----------|-------|---------|
| Google Gemini | API key from [aistudio.google.com](https://aistudio.google.com) | Aggregated metrics sent to Google |
| OpenAI | API key from [platform.openai.com](https://platform.openai.com/api-keys) | Aggregated metrics sent to OpenAI |
| Anthropic Claude | API key from [console.anthropic.com](https://console.anthropic.com) | Aggregated metrics sent to Anthropic |
| Ollama (local) | See below — no key required | Nothing leaves your machine |

### Local AI with Ollama

No API key required. Runs entirely on your machine.

**1. Install Ollama**

Download from [ollama.ai](https://ollama.ai) and follow the installer for your OS.

**2. Start the server**

```bash
ollama serve
```

Ollama must be running before you use the chat panel. If you see a connection error, run this command first.

**3. Pull a model**

```bash
# Recommended — works on most laptops (~4.7 GB)
ollama pull qwen2.5:7b

# Better output quality if you have a GPU or more RAM (~9 GB)
ollama pull qwen2.5:14b
```

**4. Enable in the app**

Open the chat panel, select **Ollama (local)**, and click **Enable local AI**.

**Notes:**
- The first response after pulling a model may be slow (~30s) while it loads into memory. Subsequent responses are faster.
- Structured query planning requires solid instruction-following ability. `qwen2.5:7b` is the minimum recommended size; `qwen2.5:14b` is more reliable for complex questions.
- If the model returns malformed output, the app retries once with a corrected prompt before surfacing an error suggesting a larger model.
- If Ollama is running on a non-default port or host, edit `src/config/llm.json` and change the `endpoints.ollama` value.

## File schema

All files are optional — insights are generated from whatever is loaded. Files are organized into three sections in the loader.

### DOC data

Exported from the Maine DOC / OMS system. Accepts CSV or Excel (`.xlsx`).

**Core:**

| Dataset | Key columns |
|---------|-------------|
| Resident Roster | `ID`, `STATUS_DESC`, `LOCATION_TO`, `LOC_TYPE_TO`, `ODARA_SCORE`, `STATIC99R_SCORE`, `LSI_RATING`, `HIGHEST_ED_LVL`, `CUSTODY_LVL`, `Earliest Release Date`, `SENTENCE_OFFENSE`, `Offense Date`, `COUNTY`, `LAST_PROGRAM_COMPLETED`, `JOB_ASSIGN` |
| Program Participation | `ID`, `GENDER`, `Facility`, `CUSTODY LEVEL`, `Housing Unit`, `Housing Pod`, `Room`, `Bed`, `Program`, `Program Status`, `Program Termination Or Completion Date`, `Earliest Release Date` |

**Advanced** (toggle "Advanced data" in the loader to reveal):

> **Note:** Advanced dataset column schemas are based on expected data structure and have not been cross-referenced against actual Maine DOC exports. Column names may differ from real exports.

| Dataset | Key columns |
|---------|-------------|
| Incident Records | `ID`, `Incident Date`, `Incident Type`, `Severity`, `Sanction`, `Facility` |
| Work Assignments | `ID`, `Assignment Type`, `Role Title`, `Start Date`, `End Date`, `Transferable Skills Flag`, `Facility` |
| Case Plans | `ID`, `State ID Obtained`, `State ID Date`, `Job Lined Up`, `Housing Plan`, `Trust Account Balance`, `Savings Goal Met` |
| Credentials | `ID`, `Credential Type`, `Credential Name`, `Date Earned`, `Issuing Body`, `Verified` |
| Housing History | `ID`, `Move Date`, `Housing Unit`, `Pod`, `Room`, `Bed`, `Custody Level`, `Move Reason`, `Facility` |

### UnlockEd platform data

> **Note:** This is a subset of the data available from the UnlockEd platform, converted to CSV files for local use.

CSV exports from the UnlockEd database. Drop multiple files at once onto the UnlockEd section.

`users.csv`, `facilities.csv`, `programs.csv`, `program_classes.csv`, `program_class_enrollments.csv`, `program_completions.csv`, `program_class_events.csv`, `program_class_event_attendance.csv`, `user_session_tracking.csv`, `program_credit_types.csv`

### Mapping & reference

Optional. Shown when "Advanced data" is enabled.

| File | Purpose |
|------|---------|
| `program_crosswalk.csv` | Maps DOC program names to UnlockEd program IDs; enables fuzzy-match fallback |
| `field_mapping.csv` | Maps OMS/DOC column names to internal tool fields; handles future export schema changes |

## Project structure

```
maine-doc-insight/
├── src/
│ ├── chat/
│ │ ├── providers.ts # Provider registry (Gemini, OpenAI, Claude, Ollama)
│ │ ├── router.ts # Planner — routes questions to metric queries
│ │ ├── writer.ts # Writer — synthesizes cited responses
│ │ ├── prompts.ts # System instructions and user turn builders
│ │ ├── aggregator.ts # Executes DataPlan against loaded data
│ │ ├── contextPacket.ts # Assembles metric context for the writer
│ │ ├── flatTables.ts # Flat table projections for the aggregator
│ │ ├── actionItems.ts # Action item extraction
│ │ ├── refusal.ts # Refusal message builder
│ │ └── fetchWithRetry.ts # Fetch wrapper with retry + network error handling
│ ├── lib/
│ │ ├── ingest.ts # CSV/Excel parsing, buildCoreFromDOC, buildCoreFromUL
│ │ ├── analytics.ts # All metric computations
│ │ ├── dataQuality.ts # Data quality checks and grading
│ │ └── demoExport.ts # Exports synthetic demo data as downloadable CSVs
│ ├── data/
│ │ ├── mockData.ts # Synthetic demo data generator
│ │ ├── schemaRegistry.ts # Metric registry (IDs, labels, table mappings)
│ │ └── demoResponses.ts # Canned AI responses for demo mode
│ ├── config/
│ │ ├── chat.json # Chat panel UI settings (confidence threshold, drawer width)
│ │ ├── llm.json # Provider endpoints and generation parameters
│ │ ├── models.json # Default model lists per provider
│ │ ├── data_quality.json # Missing-data warn/crit thresholds and LSI clustering parameters
│ │ ├── analytics.json # Algorithm parameters: top-N, mix-adjustment, cohort windows, statistical constants
│ │ ├── thresholds.json # Display thresholds: attention flag scoring, QoL green/yellow bands, color cutoffs
│ │ └── facilities.json # Facility code → display name pairs (10 Maine DOC facilities)
│ ├── components/
│ │ ├── chat/
│ │ │ └── ChatDrawer.tsx # AI chat panel with provider selection
│ │ ├── FileLoader.tsx # Drag-and-drop CSV ingestion UI
│ │ ├── OmsCards.tsx # OMS metric cards
│ │ ├── ChartFootnote.tsx # Per-chart data quality footnotes
│ │ ├── DataUnavailableCard.tsx # Placeholder when required data is missing
│ │ ├── DataQualityBanner.tsx
│ │ └── SmartActionItem.tsx
│ ├── contexts/
│ │ └── DataQualityContext.tsx
│ ├── views/
│ │ ├── HomeView.tsx # Dashboard — KPIs, top programs, near-release
│ │ ├── ProgramsView.tsx # Completion rates, waitlists, second-enrollment, breakdowns
│ │ ├── ResidentsView.tsx # Near-release, never-engaged, no-diploma roster
│ │ ├── FacilitiesView.tsx # Facility completion rates (raw + mix-adjusted)
│ │ ├── ComparisonView.tsx # Side-by-side facility comparisons
│ │ └── DataQualityView.tsx # Field-level data quality report
│ ├── types.ts # All TypeScript types
│ ├── App.tsx # Shell + navigation + quality indicators
│ └── app.css # Dark analytics theme
├── src-tauri/ # Rust/Tauri backend (minimal)
├── index.html
├── vite.config.ts
└── package.json
```

## Notes on mix-adjustment

The mix-adjusted completion rate applies a correction for LSI risk composition.
If a facility has a higher share of High/Maximum LSI residents than the system average,
its raw rate is adjusted upward proportionally — surfacing true operational performance
independent of resident population composition.

Adjustment formula:
```
mixAdjustedRate = rawRate + (facilityHighRiskShare - systemHighRiskShare) × 15pp
```
This is a simplified first-order adjustment. A regression-based approach would be more
rigorous for formal reporting.
36 changes: 36 additions & 0 deletions eslint.config.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
import tsPlugin from "@typescript-eslint/eslint-plugin";
import tsParser from "@typescript-eslint/parser";
import reactPlugin from "eslint-plugin-react";
import reactHooksPlugin from "eslint-plugin-react-hooks";

export default [
{
ignores: ["dist/**", "src-tauri/**", "node_modules/**"],
},
{
files: ["**/*.{ts,tsx}"],
languageOptions: {
parser: tsParser,
parserOptions: {
ecmaVersion: "latest",
sourceType: "module",
ecmaFeatures: { jsx: true },
},
},
plugins: {
"@typescript-eslint": tsPlugin,
react: reactPlugin,
"react-hooks": reactHooksPlugin,
},
rules: {
...tsPlugin.configs.recommended.rules,
...reactHooksPlugin.configs.recommended.rules,
"@typescript-eslint/no-unused-vars": ["error", { varsIgnorePattern: "^_", argsIgnorePattern: "^_", destructuredArrayIgnorePattern: "^_" }],
"react/jsx-uses-react": "off",
"react/react-in-jsx-scope": "off",
},
settings: {
react: { version: "detect" },
},
},
];
12 changes: 12 additions & 0 deletions index.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Maine DOC Insight</title>
</head>
<body>
<div id="root"></div>
<script type="module" src="/src/main.tsx"></script>
</body>
</html>
Loading