Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ jobs:
- run: corepack prepare pnpm@9 --activate

- name: Build and export to Docker
uses: docker/build-push-action@d08e5c354a6adb9ed34480a06d141179aa583294 # v7
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7
with:
context: .
tags: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/playwright.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ jobs:
POSTGRES_QUERY_TIMEOUT: 100000
POSTGRES_STATEMENT_TIMEOUT: 100000
BINDING: localhost
- uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
if: ${{ !cancelled() }}
with:
name: playwright
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ jobs:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

- name: Build and push Docker image
uses: docker/build-push-action@d08e5c354a6adb9ed34480a06d141179aa583294 # v7
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7
with:
platforms: linux/amd64
target: production
Expand Down
5 changes: 4 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,10 @@
"undici@>=7.0.0 <7.24.0": "7.24.0",
"file-type@>=13.0.0 <=21.3.1": "21.3.2",
"socket.io-parser@>=4.0.0 <4.2.6": "4.2.6",
"picomatch@>=4.0.0 <4.0.4": ">=4.0.4"
"picomatch@>=4.0.0 <4.0.4": ">=4.0.4",
"lodash@<4.18.0": ">=4.18.0",
"follow-redirects@<1.16.0": ">=1.16.0",
"hono@<4.12.14": ">=4.12.14"
}
}
}
1,553 changes: 765 additions & 788 deletions pnpm-lock.yaml

Large diffs are not rendered by default.

80 changes: 80 additions & 0 deletions teammapper-backend/benchmark/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# AI Benchmark

Automated benchmark suite that evaluates the quality of AI-generated mind maps. It sends a set of predefined prompts (fixtures) through the mind map generation pipeline, then grades each output on two dimensions using an LLM-as-judge approach plus static Mermaid syntax validation.

## Running

```bash
# Requires AI_LLM_TOKEN and AI_LLM_MODEL environment variables
pnpm run benchmark:ai
```

## How it works

1. Each fixture describes a mind map topic, language, and category.
2. The `AiService` generates a Mermaid mind map for the fixture.
3. The `BenchmarkGraderService` scores the output on two dimensions:
- **Semantic Quality** — An LLM judges relevance, depth, and structure (0–10).
- **Mermaid Syntax** — A static validator checks structural correctness (0–10).
4. Results are printed to the console and saved as a timestamped JSON report in `benchmark/reports/`.

## Fixtures

Defined in `fixtures.ts`. Each fixture has a topic category:

| Category | Description |
|-----------------|--------------------------------------------------|
| `informational` | Factual topics testing accuracy and depth |
| `technical` | Domain-specific jargon testing clean node labels |
| `creative` | Open-ended topics testing node diversity |
| `edge-case` | Minimal or extreme inputs testing graceful handling|

Fixtures cover both English (`en`) and German (`de`) prompts.

## Output JSON format

Reports are written to `benchmark/reports/<timestamp>.json`. The top-level structure is a `GradingReport`:

| Field | Type | Description |
|------------------|----------------------------|--------------------------------------------------------------------|
| `timestamp` | `string` | ISO 8601 timestamp of the benchmark run. |
| `provider` | `string` | LLM provider used (e.g. `"openai-compatible"`). |
| `model` | `string` | Model identifier (e.g. `"mistral-large-latest"`). |
| `systemPrompt` | `string` | The full system prompt sent to the AI for mind map generation. |
| `results` | `BenchmarkResult[]` | Per-fixture results (see below). |
| `averages` | `Record<string, number>` | Average score per grading dimension across all fixtures. |
| `overallAverage` | `number` | Grand average of all dimension scores across all fixtures. |

### `BenchmarkResult`

Each entry in the `results` array:

| Field | Type | Description |
|--------------|----------------------|----------------------------------------------------------------|
| `fixtureId` | `string` | Unique fixture identifier (e.g. `"en-info-photosynthesis"`). |
| `fixture` | `BenchmarkFixture` | Full fixture definition (id, description, language, topic). |
| `rawOutput` | `string` | The raw Mermaid syntax returned by the AI. |
| `durationMs` | `number` | Time taken for generation in milliseconds. |
| `error` | `string \| null` | Error message if generation failed or timed out, otherwise `null`. |
| `dimensions` | `GradingDimension[]` | Array of scored grading dimensions (typically 2). |

### `GradingDimension`

Each entry in the `dimensions` array:

| Field | Type | Description |
|--------------|----------|-------------------------------------------------------|
| `name` | `string` | Dimension name (`"Semantic Quality"` or `"Mermaid Syntax"`). |
| `score` | `number` | Score from 0 to 10. |
| `strengths` | `string` | One-sentence summary of what the output did well. |
| `weaknesses` | `string` | One-sentence summary of issues found. |

### `BenchmarkFixture`

| Field | Type | Description |
|---------------------|----------|----------------------------------------------------------|
| `id` | `string` | Unique identifier (e.g. `"de-tech-softwarearchitektur"`). |
| `description` | `string` | Human-readable description of what the fixture tests. |
| `mindmapDescription`| `string` | The prompt sent to the AI for mind map generation. |
| `language` | `string` | Language code (`"en"` or `"de"`). |
| `topic` | `string` | Fixture category: `informational`, `technical`, `creative`, or `edge-case`. |
26 changes: 13 additions & 13 deletions teammapper-backend/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -33,21 +33,21 @@
"prod:typeorm:migrate": "typeorm migration:run --dataSource dist/data-source.js"
},
"dependencies": {
"@ai-sdk/openai": "3.0.49",
"@ai-sdk/openai-compatible": "2.0.37",
"@ai-sdk/openai": "3.0.52",
"@ai-sdk/openai-compatible": "2.0.41",
"@nestjs/cache-manager": "^3.1.0",
"@nestjs/cli": "^11.0.17",
"@nestjs/common": "^11.1.17",
"@nestjs/config": "4.0.3",
"@nestjs/core": "^11.1.17",
"@nestjs/platform-express": "^11.1.17",
"@nestjs/platform-socket.io": "^11.1.17",
"@nestjs/cli": "^11.0.19",
"@nestjs/common": "^11.1.18",
"@nestjs/config": "4.0.4",
"@nestjs/core": "^11.1.18",
"@nestjs/platform-express": "^11.1.18",
"@nestjs/platform-socket.io": "^11.1.18",
"@nestjs/schedule": "^6.1.1",
"@nestjs/serve-static": "^5.0.4",
"@nestjs/typeorm": "^11.0.0",
"@nestjs/websockets": "^11.1.17",
"@nestjs/serve-static": "^5.0.5",
"@nestjs/typeorm": "^11.0.1",
"@nestjs/websockets": "^11.1.18",
"@types/uuid": "^11.0.0",
"ai": "6.0.143",
"ai": "6.0.158",
"cache-manager": "^7.2.8",
"class-validator": "^0.15.1",
"cookie-parser": "^1.4.7",
Expand All @@ -60,7 +60,7 @@
"reflect-metadata": "^0.2.2",
"rimraf": "^6.1.3",
"rxjs": "^7.8.2",
"sanitize-html": "^2.17.2",
"sanitize-html": "^2.17.3",
"socket.io": "4.8.3",
"typeorm": "^0.3.28",
"uuid": "11.1.0",
Expand Down
14 changes: 12 additions & 2 deletions teammapper-frontend/mmp/src/map/handlers/nodes.ts
Original file line number Diff line number Diff line change
Expand Up @@ -186,8 +186,18 @@ export default class Nodes {

background.style.stroke = color.toString();

Utils.removeAllRanges();
this.selectedNode.getNameDOM().blur();
// Don't blur the node that's currently being edited (#1249): on
// mobile, d3-drag's `started` callback fires on the second tap
// that enters edit mode and calls selectNode for the same node,
// which used to steal focus from the just-focused contenteditable
// and stop the soft keyboard from opening.
const prevName = this.selectedNode.getNameDOM();
const wouldBlurActiveEdit =
this.selectedNode === node && document.activeElement === prevName;
if (!wouldBlurActiveEdit) {
Utils.removeAllRanges();
prevName.blur();
}

this.map.events.call(
Event.nodeDeselect,
Expand Down
38 changes: 19 additions & 19 deletions teammapper-frontend/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -41,19 +41,19 @@
"test:e2e": "npx playwright test"
},
"dependencies": {
"@angular-devkit/build-angular": "21.2.6",
"@angular/animations": "21.2.7",
"@angular/cdk": "21.2.5",
"@angular/cli": "21.2.6",
"@angular/common": "21.2.7",
"@angular/compiler": "21.2.7",
"@angular/compiler-cli": "21.2.7",
"@angular/core": "21.2.7",
"@angular/forms": "21.2.7",
"@angular/material": "21.2.5",
"@angular/platform-browser": "21.2.7",
"@angular/platform-browser-dynamic": "21.2.7",
"@angular/router": "21.2.7",
"@angular-devkit/build-angular": "21.2.7",
"@angular/animations": "21.2.8",
"@angular/cdk": "21.2.6",
"@angular/cli": "21.2.7",
"@angular/common": "21.2.8",
"@angular/compiler": "21.2.8",
"@angular/compiler-cli": "21.2.8",
"@angular/core": "21.2.8",
"@angular/forms": "21.2.8",
"@angular/material": "21.2.6",
"@angular/platform-browser": "21.2.8",
"@angular/platform-browser-dynamic": "21.2.8",
"@angular/router": "21.2.8",
"@fortawesome/angular-fontawesome": "^4.0.0",
"@fortawesome/fontawesome-svg-core": "^7.2.0",
"@fortawesome/free-brands-svg-icons": "^7.2.0",
Expand All @@ -62,11 +62,11 @@
"@ngx-translate/core": "^17.0.0",
"@ngx-translate/http-loader": "^17.0.0",
"@teammapper/mermaid-mindmap-parser": "workspace:^",
"ai": "^6.0.143",
"ai": "^6.0.158",
"angular2-hotkeys": "^16.0.1",
"d3": "7.9.0",
"deep-object-diff": "^1.1.9",
"dompurify": "3.3.3",
"dompurify": "3.4.0",
"jspdf": "^4.2.1",
"localforage": "1.10.0",
"ngx-color-picker": "^20.1.1",
Expand Down Expand Up @@ -118,10 +118,10 @@
"typescript-eslint": "^8.57.2"
},
"optionalDependencies": {
"@nx/nx-darwin-arm64": "22.6.4",
"@nx/nx-darwin-x64": "22.6.4",
"@nx/nx-linux-x64-gnu": "22.6.4",
"@nx/nx-win32-x64-msvc": "22.6.4"
"@nx/nx-darwin-arm64": "22.6.5",
"@nx/nx-darwin-x64": "22.6.5",
"@nx/nx-linux-x64-gnu": "22.6.5",
"@nx/nx-win32-x64-msvc": "22.6.5"
},
"workspaces": [
"packages/*"
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,11 @@
"dist"
],
"dependencies": {
"dompurify": "3.3.3"
"dompurify": "3.4.0"
},
"devDependencies": {
"jison": "0.4.18",
"typescript": "~5.9.3",
"vite": "^6.4.1"
"vite": "^8.0.8"
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ export class SettingsService {
'zh-cn',
'es',
'pt-br',
'ja',
];

public userSettings: Observable<UserSettings | null>;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ describe('FooterComponent', () => {
'zh-cn',
'es',
'pt-br',
'ja',
]);
expect(component.currentYear).toBe(new Date().getFullYear().toString());
});
Expand Down
3 changes: 2 additions & 1 deletion teammapper-frontend/src/assets/i18n/de.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
"ZH-CN": "Vereinfachtes Chinesisch",
"PT-BR": "Portugiesisch (Brasilien)",
"ES": "Spanisch",
"DE": "Deutsch"
"DE": "Deutsch",
"JA": "Japanisch"
},
"TOOLTIPS": {
"IMPORT_MAP": "Mindmap importieren",
Expand Down
3 changes: 2 additions & 1 deletion teammapper-frontend/src/assets/i18n/en.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
"ZH-CN": "Simplified Chinese",
"PT-BR": "Portuguese Brazil",
"ES": "Spanish",
"DE": "German"
"DE": "German",
"JA": "Japanese"
},
"TOOLTIPS": {
"IMPORT_MAP": "Imports the map",
Expand Down
3 changes: 2 additions & 1 deletion teammapper-frontend/src/assets/i18n/es.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
"ZH-CN": "chino simplificado",
"PT-BR": "Portugués Brasil",
"ES": "Español",
"DE": "Alemán"
"DE": "Alemán",
"JA": "Japonés"
},
"TOOLTIPS": {
"IMPORT_MAP": "Importa el mapa",
Expand Down
3 changes: 2 additions & 1 deletion teammapper-frontend/src/assets/i18n/fr.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
"ZH-CN": "Chinois simplifié",
"PT-BR": "Portuguese Brazil",
"ES": "Espagnol",
"DE": "Allemand"
"DE": "Allemand",
"JA": "Japonais"
},
"TOOLTIPS": {
"IMPORT_MAP": "Importe la carte",
Expand Down
3 changes: 2 additions & 1 deletion teammapper-frontend/src/assets/i18n/it.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@
"ZH-CN": "Cinese semplificato",
"PT-BR": "Portoghese Brasiliano",
"ES": "Spagnolo",
"DE": "Tedesco"
"DE": "Tedesco",
"JA": "Giapponese"
},
"TOOLTIPS": {
"IMPORT_MAP": "Importa mappa",
Expand Down
Loading
Loading