Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
# statt abgewiesen.
# make setup Nur die lokale Umgebung anlegen/aktualisieren.
# make generate KI-Aufgaben generieren (braucht einen API-Schluessel, siehe README).
# make export Ein Set fuer KI-Review exportieren (ARGS="<slug> [--split-size N] ...").
# make audit Ueberblick ueber deine Inhalte ausgeben.
# make clean Die lokale Umgebung entfernen.
#
Expand All @@ -38,14 +39,15 @@ PIP := $(VENV)/bin/pip
ENGINE_PIN := $(shell cat schema/engine-version.txt)
ENGINE_STAMP := node_modules/.engine-$(ENGINE_PIN)

.PHONY: validate lint lint-warnings setup generate audit clean help
.PHONY: validate lint lint-warnings setup generate export audit clean help

help:
@echo "make validate - Inhalte pruefen (richtet sich beim ersten Mal selbst ein)"
@echo "make lint - Engine-Gate lokal (Selbsttest + alle Lektionen/Manifeste)"
@echo "make lint-warnings - derselbe Lauf, zusätzlich mit Warnungen (W-*)"
@echo "make setup - lokale Umgebung anlegen"
@echo "make generate - KI-Aufgaben generieren (API-Schluessel noetig; ARGS=\"--topic ...\")"
@echo "make export - Set fuer KI-Review exportieren (ARGS=\"<slug> [--split-size N] ...\")"
@echo "make audit - Inhalts-Ueberblick"
@echo "make clean - lokale Umgebung entfernen"

Expand Down Expand Up @@ -86,6 +88,12 @@ lint-warnings: $(ENGINE_STAMP)
generate: $(VENV)/.ready
@$(PY) scripts/generate_exercises.py $(ARGS)

# Ein Set fuer KI-Review exportieren, z. B.:
# make export ARGS="<set-slug>"
# make export ARGS="<set-slug> --split-size 5"
export: $(VENV)/.ready
@$(PY) scripts/export_set.py $(ARGS)

audit: $(VENV)/.ready
@$(PY) scripts/audit_content.py

Expand Down
18 changes: 16 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,11 +134,19 @@ Full walkthrough: [docs/GETTING-STARTED.md](docs/GETTING-STARTED.md).

`scripts/export_set.py` writes all lessons of ONE set into a single
YAML (or JSON) file so an AI assistant or a human can review the whole
set in one pass (syntax, correctness, consistency across lessons):
set in one pass (syntax, correctness, consistency across lessons).

**Recommended (via make; reuses the local environment `make validate` set up):**

```bash
python3 scripts/export_set.py react-grundlagen
make export ARGS="react-grundlagen"
# -> exports/react-grundlagen-de-<timestamp>.yaml
```

**Direct (fallback; run it inside the venv from the Quick start):**

```bash
python3 scripts/export_set.py react-grundlagen
python3 scripts/export_set.py react-grundlagen --format json --out /tmp/review.json
```

Expand All @@ -149,6 +157,12 @@ source-language directories, `--lang` (default `de`) picks the
`sets/<lang>/` directory. Non-ASCII characters stay real UTF-8. An
unknown slug aborts with a list of the available sets.

For a large set, `--split-size N` writes multiple self-contained files
of at most N lessons each instead of one huge file, e.g.
`make export ARGS="python-basics --split-size 5"` (each part keeps its
own `review_instructions` copy, so any one file can be handed to an AI
on its own). Cannot be combined with `--out`.

The export is self-contained: its first field `review_instructions`
holds the complete review prompt from
[`docs/ai-review-prompt-template.md`](docs/ai-review-prompt-template.md)
Expand Down
35 changes: 26 additions & 9 deletions docs/export-set-usage.de.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,31 +20,47 @@ Lektions-JSONs unter `sets/` ein.
## Nutzung

```bash
python3 scripts/export_set.py <set-slug> [--lang <lang>] [--format yaml|json] [--out PFAD]
make export ARGS="<set-slug> [--lang <lang>] [--format yaml|json] [--out PFAD] [--split-size N]"
# oder direkt (Fallback; innerhalb der venv aus dem Quick Start):
python3 scripts/export_set.py <set-slug> [--lang <lang>] [--format yaml|json] [--out PFAD] [--split-size N]
```

| Argument | Bedeutung | Default |
| --- | --- | --- |
| `<set-slug>` | Set-Id aus dem Wurzel-`manifest.yaml` (z. B. `react-grundlagen-from-de`) oder der Ordnername des Set-Pfads (z. B. `react-grundlagen` für `sets/de/react-grundlagen`) | Pflicht |
| `--lang` | Quellsprachen-Verzeichnis (`sets/<lang>/`), das einen Ordnernamen-Slug eindeutig macht, der unter mehreren Quellsprachen existiert | `de` |
| `--format` | Ausgabeformat: `yaml` oder `json` | `yaml` |
| `--out` | Pfad der Ausgabedatei | `exports/<set-slug>-<lang>-<timestamp>.<format>` |
| `--out` | Pfad der Ausgabedatei (nicht kombinierbar mit `--split-size`) | `exports/<set-slug>-<lang>-<timestamp>.<format>` |
| `--split-size` | Export in mehrere Dateien von je hoechstens N Lektionen aufteilen, statt einer Datei | aus (eine Datei) |

Beispiele:

```bash
# Standardfall: YAML-Export nach exports/ (die Sets liegen unter sets/de/)
python3 scripts/export_set.py react-grundlagen
make export ARGS="react-grundlagen"
# -> exports/react-grundlagen-de-<timestamp>.yaml

# Sonderfall: JSON an einen eigenen Pfad (nur wenn ein Tooling explizit JSON braucht)
python3 scripts/export_set.py react-grundlagen --format json --out /tmp/review.json
make export ARGS="react-grundlagen --format json --out /tmp/review.json"

# Grosses Set: in Teile von je hoechstens 5 Lektionen aufteilen, fuer eine
# KI mit begrenztem Kontextfenster
make export ARGS="python-basics --split-size 5"
# -> exports/python-basics-de-<timestamp>-part01-of-3.yaml, part02-of-3, part03-of-3
```

Ohne `--out` landet die Datei in `exports/` nach dem Muster
`<set-slug>-<lang>-<timestamp>.<format>`. Das Verzeichnis `exports/`
wird bei Bedarf angelegt und ist **gitignored**: Exportdateien sind
Wegwerf-Artefakte fürs Review und werden nie committet.
`<set-slug>-<lang>-<timestamp>.<format>` (bei `--split-size` je eine
Datei pro Teil nach `<set-slug>-<lang>-<timestamp>-partNN-of-MM.<format>`).
Das Verzeichnis `exports/` wird bei Bedarf angelegt und ist
**gitignored**: Exportdateien sind Wegwerf-Artefakte fürs Review und
werden nie committet.

Jeder von `--split-size` geschriebene Teil ist eigenstaendig: er traegt
seine eigene `review_instructions`-Kopie sowie die Felder
`part`/`of`/`lesson_count`/`total_lesson_count`, sodass jeder einzelne
Teil fuer sich, in beliebiger Reihenfolge, an eine KI zum Review
gegeben werden kann.

Ein unbekannter oder mehrdeutiger Slug bricht mit Exit-Code 2 und
einer Liste der verfügbaren Sets ab. Umlaute und alle anderen
Expand All @@ -55,7 +71,7 @@ Nicht-ASCII-Zeichen bleiben echtes UTF-8.
1. **Export erzeugen:**

```bash
python3 scripts/export_set.py react-grundlagen
make export ARGS="react-grundlagen"
```

2. **Exportdatei öffnen** und im `review_instructions`-Block am Anfang
Expand Down Expand Up @@ -100,7 +116,8 @@ Nicht-ASCII-Zeichen bleiben echtes UTF-8.
- **Quellkapitel bei jedem Review neu einfügen**, wenn es sich
geändert hat; nicht aus einem alten Export kopieren.
- **Große Sets in Portionen prüfen** (z. B. 8-10 Lektionen pro
Durchgang), wenn der Kontext der verwendeten KI begrenzt ist.
Durchgang), wenn der Kontext der verwendeten KI begrenzt ist - dafür
`--split-size` nutzen, statt den Export von Hand zu zerschneiden.
- **YAML als Standard belassen**; JSON nur, wenn ein Tooling das
explizit braucht.
- **Kein Copy-Paste von KI-Vorschlägen ohne Gegenlesen.** Die KI
Expand Down
30 changes: 23 additions & 7 deletions docs/export-set-usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,32 +20,47 @@ JSON files under `sets/`.
## Usage

```bash
python3 scripts/export_set.py <set-slug> [--lang <lang>] [--format yaml|json] [--out PATH]
make export ARGS="<set-slug> [--lang <lang>] [--format yaml|json] [--out PATH] [--split-size N]"
# or, direct (fallback; run it inside the venv from the Quick start):
python3 scripts/export_set.py <set-slug> [--lang <lang>] [--format yaml|json] [--out PATH] [--split-size N]
```

| Argument | Meaning | Default |
| --- | --- | --- |
| `<set-slug>` | Set id from the root `manifest.yaml` (e.g. `react-grundlagen-from-de`) or the folder name of the set path (e.g. `react-grundlagen` for `sets/de/react-grundlagen`) | required |
| `--lang` | Source-language directory (`sets/<lang>/`) that disambiguates a folder-name slug existing under several source languages | `de` |
| `--format` | Output format: `yaml` or `json` | `yaml` |
| `--out` | Output file path | `exports/<set-slug>-<lang>-<timestamp>.<format>` |
| `--out` | Output file path (cannot be combined with `--split-size`) | `exports/<set-slug>-<lang>-<timestamp>.<format>` |
| `--split-size` | Split the export into multiple files of at most N lessons each, instead of one file | off (one file) |

Examples:

```bash
# Standard case: YAML export into exports/ (the sets live under sets/de/)
python3 scripts/export_set.py react-grundlagen
make export ARGS="react-grundlagen"
# -> exports/react-grundlagen-de-<timestamp>.yaml

# Special case: JSON to a custom path (only when a tool explicitly needs JSON)
python3 scripts/export_set.py react-grundlagen --format json --out /tmp/review.json
make export ARGS="react-grundlagen --format json --out /tmp/review.json"

# Large set: split into parts of at most 5 lessons each for an AI with a
# limited context window
make export ARGS="python-basics --split-size 5"
# -> exports/python-basics-de-<timestamp>-part01-of-3.yaml, part02-of-3, part03-of-3
```

Without `--out`, the file is written to `exports/` following the
pattern `<set-slug>-<lang>-<timestamp>.<format>`. The `exports/`
pattern `<set-slug>-<lang>-<timestamp>.<format>` (or, with
`--split-size`, one file per part following
`<set-slug>-<lang>-<timestamp>-partNN-of-MM.<format>`). The `exports/`
directory is created on demand and is **gitignored**: export files are
throwaway review artifacts and are never committed.

Each part written by `--split-size` is self-contained: it carries its
own `review_instructions` copy plus `part`/`of`/`lesson_count`/
`total_lesson_count` fields, so any single part can be handed to an AI
reviewer on its own, in any order, without the others.

An unknown or ambiguous slug aborts with exit code 2 and a list of the
available sets. Umlauts and all other non-ASCII characters stay real
UTF-8.
Expand All @@ -55,7 +70,7 @@ UTF-8.
1. **Create the export:**

```bash
python3 scripts/export_set.py react-grundlagen
make export ARGS="react-grundlagen"
```

2. **Open the export file** and find the section "Quellkapitel"
Expand Down Expand Up @@ -99,7 +114,8 @@ UTF-8.
- **Re-insert the source chapter for every review** when it has
changed; do not copy it out of an old export.
- **Review large sets in slices** (e.g. 8-10 lessons per pass) when
the AI you use has a limited context window.
the AI you use has a limited context window - use `--split-size`
instead of manually cutting the export down.
- **Keep YAML as the default**; use JSON only when a tool explicitly
requires it.
- **No copy-paste of AI suggestions without cross-reading.** The AI
Expand Down
133 changes: 130 additions & 3 deletions scripts/export_set.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,17 @@
Usage:

python3 scripts/export_set.py <set-slug> [--lang de] [--format yaml|json] [--out PATH]
python3 scripts/export_set.py <set-slug> --split-size N [--lang de] [--format yaml|json]

``--split-size N`` splits a large set into multiple files of at most N
lessons each (e.g. for handing a course to an AI reviewer one digestible
chunk at a time instead of one huge file - a 115-lesson set has no business
being reviewed in one context window). Each part is self-contained: it
carries its own ``review_instructions`` copy plus ``part``/``of`` and
``total_lesson_count`` fields so a reviewer knows where the chunk sits in
the whole set. Incompatible with ``--out`` (a split export always writes
multiple files under ``exports/``, named
``<set-slug>-<lang>-<timestamp>-part<NN>-of-<MM>.<format>``).

``<set-slug>`` matches either a set id from the root manifest (e.g.
``fuehrerschein-uebung-from-de``) or the basename of a set path (e.g.
Expand Down Expand Up @@ -212,6 +223,52 @@ def build_export(set_slug: str, lang: str) -> dict:
}


def chunk_lessons(lesson_documents: list[dict], split_size: int) -> list[list[dict]]:
"""Split ``lesson_documents`` into consecutive chunks of at most
``split_size`` lessons each, preserving order. The last chunk may be
smaller. A ``split_size`` at or above the lesson count yields a single
chunk. Pure and independent of I/O so it is trivially unit-testable."""
if split_size < 1:
raise ValueError(f"split_size must be >= 1, got {split_size}")
return [
lesson_documents[start : start + split_size]
for start in range(0, len(lesson_documents), split_size)
] or [[]]


def build_export_parts(set_slug: str, lang: str, split_size: int) -> list[dict]:
"""Assemble one export payload PER CHUNK of at most ``split_size``
lessons, for AI review sessions that cannot fit an entire large set in
one context window (e.g. a 115-lesson set). Every part is
self-contained: it carries its own copy of ``review_instructions`` so
each file can be handed to a reviewer independently, plus ``part``/``of``
and ``total_lesson_count`` so the reviewer knows where a chunk sits in
the whole set. All parts share one ``generated_at`` timestamp (one
export run)."""
root_manifest = yaml.safe_load(ROOT_MANIFEST_PATH.read_text(encoding="utf-8"))
set_entry = resolve_set(root_manifest, set_slug, lang)
lesson_documents = load_lessons(REPO_ROOT / set_entry["path"])
generated_at = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
engine_version = ENGINE_VERSION_PATH.read_text(encoding="utf-8").strip()
review_instructions = load_review_instructions()
chunks = chunk_lessons(lesson_documents, split_size)
return [
{
"review_instructions": review_instructions,
"set": set_slug,
"language": lang,
"engine_version": engine_version,
"generated_at": generated_at,
"part": part_index + 1,
"of": len(chunks),
"lesson_count": len(chunk),
"total_lesson_count": len(lesson_documents),
"lessons": chunk,
}
for part_index, chunk in enumerate(chunks)
]


def render_export(export_payload: dict, export_format: str) -> str:
"""Serialize the payload; real UTF-8 in both formats, keys in insert order,
multi-line strings (notably ``review_instructions``) as YAML block scalars."""
Expand All @@ -233,6 +290,20 @@ def default_output_path(set_slug: str, lang: str, export_format: str) -> Path:
return EXPORTS_DIR / f"{set_slug}-{lang}-{file_timestamp}.{export_format}"


def default_split_output_path(
set_slug: str, lang: str, export_format: str, part: int, of: int, file_timestamp: str
) -> Path:
"""Return ``exports/<set-slug>-<lang>-<timestamp>-part<NN>-of-<MM>.<format>``,
one file per chunk of a ``--split-size`` export. ``file_timestamp`` is
computed ONCE by the caller and shared across every part of one export
run, so the parts sort together and are recognisable as one batch."""
width = max(2, len(str(of)))
return (
EXPORTS_DIR
/ f"{set_slug}-{lang}-{file_timestamp}-part{part:0{width}d}-of-{of}.{export_format}"
)


def parse_arguments(argv: list[str] | None) -> argparse.Namespace:
argument_parser = argparse.ArgumentParser(
description=(
Expand All @@ -259,13 +330,23 @@ def parse_arguments(argv: list[str] | None) -> argparse.Namespace:
)
argument_parser.add_argument(
"--out",
help="output file path (default: exports/<set-slug>-<lang>-<timestamp>.<format>)",
help="output file path (default: exports/<set-slug>-<lang>-<timestamp>.<format>). "
"Incompatible with --split-size (which writes multiple files).",
)
argument_parser.add_argument(
"--split-size",
type=int,
default=None,
help="split the export into multiple files of at most N lessons each, "
"e.g. for AI review sessions that cannot fit a large set in one "
"context window. Each file is self-contained (its own "
"review_instructions) and carries part/of/total_lesson_count. "
"Default: one file for the whole set. Incompatible with --out.",
)
return argument_parser.parse_args(argv)


def main(argv: list[str] | None = None) -> int:
cli_arguments = parse_arguments(argv)
def _write_single_export(cli_arguments: argparse.Namespace) -> int:
try:
export_payload = build_export(cli_arguments.set_slug, cli_arguments.lang)
except SetResolutionError as resolution_error:
Expand All @@ -290,5 +371,51 @@ def main(argv: list[str] | None = None) -> int:
return 0


def _write_split_export(cli_arguments: argparse.Namespace) -> int:
try:
export_parts = build_export_parts(
cli_arguments.set_slug, cli_arguments.lang, cli_arguments.split_size
)
except (SetResolutionError, ValueError) as build_error:
print(f"ERROR: {build_error}", file=sys.stderr)
return 2

EXPORTS_DIR.mkdir(parents=True, exist_ok=True)
file_timestamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
of = len(export_parts)
for export_payload in export_parts:
output_path = default_split_output_path(
cli_arguments.set_slug,
cli_arguments.lang,
cli_arguments.export_format,
export_payload["part"],
of,
file_timestamp,
)
output_path.write_text(
render_export(export_payload, cli_arguments.export_format), encoding="utf-8"
)
print(
f"Exported part {export_payload['part']}/{of} "
f"({export_payload['lesson_count']} of {export_payload['total_lesson_count']} "
f"lessons) of set '{export_payload['set']}' to {output_path}"
)
return 0


def main(argv: list[str] | None = None) -> int:
cli_arguments = parse_arguments(argv)
if cli_arguments.split_size is not None and cli_arguments.out:
print(
"ERROR: --out cannot be combined with --split-size "
"(a split export writes multiple files)",
file=sys.stderr,
)
return 2
if cli_arguments.split_size is not None:
return _write_split_export(cli_arguments)
return _write_single_export(cli_arguments)


if __name__ == "__main__":
sys.exit(main())
Loading
Loading