Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions web-astro/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
node_modules/
dist/
.astro/

# Generated by tools/convert-book.mjs (RST→MDX whole-book conversion).
# Regenerate with: npm run convert (also runs automatically on predev/prebuild)
/src/content/docs/behavior/
/src/content/docs/components/
/src/content/docs/front/
/src/content/docs/advanced.mdx
/src/content/docs/todo.mdx
/tools/sidebar.gen.json
/src/content/docs/genindex.mdx
/public/figures/

# Generated by tools/build-locales.mjs, which applies the tracked gettext
# catalogs in text/locale/ onto the English MDX. Both inputs are in the repo,
# so these are cheap to rebuild: npm run convert
/src/content/docs/ar/
/src/content/docs/cn/
/src/content/docs/de/
/src/content/docs/es/
/src/content/docs/fr/
/src/content/docs/it/
/src/content/docs/kr/
/src/content/docs/pt_BR/
/i18n/po/

# NOT ignored, deliberately: public/plots/*.svg and src/cases/*.json.
# Both derive from text/results/ (DVC output) and text/plots/ (itself
# gitignored), so neither can be rebuilt from a clean clone without a DVC
# pull and a containerised simulation run. They are committed as artifacts.
56 changes: 56 additions & 0 deletions web-astro/FEATURE-MAP.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Sphinx → Astro/Starlight feature parity map

Every Sphinx feature the book actually uses (counted across `text/source/**/*.rst`)
mapped to its Astro + Starlight equivalent. Status legend:

- **Native** — built into Astro/Starlight, no extra work.
- **Plugin** — an npm remark/rehype plugin + config (not a fork).
- **Component** — an MDX component we author (like `SimFigure`).
- **Gap** — no drop-in; needs a custom plugin or a product decision. Flagged.

## Directives

| Sphinx directive | Uses | Astro/Starlight equivalent | Status | Notes |
|---|---:|---|---|---|
| `code-block` | 293 | Fenced ` ```lang ` via Expressive Code | **Native** | Shipped with Starlight. Modelica has no Shiki grammar → renders as plain text (add a TextMate grammar later; mystmd is no better). |
| `literalinclude` | 285 | `?raw` import + `<Code>` | **Native** | Demonstrated in the slice — code stays sourced from the `.mo` files. `:lines:`/`:emphasize-lines:` = a slice/mark helper. |
| `index` | 232 | Pagefind full-text search (built in) | **Gap→decision** | Starlight ships site search, which largely supersedes a hand-maintained index. A literal back-of-book index page would need a small remark plugin. Recommend: drop in favor of search unless a print index is required. |
| `math` (block) / `:math:` (inline) | 106 / 404 | `remark-math` + `rehype-katex` | **Plugin** | Wired in the slice; renders via KaTeX. |
| `plot` + `:class: interactive` | 104 | `<SimFigure>` MDX component | **Component** | The bake-off centerpiece — native MDX component, no fork. |
| `figure` | 80 | Markdown image / `<Figure>` component | **Native** | Rendering is native; **auto-numbering** ties to the cross-ref gap below. |
| `toctree` | 21 | Starlight `sidebar` (manual or `autogenerate`) | **Native** | Different model — nav is site config, not in-content. |
| `topic` | 18 | `:::note` / `<Aside>` | **Native** | Starlight asides. |
| `todo` / `todolist` | 18 / 1 | — | **Decision** | Author-only tooling. Recommend dropping (or a remark comment stripped at build). |
| `note` | 4 | `:::note` aside | **Native** | |
| `only` | 3 | MDX conditional / build flag | **Native** | MDX handles conditional content directly. |
| `raw` | 2 | Inline HTML in MDX | **Native** | Astro passes HTML through (no sanitization). |
| `ifconfig` | 1 | MDX conditional | **Native** | |

## Roles

| Sphinx role | Uses | Astro/Starlight equivalent | Status | Notes |
|---|---:|---|---|---|
| `:math:` | 404 | `remark-math` inline `$...$` | **Plugin** | Same plugin as block math. |
| `:ref:` | 310 | Cross-references via `plugins/remark-xref.mjs` | **Resolved (plugin built)** | The custom remark plugin now: records `{/* #label */}` heading anchors (RST labels), numbers sections + figures, and resolves empty-text `[](#label)` refs to the target **title** (Sphinx `:ref:` default) or "Figure N". Verified: `[](#first-order-doc)`→"Adding Some Documentation", `[](#FO)`→"Figure 1". Remaining scale-up: persist the label map across pages for cross-**document** refs (trivial extension). |

## Cross-cutting

| Concern | Astro/Starlight | Status |
|---|---|---|
| Site search | Pagefind (built in) | **Native** |
| Sidebar / nav / prev-next | Starlight built-ins | **Native** |
| Syntax highlighting | Expressive Code (Shiki) | **Native** (no Modelica grammar) |
| Dark/light mode | Starlight built-in + our `data-mode` tokens | **Native** |
| i18n / translations | po4a bridge to gettext (see `i18n/`) | **Tooling** — see `i18n/README.md` |
| PDF / ebook export | Pandoc (see `print/`) | **Resolved (scaffolded)** — not Astro's job; the same Markdown source → Pandoc → PDF (Typst/xelatex) + EPUB. `print/build-print-md.mjs` lowers MDX components to print form (interactive figure → static plot). Print toolchain is a dependency exactly as xelatex is today (MIC-85). |

## Summary

- **Native or one-plugin-install:** code, literalinclude, math, figures, admonitions, toctree/nav, search, conditional content, raw HTML — the bulk of the book.
- **Author once as a component:** interactive figures (done).
- **Previously-open items, now closed:**
1. **`:ref:` cross-references + auto-numbering** (310 uses) — `plugins/remark-xref.mjs`, built + verified.
2. **PDF / eBook** — Pandoc pipeline scaffolded in `print/` (MDX lowered to print Markdown; components → static figures).
3. **i18n** — the po4a bridge (`i18n/`), reusing existing translations with zero re-translation.
- **Decisions, not work:** `index` (→ search), `todo` (→ drop).
- **Remaining true gap:** none of the *used* features. The only scale-ups are cross-**document** ref numbering (persist the label map across pages) and a Modelica syntax grammar.
79 changes: 79 additions & 0 deletions web-astro/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# `web-astro/` — Astro + Starlight slice (MIC-84 bake-off)

The same first-order chapter as the mystmd slice (`../web`), rebuilt on **Astro +
Starlight** so the two platforms can be compared head-to-head — tracked in
[MIC-126](https://linear.app/mtiller/issue/MIC-126).

## Quick start

```bash
cd web-astro
npm install
npm run dev # http://localhost:4321
npm run build # static site into dist/
```

Node 20+ (CI uses 26).

## Why this exists / what it proves

The open question from the mystmd slice was the **interactive figure**. In mystmd
it can't be a drop-in component (no MDX; the "renderer" plugin type is
unimplemented; the theme sanitizes raw HTML), so it needed either a theme fork or
a post-build injection hack.

In Astro it's a **native MDX component** — the thing you wanted:

```mdx
import SimFigure from '../../components/SimFigure.astro';

<SimFigure id="FO" interactive caption="Simulation of FirstOrder" />
```

Verified in the built HTML (`dist/first-order/index.html`):

- the structured contract rides straight through as `data-case` on the figure
(Astro does **not** sanitize component output — this is precisely what mystmd
stripped);
- the island `<script>` is **bundled and shipped by Astro** (no injection step);
- all four static plots, KaTeX math, and Modelica code sourced from the actual
`.mo` files via `?raw` import (single source of truth, parity with
`literalinclude`).

**No theme fork. No post-build step. No sanitizer fight.** Additional figure or
island types = more components you import — upstream is never touched.

## Layout

| Path | Purpose |
|------|---------|
| `astro.config.mjs` | Starlight + math plugins (`remark-math`/`rehype-katex`) + `?raw` fs access |
| `src/content/docs/first-order.mdx` | The converted chapter (MDX; imports the figure component + `.mo` code) |
| `src/components/SimFigure.astro` | The interactive figure island (Jony's skeleton; `data-skin`×`data-mode` skinnable) |
| `src/cases/*.json` + `index.ts` | The structured figure contract (from `_generate_casedata`, MIC-87) |
| `src/styles/skins.css` | Jony's A/B/C token system + structural CSS |
| `public/plots/*.svg` | Static plots (from the sim pipeline; fetched from the live site for the slice) |

## Native-component win vs. plugin-fill gaps

Native / out-of-the-box (no forks):

- MDX custom components (the island) — the decisive win.
- Math — `remark-math` + `rehype-katex` (npm install + config).
- Code from source files — `?raw` import + Starlight `<Code>`.
- Skinning — CSS custom properties (`data-skin` × `data-mode`).
- i18n, search (Pagefind), sidebar/nav — Starlight built-ins.

Gaps vs. mystmd's native scientific authoring (bridgeable, but real):

- **Auto figure/equation numbering + `:ref:`-style cross-references.** mystmd does
this natively; here it needs a remark plugin (a build-time plugin, not a fork).
The heaviest known gap — validate before full migration.
- **Modelica syntax highlighting.** Shiki has no Modelica grammar, so code is
rendered as plain text (`lang="text"`); mystmd is no better here. A TextMate
grammar can be added later.

## Deferred (same as the mystmd slice)

Bulk chapter conversion; i18n (see the PR for the translation analysis); live sim
API (MIC-86); final visual design pick.
53 changes: 53 additions & 0 deletions web-astro/astro.config.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
// @ts-check
import { defineConfig } from 'astro/config';
import starlight from '@astrojs/starlight';
import remarkMath from 'remark-math';
import rehypeKatex from 'rehype-katex';
import remarkModelicaXref from './plugins/remark-xref.mjs';
import bookSidebar from './tools/sidebar.gen.json' with { type: 'json' };
import modelicaGrammar from './src/langs/modelica.tmLanguage.json' with { type: 'json' };

// MIC-84 bake-off: Astro + Starlight rendering of the first-order chapter.
// Contrast with the mystmd slice in ../web: here the interactive figure is a
// first-class Astro island (a component imported in MDX + a bundled <script>) —
// no theme fork, no post-build injection, no sanitizer fight.
export default defineConfig({
// Math via remark/rehype plugins — an npm install + config, not a theme fork.
// This is the "extend via plugins" story that replaces mystmd's directive gap.
markdown: {
remarkPlugins: [remarkModelicaXref, remarkMath],
rehypePlugins: [rehypeKatex],
},
// Allow importing the .mo sources from the repo root via ?raw, so code blocks
// stay sourced from the single source of truth (parity with literalinclude).
vite: { server: { fs: { allow: ['..'] } } },
integrations: [
starlight({
title: 'Modelica by Example',
description: 'Astro + Starlight vertical slice (MIC-84)',
customCss: ['katex/dist/katex.min.css', './src/styles/skins.css'],
// Register a Modelica TextMate grammar so code blocks highlight (MIC-128).
// Shiki has no built-in Modelica lexer; this restores the old site's
// syntax highlighting for .mo listings and `modelica` fenced blocks.
expressiveCode: { shiki: { langs: [modelicaGrammar] } },
// Foldable Modelica annotations, progressive enhancement (MIC-129).
head: [{ tag: 'script', attrs: { src: '/annotation-fold.js', defer: true } }],
// All available languages. Root = English; other locales fall back to the
// English page when a translated MDX is absent (MIC-130). Keys match the
// gettext locale dirs so src/content/docs/<key>/… maps to /<key>/….
defaultLocale: 'root',
locales: {
root: { label: 'English', lang: 'en' },
cn: { label: '中文', lang: 'zh' },
kr: { label: '한국어', lang: 'ko' },
ar: { label: 'العربية', lang: 'ar', dir: 'rtl' },
de: { label: 'Deutsch', lang: 'de' },
es: { label: 'Español', lang: 'es' },
fr: { label: 'Français', lang: 'fr' },
it: { label: 'Italiano', lang: 'it' },
pt_BR: { label: 'Português (BR)', lang: 'pt-BR' },
},
sidebar: bookSidebar,
}),
],
});
89 changes: 89 additions & 0 deletions web-astro/i18n/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Translations — reusing the existing gettext catalogs (MIC-92)

**Goal:** move content to MDX **without** re-translating anything and **without**
switching to a hand-maintained parallel tree of translated Markdown files. Keep
the current gettext model — one English source + per-language `.po` overlays,
string-delta translations, fuzzy-marking on change — and just retarget it at
Markdown.

## Why the naive path fails

Starlight's *native* i18n is a parallel per-locale file tree — you'd maintain a
full translated `.mdx` per language. That's a real regression from gettext.

And you can't drop the existing `.po` files in unchanged: they were extracted by
**Sphinx from RST**, so the `msgid`s (and many `msgstr`s) contain RST markup —
`` :math:`x` ``, double-backtick ``` ``literal`` ```, `` :ref:`label` ``, RST
`\ ` null-escapes. po4a extracts from **Markdown**, whose msgids use `$x$`,
`` `code` ``, `[text](#label)`. Mismatched msgids ⇒ everything looks untranslated.

## The two-part solution

### 1. One-time migration — `migrate-gettext.mjs`

Rewrites the RST markup inside each `.po` (both `msgid` and `msgstr`) to the
Markdown forms po4a expects. Prose units are already identical and pass through
untouched; marked-up units get re-keyed; **all translated text is preserved.**

```bash
# one file
node i18n/migrate-gettext.mjs ../text/locale/cn/LC_MESSAGES/behavior/equations/first_order.po \
--out i18n/po/cn/first-order.po --stats

# whole tree (example)
for f in $(find ../text/locale -name '*.po'); do
lang=$(echo "$f" | sed -E 's#.*/locale/([^/]+)/.*#\1#')
base=$(basename "$f" .po)
mkdir -p "i18n/po/$lang"
node i18n/migrate-gettext.mjs "$f" --out "i18n/po/$lang/$base.po"
done
```

**Measured on `cn/first_order.po`** (a fully-translated chapter, 35 units):

| | count |
|---|---:|
| translation units | 35 |
| already-matching prose units (untouched) | 20 |
| marked-up msgids re-keyed to MD | 15 |
| msgstrs with RST markup rewritten to MD | 25 |
| **human re-translations required** | **0** |

Example: `` 变量\ :math:`x`\ 。 `` → `变量$x$。`, and `` ``model`` `` → `` `model` ``.

### 2. Reconcile + go forward — po4a (`po4a.cfg`)

After migration, po4a owns the lifecycle against the MDX source:

```bash
po4a --no-translations i18n/po4a.cfg # refresh .pot/.po from English MDX
po4a i18n/po4a.cfg # emit translated MDX for Starlight
```

`po4a`/`msgmerge` aligns the migrated `.po` with the freshly-extracted `.pot`.
Exact msgid matches reuse silently; near-misses become **fuzzy** (translation is
already present — a one-click confirm, not a re-translation). The generated
`src/content/docs/<lang>/*.mdx` are build artifacts (gitignore them); Starlight
serves them as its locale tree.

## Honest caveats (the "fuzzy", not "re-translate", bucket)

- **`:ref:` with translated link text.** The demo transform mangled one case
(`[](#变量\\)`). Cross-references are also the biggest feature-parity gap
(`FEATURE-MAP.md`) and need a custom remark plugin; that plugin should define
the canonical MD cross-ref syntax, and `rstToMd()` should target it. Until
then these surface as fuzzies.
- **Segmentation granularity.** Sphinx and po4a segment paragraphs almost
identically, but lists/tables/admonitions can differ. Mismatches show up as
fuzzies on first `msgmerge`, never as lost translations.
- **`migrate-gettext.mjs` is a scaffold**, proven on `first_order`. Before a full
run, spot-check a couple of languages and tighten the `:ref:` rule alongside
the cross-ref plugin.

## Net

The existing human translation work is **fully reused** — zero re-translation.
The cost is a one-time mechanical migration + a fuzzy-confirm pass, after which
the team keeps working in exactly today's gettext `.po` workflow, now targeting
Markdown. This is **platform-independent** across MDX frameworks (Docusaurus,
Astro, Nextra) — po4a is the bridge regardless.
52 changes: 52 additions & 0 deletions web-astro/i18n/VERIFICATION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# i18n pipeline verification (MIC-130)

End-to-end verification that the Astro build reuses the existing gettext
catalogs — no re-translation, no parallel hand-maintained MDX tree.

## Result: verified

- **All 8 locales build and route.** `dist/{cn,kr,ar,de,es,fr,it,pt_BR}/…` render;
`<html lang>` is set per locale (e.g. `ko`, `zh`), Arabic is `dir="rtl"`.
- **Translated pages come from reused `.po` data**, applied onto the English MDX
via `migrate-gettext.mjs` (RST-shaped → Markdown-shaped) + `apply-translation.mjs`.
Chinese and Korean each render ~91 translated chapters; sparse locales fall back
to English (Starlight i18n fallback).
- **Math, code-from-`.mo`, figures, and cross-references render identically** in
translated pages (verified on `kr/behavior/arrays/state-space`).

## Reuse per language (full-corpus dry run)

| Lang | Translated units | Reuse ceiling |
|------|-----------------:|--------------:|
| cn | 2,659 | 89% |
| kr | 2,422 | 78% |
| es | 502 | 17% |
| ar | 255 | 8% |
| pt_BR| 31 | 1% |
| fr/it| 7 each | ~0% |
| de | 0 | 0% |

Only cn/kr are substantially translated **in the source catalogs**; the rest is a
pre-existing content gap (see MIC-137), not a conversion loss. ~99% of *existing*
translated units migrate cleanly; residuals (footnotes, multi-line hyperlinks) are
handled by the migrator rules (MIC-131).

## Repeatable command sequence

```bash
# whole book, all locales (also runs on npm run build via prebuild):
npm run convert # convert-book.mjs + build-cases.mjs + build-locales.mjs

# one chapter, one language, by hand:
node i18n/migrate-gettext.mjs ../text/locale/kr/LC_MESSAGES/behavior/equations/first_order.po --out i18n/po/kr/first-order.po
node i18n/apply-translation.mjs src/content/docs/behavior/equations/first-order.mdx i18n/po/kr/first-order.po src/content/docs/kr/behavior/equations/first-order.mdx
```

`build-locales.mjs` skips catalogs with zero real translations, so Starlight
serves the English page under the localized route for those.

## Coverage measured on the built site

- Korean: ~74–76% of prose blocks are Korean (near the 78% catalog ceiling); the
remainder is untranslated **in the source catalog** (MIC-137 part 1).
- No page fails to build in any locale.
Loading