feat(game): deep-embed GAME ggml transcription (M2-M4 + desktop Vulkan)

.agent/DECISIONS.md

@@ -1,4 +1,4 @@
-# Decisions Log
+# Decisions Log
 
 Record meaningful technical decisions here. Use one entry per decision.
 
@@ -53,6 +53,53 @@ Record meaningful technical decisions here. Use one entry per decision.
 - Alternatives considered: Give the parameter panel a second magnifier; change the magnifier source to the parameter canvas; reuse pitch events while applying fixed row offsets.
 - Impacted areas: Parameter-curve pointer lifecycle, phoneme-parameter panel event forwarding, and editor magnifier coordinate routing. Existing pitch editing and magnifier source selection are unchanged.
 
+- Date: 2026-08-24
+- Decision: M4 — wire GAME transcription into the Mobile layer by subclassing Core's `MidiExtractor<GameOptions>` (`GameGgmlMidiExtractor`), replacing only `TranscribeWaveform` to call our `GameGgml.Infer`; add an EditorMore "Transcribe Audio" action + progress popup.
+- Rationale: Core's base MidiExtractor already owns the battle-tested orchestration (mono conversion, 44.1k resample, AudioSlicer chunking, tick conversion, UVoicePart assembly). Subclassing lets us reuse all of it without touching Core, swapping only the inference backend ONNX→ggml C ABI. This is consistent with the "don't modify Core" constraint (read-only reuse).
+- Alternatives considered: A standalone mobile-side transcriber duplicating mono/resample/part-assembly (more code, risk of subtle timing divergence from Core); modifying Core to swap backend (violates constraint).
+- Impacted areas: OpenUtauMobile/Services/Game/GameGgmlMidiExtractor.cs; EditorMoreAction enum; EditorMorePopup.axaml; EditorViewModel; 5 resx files; M4 UI flow.
+
+- Date: 2026-08-24
+- Decision: (SUPERSEDED below) M3 originally bundled the Q8 GAME weights directly into the shared project as an EmbeddedResource copied from a checked-in `Models/game_medium.gguf`. This proved unworkable for repo hygiene (55 MB binary in git), so it was replaced by the build-time `.oudep` extraction decision immediately below. Net effect on impacted areas unchanged: model still ships with the app, still materialized to `LocalApplicationData/game/` at runtime, same `GameModelResolver` code path.
+- Rationale: See superseding decision below.
+- Alternatives considered: AndroidAsset + platform copy; runtime download.
+- Impacted areas: OpenUtauMobile.csproj, Services/Game/*, M4 wiring, M5 packaging/APK size.
+
+- Date: 2026-08-24
+- Decision: Do NOT commit the model binary. Extract `game_medium.gguf` at **build time** from the local game.cpp release `.oudep` (zip; `J:\GGML-GAME\vendor\game-cpp-release\game_ggml-windows-x64-vulkan-q8.oudep`) into `obj/` and register that copy as an EmbeddedResource so the shared assembly still carries the model. If the local `.oudep` is absent, download it from the pinned GitHub release URL (HEAD-verified 200) via `tools/extract_game_model.ps1` (property `GameOudepLocalPath`/`GameOudepUrl`/`GameModelLogicalName` + `EnsureGameEmbeddedModel` MSBuild target). Delete the `Models/` directory. Desktop C# smoke verified: deleting the local model reproduces a fresh unpack from the assembly (57.7 MB) — packaging chain is complete.
+- Rationale: The app needs the model in the assembly for zero-network installs, but a 57 MB binary doesn't belong in git. Building from the release `.oudep` recovers it deterministically and reproducibly without a checked-in blob, keyed to the pinned game.cpp v0.1.3.
+- Alternatives considered (superseded): committing Models/game_medium.gguf into git; AndroidAsset-only packing; runtime download (needs network at first run, rejected).
+- Impacted areas: OpenUtauMobile.csproj (`EnsureGameEmbeddedModel` target + properties), `tools/extract_game_model.ps1` (new), Services/Game/GameModelResolver.cs, M5 packaging/APK size (~55 MB in APK).
+
+- Date: 2026-08-24
+- Decision: Use `.NET 10 [LibraryImport]` source-generated P/Invoke with explicit `EntryPoint = "game_capi_*"` for the 9 C exports; enable `<AllowUnsafeBlocks>true</AllowUnsafeBlocks>` in the shared project.
+- Rationale: LibraryImport is AOT/trim-safe on mobile, and sourcegen needs AllowUnsafeBlocks. Without `EntryPoint`, the generator resolves the native symbol from the C# method name (`Version`→`game_capi_version`), causing EntryPointNotFoundException at runtime — caught by the C# desktop smoke.
+- Alternatives considered: Classic DllImport (works but trimming/AOT riskier on Android); no-entry-point naming trick (would obscure intent).
+- Impacted areas: GameGgmlNative.cs, OpenUtauMobile/OpenUtauMobile.csproj, future Android interop.
+
+- Date: 2026-08-24
+- Decision: Installed user-scoped .NET SDK 10.0.300 (+10.0.400 via channel) to `%USERPROFILE%\.dotnet` using the official dotnet-install script (no admin), because the machine only had .NET 8.0.424/9.0.317 while global.json pins 10.0.300.
+- Rationale: Without the matching SDK the shared project (net10.0) cannot compile at all, blocking M3+; the user asked to continue M3. User-scoped install avoids admin and Program Files writes.
+- Alternatives considered: Relaxing global.json to accept 9.0.x (breaks the .NET 10 migration decision); changing global.json band to 10.0.4xx (premature — upstream pins 10.0.300).
+- Impacted areas: Local toolchain, all future `dotnet build` commands must use `%USERPROFILE%\.dotnet\dotnet.exe` or PATH.
+
+- Date: 2026-08-24
+- Decision: Configure all Android ABIs through `native/CMakeUserPresets.json` + `cmake --preset`, with NDK toolchain / ninja / all `-D` (ANDROID_PLATFORM=24, ANDROID_SUPPORT_FLEXIBLE_PAGE_SIZES=OFF, GGML_NATIVE=OFF, GGML_OPENMP=OFF, BUILD_SHARED_LIBS=OFF, GGML_BACKEND_DL=OFF, GAME_GGML_LLAMAFILE=OFF, accelerators OFF) baked into one hidden `android-common` preset; `--fresh` + retry loop around each configure.
+- Rationale: PowerShell 5.1 passing `-DCMAKE_TOOLCHAIN_FILE=...` on the cmake command line split/quoted args, producing empty CMAKE_MAKE_PROGRAM and broken configure. The preset route writes every option as cache variables in one validated block, is reproducible, and avoids fragile array/quoting.
+- Alternatives considered: Start-Process -ArgumentList array; temporary .bat with quoted args — all more fragile than a checked-in preset.
+- Impacted areas: native/CMakeUserPresets.json; build-android-{arm64,arm,x86,x64}; OpenUtauMobile.Android/Libs/<abi>/libgame_ggml_shared.so (4 files); HANDOFF.md.
+
+- Date: 2026-08-24
+- Decision: In ggml v0.19 use standard `BUILD_SHARED_LIBS=OFF` (not `GGML_SHARED`, which ggml v0.19 does not define; it was silently ignored). Also disable `GAME_GGML_LLAMAFILE` for all Android ABIs.
+- Rationale: `GGML_SHARED=OFF` had no effect — ggml still emitted shared libs, contradicting the static-link plan. Setting the real option `BUILD_SHARED_LIBS=OFF` (plus GGML_BACKEND_DL=OFF) makes ggml static (.a) and merges everything into a single self-contained game_ggml_shared.so whose NEEDED is only libc/libm/libdl. LLAMAFILE must be off for armeabi-v7a (ARMv7 lacks the FP16 NEON intrinsics vld1q_f16/vld1_f16 used by llamafile sgemm.cpp), and off everywhere keeps the 4 ABIs consistent and smaller.
+- Alternatives considered: Keep GGML_SHARED=OFF; build shared ggml and ship the extra .so.
+- Impacted areas: native/CMakeUserPresets.json; all Android build outputs and the delivered .so; version-consistent reproducible builds.
+
+- Date: 2026-08-22
+- Decision: Deep-embed KakaruHayate/game.cpp (GAME ggml inference) into OpenUtauMobile as a `native/` submodule plus a C ABI shim, replacing the ONNX-based GAME path entirely; no plugin/switching framework, no subprocess, every platform ships the CPU backend with GPU-first runtime fallback (Vulkan/Metal/CUDA as compiled).
+- Rationale: ONNX Runtime GAME is far too heavy for mobile and effectively infeasible to ship; embedding game.cpp natively is the only viable path. The per-platform accelerator is chosen at build time (Metal on Apple, Vulkan/CUDA on desktop, Vulkan on Android) and `init_best_backend()` provides GPU→CPU fallback automatically, so a single C ABI (`game_capi`) is all the .NET layer needs. Keep the user's fork as the build baseline and git-submodule the upstream game.cpp pinned to release v0.1.3.
+- Alternatives considered: ONNX Runtime Electron/CLI subprocess (.oudep Executable) — rejected for iOS sandbox and mobile memory; plugin-switching framework — rejected as over-engineering; using only prebuilt release .oudep binaries — no long-term mobile control.
+- Impacted areas: New `native/` tree (game.cpp submodule + shim + CMake host); C# `GameGgml` integration code placed in the Mobile layer (not `OpenUtau.Core`); future Android/Windows/Linux/macOS/iOS ship artifacts; CI additions.
 - Date: 2026-08-22
 - Decision: Give each rendered note pitch-bend curve its own finalized `StreamGeometry` instead of submitting the shared mutable `Points` and `PolylineGeometry` caches.
 - Rationale: Every `RenderPitchBend` call cleared and repopulated the same point collection retained by earlier drawing commands, so the last visible note replaced the curves submitted for all preceding notes.

.github/workflows/build-all-platforms.yml

@@ -57,6 +57,9 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@v4
+        with:
+          submodules: recursive
+          persist-credentials: false
 
       - name: Setup .NET
         uses: actions/setup-dotnet@v4
@@ -186,6 +189,9 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@v4
+        with:
+          submodules: recursive
+          persist-credentials: false
 
       - name: Setup .NET
         uses: actions/setup-dotnet@v4
@@ -282,6 +288,9 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@v4
+        with:
+          submodules: recursive
+          persist-credentials: false
 
       - name: Setup .NET
         uses: actions/setup-dotnet@v4
@@ -386,6 +395,9 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@v4
+        with:
+          submodules: recursive
+          persist-credentials: false
 
       - name: Setup .NET
         uses: actions/setup-dotnet@v4

.gitignore

@@ -454,5 +454,19 @@ $RECYCLE.BIN/
 ## 质量分析
 analysis/
 
+# OpenUtauMobile native/ (game_capi shim) build artifacts
+native/build*/
+native/*/build*/
+native/**/_deps/
+native/**/CMakeCache.txt
+native/**/CMakeFiles/
+native/**/cmake_install.cmake
+native/**/CTestTestfile.cmake
+native/**/Testing/
+
+# Machine-specific CMake user presets (keep local absolute paths out of the repo)
+native/CMakeUserPresets.json
+
+
 ## codex
 .codex/

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "native/game.cpp"]
+	path = native/game.cpp
+	url = https://github.com/KakaruHayate/game.cpp

OpenUtauMobile/Assets/Lang/Strings.en.resx

@@ -1110,6 +1110,18 @@
   <data name="EditorMore.Toast.ExportAudio" xml:space="preserve">
     <value>TODO: ExportAudio</value>
   </data>
+  <data name="EditorMore.TranscribeAudio" xml:space="preserve">
+    <value>Transcribe Audio</value>
+  </data>
+  <data name="EditorMore.TranscribeProgress" xml:space="preserve">
+    <value>Transcribing audio…</value>
+  </data>
+  <data name="EditorMore.Toast.TranscribeDone" xml:space="preserve">
+    <value>Transcription complete</value>
+  </data>
+  <data name="EditorMore.Toast.TranscribeFailed" xml:space="preserve">
+    <value>Transcription failed</value>
+  </data>
   <data name="ExportAudioPopup.Title" xml:space="preserve">
     <value>Export Audio</value>
   </data>

OpenUtauMobile/Assets/Lang/Strings.ja.resx

@@ -1107,6 +1107,18 @@
   <data name="EditorMore.Toast.ExportAudio" xml:space="preserve">
     <value>TODO: オーディオをエクスポート</value>
   </data>
+  <data name="EditorMore.TranscribeAudio" xml:space="preserve">
+    <value>音声書き起こし</value>
+  </data>
+  <data name="EditorMore.TranscribeProgress" xml:space="preserve">
+    <value>音声書き起こし中…</value>
+  </data>
+  <data name="EditorMore.Toast.TranscribeDone" xml:space="preserve">
+    <value>書き起こし完了</value>
+  </data>
+  <data name="EditorMore.Toast.TranscribeFailed" xml:space="preserve">
+    <value>書き起こしに失敗しました</value>
+  </data>
   <data name="ExportAudioPopup.Title" xml:space="preserve">
     <value>オーディオを書き出し</value>
   </data>

OpenUtauMobile/Assets/Lang/Strings.ru.resx

@@ -1033,6 +1033,18 @@
   <data name="EditorMore.Toast.ExportAudio" xml:space="preserve">
     <value>TODO: Экспорт аудио</value>
   </data>
+  <data name="EditorMore.TranscribeAudio" xml:space="preserve">
+    <value>Транскрипция аудио</value>
+  </data>
+  <data name="EditorMore.TranscribeProgress" xml:space="preserve">
+    <value>Идёт транскрипция…</value>
+  </data>
+  <data name="EditorMore.Toast.TranscribeDone" xml:space="preserve">
+    <value>Транскрипция завершена</value>
+  </data>
+  <data name="EditorMore.Toast.TranscribeFailed" xml:space="preserve">
+    <value>Не удалось выполнить транскрипцию</value>
+  </data>
   <data name="ExportAudioPopup.Title" xml:space="preserve">
     <value>Экспорт аудио</value>
   </data>

OpenUtauMobile/Assets/Lang/Strings.uk.resx

@@ -1033,6 +1033,18 @@
   <data name="EditorMore.Toast.ExportAudio" xml:space="preserve">
     <value>TODO: Експорт аудіо</value>
   </data>
+  <data name="EditorMore.TranscribeAudio" xml:space="preserve">
+    <value>Транскрипція аудіо</value>
+  </data>
+  <data name="EditorMore.TranscribeProgress" xml:space="preserve">
+    <value>Виконується транскрипція…</value>
+  </data>
+  <data name="EditorMore.Toast.TranscribeDone" xml:space="preserve">
+    <value>Транскрипцію завершено</value>
+  </data>
+  <data name="EditorMore.Toast.TranscribeFailed" xml:space="preserve">
+    <value>Не вдалося виконати транскрипцію</value>
+  </data>
   <data name="ExportAudioPopup.Title" xml:space="preserve">
     <value>Експорт аудіо</value>
   </data>

OpenUtauMobile/Assets/Lang/Strings.zh-Hans.resx

@@ -1110,6 +1110,18 @@
   <data name="EditorMore.Toast.ExportAudio" xml:space="preserve">
     <value>TODO: 导出音频功能</value>
   </data>
+  <data name="EditorMore.TranscribeAudio" xml:space="preserve">
+    <value>音频转写</value>
+  </data>
+  <data name="EditorMore.TranscribeProgress" xml:space="preserve">
+    <value>正在转写音频…</value>
+  </data>
+  <data name="EditorMore.Toast.TranscribeDone" xml:space="preserve">
+    <value>转写完成</value>
+  </data>
+  <data name="EditorMore.Toast.TranscribeFailed" xml:space="preserve">
+    <value>转写失败</value>
+  </data>
   <data name="ExportAudioPopup.Title" xml:space="preserve">
     <value>导出音频</value>
   </data>

OpenUtauMobile/Controls/EditorMorePopup.axaml

@@ -1,4 +1,4 @@
-<UserControl xmlns="https://github.com/avaloniaui"
+<UserControl xmlns="https://github.com/avaloniaui"
              xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
              xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
              xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
@@ -68,6 +68,13 @@
                                 <vm:EditorMoreAction>ImportTrack</vm:EditorMoreAction>
                             </Button.CommandParameter>
                         </Button>
+                        <Button Classes="ActionBtn"
+                                Content="{DynamicResource EditorMore.TranscribeAudio}"
+                                Command="{Binding ConfirmCommand}">
+                            <Button.CommandParameter>
+                                <vm:EditorMoreAction>TranscribeAudio</vm:EditorMoreAction>
+                            </Button.CommandParameter>
+                        </Button>
                     </StackPanel>
                 </TabItem>
                 <!-- 导出 -->

OpenUtauMobile/OpenUtauMobile.csproj

@@ -1,8 +1,9 @@
-<Project Sdk="Microsoft.NET.Sdk">
+<Project Sdk="Microsoft.NET.Sdk">
     <PropertyGroup>
         <TargetFramework>net10.0</TargetFramework>
         <Nullable>enable</Nullable>
         <LangVersion>preview</LangVersion>
+        <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
         <AvaloniaUseCompiledBindingsByDefault>true</AvaloniaUseCompiledBindingsByDefault>
     </PropertyGroup>
 
@@ -20,6 +21,58 @@
         <AvaloniaResource Include="Assets\Lang\Strings.uk.resx"/>
     </ItemGroup>
 
+    <!--
+    GAME 模型（game_medium.gguf Q8 ~55MB）构建期来源：
+    * 不在仓库提交 55MB 二进制；从 game.cpp 官方 release 的 .oudep 里"扒"出来。
+    * 优先用本机 vendor 的 .oudep（免网络）；没有则从 GitHub release 下载到 obj 缓存。
+    * 解出的 gguf 由 EnsureGameEmbeddedModel 目标动态注册为 EmbeddedResource。
+    -->
+    <PropertyGroup>
+        <!-- 本地 vendor .oudep 存在才使用（CI runner 无该路径则留空 → 走下载） -->
+        <GameOudepLocalPath Condition="'$(GameOudepLocalPath)' == '' and Exists('J:\GGML-GAME\vendor\game-cpp-release\game_ggml-windows-x64-vulkan-q8.oudep')">J:\GGML-GAME\vendor\game-cpp-release\game_ggml-windows-x64-vulkan-q8.oudep</GameOudepLocalPath>
+        <GameOudepUrl Condition="'$(GameOudepUrl)' == ''">https://github.com/KakaruHayate/game.cpp/releases/download/v0.1.3/game_ggml-windows-x64-vulkan-q8.oudep</GameOudepUrl>
+        <GameOudepSha256 Condition="'$(GameOudepSha256)' == ''">895A85B56AAEEB15CD541469ACFA1AEDB2D3DC58B1A6890A2F8B6FB629AD990D</GameOudepSha256>
+        <GameModelLogicalName>OpenUtauMobile.Models.game_medium.gguf</GameModelLogicalName>
+    </PropertyGroup>
+
+    <!-- 构建期拉取模型：下载/拷贝 .oudep -> 解出 game_medium.gguf -> 动态加入 EmbeddedResource -->
+    <Target Name="EnsureGameEmbeddedModel" BeforeTargets="PrepareResourceNames;CoreCompile">
+        <PropertyGroup>
+            <GameOudepCache>$(IntermediateOutputPath)game.oudep</GameOudepCache>
+            <GameModelFile>$(IntermediateOutputPath)game_medium.gguf</GameModelFile>
+        </PropertyGroup>
+
+        <!-- 1) 已有解出的模型则跳过 -->
+        <MakeDir Directories="$(IntermediateOutputPath)"/>
+        <Message Condition="Exists('$(GameModelFile)')" Text="GAME 模型已存在: $(GameModelFile)"/>
+
+        <!-- 2) 没有模型：确保有 .oudep（优先本地 vendor，否则从 release 下载到 obj） -->
+        <Copy Condition="!Exists('$(GameModelFile)') and Exists('$(GameOudepLocalPath)') and '$(GameOudepLocalPath)' != ''"
+              SourceFiles="$(GameOudepLocalPath)" DestinationFiles="$(GameOudepCache)"/>
+        <DownloadFile Condition="!Exists('$(GameModelFile)') and !Exists('$(GameOudepCache)') and !Exists('$(GameOudepLocalPath)')"
+                      SourceUrl="$(GameOudepUrl)" DestinationFolder="$(IntermediateOutputPath)" DestinationFileName="game.oudep"/>
+        <Error Condition="!Exists('$(GameOudepCache)') and !Exists('$(GameModelFile)')"
+               Text="无法获得 GAME 模型的 .oudep（本地未找到且下载失败）。请先手动放置 $(GameOudepLocalPath) 或设置 GameOudepUrl"/>
+
+        <!-- 3) 校验 .oudep 哈希（固定到 release 资产，防篡改；本地 vendor 与下载均校验） -->
+        <GetFileHash Condition="Exists('$(GameOudepCache)')" Files="$(GameOudepCache)" Algorithm="SHA256">
+            <Output TaskParameter="Hash" PropertyName="GameOudepActualHash"/>
+        </GetFileHash>
+        <Error Condition="Exists('$(GameOudepCache)') and '$(GameOudepActualHash)' != '' and '$(GameOudepActualHash)' != '$(GameOudepSha256)'"
+               Text="GAME 模型 .oudep SHA-256 校验失败：预期 $(GameOudepSha256)，实测 $(GameOudepActualHash)"/>
+
+        <!-- 4) 从 .oudep 解出 game_medium.gguf（跨平台：pwsh + 正斜杠路径修正） -->
+        <Exec Condition="!Exists('$(GameModelFile)') and Exists('$(GameOudepCache)')"
+              Command="pwsh -NoProfile -ExecutionPolicy Bypass -File &quot;$(MSBuildProjectDirectory)/Tools/extract_game_model.ps1&quot; -Oudep &quot;$(GameOudepCache)&quot; -Target &quot;$(GameModelFile)&quot;"/>
+
+        <Error Condition="!Exists('$(GameModelFile)')" Text="解压 GAME 模型失败"/>
+
+        <!-- 5) 动态注册为嵌入式资源（只在 obj 下生成，不进 git） -->
+        <ItemGroup>
+            <EmbeddedResource Include="$(GameModelFile)" LogicalName="$(GameModelLogicalName)"/>
+        </ItemGroup>
+    </Target>
+
     <ItemGroup>
         <PackageReference Include="Avalonia" Version="$(AvaloniaVersion)"/>
         <PackageReference Include="Avalonia.Themes.Fluent" Version="$(AvaloniaVersion)"/>