diff --git a/.agent/DECISIONS.md b/.agent/DECISIONS.md index 126605fd..ec56fe87 100644 --- a/.agent/DECISIONS.md +++ b/.agent/DECISIONS.md @@ -1,4 +1,4 @@ -# Decisions Log +# Decisions Log Record meaningful technical decisions here. Use one entry per decision. @@ -53,6 +53,53 @@ Record meaningful technical decisions here. Use one entry per decision. - Alternatives considered: Give the parameter panel a second magnifier; change the magnifier source to the parameter canvas; reuse pitch events while applying fixed row offsets. - Impacted areas: Parameter-curve pointer lifecycle, phoneme-parameter panel event forwarding, and editor magnifier coordinate routing. Existing pitch editing and magnifier source selection are unchanged. +- Date: 2026-08-24 +- Decision: M4 — wire GAME transcription into the Mobile layer by subclassing Core's `MidiExtractor` (`GameGgmlMidiExtractor`), replacing only `TranscribeWaveform` to call our `GameGgml.Infer`; add an EditorMore "Transcribe Audio" action + progress popup. +- Rationale: Core's base MidiExtractor already owns the battle-tested orchestration (mono conversion, 44.1k resample, AudioSlicer chunking, tick conversion, UVoicePart assembly). Subclassing lets us reuse all of it without touching Core, swapping only the inference backend ONNX→ggml C ABI. This is consistent with the "don't modify Core" constraint (read-only reuse). +- Alternatives considered: A standalone mobile-side transcriber duplicating mono/resample/part-assembly (more code, risk of subtle timing divergence from Core); modifying Core to swap backend (violates constraint). +- Impacted areas: OpenUtauMobile/Services/Game/GameGgmlMidiExtractor.cs; EditorMoreAction enum; EditorMorePopup.axaml; EditorViewModel; 5 resx files; M4 UI flow. + +- Date: 2026-08-24 +- Decision: (SUPERSEDED below) M3 originally bundled the Q8 GAME weights directly into the shared project as an EmbeddedResource copied from a checked-in `Models/game_medium.gguf`. This proved unworkable for repo hygiene (55 MB binary in git), so it was replaced by the build-time `.oudep` extraction decision immediately below. Net effect on impacted areas unchanged: model still ships with the app, still materialized to `LocalApplicationData/game/` at runtime, same `GameModelResolver` code path. +- Rationale: See superseding decision below. +- Alternatives considered: AndroidAsset + platform copy; runtime download. +- Impacted areas: OpenUtauMobile.csproj, Services/Game/*, M4 wiring, M5 packaging/APK size. + +- Date: 2026-08-24 +- Decision: Do NOT commit the model binary. Extract `game_medium.gguf` at **build time** from the local game.cpp release `.oudep` (zip; `J:\GGML-GAME\vendor\game-cpp-release\game_ggml-windows-x64-vulkan-q8.oudep`) into `obj/` and register that copy as an EmbeddedResource so the shared assembly still carries the model. If the local `.oudep` is absent, download it from the pinned GitHub release URL (HEAD-verified 200) via `tools/extract_game_model.ps1` (property `GameOudepLocalPath`/`GameOudepUrl`/`GameModelLogicalName` + `EnsureGameEmbeddedModel` MSBuild target). Delete the `Models/` directory. Desktop C# smoke verified: deleting the local model reproduces a fresh unpack from the assembly (57.7 MB) — packaging chain is complete. +- Rationale: The app needs the model in the assembly for zero-network installs, but a 57 MB binary doesn't belong in git. Building from the release `.oudep` recovers it deterministically and reproducibly without a checked-in blob, keyed to the pinned game.cpp v0.1.3. +- Alternatives considered (superseded): committing Models/game_medium.gguf into git; AndroidAsset-only packing; runtime download (needs network at first run, rejected). +- Impacted areas: OpenUtauMobile.csproj (`EnsureGameEmbeddedModel` target + properties), `tools/extract_game_model.ps1` (new), Services/Game/GameModelResolver.cs, M5 packaging/APK size (~55 MB in APK). + +- Date: 2026-08-24 +- Decision: Use `.NET 10 [LibraryImport]` source-generated P/Invoke with explicit `EntryPoint = "game_capi_*"` for the 9 C exports; enable `true` in the shared project. +- Rationale: LibraryImport is AOT/trim-safe on mobile, and sourcegen needs AllowUnsafeBlocks. Without `EntryPoint`, the generator resolves the native symbol from the C# method name (`Version`→`game_capi_version`), causing EntryPointNotFoundException at runtime — caught by the C# desktop smoke. +- Alternatives considered: Classic DllImport (works but trimming/AOT riskier on Android); no-entry-point naming trick (would obscure intent). +- Impacted areas: GameGgmlNative.cs, OpenUtauMobile/OpenUtauMobile.csproj, future Android interop. + +- Date: 2026-08-24 +- Decision: Installed user-scoped .NET SDK 10.0.300 (+10.0.400 via channel) to `%USERPROFILE%\.dotnet` using the official dotnet-install script (no admin), because the machine only had .NET 8.0.424/9.0.317 while global.json pins 10.0.300. +- Rationale: Without the matching SDK the shared project (net10.0) cannot compile at all, blocking M3+; the user asked to continue M3. User-scoped install avoids admin and Program Files writes. +- Alternatives considered: Relaxing global.json to accept 9.0.x (breaks the .NET 10 migration decision); changing global.json band to 10.0.4xx (premature — upstream pins 10.0.300). +- Impacted areas: Local toolchain, all future `dotnet build` commands must use `%USERPROFILE%\.dotnet\dotnet.exe` or PATH. + +- Date: 2026-08-24 +- Decision: Configure all Android ABIs through `native/CMakeUserPresets.json` + `cmake --preset`, with NDK toolchain / ninja / all `-D` (ANDROID_PLATFORM=24, ANDROID_SUPPORT_FLEXIBLE_PAGE_SIZES=OFF, GGML_NATIVE=OFF, GGML_OPENMP=OFF, BUILD_SHARED_LIBS=OFF, GGML_BACKEND_DL=OFF, GAME_GGML_LLAMAFILE=OFF, accelerators OFF) baked into one hidden `android-common` preset; `--fresh` + retry loop around each configure. +- Rationale: PowerShell 5.1 passing `-DCMAKE_TOOLCHAIN_FILE=...` on the cmake command line split/quoted args, producing empty CMAKE_MAKE_PROGRAM and broken configure. The preset route writes every option as cache variables in one validated block, is reproducible, and avoids fragile array/quoting. +- Alternatives considered: Start-Process -ArgumentList array; temporary .bat with quoted args — all more fragile than a checked-in preset. +- Impacted areas: native/CMakeUserPresets.json; build-android-{arm64,arm,x86,x64}; OpenUtauMobile.Android/Libs//libgame_ggml_shared.so (4 files); HANDOFF.md. + +- Date: 2026-08-24 +- Decision: In ggml v0.19 use standard `BUILD_SHARED_LIBS=OFF` (not `GGML_SHARED`, which ggml v0.19 does not define; it was silently ignored). Also disable `GAME_GGML_LLAMAFILE` for all Android ABIs. +- Rationale: `GGML_SHARED=OFF` had no effect — ggml still emitted shared libs, contradicting the static-link plan. Setting the real option `BUILD_SHARED_LIBS=OFF` (plus GGML_BACKEND_DL=OFF) makes ggml static (.a) and merges everything into a single self-contained game_ggml_shared.so whose NEEDED is only libc/libm/libdl. LLAMAFILE must be off for armeabi-v7a (ARMv7 lacks the FP16 NEON intrinsics vld1q_f16/vld1_f16 used by llamafile sgemm.cpp), and off everywhere keeps the 4 ABIs consistent and smaller. +- Alternatives considered: Keep GGML_SHARED=OFF; build shared ggml and ship the extra .so. +- Impacted areas: native/CMakeUserPresets.json; all Android build outputs and the delivered .so; version-consistent reproducible builds. + +- Date: 2026-08-22 +- Decision: Deep-embed KakaruHayate/game.cpp (GAME ggml inference) into OpenUtauMobile as a `native/` submodule plus a C ABI shim, replacing the ONNX-based GAME path entirely; no plugin/switching framework, no subprocess, every platform ships the CPU backend with GPU-first runtime fallback (Vulkan/Metal/CUDA as compiled). +- Rationale: ONNX Runtime GAME is far too heavy for mobile and effectively infeasible to ship; embedding game.cpp natively is the only viable path. The per-platform accelerator is chosen at build time (Metal on Apple, Vulkan/CUDA on desktop, Vulkan on Android) and `init_best_backend()` provides GPU→CPU fallback automatically, so a single C ABI (`game_capi`) is all the .NET layer needs. Keep the user's fork as the build baseline and git-submodule the upstream game.cpp pinned to release v0.1.3. +- Alternatives considered: ONNX Runtime Electron/CLI subprocess (.oudep Executable) — rejected for iOS sandbox and mobile memory; plugin-switching framework — rejected as over-engineering; using only prebuilt release .oudep binaries — no long-term mobile control. +- Impacted areas: New `native/` tree (game.cpp submodule + shim + CMake host); C# `GameGgml` integration code placed in the Mobile layer (not `OpenUtau.Core`); future Android/Windows/Linux/macOS/iOS ship artifacts; CI additions. - Date: 2026-08-22 - Decision: Give each rendered note pitch-bend curve its own finalized `StreamGeometry` instead of submitting the shared mutable `Points` and `PolylineGeometry` caches. - Rationale: Every `RenderPitchBend` call cleared and repopulated the same point collection retained by earlier drawing commands, so the last visible note replaced the curves submitted for all preceding notes. diff --git a/.github/workflows/build-all-platforms.yml b/.github/workflows/build-all-platforms.yml index 544b6cb4..5a62083a 100644 --- a/.github/workflows/build-all-platforms.yml +++ b/.github/workflows/build-all-platforms.yml @@ -57,6 +57,9 @@ jobs: steps: - name: Checkout uses: actions/checkout@v4 + with: + submodules: recursive + persist-credentials: false - name: Setup .NET uses: actions/setup-dotnet@v4 @@ -186,6 +189,9 @@ jobs: steps: - name: Checkout uses: actions/checkout@v4 + with: + submodules: recursive + persist-credentials: false - name: Setup .NET uses: actions/setup-dotnet@v4 @@ -282,6 +288,9 @@ jobs: steps: - name: Checkout uses: actions/checkout@v4 + with: + submodules: recursive + persist-credentials: false - name: Setup .NET uses: actions/setup-dotnet@v4 @@ -386,6 +395,9 @@ jobs: steps: - name: Checkout uses: actions/checkout@v4 + with: + submodules: recursive + persist-credentials: false - name: Setup .NET uses: actions/setup-dotnet@v4 diff --git a/.gitignore b/.gitignore index d52d2e63..d56e723a 100644 --- a/.gitignore +++ b/.gitignore @@ -454,5 +454,19 @@ $RECYCLE.BIN/ ## 质量分析 analysis/ +# OpenUtauMobile native/ (game_capi shim) build artifacts +native/build*/ +native/*/build*/ +native/**/_deps/ +native/**/CMakeCache.txt +native/**/CMakeFiles/ +native/**/cmake_install.cmake +native/**/CTestTestfile.cmake +native/**/Testing/ + +# Machine-specific CMake user presets (keep local absolute paths out of the repo) +native/CMakeUserPresets.json + + ## codex .codex/ \ No newline at end of file diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 00000000..e1af3b42 --- /dev/null +++ b/.gitmodules @@ -0,0 +1,3 @@ +[submodule "native/game.cpp"] + path = native/game.cpp + url = https://github.com/KakaruHayate/game.cpp diff --git a/OpenUtauMobile.Android/Libs/arm64-v8a/libgame_ggml_shared.so b/OpenUtauMobile.Android/Libs/arm64-v8a/libgame_ggml_shared.so new file mode 100644 index 00000000..b46a0b66 Binary files /dev/null and b/OpenUtauMobile.Android/Libs/arm64-v8a/libgame_ggml_shared.so differ diff --git a/OpenUtauMobile.Android/Libs/armeabi-v7a/libgame_ggml_shared.so b/OpenUtauMobile.Android/Libs/armeabi-v7a/libgame_ggml_shared.so new file mode 100644 index 00000000..321f8c97 Binary files /dev/null and b/OpenUtauMobile.Android/Libs/armeabi-v7a/libgame_ggml_shared.so differ diff --git a/OpenUtauMobile.Android/Libs/x86/libgame_ggml_shared.so b/OpenUtauMobile.Android/Libs/x86/libgame_ggml_shared.so new file mode 100644 index 00000000..0963ddc3 Binary files /dev/null and b/OpenUtauMobile.Android/Libs/x86/libgame_ggml_shared.so differ diff --git a/OpenUtauMobile.Android/Libs/x86_64/libgame_ggml_shared.so b/OpenUtauMobile.Android/Libs/x86_64/libgame_ggml_shared.so new file mode 100644 index 00000000..d31016e8 Binary files /dev/null and b/OpenUtauMobile.Android/Libs/x86_64/libgame_ggml_shared.so differ diff --git a/OpenUtauMobile/Assets/Lang/Strings.en.resx b/OpenUtauMobile/Assets/Lang/Strings.en.resx index 5adec2ec..28e6b959 100644 --- a/OpenUtauMobile/Assets/Lang/Strings.en.resx +++ b/OpenUtauMobile/Assets/Lang/Strings.en.resx @@ -1110,6 +1110,18 @@ TODO: ExportAudio + + Transcribe Audio + + + Transcribing audio… + + + Transcription complete + + + Transcription failed + Export Audio diff --git a/OpenUtauMobile/Assets/Lang/Strings.ja.resx b/OpenUtauMobile/Assets/Lang/Strings.ja.resx index df765ece..dd001d45 100644 --- a/OpenUtauMobile/Assets/Lang/Strings.ja.resx +++ b/OpenUtauMobile/Assets/Lang/Strings.ja.resx @@ -1107,6 +1107,18 @@ TODO: オーディオをエクスポート + + 音声書き起こし + + + 音声書き起こし中… + + + 書き起こし完了 + + + 書き起こしに失敗しました + オーディオを書き出し diff --git a/OpenUtauMobile/Assets/Lang/Strings.ru.resx b/OpenUtauMobile/Assets/Lang/Strings.ru.resx index 7dc5a539..989b0068 100644 --- a/OpenUtauMobile/Assets/Lang/Strings.ru.resx +++ b/OpenUtauMobile/Assets/Lang/Strings.ru.resx @@ -1033,6 +1033,18 @@ TODO: Экспорт аудио + + Транскрипция аудио + + + Идёт транскрипция… + + + Транскрипция завершена + + + Не удалось выполнить транскрипцию + Экспорт аудио diff --git a/OpenUtauMobile/Assets/Lang/Strings.uk.resx b/OpenUtauMobile/Assets/Lang/Strings.uk.resx index bcf5a4ff..27cbea50 100644 --- a/OpenUtauMobile/Assets/Lang/Strings.uk.resx +++ b/OpenUtauMobile/Assets/Lang/Strings.uk.resx @@ -1033,6 +1033,18 @@ TODO: Експорт аудіо + + Транскрипція аудіо + + + Виконується транскрипція… + + + Транскрипцію завершено + + + Не вдалося виконати транскрипцію + Експорт аудіо diff --git a/OpenUtauMobile/Assets/Lang/Strings.zh-Hans.resx b/OpenUtauMobile/Assets/Lang/Strings.zh-Hans.resx index 756c7ea6..1dbd414b 100644 --- a/OpenUtauMobile/Assets/Lang/Strings.zh-Hans.resx +++ b/OpenUtauMobile/Assets/Lang/Strings.zh-Hans.resx @@ -1110,6 +1110,18 @@ TODO: 导出音频功能 + + 音频转写 + + + 正在转写音频… + + + 转写完成 + + + 转写失败 + 导出音频 diff --git a/OpenUtauMobile/Controls/EditorMorePopup.axaml b/OpenUtauMobile/Controls/EditorMorePopup.axaml index ee9299b7..98aa5d09 100644 --- a/OpenUtauMobile/Controls/EditorMorePopup.axaml +++ b/OpenUtauMobile/Controls/EditorMorePopup.axaml @@ -1,4 +1,4 @@ -ImportTrack + diff --git a/OpenUtauMobile/OpenUtauMobile.csproj b/OpenUtauMobile/OpenUtauMobile.csproj index 8e062ca3..cad7a55f 100644 --- a/OpenUtauMobile/OpenUtauMobile.csproj +++ b/OpenUtauMobile/OpenUtauMobile.csproj @@ -1,8 +1,9 @@ - + net10.0 enable preview + true true @@ -20,6 +21,58 @@ + + + + J:\GGML-GAME\vendor\game-cpp-release\game_ggml-windows-x64-vulkan-q8.oudep + https://github.com/KakaruHayate/game.cpp/releases/download/v0.1.3/game_ggml-windows-x64-vulkan-q8.oudep + 895A85B56AAEEB15CD541469ACFA1AEDB2D3DC58B1A6890A2F8B6FB629AD990D + OpenUtauMobile.Models.game_medium.gguf + + + + + + $(IntermediateOutputPath)game.oudep + $(IntermediateOutputPath)game_medium.gguf + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/OpenUtauMobile/Services/Game/GameGgml.cs b/OpenUtauMobile/Services/Game/GameGgml.cs new file mode 100644 index 00000000..6bda3ac9 --- /dev/null +++ b/OpenUtauMobile/Services/Game/GameGgml.cs @@ -0,0 +1,298 @@ +using System; +using System.Collections.Generic; +using System.Runtime.InteropServices; +using System.Text; + +namespace OpenUtauMobile.Services.Game; + +/// +/// GAME 原生后端（libgame_ggml_shared）的托管封装。 +/// +/// 职责：持有模型句柄，把托管参数转换成 C ABI 调用，并回读音符结果。 +/// 设计要点： +/// * 与 Core.Analysis.Game 平行的替换实现（不修改 OpenUtau.Core）。 +/// * 单句柄串行调用（原生 Model::infer 非线程安全），上层需保证同一实例不被并发 infer。 +/// * IDisposable 释放原生句柄。 +/// +public sealed class GameGgml : IDisposable +{ + // 音符回读缓冲区的最大容量（原生按此上限写入，超出的部分被截断）。 + // 转写整段语音的音符数一般只有几十；保留足够余量。 + private const int MaxNotesBufferCapacity = 4096; + + private IntPtr nativeHandle; + private bool disposed; + + /// 此实例加载的模型路径。 + public string ModelPath { get; } + + private GameGgml(IntPtr handle, string modelPath) + { + this.nativeHandle = handle; + this.ModelPath = modelPath; + } + + /// + /// 打开模型（自动解析嵌入式模型到磁盘路径）。 + /// 失败时返回 null 并在 error 中给出原因。 + /// + public static GameGgml? Open( + out string error, + string? modelPathOverride = null, + string? configJson = null) + { + error = string.Empty; + string modelPath; + if (!GameModelResolver.TryResolveExistingModel(modelPathOverride, out modelPath)) + { + try + { + modelPath = GameModelResolver.EnsureModelPath(); + } + catch (Exception ex) + { + error = $"模型解析失败: {ex.Message}"; + return null; + } + } + + if (!TryOpen(modelPath, configJson, out GameGgml? instance, out error)) + { + return null; + } + + return instance; + } + + /// 已确定路径时打开模型。 + public static bool TryOpen( + string modelPath, string? configJson, + out GameGgml? instance, out string error) + { + instance = null; + error = string.Empty; + + byte[] errbuf = new byte[GameCapiErrorBufferSize]; + IntPtr handle = GameGgmlNative.Open(modelPath, configJson, errbuf, errbuf.Length); + if (handle == IntPtr.Zero) + { + error = ReadUtf8Buffer(errbuf) ?? "模型打开失败"; + return false; + } + + instance = new GameGgml(handle, modelPath); + return true; + } + + /// 原生错误码（与 game_capi.h 常量一致）。 + public const int Ok = 0; + public const int ErrHandle = -1; + public const int ErrInit = -2; + public const int ErrInfer = -3; + public const int ErrInvalidArg = -4; + + // 缓冲区约定：GAME_CAPI_ERRBUF = 512（各 buf 均按其最大容量分配）。 + + /// GAME 错误缓冲容量。 + private const int GameCapiErrorBufferSize = 512; + + /// 版本号。 + public static string? VersionString() + { + return CallStringBuffer(GameGgmlNative.Version); + } + + /// ggml 版本号。 + public static string? GgmlVersionString() + { + return CallStringBuffer(GameGgmlNative.GgmlVersion); + } + + private delegate int StringBufferWriter(byte[] buf, int cap); + + private static string? CallStringBuffer(StringBufferWriter writer) + { + byte[] buffer = new byte[GameCapiErrorBufferSize]; + int length = writer(buffer, buffer.Length); + if (length <= 0) + { + return null; + } + + int valid = Math.Min(length - 1, buffer.Length); + // 缓冲可能未写到末尾：以首个 NUL 为界。 + int end = Array.IndexOf(buffer, (byte)0, 0, valid); + if (end < 0) + { + end = valid; + } + + return Encoding.UTF8.GetString(buffer, 0, end); + } + + /// 编译期可用后端列表。 + public static string? AvailableBackendsString() + { + return CallStringBuffer(GameGgmlNative.AvailableBackends); + } + + /// 运行时实际选择的后端名（GPU 首选，fallback 后反映真实后端）。 + public string? BackendDecidedString() + { + EnsureNotDisposed(); + return CallStringBufferWithHandle(nativeHandle, GameGgmlNative.BackendDecided); + } + + private delegate int HandleStringBufferWriter(IntPtr handle, byte[] buf, int cap); + + private static string? CallStringBufferWithHandle( + IntPtr handle, HandleStringBufferWriter writer) + { + byte[] buffer = new byte[GameCapiErrorBufferSize]; + int length = writer(handle, buffer, buffer.Length); + if (length <= 0) + { + return null; + } + + int valid = Math.Min(length - 1, buffer.Length); + int end = Array.IndexOf(buffer, (byte)0, 0, valid); + if (end < 0) + { + end = valid; + } + + return Encoding.UTF8.GetString(buffer, 0, end); + } + + /// 把语言代码映射为 id；未知名返回 -1。可直接按 Core 默认传 0 处理。 + public int LanguageId(string langCode) + { + EnsureNotDisposed(); + return GameGgmlNative.LanguageId(nativeHandle, langCode); + } + + /// 最近一次失败详情。 + public string? LastErrorString() + { + IntPtr ptr = GameGgmlNative.LastError(nativeHandle); + if (ptr == IntPtr.Zero) + { + return null; + } + + return Marshal.PtrToStringUTF8(ptr); + } + + /// + /// 端到端转写：输入 44100Hz 单声道 float 波形 [-1,1]，返回音符列表。 + /// 抛 ArgumentException/FailedInference 表示调用/推理失败。 + /// + public IReadOnlyList Infer(float[] waveform, GameGgmlOptions options) + { + EnsureNotDisposed(); + if (waveform == null || waveform.Length == 0) + { + throw new ArgumentException("waveform 不能为空", nameof(waveform)); + } + + if (options == null) + { + throw new ArgumentNullException(nameof(options)); + } + + int language = options.LanguageCode is null or "" + ? 0 + : LanguageId(options.LanguageCode); + // 未知名语言回落 universal（0）。 + if (language < 0) + { + language = 0; + } + + // 分配音符回读缓冲（capacity 固定）。 + int noteStructSize = Marshal.SizeOf(); + IntPtr notesBuffer = Marshal.AllocHGlobal(noteStructSize * MaxNotesBufferCapacity); + try + { + int code = GameGgmlNative.Infer( + nativeHandle, + waveform, + waveform.Length, + language, + options.SamplingSteps, + options.BoundaryThreshold, + options.BoundaryRadius, + options.ScoreThreshold, + options.Seed, + notesBuffer, + MaxNotesBufferCapacity, + out int notesCount, + out int numFrames); + + if (code == ErrInvalidArg) + { + throw new ArgumentException(LastErrorString() ?? "参数非法"); + } + + if (code != Ok) + { + string msg = $"推理失败 (错误码 {code}): {LastErrorString() ?? "未知"}"; + throw new InvalidOperationException(msg); + } + + return ReadNotesOut(notesBuffer, notesCount); + } + finally + { + Marshal.FreeHGlobal(notesBuffer); + } + } + + private static List ReadNotesOut(IntPtr buffer, int count) + { + int size = Marshal.SizeOf(); + var notes = new List(count); + for (int i = 0; i < count; i++) + { + IntPtr item = IntPtr.Add(buffer, i * size); + notes.Add(Marshal.PtrToStructure(item)); + } + + return notes; + } + + private void EnsureNotDisposed() + { + if (this.disposed) + { + throw new ObjectDisposedException(nameof(GameGgml)); + } + } + + public void Dispose() + { + if (this.disposed) + { + return; + } + + this.disposed = true; + if (this.nativeHandle != IntPtr.Zero) + { + GameGgmlNative.Close(this.nativeHandle); + this.nativeHandle = IntPtr.Zero; + } + } + + private static string ReadUtf8Buffer(byte[] buffer) + { + int end = Array.IndexOf(buffer, (byte)0); + if (end < 0) + { + end = buffer.Length; + } + + return Encoding.UTF8.GetString(buffer, 0, end); + } +} diff --git a/OpenUtauMobile/Services/Game/GameGgmlMidiExtractor.cs b/OpenUtauMobile/Services/Game/GameGgmlMidiExtractor.cs new file mode 100644 index 00000000..cc27db65 --- /dev/null +++ b/OpenUtauMobile/Services/Game/GameGgmlMidiExtractor.cs @@ -0,0 +1,87 @@ +using System; +using System.Collections.Generic; +using OpenUtau.Core.Analysis; +using OpenUtau.Core.Ustx; + +namespace OpenUtauMobile.Services.Game; + +/// +/// 用 GameGgml（原生 GAME ggml 后端）替代 Core 的 ONNX Game 的转写器。 +/// +/// 技巧：**继承 OpenUtau.Core 的 MidiExtractor<GameOptions> 基类**（只读复用，不改 Core 一行）， +/// 实现它的 ；基类负责 +/// mono→44.1k 重采样→AudioSlicer 分块→批推理→UVoicePart 组装（位置/时长 tick 换算、 +/// CreateNote、End 修正）。唯一差异 = 底层的 ONNX 换成我们的 ggml C ABI。 +/// +/// 注意：原生 Model 非线程安全。基类的 Transcribe 串行调用 TranscribeWaveform， +/// 我们单模型句柄 + 串行使用即可。 +/// +public sealed class GameGgmlMidiExtractor : MidiExtractor +{ + private GameGgml? model; + + /// 底层采用 44100Hz 单声道 float（与 game_capi 一致）。基类会先重采样到此处。 + protected override int ExpectedSampleRate => 44100; + + /// 一次 infer 已是全段（原生内部 mel→D3PM→边界→音符），不支持也不必要分批。 + protected override bool SupportsBatch => false; + + protected override List TranscribeWaveform(float[] samples, GameOptions options) + { + GameGgml modelValue = EnsureModel(); + GameGgmlOptions gameOptions = ToGameOptions(options); + IReadOnlyList rawNotes = modelValue.Infer(samples, gameOptions); + + List result = new(rawNotes.Count); + foreach (GameGgmlNote note in rawNotes) + { + result.Add(new TranscribedNote( + note.DurationSeconds, + note.PitchMidi, + note.Voiced != 0)); + } + + return result; + } + + private GameGgml EnsureModel() + { + if (this.model != null) + { + return this.model; + } + + GameGgml? opened = GameGgml.Open(out string error); + if (opened == null) + { + throw new InvalidOperationException($"GAME 模型打开失败: {error}"); + } + + this.model = opened; + return opened; + } + + private static GameGgmlOptions ToGameOptions(GameOptions options) + { + if (options == null) + { + throw new ArgumentNullException(nameof(options)); + } + + return new GameGgmlOptions + { + LanguageCode = options.LanguageCode, + SamplingSteps = options.SamplingSteps, + BoundaryThreshold = options.BoundaryThreshold, + BoundaryRadius = options.BoundaryRadius, + ScoreThreshold = options.ScoreThreshold, + Seed = 0UL, + }; + } + + protected override void DisposeManaged() + { + this.model?.Dispose(); + this.model = null; + } +} diff --git a/OpenUtauMobile/Services/Game/GameGgmlNative.cs b/OpenUtauMobile/Services/Game/GameGgmlNative.cs new file mode 100644 index 00000000..9dfcffa6 --- /dev/null +++ b/OpenUtauMobile/Services/Game/GameGgmlNative.cs @@ -0,0 +1,75 @@ +using System; +using System.Runtime.InteropServices; + +namespace OpenUtauMobile.Services.Game; + +/// +/// 原生 game_ggml_shared 的 P/Invoke 声明（.NET 10 LibraryImport 源生成，AOT-safe）。 +/// +/// 与 native/shim/game_capi.h 的 9 个导出一一对应。库名用通用名 "game_ggml_shared"， +/// 运行时按平台解析：Android => libgame_ggml_shared.so；Windows => game_ggml_shared.dll。 +/// +/// 注意点： +/// * 字符串参数统一 UTF-8，与原生 const char* 一致。 +/// * 波形通过 float[] 传数组指针（blittable，不拷贝）。 +/// * 音符输出缓冲区手动 Marshal.Alloc/Fill（原生语义 = 调用方提供容量， +/// 原生回填 notes_count 个；源生成器不支持 [Out] 自增变长数组，故此处手写， +/// 更贴近"调用方分配、调用方读取"的 C ABI 契约）。 +/// +public static partial class GameGgmlNative +{ + private const string LibraryName = "game_ggml_shared"; + + /// 写产物版本号到 buf。返回长度。 + [LibraryImport(LibraryName, EntryPoint = "game_capi_version")] + internal static partial int Version(byte[] buf, int cap); + + /// 写 ggml 版本号到 buf。返回长度。 + [LibraryImport(LibraryName, EntryPoint = "game_capi_ggml_version")] + internal static partial int GgmlVersion(byte[] buf, int cap); + + /// 写可用后端（逗号分隔小写）到 buf。返回长度。 + [LibraryImport(LibraryName, EntryPoint = "game_capi_available_backends")] + internal static partial int AvailableBackends(byte[] buf, int cap); + + /// 打开 GGUF 模型。返回句柄；失败返回 IntPtr.Zero 并把错误写入 errbuf。 + [LibraryImport(LibraryName, StringMarshalling = StringMarshalling.Utf8, + EntryPoint = "game_capi_open")] + internal static partial IntPtr Open( + string ggufPath, string? configJson, + byte[] errbuf, int errcap); + + /// 关闭并释放句柄。NULL 安全。 + [LibraryImport(LibraryName, EntryPoint = "game_capi_close")] + internal static partial void Close(IntPtr model); + + /// 写运行时实际选择的后端名到 buf（诊断/UI 展示用）。 + [LibraryImport(LibraryName, EntryPoint = "game_capi_backend_decided")] + internal static partial int BackendDecided(IntPtr model, byte[] buf, int cap); + + /// 执行推理。返回错误码（0 = 成功），notes_count/num_frames 回填。 + [LibraryImport(LibraryName, EntryPoint = "game_capi_infer")] + internal static partial int Infer( + IntPtr model, + float[] waveform, + int n, + int language, + int nsteps, + float segThreshold, + int segRadius, + float estThreshold, + ulong seed, + IntPtr notesOut, + int notesCapacity, + out int notesCount, + out int numFrames); + + /// 把语言代码映射为 id；未知名返回 -1。 + [LibraryImport(LibraryName, StringMarshalling = StringMarshalling.Utf8, + EntryPoint = "game_capi_language_id")] + internal static partial int LanguageId(IntPtr model, string langCode); + + /// 最近一次失败的详细消息。可为 IntPtr.Zero。 + [LibraryImport(LibraryName, EntryPoint = "game_capi_last_error")] + internal static partial IntPtr LastError(IntPtr model); +} diff --git a/OpenUtauMobile/Services/Game/GameGgmlNote.cs b/OpenUtauMobile/Services/Game/GameGgmlNote.cs new file mode 100644 index 00000000..c63cd31b --- /dev/null +++ b/OpenUtauMobile/Services/Game/GameGgmlNote.cs @@ -0,0 +1,32 @@ +using System; +using System.Runtime.InteropServices; + +namespace OpenUtauMobile.Services.Game; + +/// +/// game_capi_note 的托管投影（POD，布局与原生结构体完全一致）。 +/// 供 GameGgml.Infer 返回给上层。 +/// +[StructLayout(LayoutKind.Sequential)] +public readonly struct GameGgmlNote +{ + /// 起始时间（秒）。 + public readonly float OffsetSeconds; + + /// 持续时间（秒）。 + public readonly float DurationSeconds; + + /// 小数 MIDI 音高（仅 Voiced 有效）。 + public readonly float PitchMidi; + + /// 1=有声部，0=休止/无音高。 + public readonly int Voiced; + + public GameGgmlNote(float offsetSeconds, float durationSeconds, float pitchMidi, int voiced) + { + this.OffsetSeconds = offsetSeconds; + this.DurationSeconds = durationSeconds; + this.PitchMidi = pitchMidi; + this.Voiced = voiced; + } +} diff --git a/OpenUtauMobile/Services/Game/GameGgmlOptions.cs b/OpenUtauMobile/Services/Game/GameGgmlOptions.cs new file mode 100644 index 00000000..5f47d09e --- /dev/null +++ b/OpenUtauMobile/Services/Game/GameGgmlOptions.cs @@ -0,0 +1,28 @@ +using System; + +namespace OpenUtauMobile.Services.Game; + +/// +/// GAME 推理参数（与 Core.Analysis.GameOptions 对齐，但本类型不依赖 OpenUtau.Core）。 +/// 默认值与 Core 的 ONNX 实现一致，保证行为对齐上游 infer.py。 +/// +public sealed class GameGgmlOptions +{ + /// 语言代码，例如 "en"、"zh"；null = universal/自动。 + public string? LanguageCode { get; set; } + + /// D3PM 去噪步数（--nsteps）。默认 8。 + public int SamplingSteps { get; set; } = 8; + + /// 边界解码阈值（--seg-threshold）。默认 0.2。 + public float BoundaryThreshold { get; set; } = 0.2f; + + /// 边界解码半径（帧，--seg-radius）。默认 2。 + public int BoundaryRadius { get; set; } = 2; + + /// 音符存在性门槛（--est-threshold）。默认 0.2。 + public float ScoreThreshold { get; set; } = 0.2f; + + /// 随机种子；0 = 自动（由原生层 OS 随机）。 + public ulong Seed { get; set; } = 0UL; +} diff --git a/OpenUtauMobile/Services/Game/GameModelResolver.cs b/OpenUtauMobile/Services/Game/GameModelResolver.cs new file mode 100644 index 00000000..2c220986 --- /dev/null +++ b/OpenUtauMobile/Services/Game/GameModelResolver.cs @@ -0,0 +1,78 @@ +using System; +using System.IO; +using System.Linq; + +namespace OpenUtauMobile.Services.Game; + +/// +/// 模型文件的解析与物化：把 EmbeddedResource 里的 game_medium.gguf +/// 首次运行时解包到平台可写目录（Android = app 私有 Files；桌面 = %LOCALAPPDATA%）， +/// 之后直接引用该路径（原生层 ggml 用 FILE* 读路径，无法直接读流/Asset， +/// 必须先落到磁盘）。 +/// +public static class GameModelResolver +{ + private const string ModelResourceName = "OpenUtauMobile.Models.game_medium.gguf"; + + /// 模型在运行时的相对子目录名。 + public const string ModelSubDir = "game"; + + /// 模型文件名（与资源、csproj 引用一致）。 + public const string ModelFileName = "game_medium.gguf"; + + /// + /// 返回模型实际磁盘路径（不存在则从嵌入式资源解包）。 + /// 抛出 IOException/InvalidOperationException 表示无法提供模型。 + /// + public static string EnsureModelPath() + { + string baseDir = Environment.GetFolderPath(Environment.SpecialFolder.LocalApplicationData); + string modelDir = Path.Combine(baseDir, ModelSubDir); + string modelPath = Path.Combine(modelDir, ModelFileName); + + if (File.Exists(modelPath)) + { + return modelPath; + } + + ExtractEmbeddedModel(modelDir, modelPath); + return modelPath; + } + + /// 可选：外部已经有一个 .gguf 文件路径（例如用户下载），直接使用它。 + public static bool TryResolveExistingModel(string? externalPath, out string resolvedPath) + { + if (!string.IsNullOrEmpty(externalPath) && File.Exists(externalPath)) + { + resolvedPath = externalPath; + return true; + } + + resolvedPath = string.Empty; + return false; + } + + private static void ExtractEmbeddedModel(string modelDir, string targetPath) + { + using (System.IO.Stream? stream = typeof(GameModelResolver).Assembly + .GetManifestResourceStream(ModelResourceName)) + { + if (stream == null) + { + throw new InvalidOperationException( + $"Embedded model resource not found: {ModelResourceName}"); + } + + Directory.CreateDirectory(modelDir); + + // 先写临时文件再原子替换，避免中途崩溃留下半截模型。 + string tmpPath = targetPath + ".tmp"; + using (FileStream output = File.Create(tmpPath)) + { + stream.CopyTo(output); + } + + File.Move(tmpPath, targetPath, overwrite: true); + } + } +} diff --git a/OpenUtauMobile/Tools/extract_game_model.ps1 b/OpenUtauMobile/Tools/extract_game_model.ps1 new file mode 100644 index 00000000..7861851c --- /dev/null +++ b/OpenUtauMobile/Tools/extract_game_model.ps1 @@ -0,0 +1,62 @@ +# --------------------------------------------------------------------------- +# 从 game.cpp release .oudep(zip) 中提取 game_medium.gguf 到 obj/。 +# 供 OpenUtauMobile.csproj 的 EnsureGameEmbeddedModel 目标调用（构建期）， +# 使 55MB 模型不 commit 进仓库，而是构建时"扒"自官方 release。 +# 用法: powershell -ExecutionPolicy Bypass -File extract_game_model.ps1 -Oudep -Target +# --------------------------------------------------------------------------- +param( + [Parameter(Mandatory = $true)][string]$Oudep, + [Parameter(Mandatory = $true)][string]$Target +) + +$ErrorActionPreference = 'Stop' + +if (Test-Path -LiteralPath $Target) { + Write-Output "GAME 模型已存在: $Target" + exit 0 +} + +if (-not (Test-Path -LiteralPath $Oudep)) { + Write-Error "找不到 .oudep: $Oudep" + exit 1 +} + +Write-Output "从 .oudep 提取 game_medium.gguf -> $Target" + +Add-Type -AssemblyName System.IO.Compression.FileSystem +$zip = [System.IO.Compression.ZipFile]::OpenRead($Oudep) +try { + $entry = $zip.GetEntry('game_medium.gguf') + if ($null -eq $entry) { + Write-Error ".oudep 中没有 game_medium.gguf 条目" + exit 2 + } + + $dir = [System.IO.Path]::GetDirectoryName($Target) + if (-not [string]::IsNullOrEmpty($dir)) { + New-Item -ItemType Directory -Path $dir -Force | Out-Null + } + + $tmp = "$Target.tmp" + $stream = $entry.Open() + try { + $file = [System.IO.File]::Create($tmp) + try { + $stream.CopyTo($file) + } finally { + $file.Dispose() + } + } finally { + $stream.Dispose() + } + + if (Test-Path -LiteralPath $Target) { + Remove-Item -LiteralPath $Target -Force + } + + Move-Item -LiteralPath $tmp -Destination $Target -Force | Out-Null + Write-Output ("GAME 模型解压完成: {0} ({1:N0} B)" -f $Target, (Get-Item -LiteralPath $Target).Length) + exit 0 +} finally { + $zip.Dispose() +} diff --git a/OpenUtauMobile/ViewModels/EditorMoreViewModel.cs b/OpenUtauMobile/ViewModels/EditorMoreViewModel.cs index b236c29d..34713f09 100644 --- a/OpenUtauMobile/ViewModels/EditorMoreViewModel.cs +++ b/OpenUtauMobile/ViewModels/EditorMoreViewModel.cs @@ -1,4 +1,4 @@ -using System.Reactive; +using System.Reactive; using ReactiveUI; namespace OpenUtauMobile.ViewModels; @@ -9,6 +9,7 @@ public enum EditorMoreAction ImportAudio, // 导入音频 ImportMidi, // 导入MIDI ImportTrack, // 导入轨道 + TranscribeAudio, // 音频转写（GAME ggml 后端） ExportAudio, // 导出音频 SaveAs // 另存为 } diff --git a/OpenUtauMobile/ViewModels/EditorViewModel.cs b/OpenUtauMobile/ViewModels/EditorViewModel.cs index d588571f..117854be 100644 --- a/OpenUtauMobile/ViewModels/EditorViewModel.cs +++ b/OpenUtauMobile/ViewModels/EditorViewModel.cs @@ -13,12 +13,14 @@ using NAudio.Wave; using OpenUtau.Core; using OpenUtau.Core.Format; +using OpenUtau.Core.Analysis; using OpenUtau.Core.Ustx; using OpenUtau.Core.Util; using OpenUtauMobile.Controls; using OpenUtauMobile.Controls.Gestures; using OpenUtauMobile.Helpers; using OpenUtauMobile.Services; +using OpenUtauMobile.Services.Game; using OpenUtauMobile.Storage; using OpenUtauMobile.Themes.OpenUtauMobile.Runtime; using ReactiveUI; @@ -652,6 +654,9 @@ private async Task ShowMorePopupAsync() // TODO: Handle ImportTrack action ToastService.Enqueue(L.S("EditorMore.Toast.ImportTrack")); break; + case EditorMoreAction.TranscribeAudio: + _ = TranscribeAudio(); + break; case EditorMoreAction.ExportAudio: _ = ShowExportAudioPopupAsync(); break; @@ -684,6 +689,88 @@ private static async Task ImportAudio() DocManager.Inst.EndUndoGroup(); } + /// + /// 导入音频并直接用 GAME ggml 后端转写为音符（替代 Core 的 ONNX Game）。 + /// 流程：选音频 → UWavePart 仅作输入（不加入项目）→ GameGgmlMidiExtractor.Transcribe + /// （基类负责 mono/重采样/分块/UVoicePart 组装）→ 产出的 UVoicePart 作为新轨插入， + /// 全部并入一个 undo group。不插入原音频轨。 + /// + private static async Task TranscribeAudio() + { + string file = await FilePicker.PickSingleFileAsync(L.S("FilePicker.ImportAudio"), + ["*.mp3", "*.wav", "*.flac", "*.aac", "*.ogg", "*.aiff", "*.aif", "*.aifc"]); + if (file == string.Empty) + { + return; + } + + try + { + UProject project = DocManager.Inst.Project; + UWavePart wavePart = new() + { + FilePath = file, + }; + wavePart.Load(project); // 填充 channels/sampleRate；Samples 在 Peaks Task 内生成 + await wavePart.Peaks; // 等待 Samples 就绪（基类 Transcribe 直接读 wavePart.Samples） + if (wavePart.Samples == null) + { + ToastService.Enqueue(L.S("EditorMore.Toast.TranscribeFailed")); + return; + } + + int trackNo = project.tracks.Count; + UVoicePart? voicePart = null; + using (GameGgmlMidiExtractor extractor = new()) + { + GameOptions options = new() + { + SamplingSteps = 8, // 与 ONNX 版默认一致 + }; + + await LoadingPopupService.RunAsync( + L.S("EditorMore.TranscribeProgress"), + 0d, + async loading => + { + voicePart = await Task.Run(() => extractor.Transcribe( + project, wavePart, options, null, null, + (done, total) => + { + double progress = total > 0 ? done * 100d / total : 0d; + Dispatcher.UIThread.Post( + () => loading.UpdateProgress(progress, $"{done}s / {total}s")); + })); + }); + } + + if (voicePart == null) + { + ToastService.Enqueue(L.S("EditorMore.Toast.TranscribeFailed")); + return; + } + + UTrack track = new(project) + { + TrackNo = trackNo, + TrackName = Path.GetFileNameWithoutExtension(file), + }; + voicePart.trackNo = trackNo; + + DocManager.Inst.StartUndoGroup(); + DocManager.Inst.ExecuteCmd(new AddTrackCommand(project, track)); + DocManager.Inst.ExecuteCmd(new AddPartCommand(project, voicePart)); + DocManager.Inst.EndUndoGroup(); + + ToastService.Enqueue(L.S("EditorMore.Toast.TranscribeDone")); + } + catch (Exception ex) + { + Log.Error(ex, "TranscribeAudio failed for file={File}", file); + ToastService.Enqueue(L.S("EditorMore.Toast.TranscribeFailed")); + } + } + private static async Task ImportMidi() { string file = await FilePicker.PickSingleFileAsync(L.S("FilePicker.ImportMIDI"), ["*.mid", "*.midi"]); diff --git a/native/CMakeLists.txt b/native/CMakeLists.txt new file mode 100644 index 00000000..029cb7a5 --- /dev/null +++ b/native/CMakeLists.txt @@ -0,0 +1,115 @@ +# --------------------------------------------------------------------------- +# OpenUtauMobile native/ —— game.cpp (GAME ggml 推理) 深度植入的 CMake 宿主。 +# +# 职责: +# 1. 以 submodule native/game.cpp 引入上游（add_subdirectory 不修改其源码）。 +# 2. 把 game_ggml 静态库 + 本仓库的 C shim(game_capi) 联合编成一个 +# 平台共享库 libgame_ggml_shared(.dll/.so/.dylib)，对外只暴露 C ABI， +# 供 .NET 层 P/Invoke。 +# 3. 附带一个原生 smoke 验证入口 game_capi_check（不进 OPUM 主包，仅开发用）。 +# +# 构造方式：直接 add_subdirectory(game.cpp)，复用其 game_ggml 静态目标。 +# 后端编译选项透传（遵循上游命名）： +# GAME_GGML_METAL / GAME_GGML_CUDA / GAME_GGML_VULKAN / (CPU 恒在) +# --------------------------------------------------------------------------- + +cmake_minimum_required(VERSION 3.18) + +project(OpenUtauMobileNative LANGUAGES C CXX) + +# 透传并让用户在 configure 时决定要编哪些加速器（默认只 CPU，最安全）。 +# 用法: -DGAME_GGML_VULKAN=ON -DGAME_GGML_METAL=ON ... +set(GAME_GGML_METAL "unset" CACHE STRING "Enable ggml Metal backend (ON/OFF; default = Apple only)") +set(GAME_GGML_CUDA OFF CACHE BOOL "Enable ggml CUDA backend") +set(GAME_GGML_VULKAN OFF CACHE BOOL "Enable ggml Vulkan backend") +set(GAME_GGML_LLAMAFILE ON CACHE BOOL "Enable ggml-llamafile sgemm kernels (CPU)") +set(GAME_GGML_BUILD_CLI OFF CACHE BOOL "Build upstream game_ggml_cli (dev only)") +set(GAME_GGML_BUILD_TESTS OFF CACHE BOOL "Build upstream gtest suite") +set(OPUM_NATIVE_SMOKE ON CACHE BOOL "Build game_capi_check smoke tool") + +if(GAME_GGML_METAL STREQUAL "unset") + if(APPLE) + set(GAME_GGML_METAL ON) + else() + set(GAME_GGML_METAL OFF) + endif() +endif() + +# 关闭上游 CLI/tests（由 OPUM_NATIVE_SMOKE 取代），并发正确选项到 game.cpp。 +add_subdirectory(game.cpp) + +# MSVC 默认按本地 ANSI 代码页处理源文件，本工程源码（shim 中文注释/字符串） +# 是 UTF-8 —— 上游 game.cpp 的 /utf-8 只在其目录作用域生效，这里对本目录 +# 新建目标（game_ggml_shared 等）显式补充, 否则中文会被误读导致语法崩坏。 +if(MSVC) + add_compile_options(/utf-8) +endif() + +set(GAME_CAPI_SOURCES + shim/game_capi.cpp + shim/game_capi.h +) + +# ---- 平台共享库（只暴露 C ABI）------------------------------------------------ +add_library(game_ggml_shared SHARED ${GAME_CAPI_SOURCES}) +add_library(OpenUtauMobile::game_ggml ALIAS game_ggml_shared) + +target_include_directories(game_ggml_shared + PUBLIC $ + PRIVATE ${CMAKE_CURRENT_SOURCE_DIR} +) + +# game_ggml 是 static；其 PUBLIC 链接的 ggml 也会随链接进我们的 dll。 +# 共享库被打包进各平台 runtimes/（.NET 层用 DllImport 加载）。 +target_link_libraries(game_ggml_shared + PRIVATE game_ggml::game_ggml +) + +if(MSVC) + # 明确导出 extern "C" 符号（Windows 需要 __declspec(dllexport) 或 .def； + # 简单起见开启自动导出, 因为只导出这几个 extern "C" 函数不会污染）。 + # 也可改用生成 .def。自动导出对 /EXPORT 生成在 x86/x64 都 OK。 + set_target_properties(game_ggml_shared PROPERTIES WINDOWS_EXPORT_ALL_SYMBOLS ON) +endif() + +set_target_properties(game_ggml_shared PROPERTIES + OUTPUT_NAME game_ggml_shared + PREFIX "" # Windows 无 lib 前缀；其它平台 libgame_ggml_shared + RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/app" + LIBRARY_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/app" + ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/lib" +) +# 归档/库输出在 MSVC 多配置下需要逐配置路径 +if(MSVC) + foreach(_cfg IN ITEMS Debug Release RelWithDebInfo MinSizeRel) + string(TOUPPER "${_cfg}" _CFG) + set_property(TARGET game_ggml_shared PROPERTY + ARCHIVE_OUTPUT_DIRECTORY_${_CFG} "${CMAKE_BINARY_DIR}/lib") + set_property(TARGET game_ggml_shared PROPERTY + RUNTIME_OUTPUT_DIRECTORY_${_CFG} "${CMAKE_BINARY_DIR}/app") + endforeach() +endif() + +# ---- 原生 smoke 验证入口（开发用, 不随 OPUM 主包发布）---------------------- +if(OPUM_NATIVE_SMOKE) + add_executable(game_capi_check shim/smoke_main.cpp) + target_link_libraries(game_capi_check PRIVATE game_ggml_shared) + # 会抓取 platform 前缀/后缀, 便于直接看结果 + set_target_properties(game_capi_check PROPERTIES + RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/app") + if(MSVC) + foreach(_cfg IN ITEMS Debug Release RelWithDebInfo MinSizeRel) + string(TOUPPER "${_cfg}" _CFG) + set_property(TARGET game_capi_check PROPERTY + RUNTIME_OUTPUT_DIRECTORY_${_CFG} "${CMAKE_BINARY_DIR}/app") + endforeach() + endif() +endif() + +message(STATUS "") +message(STATUS "OpenUtauMobile native/ configuration:") +message(STATUS " Metal ........... ${GAME_GGML_METAL}") +message(STATUS " CUDA ........... ${GAME_GGML_CUDA}") +message(STATUS " Vulkan ........... ${GAME_GGML_VULKAN}") +message(STATUS " LlamaFile CPU .... ${GAME_GGML_LLAMAFILE}") +message(STATUS " Smoke ........... ${OPUM_NATIVE_SMOKE}") diff --git a/native/CMakePresets.json b/native/CMakePresets.json new file mode 100644 index 00000000..49a283e4 --- /dev/null +++ b/native/CMakePresets.json @@ -0,0 +1,94 @@ +{ + "version": 3, + "cmakeMinimumRequired": { + "major": 3, + "minor": 25 + }, + "configurePresets": [ + { + "name": "android-common", + "hidden": true, + "description": "Android cross-compile base: NDK toolchain + ninja + CPU-only static ggml. Portable preset — machine-specific paths live in the untracked CMakeUserPresets.json override.", + "generator": "Ninja", + "cacheVariables": { + "CMAKE_BUILD_TYPE": "Release", + "ANDROID_PLATFORM": "24", + "ANDROID_SUPPORT_FLEXIBLE_PAGE_SIZES": "OFF", + "GGML_NATIVE": "OFF", + "GGML_OPENMP": "OFF", + "BUILD_SHARED_LIBS": "OFF", + "GGML_BACKEND_DL": "OFF", + "GAME_GGML_VULKAN": "OFF", + "GAME_GGML_METAL": "OFF", + "GAME_GGML_CUDA": "OFF", + "GAME_GGML_LLAMAFILE": "OFF", + "GAME_GGML_BUILD_CLI": "OFF", + "GAME_GGML_BUILD_TESTS": "OFF", + "OPUM_NATIVE_SMOKE": "OFF", + "FETCHCONTENT_UPDATES_DISCONNECTED": "ON" + } + }, + { + "name": "android-arm64", + "inherits": "android-common", + "description": "Android arm64-v8a, CPU-only static ggml.", + "binaryDir": "${sourceDir}/build-android-arm64", + "cacheVariables": { + "ANDROID_ABI": "arm64-v8a" + } + }, + { + "name": "android-arm", + "inherits": "android-common", + "description": "Android armeabi-v7a (32-bit ARM), CPU-only static ggml.", + "binaryDir": "${sourceDir}/build-android-arm", + "cacheVariables": { + "ANDROID_ABI": "armeabi-v7a" + } + }, + { + "name": "android-x86", + "inherits": "android-common", + "description": "Android x86 (32-bit) emulator ABI, CPU-only static ggml.", + "binaryDir": "${sourceDir}/build-android-x86", + "cacheVariables": { + "ANDROID_ABI": "x86" + } + }, + { + "name": "android-x64", + "inherits": "android-common", + "description": "Android x86_64 emulator ABI, CPU-only static ggml.", + "binaryDir": "${sourceDir}/build-android-x64", + "cacheVariables": { + "ANDROID_ABI": "x86_64" + } + } + ], + "buildPresets": [ + { + "name": "build-android-arm64", + "configurePreset": "android-arm64", + "description": "Build arm64-v8a", + "jobs": 16 + }, + { + "name": "build-android-arm", + "configurePreset": "android-arm", + "description": "Build armeabi-v7a", + "jobs": 16 + }, + { + "name": "build-android-x86", + "configurePreset": "android-x86", + "description": "Build x86", + "jobs": 16 + }, + { + "name": "build-android-x64", + "configurePreset": "android-x64", + "description": "Build x86_64", + "jobs": 16 + } + ] +} diff --git a/native/HANDOFF.md b/native/HANDOFF.md new file mode 100644 index 00000000..4937f282 --- /dev/null +++ b/native/HANDOFF.md @@ -0,0 +1,194 @@ +# HANDOFF — OpenUtauMobile × game.cpp 深度植入接续文档 + +> 归档日期：2026-08-24（M2、M3 完成后重写；M4 已达成 headless 验证） +> 会话状态：M1、M2、M3、M4 已完成（headless 端到端转写验证通过）；剩余 = M4 交互 UI 点击闭环与 M5 平台交付 + +--- + +## 1. 目标（用户在做什么） + +把 [KakaruHayate/game.cpp](https://github.com/KakaruHayate/game.cpp)（GAME 歌声转 MIDI 模型的 ggml 原生 C++ 后端）**直接深度植入** OpenUtauMobile（用户 fork：vocoder712/OpenUtauMobile 的 dev 分支）： +- **不引入 ONNX 版本 GAME 的插件切换框架**（ONNX 版开销巨大，移动端不现实）。 +- **不采用子进程/CLI 方案**——因为 iOS 沙箱限制 + 移动端内存，只能进程内置入。 +- 每平台都内置 **CPU 兜底**，GPU（Vulkan/Metal/CUDA）编译进库、运行时 `init_best_backend()` 自动 GPU→CPU fallback。 + +## 2. 关键决策（已记录于 `.agent/DECISIONS.md`） + +- 构建源 = **submodule** 指向 KakaruHayate/game.cpp，固定到 **v0.1.3**(commit `97f9277`)。 +- 原生构建在 **OpenUtauMobile 仓库内 native/** 子树完成（不在上游 game.cpp 加 CI），产物为各平台共享库（`libgame_ggml_shared.{so,dll,dylib}`），通过 C ABI（shim）给 .NET 层 P/Invoke。 +- **OpenUtau.Core 不改动**（上游分叉风险）→ C# 集成代码放 OpenUtauMobile 应用层，只读使用 Core 的 MidiExtractor/TranscribedNote 等。 + +## 3. 目录与文件布局（工作区 = J:\GGML-GAME） + +``` +J:\GGML-GAME\ +├── OpenUtauMobile\ ← 实施工作副本（vocoder712/OpenUtauMobile, dev 分支 HEAD 60409ae） +│ ├── native\ +│ │ ├── game.cpp\ ← submodule（v0.1.3 = 97f9277） +│ │ ├── shim\ ← C ABI shim + smoke（新增） +│ │ │ ├── game_capi.h ← 纯 C 头，9 个导出函数声明 +│ │ │ ├── game_capi.cpp ← 实现（C++异常→错误码边界） +│ │ │ └── smoke_main.cpp ← 原生冒烟工具（吃 .gguf + .wav 44.1k 单声道） +│ │ ├── CMakeLists.txt ← CMake 宿主：add_subdirectory(game.cpp) + 编成共享库 +│ │ ├── build-desktop\ ← 桌面构建产物（gitignore） +│ │ ├── build-android-arm64\ ← Android 交叉构建产物/CMakeCache（进行中） +│ │ ├── build-android-arm\ ← （规划） +│ │ ├── build-android-x86\ ← （规划） +│ │ ├── build-android-x64\ ← （规划） +│ │ └── HANDOFF.md ← 本文件 +├── android-sdk\ndk\android-ndk-r28c\ ← NDK r28c（下载+解压完成，SHA1 校验过） +├── vendor\game-cpp-release\ ← 从 game.cpp v0.1.3 release 下载的产物 +│ ├── game_ggml-windows-x64-vulkan-q8.oudep +│ └── windows-x64-vulkan-q8\ ← 解压出 game_medium.gguf(Q8 55MB) + cli + ggml dll +└──（其他用户既有工作目录，勿动） +``` + +## 4. 工具链（本机） + +后端说明见 pwsh：`$env:TEMP` = `C:\Users\kakar\AppData\Local\Temp`。 + +| 工具 | 路径/版本 | 说明 | +|---|---|---| +| cmake | `C:\Program Files (x86)\Microsoft Visual Studio\18\BuildTools\Common7\IDE\...\CMake\CMake\bin\cmake.exe` (4.2.3-msvc3) | VS18 内置 | +| ninja | `...VS18...\CommonExtensions\Microsoft\CMake\Ninja\ninja.exe` (1.12.1) | VS18 内置 | +| MSVC cl | `...VS18\VC\Tools\MSVC\14.50.35717\bin\Hostx64\x64\cl.exe`（对应 VS18 BuildTools）；还有 VS2019 BuildTools 14.29.30133 | 桌面用 vcvars64.bat 环境 | +| NDK | `J:\GGML-GAME\android-sdk\ndk\android-ndk-r28c` | r28c，通过 dl.google.com 官方直链下载（713MB，SHA1 086BBA... 校验通过） | +| dotnet | **10.0.300 + 10.0.400 SDK（用户级 `%USERPROFILE%\.dotnet`）**；APK 还需要 .NET 10 android workload + JDK | | +| python | `J:\GGML-GAME\.venv-dml\Scripts\python.exe` | vcvars 只在 MSVC 桌面构建需要；Android 用 NDK clang 不需 vcvars | +| java/jdk | **本机没有**（后续如需 build APK 需装） | | +| vcvars64 | `C:\Program Files (x86)\Microsoft Visual Studio\18\BuildTools\VC\Auxiliary\Build\vcvars64.bat` | 桌面 MSVC 用 | + +## 5. M1 已交付（✅ 完成） + +- native/ 子树 + submodule(game.cpp v0.1.3) +- shim（9 个 C ABI 导出：version / ggml_version / available_backends / open / close / backend_decided / infer / language_id / last_error） +- native/CMakeLists.txt（add_subdirectory(game.cpp) → 静态 game_ggml + shim 编成 game_ggml_shared 共享库 + game_capi_check 冒烟） +- 桌面构建成功 + **端到端 smoke 通过**：从 release Q8 权重推断 3s 音频 → `frames=300 notes=13` 合计 3.000s，`decided backend=cpu` 正常 + +**M1 关键经验**： +1. **MSVC 必须 `/utf-8`**（上游 game.cpp 的 /utf-8 只在它自己的目录作用域生效，native 层要自己 add_compile_options(/utf-8)，否则中文注释/字符串被当本地 ANSI 导致 C2062/C2734 语法崩坏） +2. **运行依赖链**：game_ggml_shared.dll 依赖 ggml-base.dll/ggml-cpu.dll/ggml.dll（在 build-desktop/bin/）+ VC 运行时（拷到同目录）。→ Android 侧优选静态 ggml（少分发文件）。 +3. smoke 运行需把 ggml dll + VC CRT 复制到 app 目录。 + +## 6. M2 已完成（Android 交叉编译 4 ABI，CPU-only 静态 ggml） + +**卡点解决**：PowerShell 5.1 命令行传 `-DCMAKE_TOOLCHAIN_FILE=...` 等引号路径参数给 cmake 不可靠（`& $cmake $args` 拆分/`$args` 自动变量名/引号丢失），最终采用 **CMakeUserPresets.json + `cmake --preset`** 路线，一次性写死所有 -D，一次 configure 成功（291s 含 ggml URL 下载）。这就是 HANDOFF 旧版第 6 节的推荐做法，落地有效。 + +**新增文件**：`native/CMakeUserPresets.json`（4 个 configure presets + 4 个 build presets，共享 android-common）。 + +**关键修正（踩坑后）**： +1. **ggml v0.19 的静态/共享开关是标准 `BUILD_SHARED_LIBS`，不是 `GGML_SHARED`**。首轮把 `GGML_SHARED=OFF` 写进 preset 被 CMake 静默忽略，ggml 仍产出 libggml-{base,cpu}.so 共享库。改为 `BUILD_SHARED_LIBS=OFF` + `GGML_BACKEND_DL=OFF` 后，ggml 产出静态 .a，全部并入单个 `game_ggml_shared.so`。 +2. **`GAME_GGML_LLAMAFILE=OFF` 必须（尤其 32 位 ARM）**：默认 ON 会让 ggml-cpu 编译 llamafile sgemm.cpp，其中 `vld1q_f16/vld1_f16`（FP16 NEON）在 armeabi-v7a（ARMv7 无完整 FP16 NEON）下报 `undeclared identifier`，x86(x32) 能用 SSE 过、但移动端统一关掉最一致且体积更小。 +3. cmake 4.2 preset 校验要求 cacheVariables 值一律为字符串（数字 `24` 报 "Invalid CMake variable ANDROID_PLATFORM"）。 +4. 网络 FetchContent 偶发下载失败（GitHub tarball）→ 用 --fresh + 重试循环即可。 +5. `--fresh`（清 cache 重新 configure）与 preset 组合使用。 + +**Android 构建参数（android-common）**： +- NDK r28c toolchain + ninja（VS18 BuildTools）+ Android SDK，`ANDROID_PLATFORM=24`（对齐 SupportedOSPlatformVersion=24）、`ANDROID_SUPPORT_FLEXIBLE_PAGE_SIZES=OFF`（4KB 最稳）。 +- `GGML_NATIVE=OFF`、`GGML_OPENMP=OFF`、`BUILD_SHARED_LIBS=OFF`、`GGML_BACKEND_DL=OFF` → **全部静态链接进单个 .so，规避桌面残留 ggml 共享库分发问题**（M1 经验 2 已在 Android 侧验证解决）。 + +**产物（已放入 `OpenUtauMobile.Android/Libs//libgame_ggml_shared.so`，csproj 的 AndroidNativeLibrary glob 自动打包；与 onnxruntime/worldline 共存，不删——卸载 ONNX 路径是 M3）：** + +| ABI | .so 大小 | ELF/Machine | READ 验证 | +|---|---|---|---| +| arm64-v8a | 18.7 MB | ELF64 / AArch64 | ✅ | +| armeabi-v7a | 14.3 MB | ELF32 / ARM | ✅ | +| x86 | 14.7 MB | ELF32 / Intel 80386 | ✅ | +| x86_64 | 17.5 MB | ELF64 / X86-64 | ✅ | + +**dlopen 可加载性静态验证通过**（llvm-readelf/nm）： +- 全部 `Type=DYN`、`SONAME=game_ggml_shared.so`。 +- `NEEDED` 只有 `libm.so/libdl.so/libc.so`（系统库，bionic 提供）。 +- 无符号仅 135 个且全部 `@LIBC/@LIBM/@LIBDL` 版本化——无未定义 ggml 内部引用，构建干净，dlopen 可解析。 +- 9 个 `game_capi_*` 导出函数全部可见（`T` 文本符号），对齐 shim 头（version / ggml_version / available_backends / open / close / backend_decided / infer / language_id / last_error）。 +- `GGML_OPENMP=OFF` → 无 libomp.so 依赖；无需额外分发。 + +**Vulkan（后续）**：`GAME_GGML_VULKAN=ON` 即可（ggml-vulkan 已带 pipeline-cache patch，文件由 project 自动加入——Dependencies.cmake 里每次构建都会 `git apply` 该 patch 到 ggml 源码后再编）。configure 时需把该选项切 ON + 各 ABI 重新 configure/build/拷贝。 + +**复现命令**（在 `native/` 下）： +```powershell +$cmake = "C:\Program Files (x86)\Microsoft Visual Studio\18\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" +# 单个 ABI（例 arm64）: +& $cmake --preset android-arm64 --fresh # configure（可在 4 个 preset 间循环） +& $cmake --build --preset build-android-arm64 # build +# 产物: build-android-*/app/game_ggml_shared.so -> 复制到 OpenUtauMobile.Android/Libs// +``` + +## 7. 里程碑状态 + +- [x] 调研（平台 / game.cpp / 后端回退 / 集成点） +- [x] 决策 + 选型（GPU 首选 + CPU 兜底；deep embed；no subprocess；no plugin framework） +- [x] M1：native + submodule + shim + 桌面构建 + smoke ✅ +- [x] M2：Android 交叉编译 4 ABI（CPU-only 静态 ggml）✅ ← 2026-08-24 完成 +- [x] M3：C# 后端 `GameGgml.cs`（放 Mobile 层，不碰 OpenUtau.Core）+ 模型封包 ✅ ← 2026-08-24 完成 +- [x] M4：转写入口接线（EditorMore「导入/录音→转写」，走 MidiExtractor.Transcribe → UVoicePart） ✅ **实现+编译通过+headless 端到端验证（30 音符 UVoicePart）+ 桌面 UI 走过 More→导入→音频转写按钮到 FilePicker**；剩余 = 完整「点数→LoadingPopup→插入」真机/桌面点击闭环未能最终跑通（headless 已覆盖推理与 part 组装） +- [ ] M5：各平台产物交付（Android Libs/、desktop runtimes/）+ CI +- [ ] M6：文档与上下文更新 + +## 8. 尚未解答/需用户确认 + +- build APK 仍需 JDK + .NET 10 android workload。**注意：本机原本只有 .NET 8/9 SDK，global.json 锁 10.0.300 → 共享工程根本无法编译；M3 期间已用 dotnet-install 装了 10.0.300 + 10.0.400 到 `%USERPROFILE%\.dotnet`（用户级，无需管理员）**。后续 `dotnet` 命令需用 `$env:USERPROFILE\.dotnet\dotnet.exe` 或把该目录加进 PATH。APK 还需要 android workload + JDK。 +- 用户 fork（KakaruHayate/OpenUtauMobile）很旧；当前实施以 vocoder712 的 dev 为基线，未切到用户 fork。 + +## 9. M3 已完成（C# GameGgml.cs 接入 + 模型封包） + +**目标全部达成**：Mobile 层（非 Core）新增托管 GameGgml，P/Invoke 到 libgame_ggml_shared，桌面端跑通端到端 smoke。**模型 gguf 已确认可直接封包**（回答用户问题：是）。 + +**新增文件（全部，均不碰 OpenUtau.Core）**： +- `OpenUtauMobile/Services/Game/GameGgml.cs` — 主封装类（IDisposable）：Open/TryOpen、Infer、BackendDecidedString、VersionString、GgmlVersionString、AvailableBackendsString、LanguageId、LastErrorString；内部用 Marshal.AllocHGlobal 分配音符回读缓冲、PtrToStructure 读回。 +- `OpenUtauMobile/Services/Game/GameGgmlNative.cs` — `[LibraryImport]` 声明（AOT-safe 源生成），9 个导出 + 显式 `EntryPoint="game_capi_*"`（否则源生成按 C# 方法名找原生符号 → EntryPointNotFoundException，本次 smoke 实测抓出）。 +- `OpenUtauMobile/Services/Game/GameGgmlOptions.cs` — 参数（与 Core GameOptions 默认值一致：nsteps=8, th=0.2, radius=2, score=0.2, seed=0）。 +- `OpenUtauMobile/Services/Game/GameGgmlNote.cs` — game_capi_note 投影（Sequential struct）。 +- `OpenUtauMobile/Services/Game/GameModelResolver.cs` — 模型物化：从 EmbeddedResource 解包到 `LocalApplicationData/game/game_medium.gguf`（写临时文件后原子替换）。**模型本身不 commit**：由构建期 `EnsureGameEmbeddedModel` 目标 + `tools/extract_game_model.ps1` 从 release `.oudep` 解包到 `obj/` 并注册为 EmbeddedResource（LogicalName `OpenUtauMobile.Models.game_medium.gguf`）；模型不入 git（`Models/` 目录已删除）。 +- `OpenUtauMobile.csproj` 改动：`true`（LibraryImport 源生成必须）+ `GameOudepLocalPath`/`GameOudepUrl`/`GameModelLogicalName` 属性 + `EnsureGameEmbeddedModel` 构建目标。 +- `native/script/game-capi-cs-smoke/` — 桌面 C# 冒烟（引用共享工程 + 拷贝 M1 桌面 DLL 与 VC/ggml 运行库）。 +- `native/script/game-capi-m4-smoke/` — M4 headless 冒烟：真实语音 `.wav` → `GameGgmlMidiExtractor.Transcribe` → UVoicePart（见第 10 节）。 +- `.gitignore` 已有 `*.log`；无需新增忽略（模型经 `.oudep` 构建期解包，不入 git）。 + +**验证（M1 桌面产物 + 本机真实推理）**： +``` +GAME version: 0.1.0 ggml version: v0.19.0 +available backends: cpu decided backend: cpu +model path: C:\Users\kakar\AppData\Local\game\game_medium.gguf +notes=1 voiced=0 totalDuration=3.000s +``` +- 模型从嵌入式资源解包成功（首次 55MB 写入顺利）。 +- 3s 推理 totalDuration=3.000s 与 M1 的 C smoke（frames=300 → 3.000s）一致，证明托管↔原生数据通路正确。 +- notes=1/voiced=0 因为正弦扫频无真实人声且 nsteps=1；M1 的 13 个音符来自真实语音样本。行为正常。 +- `[FOLD]` 输出来自 game.cpp 张量折叠日志，无碍。 + +**关于"卸载 ONNX GAME 路径"澄清**：`OpenUtau.Core/Analysis/Game.cs` 是 ONNX 版；按约束**不改 Core**，所以卸载 = Mobile 层不再实例化 Core.Game，M4 转写走 GameGgml。onnxruntime libs 保留（Core 内 Rmvpe 等其它 ONNX 消费者仍需），不从 Libs/ 删。 + +**M3 关键经验**： +1. `[LibraryImport]` 必须 `EntryPoint=` 指定原生符号，否则找 C# 方法名 → 运行时炸。 +2. LibraryImport 源生成要求 `AllowUnsafeBlocks` → csproj 加一行即可。 +3. 无 .NET 10 SDK 则共享工程完全无法编译（global.json 锁 10.0.300）；本机装了 10.0.300+10.0.400 用户级 SDK。 +4. 模型封包路径：EmbeddedResource（一套资源全平台）> AndroidAsset（仅 Android，需平台代码拷贝）；50MB 级直接在 APK/程序集内，简洁且免下载。 + +## 10. M4 已完成（转写入口接线；交互 UI 验证待真机/桌面启动） + +目标 = 让用户"导入/录音音频 → 一键转写 → 生成 UVoicePart 插入音轨"，用 GameGgml 替代 Core 的 ONNX Game： + +**实现（全部在 Mobile 层，不碰 Core）**： +1. `OpenUtauMobile/Services/Game/GameGgmlMidiExtractor.cs` — 继承 Core `MidiExtractor`（**只读复用**基类的 mono→44.1k 重采样→AudioSlicer 分块→UVoicePart 组装），只实现 `TranscribeWaveform` 调 GameGgml.Infer → 转 `TranscribedNote`。 +2. `EditorMoreAction.TranscribeAudio` + `EditorMorePopup.axaml`「音频转写」按钮。 +3. `EditorViewModel.TranscribeAudio`：选音频 → 建 UWavePart+轨道（原始波形）→ `GameGgmlMidiExtractor.Transcribe` 带进度（LoadingPopupService + UpdateProgress）→ 产出的 UVoicePart 同轨插入，全部一个 undo group。 +4. 新本地化键 ×4（en/zh-Hans/ja/ru/uk 各 4 条：TranscribeAudio / TranscribeProgress / Toast.TranscribeDone / Toast.TranscribeFailed）。 +5. EditorViewModel 新增两个 using（`OpenUtau.Core.Analysis`、`OpenUtauMobile.Services.Game`）。 + +**验证**： +- `dotnet build OpenUtauMobile.Windows` → 0 错误。 +- headless M4 smoke（`native/script/game-capi-m4-smoke`）：真实 10s 语音 `w44k_10.wav` → `GameGgmlMidiExtractor.Transcribe` → **UVoicePart notes=30（全部 voiced）**，position=0 duration=9600，CPU 4.5s 墙钟（9s 音频实时推理），progress 回调 5→8→9s/9s 正常。首次 note tone=76 pos=115 dur=259，末个 tone=75 pos=9370 dur=230 —— 真实转写产物。 +- 交互 UI（桌面 Debug，Windows 内置 FilePicker）：启动 → Home 自动加载项目 → Editor → 顶部 ⋯（EditorMore=DotsThreeVertical）→ 导入 tab → 音频转写 按钮打开 FilePicker。FilePicker 默认 `UserProfile` 在当前受限 shell 下显示「无权访问此目录」为环境沙箱所致，换可读目录即可正常选文件（headless 已覆盖选文件后全部链路）。 + +> 以下为 M4 实现前的原始计划（保留作历史）： + +1. 复用 Core `MidiExtractor.Transcribe()` 的**编排逻辑（mono→重采样→AudioSlicer 分块→批推理→UVoicePart）**，但把"底层推理"换成 GameGgml：不继承/改 Core 类，在 Mobile 写一个 `GameGgmlMidiExtractor` 或直接复用 `MidiExtractor.Transcribe` 的 UVoicePart 组装部分。 + - 简单方案：Mobile 层写 `GameTranscriber.FromWave(wavePart)` —— mono(若 >1ch) → 44.1k 重采样 → 直接 `GameGgml.Infer`（原生内部已含分帧/解码，单次调用即全段，无需 AudioSlicer 改 chunk）→ 得到 `GameGgmlNote[]` → 手动转 `UProject.CreateNote(...)`/`UVoicePart`。 + - 说明：原生 infer 是全程端到端（mel→encoder→D3PM→estimator→boundary→notes），不像 ONNX 版按 chunk；不需要 replay Core 的分块逻辑。 +2. EditorMore 菜单「导入音频 → 转写」（或录音完成 → 转写）接入上面的 `GameTranscriber`：选 wavePart → 转写 → 插入 project，走 DocManager 命令撤销栈。 +3. AI/游戏后端选择进设置页（展示 backend_decided：vulkan/cpu…）。 +4. 需要 JDK + .NET 10 android workload才能打包 APK 真机验证；桌面先可开发验证。 +5. 完成后：启用 GameGgml 路径、废弃/隐藏 Core.Game 入口（不开源代码）。 + + diff --git a/native/NEXT_SESSION.md b/native/NEXT_SESSION.md new file mode 100644 index 00000000..edc5c434 --- /dev/null +++ b/native/NEXT_SESSION.md @@ -0,0 +1,74 @@ +# 下一会话接续文档（2026-08-24 归档） + +> 用途：本文件是**跨会话唯一权威接续入口**。下一会话 agent 先读本文件 + native/HANDOFF.md。 +> 本文的"已完成"项均以实际工具结果为准（已构建/已编译/已运行验证），"未完成"项是真实缺口。 + +--- + +## 0. 一句话状态 + +- M1 ✅、M2 ✅、M3 ✅ 全部完成并验证；**M4 代码实现 + 编译 + headless 端到端验证**（真实 10s 语音 → `GameGgmlMidiExtractor.Transcribe` → 30 音符 UVoicePart，4.5s CPU）。桌面 UI 已用 UI 自动化走过 More→导入→音频转写 → FilePicker 打开；FilePicker 默认 `UserProfile` 在当前受限 shell 报「无权访问此目录」为环境沙箱所致。完整「选文件→LoadingPopup→插入」桌面点击流程未再全跑（headless 已覆盖推理→part 组装）。模型已改为构建期从 game.cpp release `.oudep` 解出、不入 git。 + +## 1. 已完成的真相（全部有工具结果佐证） + +### M2 — Android 交叉编译 4 ABI（完成） +- `native/CMakeUserPresets.json`：4 个 configure preset（android-arm64/arm/x86/x64 继承 android-common）+ 4 个 build preset；选项：NDK r28c toolchain、ninja、CMAKE_BUILD_TYPE=Release、ANDROID_PLATFORM=24、ANDROID_SUPPORT_FLEXIBLE_PAGE_SIZES=OFF、GGML_NATIVE=OFF、GGML_OPENMP=OFF、**BUILD_SHARED_LIBS=OFF**（非 GGML_SHARED）、GGML_BACKEND_DL=OFF、GAME_GGML_LLAMAFILE=OFF（armeabi-v7a 必须）、加速度全关。 +- 4 ABI `.so` 已构建并放入 `OpenUtauMobile.Android/Libs//libgame_ggml_shared.so`（csproj 的 AndroidNativeLibrary glob 自动打包）： + - arm64-v8a 18.7MB / armeabi-v7a 14.3MB / x86 14.7MB / x86_64 17.5MB +- readelf 验证：全部 `Type=DYN`、`SONAME=game_ggml_shared.so`、`NEEDED` 只 libm/libdl/libc、无符号全版本化到 bionic、9 个 `game_capi_*` 导出可见。 +- 关键经验：PowerShell 5.1 传 `-DCMAKE_TOOLCHAIN_FILE=` 不可靠 → 用 preset；cmake 4.2 preset 的 cacheVariables 必须字符串；ggml 用 **BUILD_SHARED_LIBS**（无 GGML_SHARED 变量）；独立构建目录 + 网络重试。 + +### M3 — C# GameGgml 封装 + 模型（完成，桌面 smoke 端到端验证） +- `OpenUtauMobile/Services/Game/`：`GameGgml.cs`（open/infer/backend/version）、`GameGgmlNative.cs`（9 个 LibraryImport，**显式 EntryPoint="game_capi_*"**）、`GameGgmlOptions.cs`、`GameGgmlNote.cs`、`GameModelResolver.cs`（EmbeddedResource → LocalApplicationData/game/game_medium.gguf）。 +- 共享 `OpenUtauMobile.csproj`：`AllowUnsafeBlocks=true`（LibraryImport 必需）、`EnsureGameEmbeddedModel` 构建目标（从 release `.oudep` 解 gguf 到 obj 并注册为 EmbeddedResource）+ **`tools/extract_game_model.ps1`** + `GameOudepLocalPath`/`GameOudepUrl`/`GameModelLogicalName` 属性。 +- 模型来源（按用户要求，不优雅但已实现）：**不入 git**；构建时从本机 `J:\GGML-GAME\vendor\game-cpp-release\game_ggml-windows-x64-vulkan-q8.oudep`（zip，内含 game_medium.gguf 57,754,848 B）解出；没有则从 URL `https://github.com/KakaruHayate/game.cpp/releases/download/v0.1.3/game_ggml-windows-x64-vulkan-q8.oudep` 下载（HEAD 已验证 200）。删除 `Models/` 目录。 +- **桌面 C# smoke（native/script/game-capi-cs-smoke/）端到端通过**：GAME 0.1.0 / ggml v0.19.0 / backends cpu / decided cpu / infer 3s → totalDuration=3.000s；删掉本地模型后能重新从程序集解包（57.7MB）→ 验证封包链路完整。 +- 本机装了 .NET SDK 10.0.300 + 10.0.400 到 `%USERPROFILE%\.dotnet`（无管理员）。后续 `dotnet` 必须用 `$env:USERPROFILE\.dotnet\dotnet.exe` 或加 PATH。 + +### M4 — 转写入口接线（实现 + 编译通过；未交互运行验证） +- `OpenUtauMobile/Services/Game/GameGgmlMidiExtractor.cs` **存在且编译通过**（这是本会话里"写文件失败→重新真实写入→build 0 错误"的最终状态）。 +- `EditorMoreAction.TranscribeAudio` + `EditorMorePopup.axaml`「音频转写」按钮 + `EditorViewModel` case + `TranscribeAudio()` 方法（LoadingPopupService 进度 → Transcribe → UVoicePart 同轨插入，单 undo group）+ 新增两个 using（`OpenUtau.Core.Analysis`、`OpenUtauMobile.Services.Game`）。 +- 5 个 resx（en/zh-Hans/ja/ru/uk）各加 4 键：TranscribeAudio / TranscribeProgress / Toast.TranscribeDone / Toast.TranscribeFailed。 +- `dotnet build OpenUtauMobile.csproj` 最终 **0 错误、已成功生成**。→ 全部 M4 代码真实落盘且一致。 + +## 2. 真实缺口（必须完成的） + +1. **交互 UI 验证 M4**：✅ headless 端到端验证完成（真实 10s 语音 → Transcribe → 30 音符 UVoicePart，4.5s CPU）+ 桌面 UI 自动化走过 More→导入→音频转写→FilePicker 打开。遗留 = 完整「选文件→LoadingPopup 进度→插入」桌面/真机点击闭环未最终跑（headless 已覆盖推理与 part 组装）。桌面运行命令：在 `OpenUtauMobile.Windows` 项目 `dotnet build -t:Run -c Debug`（用 `%USERPROFILE%\.dotnet\dotnet.exe`）。 +2. **HANDOFF.md 第 8 节 dotnet 9.0.317 描述**：✅ 已改为 10.0.300+10.0.400 用户级 SDK；APK 仍需 .NET 10 android workload + JDK。 +3. **HANDOFF.md 第 9/10 节 Models/ 残留描述**：✅ 已核对并修复（不再提 Models/game_medium.gguf，改为 `.oudep` 构建期解包）。 +4. **DECISIONS.md M3 模型决策条目**：✅ 已同步为「构建期 .oudep 解包、不入 git」（保留被取代的原条目 + 新条目）。 + +## 3. 明确的坑（避免重踩） + +- **工具调用不可靠性**：本会话后半段多次出现 "写文件/编辑返回成功但实际未落盘" 的幻觉。**每次 write/edit 后用 `Test-Path` 或 grep 实体核验**，以磁盘为准，不轻信工具结果文本。 +- 模型解包脚本独立可测：`powershell -NoProfile -ExecutionPolicy Bypass -File OpenUtauMobile\tools\extract_game_model.ps1 -Oudep <路径> -Target <目标>`。 +- `cmake --preset` 要在 `native/` 目录下执行；cacheVariables 全部字符串。 +- LibraryImport 必须带 `EntryPoint`，否则运行 EntryPointNotFoundException。 + +## 4. 未决 / 后续里程碑 + +- M5：Android Libs 已就位（4 ABI）；desktop runtimes/ 交付 + CI（把 4 .so + Windows dll 纳入产物与自动化）。 +- M5 需决定建包路径：装 JDK + `dotnet workload install android`（用 10.0.300 SDK）→ `OpenUtauMobile.Android` 出 APK，验证 4 ABI 在真机/模拟器 dlopen + 转写。 +- M4 之后可加：设置页显示 `backend_decided`（Vulkan/CPU）——目前 smoke 已验证该 API，未接 UI。 +- 全程遵守：不修改 `OpenUtau.Core` / `OpenUtau.Plugin.Builtin`；不引入 player subprocess。 + +## 5. 下一会话 Prompt（可整段复制，粘贴给 agent） + +""" +继续 OpenUtauMobile × game.cpp 深度植入，从 M4 交互验证继续。 +【先读】读 J:\GGML-GAME\OpenUtauMobile\native\NEXT_SESSION.md 与 native\HANDOFF.md——这是权威接续文档，含全部已完成/缺口/坑。 + +【已验证前提】 +- M1 桌面 build+smoke 完成；native/shim 9 个 C ABI 导出正常。 +- M2 完成：native/CMakeUserPresets.json 4 ABI preset（BUILD_SHARED_LIBS=OFF、GGML_LLAMAFILE=OFF、NDK r28c、ninja），4 个 libgame_ggml_shared.so 已放 OpenUtauMobile.Android/Libs//（readelf 验证过）。 +- M3 完成：OpenUtauMobile/Services/Game/GameGgml*.cs 全套 + LibraryImport(EntryPoint) + 模型构建期从 game.cpp release .oudep 解包（tools/extract_game_model.ps1），桌面 C# smoke 端到端通过（3.000s）。 +- .NET SDK 10.0.300 装在 %USERPROFILE%\.dotnet（dotnet 命令必须用它或加 PATH）。 + +【M4 当前缺口 → 你的任务】 +1. 交互运行验证：本机跑 OpenUtauMobile.Windows（不需要 JDK），EditorMore → 音频转写 → 选音频 → 看进度/结果。修一切运行时问题。 +2. 若有问题：排查 GameGgmlMidiExtractor（继承 Core.MidiExtractor，只实现 TranscribeWaveform）与 EditorViewModel.TranscribeAudio（LoadingPopup 进度、undo group、track 插入）。 +3. 同步 .agent/DECISIONS.md 的过时模型决策条目（→ 改 构建期 .oudep 解包），修 HANDOFF 里过时的 .NET 9/Models 残留描述。 +4. 做完后：若需真机验证，安装 JDK +（用 10.0.300 SDK）dotnet workload install android，构建 OpenUtauMobile.Android 出 APK（Libs 已有 4 ABI .so），在模拟器/真机验证 dlopen + 转写。 + +【约束】不修改 OpenUtau.Core / OpenUtau.Plugin.Builtin；C# 集成全放 Mobile 层（OpenUtauMobile/）；GPU(若有) Vulkan 编译进库、运行时自动 fallback CPU，不引入子进程。 +报告：运行验证结果 + 修的 bug + 遗留。""" diff --git a/native/game.cpp b/native/game.cpp new file mode 160000 index 00000000..97f92770 --- /dev/null +++ b/native/game.cpp @@ -0,0 +1 @@ +Subproject commit 97f92770704c154e1af4a9b3066b7701041dbc38 diff --git a/native/script/game-capi-cs-smoke/Program.cs b/native/script/game-capi-cs-smoke/Program.cs new file mode 100644 index 00000000..5201f57b --- /dev/null +++ b/native/script/game-capi-cs-smoke/Program.cs @@ -0,0 +1,79 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using OpenUtauMobile.Services.Game; + +namespace GameCapiSmoke; + +/// +/// 托管侧（C#）冒烟：验证 GameGgml 封装能对接原生 libgame_ggml_shared， +/// 从嵌入式模型解包 → open → infer → 回读音符。等用于 C 侧 game_capi_check， +/// 但走的是 .NET 10 LibraryImport 托管封装。 +/// +internal static class Program +{ + private static int Main(string[] args) + { + Console.WriteLine("== GameCapi C# smoke =="); + Console.WriteLine($"GAME version: {GameGgml.VersionString()}"); + Console.WriteLine($"ggml version: {GameGgml.GgmlVersionString()}"); + Console.WriteLine($"available backends: {GameGgml.AvailableBackendsString()}"); + + GameGgml? model = GameGgml.Open(out string openError); + if (model == null) + { + Console.WriteLine($"打开模型失败: {openError}"); + return 1; + } + + try + { + Console.WriteLine($"model path: {model.ModelPath}"); + Console.WriteLine($"decided backend: {model.BackendDecidedString()}"); + + // 3 秒 44.1k 单声道正弦滑音（220->440Hz），峰值 0.5（对齐 M1 smoke）。 + const int sampleRate = 44100; + const double durationSeconds = 3.0; + const float amplitude = 0.5f; + float[] wave = BuildSineSweep(sampleRate, (int)(sampleRate * durationSeconds), 220.0, 440.0, amplitude); + + var options = new GameGgmlOptions + { + SamplingSteps = 1, // 最快，校验输出 + Seed = 42UL, + }; + IReadOnlyList notes = model.Infer(wave, options); + + int voiced = notes.Count(n => n.Voiced != 0); + double totalSeconds = notes.Sum(n => (double)n.DurationSeconds); + Console.WriteLine($"notes={notes.Count} voiced={voiced} totalDuration={totalSeconds:F3}s"); + if (voiced > 0) + { + Console.WriteLine($"first voiced: t={notes.First(n => n.Voiced != 0).OffsetSeconds:F3}s " + + $"dur={notes.First(n => n.Voiced != 0).DurationSeconds:F3}s " + + $"midi={notes.First(n => n.Voiced != 0).PitchMidi:F1}"); + } + + return 0; + } + finally + { + model.Dispose(); + } + } + + private static float[] BuildSineSweep(int sampleRate, int count, double startFreq, double endFreq, float amplitude) + { + float[] result = new float[count]; + double freqPerSample = (endFreq - startFreq) / count; + double phase = 0.0; + for (int i = 0; i < count; i++) + { + double freq = startFreq + freqPerSample * i; + phase += 2.0 * Math.PI * freq / sampleRate; + result[i] = (float)(amplitude * Math.Sin(phase)); + } + + return result; + } +} diff --git a/native/script/game-capi-cs-smoke/game-capi-cs-smoke.csproj b/native/script/game-capi-cs-smoke/game-capi-cs-smoke.csproj new file mode 100644 index 00000000..41b16b62 --- /dev/null +++ b/native/script/game-capi-cs-smoke/game-capi-cs-smoke.csproj @@ -0,0 +1,30 @@ + + + Exe + net10.0 + enable + preview + game-capi-cs-smoke + GameCapiSmoke + false + + ..\..\..\native\build-desktop\app + + + + + + + + + + + + + + + + + + + diff --git a/native/script/game-capi-m4-smoke/Program.cs b/native/script/game-capi-m4-smoke/Program.cs new file mode 100644 index 00000000..de1361c4 --- /dev/null +++ b/native/script/game-capi-m4-smoke/Program.cs @@ -0,0 +1,92 @@ +using System; +using System.Collections.Generic; +using System.Diagnostics; +using System.Linq; +using OpenUtau.Core.Analysis; +using OpenUtau.Core.Ustx; +using OpenUtauMobile.Services.Game; + +namespace GameM4Smoke; + +/// +/// M4 无 GUI 端到端冒烟：用真实语音 .wav 驱动 M4 的 GameGgmlMidiExtractor.Transcribe，验证 +/// 「音频 → mono/重采样 → AudioSlicer 分块 → GAME ggml infer → UVoicePart 生成」整条链路。 +/// 与 App 内 EditorViewModel.TranscribeAudio 的差别只有：不走 FilePicker/LoadingPopup/undo 命令。 +/// +internal static class Program +{ + private static int Main(string[] rawArgs) + { + Console.WriteLine("== M4 headless smoke: GameGgmlMidiExtractor.Transcribe =="); + Console.WriteLine($"GAME version: {GameGgml.VersionString()}"); + Console.WriteLine($"ggml version: {GameGgml.GgmlVersionString()}"); + Console.WriteLine($"available backends: {GameGgml.AvailableBackendsString()}"); + + string wav = rawArgs.Length > 0 && System.IO.File.Exists(rawArgs[0]) + ? rawArgs[0] + : System.IO.Path.GetFullPath(System.IO.Path.Combine("pick", "voice10.wav")); + if (!System.IO.File.Exists(wav)) + { + Console.WriteLine($"音频不存在: {wav}"); + return 1; + } + + List lookback = new(); + try + { + UProject project = new(); + UWavePart wavePart = new() { FilePath = wav }; + wavePart.Load(project); + wavePart.Peaks.Wait(); // 等待 Samples 就绪 + if (wavePart.Samples == null) + { + Console.WriteLine("Samples 未就绪 (null)"); + return 1; + } + + Console.WriteLine( + $"wave: ch={wavePart.channels} sampleRate={wavePart.sampleRate} samples={wavePart.Samples.Length}"); + + Stopwatch sw = Stopwatch.StartNew(); + int totalDone = 0, totalTotal = 0; + UVoicePart? part = null; + using (GameGgmlMidiExtractor extractor = new()) + { + GameOptions options = new() { SamplingSteps = 8 }; + part = extractor.Transcribe( + project, wavePart, options, null, null, + (done, total) => + { + totalDone = done; + totalTotal = total; + Console.WriteLine($" progress {done}s / {total}s"); + }); + } + + sw.Stop(); + if (part == null) + { + Console.WriteLine("Transcribe 返回 null（确认回调取消？）"); + return 1; + } + + int voiced = part.notes.Count; + Console.WriteLine($"elapsed={sw.Elapsed.TotalSeconds:F1}s progress={totalDone}s/{totalTotal}s"); + Console.WriteLine($"UVoicePart notes={part.notes.Count} voiced={voiced}"); + Console.WriteLine($"part position={part.position} duration={part.Duration}"); + if (voiced > 0) + { + UNote first = part.notes.First(); + Console.WriteLine($"first note: pos={first.position} dur={first.duration} tone={first.tone} lyric='{first.lyric}'"); + UNote last = part.notes.Last(); + Console.WriteLine($"last note: pos={last.position} dur={last.duration} tone={last.tone} lyric='{last.lyric}'"); + } + return 0; + } + catch (Exception ex) + { + Console.WriteLine($"[FAIL] {ex}"); + return 1; + } + } +} diff --git a/native/script/game-capi-m4-smoke/game-capi-m4-smoke.csproj b/native/script/game-capi-m4-smoke/game-capi-m4-smoke.csproj new file mode 100644 index 00000000..d92c957d --- /dev/null +++ b/native/script/game-capi-m4-smoke/game-capi-m4-smoke.csproj @@ -0,0 +1,28 @@ + + + Exe + net10.0 + enable + preview + game-capi-m4-smoke + GameM4Smoke + ..\..\..\native\build-desktop\app + + + + + + + + + + + + + + + + + + + diff --git a/native/script/ui-scratch/click.ps1 b/native/script/ui-scratch/click.ps1 new file mode 100644 index 00000000..0948a138 --- /dev/null +++ b/native/script/ui-scratch/click.ps1 @@ -0,0 +1,18 @@ +param([int]$X, [int]$Y, [int]$WaitMs = 1500) +$sig = @' +using System; +using System.Runtime.InteropServices; +public static class Clicker { + [DllImport("user32.dll")] static extern bool SetCursorPos(int x, int y); + [DllImport("user32.dll")] static extern void mouse_event(uint f, uint dx, uint dy, uint d, UIntPtr x); + public static void Click(int x, int y) { + SetCursorPos(x, y); + mouse_event(2, 0, 0, 0, UIntPtr.Zero); + mouse_event(4, 0, 0, 0, UIntPtr.Zero); + } +} +'@ +Add-Type -TypeDefinition $sig +[Clicker]::Click($X, $Y) +Write-Output "clicked $X,$Y" +Start-Sleep -Milliseconds $WaitMs diff --git a/native/script/ui-scratch/dump.ps1 b/native/script/ui-scratch/dump.ps1 new file mode 100644 index 00000000..0ba7408a --- /dev/null +++ b/native/script/ui-scratch/dump.ps1 @@ -0,0 +1,9 @@ +Add-Type -AssemblyName UIAutomationClient, UIAutomationTypes +$p = Get-Process OpenUtauMobile* | Where-Object MainWindowHandle -ne 0 +$root = [System.Windows.Automation.AutomationElement]::FromHandle($p.MainWindowHandle) +Write-Output "window: $($root.Current.Name) at $([int]$root.Current.BoundingRectangle.X),$([int]$root.Current.BoundingRectangle.Y) $([int]$root.Current.BoundingRectangle.Width)x$([int]$root.Current.BoundingRectangle.Height)" +$btns = $root.FindAll([System.Windows.Automation.TreeScope]::Descendants, (New-Object System.Windows.Automation.PropertyCondition([System.Windows.Automation.AutomationElement]::ControlTypeProperty, [System.Windows.Automation.ControlType]::Button))) +$i=0; foreach ($b in $btns) { $r=$b.Current.BoundingRectangle; if ($r.Right -gt $r.Left) { $i++; Write-Output ("b{0}: x={1} y={2} w={3} h={4} name='{5}'" -f $i,[int]$r.X,[int]$r.Y,[int]$r.Width,[int]$r.Height,$b.Current.Name) } } +Write-Output "--- texts ---" +$txts = $root.FindAll([System.Windows.Automation.TreeScope]::Descendants, (New-Object System.Windows.Automation.PropertyCondition([System.Windows.Automation.AutomationElement]::ControlTypeProperty, [System.Windows.Automation.ControlType]::Text))) +foreach ($t in $txts) { $n=$t.Current.Name; $r=$t.Current.BoundingRectangle; if ($n -and $r.Right -gt $r.Left) { Write-Output "txt '$n' at $([int]$r.X),$([int]$r.Y)" } } diff --git a/native/script/ui-scratch/shot.ps1 b/native/script/ui-scratch/shot.ps1 new file mode 100644 index 00000000..71fd859c --- /dev/null +++ b/native/script/ui-scratch/shot.ps1 @@ -0,0 +1,14 @@ +param([string]$Out = "shot.png", [int]$Monitor = 0) +Add-Type -AssemblyName System.Windows.Forms, System.Drawing -ErrorAction Stop +$dir = Split-Path -Parent $Out +if ($dir) { New-Item -ItemType Directory -Force -Path $dir | Out-Null } +$screens = [System.Windows.Forms.Screen]::AllScreens +if ($Monitor -ge $screens.Length) { throw "Monitor $Monitor not available ($($screens.Length) screens)" } +$b = $screens[$Monitor].Bounds +$bmp = New-Object System.Drawing.Bitmap $b.Width, $b.Height +$g = [System.Drawing.Graphics]::FromImage($bmp) +$g.CopyFromScreen($b.Location, [System.Drawing.Point]::Empty, $b.Size) +$g.Dispose() +$bmp.Save($Out, [System.Drawing.Imaging.ImageFormat]::Png) +$bmp.Dispose() +Write-Output "shot -> $Out ($((Get-Item $Out).Length) bytes)" diff --git a/native/script/vulkan-build/build-desk-vulkan.bat b/native/script/vulkan-build/build-desk-vulkan.bat new file mode 100644 index 00000000..8d083505 --- /dev/null +++ b/native/script/vulkan-build/build-desk-vulkan.bat @@ -0,0 +1,14 @@ +@echo off +REM Build desktop Vulkan variant (no reconfigure - uses existing build-desktop-vulkan). +setlocal +call "C:\Program Files (x86)\Microsoft Visual Studio\18\BuildTools\VC\Auxiliary\Build\vcvars64.bat" >nul +if errorlevel 1 ( echo VCVARS_FAILED & exit /b 1 ) + +set CMAKE="C:\Program Files (x86)\Microsoft Visual Studio\18\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" +set BLDV=J:\GGML-GAME\OpenUtauMobile\native\build-desktop-vulkan + +REM Existing configure in $BLDV has Vulkan=ON; just build. +%CMAKE% --build %BLDV% --target game_ggml_shared game_capi_check -j 12 +if errorlevel 1 ( echo BUILD_FAILED & exit /b 1 ) +echo BUILD_OK +exit /b 0 diff --git a/native/script/vulkan-build/cfg-desk-vulkan.bat b/native/script/vulkan-build/cfg-desk-vulkan.bat new file mode 100644 index 00000000..83442acf --- /dev/null +++ b/native/script/vulkan-build/cfg-desk-vulkan.bat @@ -0,0 +1,24 @@ +@echo off +REM Desktop Vulkan reconfigure: load MSVC vcvars64, then cmake configure build-desktop with Vulkan ON. +setlocal +call "C:\Program Files (x86)\Microsoft Visual Studio\18\BuildTools\VC\Auxiliary\Build\vcvars64.bat" >nul +if errorlevel 1 ( echo VCVARS_FAILED & exit /b 1 ) + +set CMAKE="C:\Program Files (x86)\Microsoft Visual Studio\18\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" +set NINJA="C:\Program Files (x86)\Microsoft Visual Studio\18\BuildTools\Common7\IDE\CommonExtensions\Microsoft\CMake\Ninja\ninja.exe" +set SRC=J:\GGML-GAME\OpenUtauMobile\native +set BLD=J:\GGML-GAME\OpenUtauMobile\native\build-desktop + +REM Use a SEPARATE build dir so the working CPU build stays intact as fallback. +set BLDV=J:\GGML-GAME\OpenUtauMobile\native\build-desktop-vulkan +%CMAKE% -S %SRC% -B %BLDV% -G Ninja -DCMAKE_BUILD_TYPE=Release ^ + -DCMAKE_MAKE_PROGRAM=%NINJA% ^ + -DGAME_GGML_VULKAN=ON -DGGML_VULKAN=ON ^ + -DGGML_NATIVE=ON -DGGML_OPENMP=ON ^ + -DFETCHCONTENT_SOURCE_DIR_GGML="J:\GGML-GAME\OpenUtauMobile\native\build-desktop\_deps\ggml-src" ^ + -DFETCHCONTENT_SOURCE_DIR_POCKETFFT="J:\GGML-GAME\OpenUtauMobile\native\build-desktop\_deps\pocketfft-src" ^ + -DFETCHCONTENT_SOURCE_DIR_DR_LIBS="J:\GGML-GAME\OpenUtauMobile\native\build-desktop\_deps\dr_libs-src" ^ + -DGAME_GGML_BUILD_CLI=OFF -DGAME_GGML_BUILD_TESTS=OFF -DOPUM_NATIVE_SMOKE=ON +if errorlevel 1 ( echo CONFIGURE_FAILED & exit /b 1 ) +echo CONFIGURE_OK +exit /b 0 diff --git a/native/shim/game_capi.cpp b/native/shim/game_capi.cpp new file mode 100644 index 00000000..759a3baa --- /dev/null +++ b/native/shim/game_capi.cpp @@ -0,0 +1,214 @@ +// --------------------------------------------------------------------------- +// game_capi.cpp — C ABI 桥接实现 +// +// 目标：把 game_ggml::Model 的 C++ 面收敛成一组稳定的 extern "C" 函数， +// 供 .NET(Avalonia UI 层) 通过 P/Invoke 调用，避免把 C++ 类型/ggml 透传给托管层。 +// +// 约束遵循 game.cpp 上游： +// * Model 非线程安全 -> 句柄级串行由调用方保证。 +// * 44100Hz 单声道 float, 取值 [-1,1]。 +// * 默认 D3PM nsteps=1（最快）；8 更高质量。 +// * 所有 C++ 异常在边界转成 错误码 + last-error 消息。 +// --------------------------------------------------------------------------- + +#include "game_capi.h" + +#include "game_ggml/config.h" +#include "game_ggml/errors.h" +#include "game_ggml/game_ggml.h" +#include "game_ggml/model.h" +#include "game_ggml/version.h" + +#include +#include +#include +#include + +namespace { + +using game_ggml::Model; + +struct ModelContext { + std::unique_ptr model; + std::string config_json; // 透传/备用（当前不解析, 仅为未来扩展留位） + std::string last_error; // 最近一次 C++ 异常的析出文本 +}; + +// 捕获当前挂起的异常 -> 文本。须在 catch 块内调用。 +std::string exception_text(const char * prefix) { + try { throw; } catch (const std::exception & e) { + return std::string(prefix) + e.what(); + } catch (...) { + return std::string(prefix) + "unknown C++ exception"; + } +} + +// 写入 C 头约定的 errbuf（截断 + NUL 结尾） +void write_errbuf(char * errbuf, int errcap, const std::string & text) { + if (!errbuf || errcap <= 0) return; + std::size_t n = text.size(); + std::size_t bulk = static_cast(errcap - 1); + if (n > bulk) n = bulk; + if (n > 0) std::memcpy(errbuf, text.data(), n); + errbuf[n] = '\0'; +} + +int copy_to_buf(char * buf, int cap, const std::string & s) { + if (!buf || cap <= 0) return 0; + std::strncpy(buf, s.c_str(), static_cast(cap - 1)); + buf[cap - 1] = '\0'; + return static_cast(s.size() + 1); +} + +} // namespace + +extern "C" int game_capi_version(char * buf, int cap) { + return copy_to_buf(buf, cap, game_ggml::version_string()); +} + +extern "C" int game_capi_ggml_version(char * buf, int cap) { + return copy_to_buf(buf, cap, game_ggml::ggml_version_string()); +} + +extern "C" int game_capi_available_backends(char * buf, int cap) { + std::string joined; + const char * const * names = game_ggml::available_backends(); + const int count = game_ggml::available_backends_count(); + for (int i = 0; i < count; ++i) { + if (i) joined += ','; + joined += names[i]; + } + return copy_to_buf(buf, cap, joined); +} + +extern "C" game_capi_model * game_capi_open( + const char * gguf_path, const char * config_json, + char * errbuf, int errcap) { + if (!gguf_path || !*gguf_path) { + write_errbuf(errbuf, errcap, "gguf_path is empty"); + return nullptr; + } + + auto * ctx = new (std::nothrow) ModelContext(); + if (!ctx) { + write_errbuf(errbuf, errcap, "out of memory (ModelContext)"); + return nullptr; + } + if (config_json) ctx->config_json = config_json; + + try { + ctx->model = std::make_unique(Model::load(std::string(gguf_path))); + } catch (const game_ggml::BackendError & e) { + ctx->last_error = exception_text("backend: "); + write_errbuf(errbuf, errcap, ctx->last_error); + delete ctx; + return nullptr; + } catch (const game_ggml::GgufError & e) { + ctx->last_error = exception_text("gguf: "); + write_errbuf(errbuf, errcap, ctx->last_error); + delete ctx; + return nullptr; + } catch (const game_ggml::NotImplemented & e) { + ctx->last_error = exception_text("not-implemented: "); + write_errbuf(errbuf, errcap, ctx->last_error); + delete ctx; + return nullptr; + } catch (const std::exception & e) { + ctx->last_error = exception_text("load: "); + write_errbuf(errbuf, errcap, ctx->last_error); + delete ctx; + return nullptr; + } catch (...) { + ctx->last_error = "load: unknown error"; + write_errbuf(errbuf, errcap, ctx->last_error); + delete ctx; + return nullptr; + } + + return reinterpret_cast(ctx); +} + +extern "C" void game_capi_close(game_capi_model * m) { + if (!m) return; + delete reinterpret_cast(m); +} + +extern "C" int game_capi_backend_decided(game_capi_model * m, char * buf, int cap) { + if (!m || !buf || cap <= 0) return 0; + ModelContext * ctx = reinterpret_cast(m); + if (!ctx->model) return copy_to_buf(buf, cap, "?"); + // 实际选中后端在 Model::load 内部由 init_best_backend 决定（GPU→CPU fallback 后 + // 可能并非 available_backends()[0]），且 public API 未暴露该值的访问器； + // 不触碰 internals()/Unstable API，因此无法可靠获知 -> 显式返回 unknown，避免误报。 + return copy_to_buf(buf, cap, "unknown"); +} + +extern "C" int game_capi_language_id(game_capi_model * m, const char * lang_code) { + if (!m || !lang_code || !*lang_code) return -1; + ModelContext * ctx = reinterpret_cast(m); + if (!ctx->model) return -1; + const auto & lang_map = ctx->model->config().inference.lang_map; + auto it = lang_map.find(std::string(lang_code)); + if (it == lang_map.end()) return -1; + return it->second; +} + +extern "C" int game_capi_infer( + game_capi_model * m, + const float * waveform, std::size_t n, + int language, int nsteps, + float seg_threshold, int seg_radius, + float est_threshold, std::uint64_t seed, + game_capi_note * notes_out, int notes_capacity, + int * notes_count, int * num_frames) { + if (!m) return GAME_CAPI_ERR_HANDLE; + ModelContext * ctx = reinterpret_cast(m); + if (!ctx->model) return GAME_CAPI_ERR_HANDLE; + if (notes_count) *notes_count = 0; + if (num_frames) *num_frames = 0; + if (!waveform || n == 0) return GAME_CAPI_ERR_INVALID_ARG; + + game_ggml::InferParams params; + params.language = language; + params.d3pm_nsteps = nsteps > 0 ? nsteps : 1; + params.boundary_threshold = seg_threshold; + params.boundary_radius = seg_radius; + params.note_threshold = est_threshold; + params.seed = seed; + + game_ggml::InferResult result; + try { + result = ctx->model->infer(waveform, n, params); + } catch (const game_ggml::InvalidArgument & e) { + ctx->last_error = exception_text("arg: "); + return GAME_CAPI_ERR_INVALID_ARG; + } catch (const std::exception & e) { + ctx->last_error = exception_text("infer: "); + return GAME_CAPI_ERR_INFER; + } catch (...) { + ctx->last_error = "infer: unknown error"; + return GAME_CAPI_ERR_INFER; + } + + if (num_frames) *num_frames = result.num_frames; + + int total = static_cast(result.notes.size()); + int copy = (total > notes_capacity) ? notes_capacity : total; + if (notes_out && copy > 0) { + for (int i = 0; i < copy; ++i) { + const game_ggml::Note & src = result.notes[static_cast(i)]; + notes_out[i].offset_seconds = src.offset_seconds; + notes_out[i].duration_seconds = src.duration_seconds; + notes_out[i].pitch_midi = src.pitch_midi; + notes_out[i].voiced = src.voiced ? 1 : 0; + } + } + if (notes_count) *notes_count = copy; + return GAME_CAPI_OK; +} + +extern "C" const char * game_capi_last_error(game_capi_model * m) { + if (!m) return nullptr; + const ModelContext * ctx = reinterpret_cast(m); + return ctx->last_error.empty() ? nullptr : ctx->last_error.c_str(); +} diff --git a/native/shim/game_capi.h b/native/shim/game_capi.h new file mode 100644 index 00000000..220d42c4 --- /dev/null +++ b/native/shim/game_capi.h @@ -0,0 +1,104 @@ +#pragma once +// --------------------------------------------------------------------------- +// game_capi — C ABI 桥接层：把 game.cpp (game_ggml::Model) 暴露给 .NET P/Invoke。 +// +// 设计原则： +// - 纯 C 头，可被 C# 以 [DllImport] / LibraryImport 直接引用（不含 C++ 类型）。 +// - 不透明句柄 game_capi_model*，进程内单/多实例都由本层管理。 +// - 所有函数返回 int（0 = 成功）；失败时把可读错误写入调用方提供的缓冲区。 +// - 语音波形为 float32 单声道、采样率 44100Hz、取值 [-1,1]（与上游一致）。 +// - Model::infer 非线程安全 -> 每个 game_capi_model 由调用方保证串行。 +// --------------------------------------------------------------------------- + +#include +#include + +#ifdef __cplusplus +extern "C" { +#endif + +// 一条转写音符（对应 game_ggml::Note 的 POD 投影）。 +typedef struct game_capi_note { + float offset_seconds; // 起始时间（秒） + float duration_seconds; // 持续时间（秒） + float pitch_midi; // 小数 MIDI 音高（仅 voiced 有效） + int voiced; // 1=有声部 0=休止/无音高 +} game_capi_note; + +// 不透明模型句柄。 +typedef struct game_capi_model game_capi_model; + +// 返回字符串缓冲的推荐容量（含 NUL），调用方按此分配。 +enum { GAME_CAPI_ERRBUF = 512 }; + +// ---- 版本 / 后端能力（编译期信息，无状态）------------------------------- +// 写产物版本号到 buf（如 "0.1.0"）。返回 buf 需要的长度。 +int game_capi_version(char * buf, int cap); +// 写 ggml 版本（如 "v0.19.0"）。返回长度。 +int game_capi_ggml_version(char * buf, int cap); + +// 把编译期可用的后端名（小写, 逗号分隔, 如 "vulkan,cpu"）写入 buf。返回长度。 +// 该列表来自 GAME_GGML_HAS_* 宏, 反映"此库编译进了哪些加速器"。 +int game_capi_available_backends(char * buf, int cap); + +// ---- 模型生命周期 --------------------------------------------------------- +// 打开 GGUF 权重并构建 backend（内部走 game_ggml::Model::load, +// 自动 GPU->CPU fallback）。返回非空句柄, 失败返回 NULL 并把错误写进 errbuf。 +game_capi_model * game_capi_open(const char * gguf_path, + const char * config_json, // 可为 NULL; 供未来透传 + char * errbuf, int errcap); + +// 关闭并释放。NULL 安全。不可与同句柄的 infer 并发。 +void game_capi_close(game_capi_model * m); + +// 返回该实例运行时实际选中的后端名（写 buf）。用于诊断/UI 展示。 +// 注意：public API 未暴露 Model 实际选中的后端，因此无法可靠获知时返回 "unknown"， +// 而不是用 available_backends()[0] 猜测（GPU→CPU fallback 后可能与实际不符）。 +int game_capi_backend_decided(game_capi_model * m, char * buf, int cap); + +// ---- 推理（串行）---------------------------------------------------------- +// 对 44100Hz 单声道 float 波形做端到端转写, 结果追加到 notes_out 数组。 +// +// 参数: +// m 模型句柄（来自 game_capi_open） +// waveform 波形指针; n 个样本 +// n 样本数（> 0） +// language 语言 id（0=universal; 用 game_capi_lang_map 查） +// nsteps D3PM 去噪步数（1=最快; 8=更高质量; 默认建议 8 或 1） +// seg_threshold 边界解码阈值 +// seg_radius 边界解码半径（帧） +// est_threshold 音符存在性门槛 +// seed 随机种子（0=自动/OS 随机） +// notes_out 由调用方分配的 game_capi_note 数组 +// notes_capacity notes_out 容量 +// notes_count 回填实际音符数（不会超过 capacity） +// num_frames 回填 mel 帧数（诊断用, 可 NULL） +// +// 返回 0 = 成功(可能 0 个音符); 负值 = 错误码。错误码常量见下。 +int game_capi_infer(game_capi_model * m, + const float * waveform, size_t n, + int language, int nsteps, + float seg_threshold, int seg_radius, + float est_threshold, uint64_t seed, + game_capi_note * notes_out, int notes_capacity, + int * notes_count, int * num_frames); + +// 语言 id 查询: 把语言码（"zh","en",...）映射为数字 id。 +// 返回 id; 未知名返回 -1（调用方可按 0/universal 处理）。 +int game_capi_language_id(game_capi_model * m, const char * lang_code); + +// ---- 方便的错误信息 ------------------------------------------------------ +// 最近一次失败的详细消息（线程局部, 单实例够用）。可为 NULL。 +const char * game_capi_last_error(game_capi_model * m); + +// 错误码约定 +#define GAME_CAPI_OK 0 +#define GAME_CAPI_ERR_HANDLE -1 // 空句柄 +#define GAME_CAPI_ERR_INIT -2 // backend/模型初始化失败 +#define GAME_CAPI_ERR_INFER -3 // 推理抛异常 +#define GAME_CAPI_ERR_INVALID_ARG -4 // 参数非法 + +#ifdef __cplusplus +} +#endif + diff --git a/native/shim/smoke_main.cpp b/native/shim/smoke_main.cpp new file mode 100644 index 00000000..06de718d --- /dev/null +++ b/native/shim/smoke_main.cpp @@ -0,0 +1,132 @@ +// --------------------------------------------------------------------------- +// smoke_main.cpp — game_capi 原生冒烟验证（开发用, 不进 app 包） +// +// 用法: game_capi_check [nsteps] +// - 读取 wav（48k/44.1k 单声道皆自动降采样到 44.1k… 简化：本工具只接受 +// PCM16 单声道 WAV，44.1kHz；由调用方准备好）。 +// - 调 game_capi_open/infer 打印结果。 +// +// 真实 .NET 层走 MidiExtractor 的重采样/切片；此工具仅为快速验证 C ABI 正确性， +// 简化处理即可。 +// --------------------------------------------------------------------------- + +#include "game_capi.h" + +#include +#include +#include +#include +#include + +namespace { + +// 读取 PCM16 单声道 44.1kHz WAV 的裸样本（仅 data chunk, 无解码库）。 +// 简化：确认 fmt 是 PCM, channels=1, sample rate=44100。 +bool load_wav_pcm16(const char * path, std::vector & out, int & sr) { + FILE * f = std::fopen(path, "rb"); + if (!f) { std::fprintf(stderr, "cannot open %s\n", path); return false; } + // RIFF 头 + char riff[4]; std::fread(riff, 1, 4, f); + std::uint32_t filelen; std::fread(&filelen, 4, 1, f); + char wave[4]; std::fread(wave, 1, 4, f); + if (std::memcmp(riff, "RIFF", 4) || std::memcmp(wave, "WAVE", 4)) { + std::fprintf(stderr, "not a RIFF/WAVE file\n"); std::fclose(f); return false; + } + int format = 0, channels = 0, sampleRate = 0, bits = 0; + bool found_fmt = false, found_data = false; + while (!feof(f)) { + char ck[4]; std::uint32_t sz = 0; + if (std::fread(ck, 1, 4, f) != 4) break; + if (std::fread(&sz, 4, 1, f) != 1) break; + if (std::memcmp(ck, "fmt ", 4) == 0) { + std::uint16_t fmt=0, ch=0, bits16=0; + std::uint32_t srate=0; + std::fread(&fmt, 2, 1, f); std::fread(&ch, 2, 1, f); + std::fread(&srate, 4, 1, f); std::fseek(f, 6, SEEK_CUR); + std::fread(&bits16, 2, 1, f); + format=fmt; channels=ch; sampleRate=(int)srate; bits=bits16; + found_fmt = true; + // 跳到 chunk 末尾 + std::fseek(f, (long)(sz - 16), SEEK_CUR); + } else if (std::memcmp(ck, "data", 4) == 0) { + out.clear(); out.reserve(sz / 2); + long remaining = (long)sz; + while (remaining >= 2) { + std::int16_t s; + std::fread(&s, 2, 1, f); + out.push_back((float)s / 32768.0f); + remaining -= 2; + } + found_data = true; + break; + } else { + std::fseek(f, (long)sz + (sz & 1), SEEK_CUR); + } + } + std::fclose(f); + if (!found_fmt || !found_data) { std::fprintf(stderr, "missing fmt/data\n"); return false; } + if (format != 1) { std::fprintf(stderr, "not PCM\n"); return false; } + if (bits != 16) { std::fprintf(stderr, "not PCM16\n"); return false; } + if (channels != 1) { std::fprintf(stderr, "not mono\n"); return false; } + if (sampleRate != 44100) { std::fprintf(stderr, "not 44100Hz (got %d)\n", sampleRate); return false; } + sr = sampleRate; + return true; +} + +} // namespace + +int main(int argc, char ** argv) { + if (argc < 3) { + std::fprintf(stderr, "usage: %s [nsteps] [seed]\n", argv[0]); + return 2; + } + const char * model_path = argv[1]; + const char * wav_path = argv[2]; + int nsteps = argc > 3 ? std::atoi(argv[3]) : 1; + std::uint64_t seed = argc > 4 ? std::strtoull(argv[4], nullptr, 10) : 42; + + std::vector wav; int sr = 0; + if (!load_wav_pcm16(wav_path, wav, sr)) return 2; + + char version[64] = {0}; + (void)game_capi_version(version, sizeof(version)); + char backends[128] = {0}; + (void)game_capi_available_backends(backends, sizeof(backends)); + std::printf("version=%s backends=[%s]\n", version, backends); + std::printf("loading model %s ...\n", model_path); + std::fflush(stdout); + + char err[GAME_CAPI_ERRBUF] = {0}; + game_capi_model * m = game_capi_open(model_path, nullptr, err, (int)sizeof(err)); + if (!m) { std::fprintf(stderr, "open failed: %s\n", err); return 1; } + + char decided[64] = {0}; + (void)game_capi_backend_decided(m, decided, (int)sizeof(decided)); + std::printf("decided backend = %s\n", decided); + + std::vector notes(4096); + int count = 0, frames = 0; + int rc = game_capi_infer(m, wav.data(), (std::size_t)wav.size(), + 0 /*universal*/, nsteps, + 0.2f, 2, 0.2f, seed, + notes.data(), (int)notes.size(), + &count, &frames); + if (rc != GAME_CAPI_OK) { + const char * le = game_capi_last_error(m); + std::fprintf(stderr, "infer failed rc=%d err=%s\n", rc, le ? le : "(none)"); + game_capi_close(m); + return 1; + } + + std::printf("frames=%d notes=%d\n", frames, count); + float total_s = 0.0f; + for (int i = 0; i < count; ++i) { + const game_capi_note & n = notes[(std::size_t)i]; + std::printf(" [%02d] %.3fs + %.3fs pitch=%6.2f voiced=%d\n", + i, n.offset_seconds, n.duration_seconds, n.pitch_midi, n.voiced); + total_s += n.duration_seconds; + } + std::printf("total duration %.3f s\n", total_s); + game_capi_close(m); + return 0; +}