Skip to content

trace : add DeepSeek V4.1 cross-runtime correctness harness - #67

Closed
ajaxdude wants to merge 3 commits into
jeromecoste-microsoft-deepseek-v41-core-runtimefrom
jeromecoste-microsoft-deepseek-v41-trace-harness
Closed

ajaxdude wants to merge 3 commits into
jeromecoste-microsoft-deepseek-v41-core-runtimefrom
jeromecoste-microsoft-deepseek-v41-trace-harness

Conversation

@ajaxdude

Copy link
Copy Markdown
Member

Layer 2 of 2 (PR B): correctness trace harness

AGENT-AUTHORED. Written end-to-end by Claude Sonnet 5 (GitHub Copilot CLI), revived from closed #59.
Stacked on #66 (Layer 1, "deepseek41 : add core DeepSeek V4.1 runtime"): base branch is
jeromecoste-microsoft-deepseek-v41-core-runtime, so this diff is additive on top of #66 and will
shrink to just this layer once #66 merges.

Background

#59/#49/#60 were closed together by a reviewer because none were stacked (all targeted master
directly, causing overlapping diffs) and because trace-tooling defaults/README examples leaked a
personal path (/home/papa/...). This PR re-applies the validated correctness-harness content from
that effort as fresh, minimal, logical commits (not a replay of the old 113-commit branch), built on
top of #66 instead of master. The personal-path leak is fixed at the source commit used
(1eeb5bd49d63d6eb3692d92ccd6e161abc3067a9 on the old recovery branch) and re-verified here with a
clean git grep -niE papa across every file this PR touches.

What this adds

  • tools/deepseek-v41-trace/: the cross-runtime correctness trace harness (native trace/prompt-builder
    binaries, Linux containment helper, host-attestation, receipt generation, preflight/watchdog-audit
    checks, Python trace-format/verification helpers) plus its CMake wiring
    (LLAMA_DEEPSEEK_V41_NATIVE_CONTAINMENT_TESTS option, tools/CMakeLists.txt subdirectory).
  • tools/server/server-child-protocol.h and matching tools/server/server-models.cpp fixes: ANSI-reset
    residue recognition for child state lines, skip the stop-monitor thread for download children, and
    notify cv_stop directly on their teardown -- needed so the trace harness can reliably read child
    process state during the correctness flow.
  • tools/server/tests/unit/test_router.py: a private_router_download_cache fixture for test
    isolation around the download-cache fixes above.
  • tests/test-server-child-protocol.cpp (new) and tests/test-deepseek41-trace.py (new, the
    cross-runtime correctness trace test suite), wired into CTest via tests/CMakeLists.txt.

Nothing from #60's scope (src/llama-host-memory-guard.*, its loader wiring, test-host-memory-guard.cpp)
is included here; that stays out of both layers of this revival.

Scope note: watchdog-revision trust anchor

The harness pins a WATCHDOG_REVISION git-commit-hash as a trust anchor that must be an ancestor of HEAD
for preflight.watchdog_audit() to pass. The value carried over from the old branch pointed at a commit
in the old, unrelated fork history, which can never be an ancestor of this repo. Updated it to #66's
actual commit that adds scripts/strix_memory_watchdog.py (02235de637ae1ffe8aeef9479628962792190b22);
script content hash is unchanged (verified byte-identical via shasum -a 256). If #66 is rebased or
amended before merge, this pin will need to be re-verified/updated
, or
test_canonical_watchdog_artifacts_embed_and_validate / test_approved_watchdog_is_exact will fail.

Cross-runtime context (no merge dependency)

This harness's manifest/receipt schema is designed to align with the unmerged draft
ggml-org#28696 (safetensors -> GGUF converter for DeepSeek V4.1), so trace artifacts stay
comparable across runtimes as that work lands upstream. It makes no performance claim: DeepSeek V4.1
performance work was, and remains, blocked pending cross-runtime correctness passing through this harness.
Any Fedora/ROCm validation referenced historically in this effort was build/install/trace/containment
evidence only -- explicitly not V4.1 ROCm inference correctness evidence.

An external contributor's independently-benchmarked ROCm DeepSeek V4.1 implementation for the same
hardware class (antirez/ds4#1036, by kyuz0) was reviewed during this revival. It validates this harness's
design assumptions with no correctness gaps surfaced against it: matching Engram row layout, matching
CED 20/20 layer-boundary metadata key LLM_KV_DSV41_CANDIDATE_SOURCE_LAYER_ID, and matching
attention-sink/split-KV concepts. It is a separate codebase and runtime, not a merge dependency for this
PR, and nothing was ported from it -- it is cited purely as external validation context.

Verified / not verified

Verified (this session, macOS, model-free):

  • cmake -S . -B build -DLLAMA_BUILD_TESTS=ON -DLLAMA_BUILD_TOOLS=ON -DCMAKE_BUILD_TYPE=Release configures
    and cmake --build build completes for this layer's targets.
  • ctest -R 'deepseek41|server-child-protocol' --output-on-failure: 8/10 pass.
  • git diff --check clean; ASCII-only; no personal-path leaks (re-grepped after every checkout).

Not verified / known gaps:

  • No GGUF loaded, no inference run, no ROCm/Vulkan backend run, no Strix Halo hardware run (model-free
    by design/policy for this session).
  • test-deepseek41-trace and test-deepseek41-trace-install (2 of the 10 above) fail on this Mac with
    "loaded unclassified module outside trusted system roots: /opt/homebrew/Cellar/openssl@3/.../libcrypto.3.dylib".
    Reproduced directly outside pytest -- it is the harness's OpenSSL/Homebrew trusted-roots check hitting a
    macOS-specific path layout gap, not a logic bug in this PR. Left as-is per scope; needs a Linux run (or a
    macOS-Homebrew-aware trusted-roots entry) to close out, tracked as NOT VERIFIED rather than fixed
    blind for one machine.
  • No performance numbers are claimed anywhere in this PR.

Assisted-by: Claude Sonnet 5
Co-authored-by: Copilot App 223556219+Copilot@users.noreply.github.com

Jerome Coste and others added 3 commits September 16, 2026 14:22
Add the llama-deepseek-v41-trace tool under tools/deepseek-v41-trace/:
manifest export, prompt builder, Linux containment helper, host
attestation, and run/verify scripts for cross-runtime (llama.cpp vs
DS4) correctness comparison. Wire the subdirectory into
tools/CMakeLists.txt and add the LLAMA_DEEPSEEK_V41_NATIVE_CONTAINMENT_TESTS
build option next to the other LLAMA_BUILD_* options.

Assisted-by: Claude Sonnet 5
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add server-child-protocol.h with server_child_state_line() to
recognize child state lines even when ANSI reset codes precede them
(colored logs), and use it in server-models.cpp instead of a raw
prefix match. Skip the stop-monitor thread for download children
(they never enter stopping_models, so joining it would hang remove())
and notify cv_stop directly on their teardown path. Fix test_router.py
download-cache test isolation so tests don't leak state across runs.

Add test-server-child-protocol.cpp covering the ANSI-reset and
malformed-line cases, wired into CTest.

Assisted-by: Claude Sonnet 5
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add test-deepseek41-trace.py, the pytest suite driving the trace
harness through model-free manifest, prompt-provenance, containment,
and cross-runtime schema-alignment checks. Wire it into CTest as
test-deepseek41-trace, feeding it the native trace/manifest/inject
binary paths and gating the containment-helper environment variable
on LLAMA_DEEPSEEK_V41_NATIVE_CONTAINMENT_TESTS (Linux only).

Also register test-deepseek41-runtime.cpp, whose registration PR A
deferred pending tools/deepseek-v41-trace/trace-components.h landing
with this layer.

Assisted-by: Claude Sonnet 5
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@ajaxdude

Copy link
Copy Markdown
Member Author

Note: the predecessor layer is now #68, not #66 -- #66 was closed and replaced with a same-repo PR (#68) from the identical branch/commit, since #66 was accidentally opened as a cross-fork PR which breaks native stacking. No branch or commit changes; this PR's base ref is unaffected.

@ajaxdude
ajaxdude added this pull request to stack #69 September 16, 2026 21:45
@ajaxdude

Copy link
Copy Markdown
Member Author

@dzannotti This is layer 2 of the same revival -- full context is on #68. Specific to this PR's original form (#59): the trace-tooling's hardcoded /home/papa/... default (argparse default in run_ds4.py, plus README examples) is fixed and re-verified clean here (re-grepped every added line; only placeholder fixture/README paths like /Users/oracle/... remain, not real ones).

What I'm asking for: review alongside #68, in the same base-to-stacked order the Stack (ggml-org#69) reflects.

@ajaxdude

Copy link
Copy Markdown
Member Author

Closing and superseding with #71 (stacked on #70, which supersedes #68 for the same reason). Same content, same 3 commits -- but this PR's commits carried a real name + machine hostname as git author/committer (Jerome Coste <jeromecoste@Jeromes-MacBook-Pro.local>), and the branch name embedded a corporate username (jeromecoste-microsoft-...). Same category of issue that got the original #49/#59/#60 closed (leaked personal configuration/pathing), just at the commit-metadata/branch-name level instead of file content. Fixed via authorship rebase (git commit --amend --reset-author, verified against this PR's tip) and a fresh push under a leak-free branch name. Note: GitHub retains this PR's old branch name in this page's own history regardless of what happens to the branch itself -- that's an inherent platform limitation of closing/re-opening rather than something further action here can fix; #71 is clean going forward.

@ajaxdude ajaxdude closed this Sep 17, 2026
@ajaxdude
ajaxdude deleted the jeromecoste-microsoft-deepseek-v41-trace-harness branch September 17, 2026 02:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build documentation Improvements or additions to documentation server testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant