Windows: native MSVC build with vcpkg-managed dependencies - #233
Open
troubadour-hell wants to merge 10 commits into
Open
troubadour-hell wants to merge 10 commits into
troubadour-hell wants to merge 10 commits into
Conversation
- NOMINMAX for every Windows translation unit (windows.h min/max macros corrupt std::min/std::max and the CUDA headers). - /utf-8 source encoding for MSVC C++ and CUDA; the conforming MSVC preprocessor (-Xcompiler=/Zc:preprocessor) is required by CCCL on CUDA 13. - Windows dependencies are declared in vcpkg.json (baseline-locked ffmpeg + curl with Schannel TLS) and resolved with find_package through the vcpkg toolchain file; CMakePresets.json wires a one-command setup (cmake --preset windows-vcpkg). pkg-config lookup stays untouched on Linux. - FFMPEG_LIBRARIES may carry optimized/debug keywords, so they flow into the interface target via target_link_libraries rather than INTERFACE_LINK_LIBRARIES. - utf8proc compiles statically on Windows (UTF8PROC_STATIC). - README: Windows build instructions (vcpkg bootstrap, manifest install, preset build). vcpkg.json baseline: a1cae005c39be7b18ba319fced856b68d7276271
- activeCodePage=UTF-8 manifest (merged through target_sources on MSVC) makes the CRT hand the CLI apps UTF-8 argv instead of the legacy ANSI code page. - SetConsoleCP/SetConsoleOutputCP(CP_UTF8) fix console display of UTF-8 output on ANSI-code-page (e.g. 936/GBK) consoles; redirected pipes keep raw UTF-8. - perplexity timestamp: gmtime_s argument order vs POSIX gmtime_r.
…hecks - winsock2/ws2tcpip replace arpa/inet.h, netdb.h, and sys/socket.h under _WIN32; the code shares the existing getaddrinfo/inet_ntop flow. - The media-root escape check compares native wide-char paths on Windows (L"..") so the traversal guard keeps working with the same semantics. - Link ws2_32 into ninfer_media_acquire on Windows.
CreateFileW/CreateFileMappingW/MapViewOfFile replace POSIX O_DIRECT mmap, and OVERLAPPED ReadFile performs the aligned direct reads. Bounds checks mirror the POSIX branch (LONGLONG offsets, DWORD transfer sizes).
MSVC has no __int128; add u128_mul (_umul128 on x64) and saturating_u64_mul, then use them in PrefillWork, the materialization-search tie-break, and context-cost Q32 accounting. PrefillWork keeps the exact saturating semantics of the old 128-bit code: attention_pairs = min(max64, prefix*suffix + suffix*(suffix+1)/2) is summed limb-wise, and the triangular term halves a factor before multiplying so it never saturates early at 2^63. q32_product_ns reproduces the (product + 2^32 - 1) >> 32 rounding limb-wise; the saturating threshold max64 << 32 is compared exactly.
MSVC rejects the alignas(128) descriptor struct as a by-value kernel parameter (C2711), so on _MSC_VER builds the launchers copy the descriptors to a per-launch device buffer (cudaMalloc + cudaMemcpyAsync on the compute stream, cudaFreeAsync after the launch) and the kernels dereference a pointer. Other platforms keep the zero-copy __grid_constant__ parameter so the hot decode path is unchanged.
…linking MSVC does not emit out-of-class defaulted explicit specializations unless they are odr-used in the defining translation unit, which left undefined references that GCC weak symbols had masked. Provide user-defined move constructors, move assignments, and destructors instead, keeping the implicit noexcept of the old defaulted declarations explicit.
… width - logging: _isatty on the redirected/console stderr instead of POSIX isatty; localtime_s argument order vs localtime_r. - startup log: CONSOLE_SCREEN_BUFFER_INFO width when the progress bar targets a console, ioctl TIOCGWINSZ otherwise. - request log and context cost: _getpid on Windows (see refactor(core) for the context-cost side).
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 91fd83ca9f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…ffered Windows reads Address Codex review feedback on PR Neroued#233: - Replace synchronous cudaMalloc with stream-ordered cudaMallocAsync for TMA descriptors on MSVC. The stream memory pool reuses the same slot on subsequent launches, eliminating per-launch allocation overhead in the prefill hot path. (nvfp4_w4a4_tma.cu, nvfp4_linear_swiglu_w4a4_tma.cu) - Add FILE_FLAG_NO_BUFFERING to the Windows artifact file handle so ReadFile bypasses the system cache, matching POSIX O_DIRECT semantics. The existing 4096-byte alignment contract satisfies the flag's sector-alignment requirements. (reader.cpp)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Windows: native MSVC build with vcpkg-managed dependencies
Summary
This PR adds a Windows (MSVC / CUDA) build path to NInfer. All Windows-specific code is isolated behind
#if defined(_MSC_VER)(compiler-specific) or#if defined(_WIN32)(platform-specific) guards in source files, andif(WIN32)in CMake (CMake variable). The existing Linux build path—pkg-configlookup, compile flags, link libraries—is unchanged.Dependencies: vcpkg manifest mode
FFmpeg and libcurl are declared in
vcpkg.jsonwith a pinned baseline. On Windows,find_package(FFMPEG)andfind_package(CURL)resolve them through the vcpkg toolchain (wired viaCMakePresets.json). On Linux, the existingpkg_check_modulespath is untouched.Setup:
vcpkg's binary cache makes subsequent configurations complete in seconds.
What this PR introduces
Build system
CMakeLists.txt:NOMINMAX,/utf-8source encoding,/Zc:preprocessor(required by CCCL on CUDA 13).find_package(FFMPEG)/find_package(CURL)on Windows, withPkgConfig::FFMPEG/PkgConfig::LIBCURLinterface targets synthesized to match the existing target names.CMakePresets.json:windows-vcpkgpreset pointingCMAKE_TOOLCHAIN_FILEat vcpkg.vcpkg.json: manifest with pinned baseline (a1cae005c39be7b18ba319fced856b68d7276271).src/CMakeLists.txt:UTF8PROC_STATICon Windows;ws2_32linked intoninfer_media_acquire(post-merge: these live insrc/text/CMakeLists.txtandsrc/product/CMakeLists.txt, where upstream splitsrc/CMakeLists.txt).README.md: Windows build instructions (vcpkg bootstrap, preset build).MSVC compatibility
_MSC_VERbuilds the launchers copy descriptors to a per-launch device buffer (cudaMalloc+cudaMemcpyAsyncon the compute stream,cudaFreeAsyncafter launch); other platforms keep the zero-copy__grid_constant__by-value parameter. All device allocations are stream-ordered on the kernel's own stream.__int128. Addedu128_mul(_umul128on x64) andsaturating_u64_mul, applied in prefill-work accounting, materialization-search tie-breaking, and context-cost Q32 arithmetic.activeCodePage=UTF-8application manifest;SetConsoleCP/SetConsoleOutputCP(CP_UTF8)for console display;gmtime_sargument order.CreateFileW/CreateFileMappingW/MapViewOfFile+OVERLAPPED ReadFilefor aligned direct reads (post-merge: upstream v3 removed mmap; the direct-read support now lives in the newsrc/artifact/file_io.{h,cpp}, see below).winsock2.h/ws2tcpip.hunder_WIN32; wide-char path comparison (L"..") for the media-root escape guard._isatty,localtime_s,CONSOLE_SCREEN_BUFFER_INFOterminal width,_getpid.Platform guards
#if defined(_WIN32)for platform checks and#if defined(_MSC_VER)for compiler-specific workarounds—both are compiler-predefined macros, not CMake variables.if(WIN32), which is a CMake variable set on all Windows generators.Upstream v3 merge adaptation
The upstream v3 refactor (artifact format v3, non-template runtime under
src/models/qwen3_5/, componentized CMake) removed or relocated files this PR had adapted. The merge re-ports the Windows support onto the new structure:src/artifact/file_io.{h,cpp}(new upstream file):CreateFileW/GetFileSizeEx+OVERLAPPED ReadFile;read_directopens a separateFILE_FLAG_NO_BUFFERINGhandle, matching the POSIXO_DIRECTsemantics.cmake/Dependencies.cmake(new upstream file): the vcpkgfind_package(FFMPEG)/find_package(CURL)branches live here;FFMPEG_LIBRARIESkeyword lists are wired throughtarget_link_libraries.src/text/CMakeLists.txt:UTF8PROC_STATICfor the newly vendored utf8proc (MSVC would otherwise dllimport symbols of its own static library).third_party/llama-jinja(newly vendored):localtime_rshimmed vialocaltime_s.src/product/CMakeLists.txt:ws2_32linked intoninfer_media_acquirehere.tools/upgrade_ninfer_v2_to_v3.py: POSIX-onlyposix_fadvise/fdatasync/os.linkshimmed so v2 artifacts can be upgraded on Windows as well.The
api_impl.h/program.htemplate-specialization workarounds were dropped with the v3 non-template runtime (the MSVC bug they worked around no longer applies).Commit structure
build:MSVC toolchain with vcpkg-managed dependenciesfeat(apps):UTF-8 code-page manifest and consolesfix(media_acquire):Windows socket stack and wide-char pathsfeat(artifact):Win32 mapped-file direct-read pathrefactor(core):portable 128-bit multiplyfix(ops):TMA descriptors by device pointer on MSVCfix(targets):qwen3_6 runtime special members for MSVCfix(product/serve):Windows console TTY, localtime, pidmerge:upstream master (v3 artifact/architecture)Verification
Build: 529/529 targets compiled with MSVC 19.42 + CUDA 13.0 on Windows.
Post-merge build (branch tip, upstream v3): Linux 686/686 targets (gcc + CUDA 13.3, WSL2, same machine); Windows clean full build, zero linker warnings (MSVC 19.44 + CUDA 13.0). End-to-end v3 artifact smoke (Qwen3.8-27B NVFP4, 8-token generate, bf16 KV): Windows 77.1 tok/s, Linux/WSL2 76.2 tok/s. The v3 reader rejects v2 artifacts; the bundled upgrade tool was used to convert (weights preserved).
All testing was done on the following machine:
Runtime — Qwen3.8-27B NVFP4 W4A4, 262K context, fp8 KV cache:
Functional test — DeepSeek agent harness with tool-calling (exercises tool-call parsing and stop-token generation):
The model correctly generates tool calls, stops at the right tokens, and produces well-formed output—confirming the TMA kernel device-pointer path produces numerically correct results on Windows.
Serve throughput — Qwen3.8-27B NVFP4 W4A4 with MTP speculative decoding, exercised through the serve corpus runner (same configuration and metric definitions as the upstream serve benchmark methodology: int8 KV cache,
--max-context 262144 --prefill-chunk 1024 --no-prefix-reuse --spec mtp --draft-tokens 3 --lm-head-draft, stochastic sampling temp=0.6 / top-p=0.95 / top-k=20 / min-p=0 / presence=1.0 / freq=0, decode phase speed = (completion_tokens − 1) / decode_seconds, 5 seeds per fixture, 3 fixtures per family). The Linux column was measured in WSL2 on the same machine (same GPU, same model artifact, same runner and seeds).MTP acceptance and tokens/round are identical between platforms across all four task families (code 73.9%, story 38.4%, translation 74.3%, structured 89.5%), confirming the Windows port produces numerically equivalent decode behavior; throughput differences stay within platform-level variance on the same GPU.