From f05442f2d62d33075159c6593ed9310e77f275a0 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Mon, 20 Jul 2026 23:02:12 +0200 Subject: [PATCH 01/37] readme --- Directory.Build.props | 4 +- Sources/Cli/Cli.csproj | 7 ++++ Templates/DevOnBike.Overfit.Templates.csproj | 9 +++++ Templates/README.md | 40 ++++++++++++++++++++ 4 files changed, 58 insertions(+), 2 deletions(-) create mode 100644 Templates/README.md diff --git a/Directory.Build.props b/Directory.Build.props index 6c7d35ab..1316f09c 100644 --- a/Directory.Build.props +++ b/Directory.Build.props @@ -1,7 +1,7 @@  - 10.0.30 - 10.0.30 + 10.0.31 + 10.0.31 release+id DevOnBike Sharp IT diff --git a/Sources/Cli/Cli.csproj b/Sources/Cli/Cli.csproj index a9825cbd..244e2638 100644 --- a/Sources/Cli/Cli.csproj +++ b/Sources/Cli/Cli.csproj @@ -37,6 +37,8 @@ DevOnBike.Overfit.Cli Overfit CLI Overfit CLI — a pure-.NET, OpenAI-compatible local LLM server (`overfit serve`) plus model pull/chat/embed/tts. No Python, no native engine. Also distributed as a Native-AOT binary and a Docker image. + README.md + LICENSE.md llm;gguf;openai;cli;ai;inference;local;dotnet-tool @@ -47,6 +49,11 @@ + + + + + all diff --git a/Templates/DevOnBike.Overfit.Templates.csproj b/Templates/DevOnBike.Overfit.Templates.csproj index e9e6d300..7ad271e7 100644 --- a/Templates/DevOnBike.Overfit.Templates.csproj +++ b/Templates/DevOnBike.Overfit.Templates.csproj @@ -19,6 +19,8 @@ `dotnet new` templates for building local-LLM apps with Overfit — a pure-.NET, in-process, on-device LLM runtime (no Python, no Ollama, no cloud). Includes `overfit-chat`: a Minimal API streaming-chat app that exposes a local GGUF model as a standard Microsoft.Extensions.AI IChatClient. dotnet-new;templates;llm;ai;local-llm;on-device;ichatclient;microsoft-extensions-ai;gguf;native-aot;overfit Copyright (c) 2026 DevOnBike + README.md + LICENSE.md https://github.com/DevOnBike/Overfit git + + + + + diff --git a/Templates/README.md b/Templates/README.md new file mode 100644 index 00000000..00b4a21a --- /dev/null +++ b/Templates/README.md @@ -0,0 +1,40 @@ +# Overfit project templates + +`dotnet new` templates for building **local, in-process LLM apps** with +[Overfit](https://github.com/DevOnBike/Overfit) — a pure-.NET, on-device LLM runtime. +No Python, no Ollama, no model server, no cloud. + +## Install + +```bash +dotnet new install DevOnBike.Overfit.Templates +``` + +## Templates + +### `overfit-chat` + +A Minimal API streaming-chat app that loads a local GGUF model and exposes it as a standard +`Microsoft.Extensions.AI.IChatClient`, so it drops into code already written against that +abstraction. + +```bash +dotnet new overfit-chat -o MyChatApp +cd MyChatApp +# point it at a GGUF model, then: +dotnet run +``` + +The scaffolded app pins the Overfit runtime packages to a known-published version, so +`dotnet new` → `dotnet run` works out of the box. + +## Adding Overfit to an EXISTING app + +Templates are for new apps. To add local inference to a project you already have, use the +`overfit-add-llm` skill from the Overfit plugin, or follow +[the integration guide](https://github.com/DevOnBike/Overfit#readme). + +## License + +Dual-licensed: AGPL-3.0-or-later for open source, commercial license available. See +[LICENSE.md](https://github.com/DevOnBike/Overfit/blob/main/LICENSE.md). From 608024f3f1359bb419f24f31d0ffa7ffdcb10200 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Tue, 21 Jul 2026 00:38:51 +0200 Subject: [PATCH 02/37] else if + else --- .editorconfig | 2 +- Sources/Main/Autograd/AutogradNode.cs | 5 +- .../Main/Autograd/ComputationGraph.Linear.cs | 3 +- .../Main/Autograd/ComputationGraph.Losses.cs | 5 +- Sources/Main/Autograd/ComputationGraph.cs | 9 ++-- Sources/Main/Data/DataAugmenter.cs | 9 +++- .../Main/Data/Normalizers/MinMaxNormalizer.cs | 15 +++--- .../Main/Data/Normalizers/ZScoreNormalizer.cs | 10 ++-- .../Data/Prepare/ConstantColumnFilterLayer.cs | 6 ++- Sources/Main/Data/Prepare/OutlierClipLayer.cs | 12 ++--- .../Main/Data/Prepare/RobustScalingLayer.cs | 3 +- .../Main/Data/Prepare/ShapSelectionLayer.cs | 3 +- .../Data/Tabular/TabularToTensorConverter.cs | 42 ++++++++------- .../Crossover/SbxCrossoverOperator.cs | 11 ++-- .../Evolutionary/Runtime/EvolutionRunner.cs | 31 ++++++----- .../Evolutionary/Storage/GridEliteArchive.cs | 27 +++------- .../GenerationalGeneticAlgorithm.cs | 7 ++- .../Strategies/OpenAiEsStrategy.cs | 21 ++++---- .../Strategies/SeparableCmaEsStrategy.cs | 5 +- Sources/Main/Onnx/OnnxGraphModel.cs | 11 ++-- Sources/Main/Onnx/OnnxProtoParser.cs | 9 ++-- Sources/Main/Onnx/Operators/AddOperator.cs | 5 +- Sources/Main/Onnx/Operators/ConvOperator.cs | 8 ++- Sources/Main/Onnx/Operators/GemmOperator.cs | 16 ++---- Sources/Main/Optimizers/Adam.cs | 54 +++++++++---------- Sources/Main/Optimizers/LRScheduler.cs | 11 ++-- Sources/Main/Optimizers/SGD.cs | 24 ++++----- 27 files changed, 177 insertions(+), 187 deletions(-) diff --git a/.editorconfig b/.editorconfig index c87b26bf..12c95bf6 100644 --- a/.editorconfig +++ b/.editorconfig @@ -240,7 +240,7 @@ dotnet_diagnostic.RS0030.severity = none # 3. The rule ID must NOT appear in in Directory.Build.props. That maps to # -warnaserror-:ID, which reverts the diagnostic to warning EVEN WHERE .editorconfig sets error — # defeating this whole section. (This is why OVERFIT008/015 are absent from that list.) -[Sources/Main/{Anomalies,Core,Exceptions,Inference,Parameters,Randomization,Serving,Statistical,Tensors}/**.cs] +[Sources/Main/{Anomalies,Autograd,Core,Data,Evolutionary,Exceptions,Inference,Optimizers,Parameters,Randomization,Serving,Statistical,Tensors}/**.cs] dotnet_diagnostic.OVERFIT021.severity = error # OVERFIT022 (recursion) — ERROR on the paths that parse untrusted, externally-authored input, where an diff --git a/Sources/Main/Autograd/AutogradNode.cs b/Sources/Main/Autograd/AutogradNode.cs index e7e8bf75..a7e3f1fb 100644 --- a/Sources/Main/Autograd/AutogradNode.cs +++ b/Sources/Main/Autograd/AutogradNode.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -119,7 +119,8 @@ private AutogradNode( _gradStorage = externalGrad; _ownsGradStorage = ownsGradStorage; } - else + + if (externalGrad == null) { // Node-owned grad storage: the default for graph temporaries. _gradStorage = TensorFactory.CloneStorage(data, clearMemory: true); diff --git a/Sources/Main/Autograd/ComputationGraph.Linear.cs b/Sources/Main/Autograd/ComputationGraph.Linear.cs index 3d1580cb..27e9d566 100644 --- a/Sources/Main/Autograd/ComputationGraph.Linear.cs +++ b/Sources/Main/Autograd/ComputationGraph.Linear.cs @@ -44,7 +44,8 @@ public AutogradNode Linear( { LinearKernels.ForwardBatched(inS, wS, outS, N, K, M); } - else + + if ((long)N * K * M >= LinearKernels.ForwardBatchedThreshold) { unsafe { diff --git a/Sources/Main/Autograd/ComputationGraph.Losses.cs b/Sources/Main/Autograd/ComputationGraph.Losses.cs index ceab5f82..5bc14623 100644 --- a/Sources/Main/Autograd/ComputationGraph.Losses.cs +++ b/Sources/Main/Autograd/ComputationGraph.Losses.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -56,7 +56,8 @@ public AutogradNode SoftmaxCrossEntropy( { Record(OpCode.SoftmaxCrossEntropy, output, logits, target, c0: probsNode, contextCount: 1); } - else + + if (!logits.RequiresGrad) { // probsNode not needed for backward — dispose immediately to free arena slot. probsNode.Dispose(); diff --git a/Sources/Main/Autograd/ComputationGraph.cs b/Sources/Main/Autograd/ComputationGraph.cs index c6568c85..d57375fc 100644 --- a/Sources/Main/Autograd/ComputationGraph.cs +++ b/Sources/Main/Autograd/ComputationGraph.cs @@ -211,12 +211,11 @@ public bool BackwardProfileEnabled { _opTicks ??= new long[OpCodeCount]; _opCount2 ??= new int[OpCodeCount]; + return; } - else - { - _opTicks = null; - _opCount2 = null; - } + + _opTicks = null; + _opCount2 = null; } } diff --git a/Sources/Main/Data/DataAugmenter.cs b/Sources/Main/Data/DataAugmenter.cs index c74e1f90..3b11e9c1 100644 --- a/Sources/Main/Data/DataAugmenter.cs +++ b/Sources/Main/Data/DataAugmenter.cs @@ -20,13 +20,18 @@ public static FastTensor AugmentBatch(FastTensor originalBatch, in var inputRow = originalBatch.GetView().AsReadOnlySpan().Slice(i * width * height, width * height); var outputRow = augmentedBatch.GetView().AsSpan().Slice(i * width * height, width * height); - if (Random.Shared.NextSingle() > 0.5f) + // Capture the coin flip: re-evaluating Random.Shared in a second `if` would draw a + // different value and could both shift AND copy (or neither). + var shift = Random.Shared.NextSingle() > 0.5f; + + if (shift) { var shiftX = Random.Shared.Next(-2, 3); var shiftY = Random.Shared.Next(-2, 3); ShiftImage(inputRow, outputRow, width, height, shiftX, shiftY); } - else + + if (!shift) { inputRow.CopyTo(outputRow); } diff --git a/Sources/Main/Data/Normalizers/MinMaxNormalizer.cs b/Sources/Main/Data/Normalizers/MinMaxNormalizer.cs index e65a359e..cff205d3 100644 --- a/Sources/Main/Data/Normalizers/MinMaxNormalizer.cs +++ b/Sources/Main/Data/Normalizers/MinMaxNormalizer.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -124,14 +124,15 @@ public void TransformInPlace(Span data) TensorPrimitives.Max(data, 0f, data); TensorPrimitives.Min(data, _clipMax, data); } - else if (ClipToRange) - { - TensorPrimitives.Max(data, _frozenMin, data); - TensorPrimitives.Min(data, _frozenMax, data); - } - else + + if (!_hasClipMax) { TensorPrimitives.Max(data, _frozenMin, data); + + if (ClipToRange) + { + TensorPrimitives.Min(data, _frozenMax, data); + } } TensorPrimitives.Subtract(data, _frozenMin, data); diff --git a/Sources/Main/Data/Normalizers/ZScoreNormalizer.cs b/Sources/Main/Data/Normalizers/ZScoreNormalizer.cs index a5d42bfc..8eaefb65 100644 --- a/Sources/Main/Data/Normalizers/ZScoreNormalizer.cs +++ b/Sources/Main/Data/Normalizers/ZScoreNormalizer.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -61,13 +61,17 @@ public void FitBatch(ReadOnlySpan data) localM2 = TensorPrimitives.SumOfSquares(diffs); - if (_count == 0) + // Capture before the first branch sets _count = n2, or a later if (_count != 0) would fire. + var wasEmpty = _count == 0; + + if (wasEmpty) { _count = n2; _mean = localMean; _m2 = localM2; } - else + + if (!wasEmpty) { var newCount = _count + n2; var delta = localMean - _mean; diff --git a/Sources/Main/Data/Prepare/ConstantColumnFilterLayer.cs b/Sources/Main/Data/Prepare/ConstantColumnFilterLayer.cs index 2d17f5f9..5558512f 100644 --- a/Sources/Main/Data/Prepare/ConstantColumnFilterLayer.cs +++ b/Sources/Main/Data/Prepare/ConstantColumnFilterLayer.cs @@ -55,7 +55,8 @@ public PipelineContext Process(PipelineContext context) { IdentifyByUniqueRatio(span, rows, cols, keptList); } - else + + if (_minUniqueRatio <= 0f) { IdentifyByVariance(span, rows, cols, keptList); } @@ -118,7 +119,8 @@ private void IdentifyByVariance(ReadOnlySpan span, int rows, int cols, Li } } } - else + + if (_epsilon != 0f) { for (var r = 1; r < rows; r++) { diff --git a/Sources/Main/Data/Prepare/OutlierClipLayer.cs b/Sources/Main/Data/Prepare/OutlierClipLayer.cs index 3a6dca2a..9b015338 100644 --- a/Sources/Main/Data/Prepare/OutlierClipLayer.cs +++ b/Sources/Main/Data/Prepare/OutlierClipLayer.cs @@ -108,7 +108,8 @@ private void Fit(ReadOnlySpan span, int rows, int cols) _lowThresholds[c] = float.MinValue; _highThresholds[c] = float.MaxValue; } - else + + if (lowVal < highVal) { _lowThresholds[c] = lowVal; _highThresholds[c] = highVal; @@ -133,14 +134,7 @@ private void ClipAll(Span span, int rows, int cols) for (var r = 0; r < rows; r++) { ref var val = ref span[r * cols + c]; - if (val < lowVal) - { - val = lowVal; - } - else if (val > highVal) - { - val = highVal; - } + val = Math.Clamp(val, lowVal, highVal); } } } diff --git a/Sources/Main/Data/Prepare/RobustScalingLayer.cs b/Sources/Main/Data/Prepare/RobustScalingLayer.cs index 425e3675..44f4d79b 100644 --- a/Sources/Main/Data/Prepare/RobustScalingLayer.cs +++ b/Sources/Main/Data/Prepare/RobustScalingLayer.cs @@ -119,7 +119,8 @@ private void Transform(Span span, int rows, int cols) val = (val - median) * invIqr; } } - else + + if (!_centerByMedian) { for (var r = 0; r < rows; r++) { diff --git a/Sources/Main/Data/Prepare/ShapSelectionLayer.cs b/Sources/Main/Data/Prepare/ShapSelectionLayer.cs index 69483f37..cfcae527 100644 --- a/Sources/Main/Data/Prepare/ShapSelectionLayer.cs +++ b/Sources/Main/Data/Prepare/ShapSelectionLayer.cs @@ -89,7 +89,8 @@ public PipelineContext Process(PipelineContext context) taken++; } } - else + + if (_targetFeatureCount <= 0) { foreach (var x in importanceRanking) { diff --git a/Sources/Main/Data/Tabular/TabularToTensorConverter.cs b/Sources/Main/Data/Tabular/TabularToTensorConverter.cs index 1b6d2091..216fc794 100644 --- a/Sources/Main/Data/Tabular/TabularToTensorConverter.cs +++ b/Sources/Main/Data/Tabular/TabularToTensorConverter.cs @@ -51,11 +51,11 @@ public void Fit(IReadOnlyList data) var categories = categoriesList.ToArray(); _categoryMaps[col.Name] = categories; _featureWidth += categories.Length; + + continue; } - else - { - _featureWidth += 1; - } + + _featureWidth += 1; } } @@ -79,23 +79,25 @@ public void Fit(IReadOnlyList data) { var val = GetValue(data[i], col.Name); - if (col.Type == ColumnType.Numeric) - { - fSpan[rowOffset + currentPos++] = System.Convert.ToSingle(val); - } - else if (col.Type == ColumnType.Binary) + switch (col.Type) { - fSpan[rowOffset + currentPos++] = System.Convert.ToBoolean(val) ? 1f : 0f; - } - else if (col.Type == ColumnType.Categorical) - { - var categories = _categoryMaps[col.Name]; - var currentVal = val?.ToString(); - - for (var c = 0; c < categories.Length; c++) - { - fSpan[rowOffset + currentPos++] = categories[c] == currentVal ? 1f : 0f; - } + case ColumnType.Numeric: + fSpan[rowOffset + currentPos++] = System.Convert.ToSingle(val); + break; + + case ColumnType.Binary: + fSpan[rowOffset + currentPos++] = System.Convert.ToBoolean(val) ? 1f : 0f; + break; + + case ColumnType.Categorical: + var categories = _categoryMaps[col.Name]; + var currentVal = val?.ToString(); + + for (var c = 0; c < categories.Length; c++) + { + fSpan[rowOffset + currentPos++] = categories[c] == currentVal ? 1f : 0f; + } + break; } } diff --git a/Sources/Main/Evolutionary/Crossover/SbxCrossoverOperator.cs b/Sources/Main/Evolutionary/Crossover/SbxCrossoverOperator.cs index 76e26acb..7503c848 100644 --- a/Sources/Main/Evolutionary/Crossover/SbxCrossoverOperator.cs +++ b/Sources/Main/Evolutionary/Crossover/SbxCrossoverOperator.cs @@ -119,14 +119,9 @@ public void Crossover( float beta; - if (u <= 0.5f) - { - beta = MathF.Pow(2f * u, inverseExponent); - } - else - { - beta = MathF.Pow(1f / (2f * (1f - u)), inverseExponent); - } + beta = u <= 0.5f + ? MathF.Pow(2f * u, inverseExponent) + : MathF.Pow(1f / (2f * (1f - u)), inverseExponent); // Symmetric recombination. child1 + child2 = p1 + p2 by construction, so the // arithmetic mean is preserved — a defining property of SBX. diff --git a/Sources/Main/Evolutionary/Runtime/EvolutionRunner.cs b/Sources/Main/Evolutionary/Runtime/EvolutionRunner.cs index 6c673ee1..74017500 100644 --- a/Sources/Main/Evolutionary/Runtime/EvolutionRunner.cs +++ b/Sources/Main/Evolutionary/Runtime/EvolutionRunner.cs @@ -38,26 +38,25 @@ public EvolutionRunner( clearMemory: false); _ownsWorkspace = true; + return; } - else - { - if (workspace.PopulationSize != _algorithm.PopulationSize) - { - throw new ArgumentException( - $"Workspace population size {workspace.PopulationSize} does not match algorithm population size {_algorithm.PopulationSize}.", - nameof(workspace)); - } - if (workspace.GenomeSize != _algorithm.ParameterCount) - { - throw new ArgumentException( - $"Workspace genome size {workspace.GenomeSize} does not match algorithm parameter count {_algorithm.ParameterCount}.", - nameof(workspace)); - } + if (workspace.PopulationSize != _algorithm.PopulationSize) + { + throw new ArgumentException( + $"Workspace population size {workspace.PopulationSize} does not match algorithm population size {_algorithm.PopulationSize}.", + nameof(workspace)); + } - _workspace = workspace; - _ownsWorkspace = false; + if (workspace.GenomeSize != _algorithm.ParameterCount) + { + throw new ArgumentException( + $"Workspace genome size {workspace.GenomeSize} does not match algorithm parameter count {_algorithm.ParameterCount}.", + nameof(workspace)); } + + _workspace = workspace; + _ownsWorkspace = false; } public IEvolutionAlgorithm Algorithm diff --git a/Sources/Main/Evolutionary/Storage/GridEliteArchive.cs b/Sources/Main/Evolutionary/Storage/GridEliteArchive.cs index f3af9d5c..2bd88ce7 100644 --- a/Sources/Main/Evolutionary/Storage/GridEliteArchive.cs +++ b/Sources/Main/Evolutionary/Storage/GridEliteArchive.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -486,25 +486,12 @@ public bool TryGetCellIndex(ReadOnlySpan descriptor, out int cellIndex) return false; } - int bin; - if (value == max) - { - bin = _binsPerDimension[d] - 1; - } - else - { - var normalized = (value - min) * _descriptorInvRange[d]; - bin = (int)(normalized * _binsPerDimension[d]); - - if (bin < 0) - { - bin = 0; - } - else if (bin >= _binsPerDimension[d]) - { - bin = _binsPerDimension[d] - 1; - } - } + // normalized is harmless when value == max (that case takes the last bin below); computing it + // unconditionally keeps `bin` a single definitely-assigned expression with no else. + var normalized = (value - min) * _descriptorInvRange[d]; + var bin = value == max + ? _binsPerDimension[d] - 1 + : Math.Clamp((int)(normalized * _binsPerDimension[d]), 0, _binsPerDimension[d] - 1); index += bin * _cellStrides[d]; } diff --git a/Sources/Main/Evolutionary/Strategies/GenerationalGeneticAlgorithm.cs b/Sources/Main/Evolutionary/Strategies/GenerationalGeneticAlgorithm.cs index abde9d3c..079bcd11 100644 --- a/Sources/Main/Evolutionary/Strategies/GenerationalGeneticAlgorithm.cs +++ b/Sources/Main/Evolutionary/Strategies/GenerationalGeneticAlgorithm.cs @@ -354,11 +354,10 @@ private void CreateChildren( if (_crossoverOperator is null) { CreateChildrenMutationOnly(currentPopulation, nextPopulation, eliteIndices); + return; } - else - { - CreateChildrenWithCrossover(currentPopulation, nextPopulation, eliteIndices); - } + + CreateChildrenWithCrossover(currentPopulation, nextPopulation, eliteIndices); } private void CreateChildrenMutationOnly( diff --git a/Sources/Main/Evolutionary/Strategies/OpenAiEsStrategy.cs b/Sources/Main/Evolutionary/Strategies/OpenAiEsStrategy.cs index 14b3dcab..b69dcf7d 100644 --- a/Sources/Main/Evolutionary/Strategies/OpenAiEsStrategy.cs +++ b/Sources/Main/Evolutionary/Strategies/OpenAiEsStrategy.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -335,7 +335,8 @@ public void Tell(ReadOnlySpan fitness) { ApplyAdamStep(); } - else + + if (!_useAdam) { ApplySgdStep(); } @@ -451,14 +452,8 @@ private void UpdateBestCandidate(ReadOnlySpan fitness) var isNegative = (bestLocalIndex & 1) == 1; var noise = _noiseTable.GetSlice(_noiseOffsets[pairIndex], paramCount); - if (isNegative) - { - TensorPrimitives.MultiplyAdd(noise, -_sigma, _mu, _bestParameters); - } - else - { - TensorPrimitives.MultiplyAdd(noise, _sigma, _mu, _bestParameters); - } + var signedSigma = isNegative ? -_sigma : _sigma; + TensorPrimitives.MultiplyAdd(noise, signedSigma, _mu, _bestParameters); _bestFitness = bestLocalFitness; } @@ -537,7 +532,8 @@ public void Load(BinaryReader reader) _rngState = NormalizeSeed(reader.ReadUInt32()); _hasPendingPopulation = reader.ReadBoolean(); } - else + + if (schemaVersion < 3) { // Legacy schema v2 had no RNG state and no pending Ask/Tell state. // We restore a deterministic fallback state so the strategy remains usable, @@ -563,7 +559,8 @@ public void Load(BinaryReader reader) _noiseOffsets[i] = reader.ReadInt32(); } } - else + + if (schemaVersion < 3) { Array.Clear(_noiseOffsets, 0, _noiseOffsets.Length); } diff --git a/Sources/Main/Evolutionary/Strategies/SeparableCmaEsStrategy.cs b/Sources/Main/Evolutionary/Strategies/SeparableCmaEsStrategy.cs index 46134bab..1ddc6a98 100644 --- a/Sources/Main/Evolutionary/Strategies/SeparableCmaEsStrategy.cs +++ b/Sources/Main/Evolutionary/Strategies/SeparableCmaEsStrategy.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -647,7 +647,8 @@ public void Load(BinaryReader reader) { ReadFloats(reader, _z); } - else + + if (!_hasPendingPopulation) { Array.Clear(_z); } diff --git a/Sources/Main/Onnx/OnnxGraphModel.cs b/Sources/Main/Onnx/OnnxGraphModel.cs index bef899aa..73c9440a 100644 --- a/Sources/Main/Onnx/OnnxGraphModel.cs +++ b/Sources/Main/Onnx/OnnxGraphModel.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -80,12 +80,11 @@ public void RunInference(ReadOnlySpan input, Span output) var left = _buffers[node.InputSlots[0]].AsSpan(); var right = _buffers[node.InputSlots[1]].AsSpan(); addLayer.ForwardInference(left, right, outBuf); + continue; } - else - { - var inBuf = _buffers[node.InputSlots[0]].AsSpan(); - node.Module.ForwardInference(inBuf, outBuf); - } + + var inBuf = _buffers[node.InputSlots[0]].AsSpan(); + node.Module.ForwardInference(inBuf, outBuf); } // Last node's output slot → caller's output span. diff --git a/Sources/Main/Onnx/OnnxProtoParser.cs b/Sources/Main/Onnx/OnnxProtoParser.cs index f5dcbb89..82a57503 100644 --- a/Sources/Main/Onnx/OnnxProtoParser.cs +++ b/Sources/Main/Onnx/OnnxProtoParser.cs @@ -288,7 +288,8 @@ private static OnnxAttribute ParseAttribute(ref ProtoReader reader) { floats.AddRange(reader.ReadPackedFloat()); } - else + + if (wireType != WireType.LengthDelimited) { floats.Add(reader.ReadFloat()); } @@ -298,7 +299,8 @@ private static OnnxAttribute ParseAttribute(ref ProtoReader reader) { ints.AddRange(reader.ReadPackedInt64()); } - else + + if (wireType != WireType.LengthDelimited) { ints.Add(reader.ReadInt64()); } @@ -363,7 +365,8 @@ private static OnnxTensor ParseTensor(ref ProtoReader reader) { dims.AddRange(reader.ReadPackedInt64()); } - else + + if (wireType != WireType.LengthDelimited) { dims.Add(reader.ReadInt64()); } diff --git a/Sources/Main/Onnx/Operators/AddOperator.cs b/Sources/Main/Onnx/Operators/AddOperator.cs index f2fae1d1..737b20b2 100644 --- a/Sources/Main/Onnx/Operators/AddOperator.cs +++ b/Sources/Main/Onnx/Operators/AddOperator.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -28,7 +28,8 @@ public static IModule Build( { shapes.SetShape(node.Outputs[0], shape0); } - else if (shape1 != null) + + if (shape0 == null && shape1 != null) { shapes.SetShape(node.Outputs[0], shape1); } diff --git a/Sources/Main/Onnx/Operators/ConvOperator.cs b/Sources/Main/Onnx/Operators/ConvOperator.cs index fe99f893..b77ad741 100644 --- a/Sources/Main/Onnx/Operators/ConvOperator.cs +++ b/Sources/Main/Onnx/Operators/ConvOperator.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -118,13 +118,17 @@ public static IModule Build( var layer = new ConvLayer(inC, outC, h, w, kH, padding, stride); // Third input is per-channel bias (optional but common in PyTorch exports) + var biasLoaded = false; + if (node.Inputs.Count >= 3 && !string.IsNullOrEmpty(node.Inputs[2]) && initializers.TryGetValue(node.Inputs[2], out var biasTensor)) { var biasData = OnnxImporter.DecodeFloatTensor(biasTensor); layer.LoadParameters(kernelData, biasData); + biasLoaded = true; } - else + + if (!biasLoaded) { layer.LoadParameters(kernelData); } diff --git a/Sources/Main/Onnx/Operators/GemmOperator.cs b/Sources/Main/Onnx/Operators/GemmOperator.cs index 8d640ddb..d3d5b4e1 100644 --- a/Sources/Main/Onnx/Operators/GemmOperator.cs +++ b/Sources/Main/Onnx/Operators/GemmOperator.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -52,17 +52,9 @@ public static IModule Build( // transB=1 → weight is [out, in], must transpose to Overfit's [in, out] // transB=0 → weight is already [in, out] - int inFeatures, outFeatures; - if (transB == 1) - { - outFeatures = (int)weightTensor.Dims[0]; - inFeatures = (int)weightTensor.Dims[1]; - } - else - { - inFeatures = (int)weightTensor.Dims[0]; - outFeatures = (int)weightTensor.Dims[1]; - } + // transB=1 → weight is [out, in]; transB=0 → already [in, out]. + var outFeatures = (int)(transB == 1 ? weightTensor.Dims[0] : weightTensor.Dims[1]); + var inFeatures = (int)(transB == 1 ? weightTensor.Dims[1] : weightTensor.Dims[0]); var weightData = OnnxImporter.DecodeFloatTensor(weightTensor); diff --git a/Sources/Main/Optimizers/Adam.cs b/Sources/Main/Optimizers/Adam.cs index e68e7ee5..3d108d2a 100644 --- a/Sources/Main/Optimizers/Adam.cs +++ b/Sources/Main/Optimizers/Adam.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -145,28 +145,28 @@ public void Step() lr, wd); } + + return; } - else + + foreach (var state in _states) { - foreach (var state in _states) + if (!state.RequiresGrad) { - if (!state.RequiresGrad) - { - continue; - } - - StepAdamState( - state, - b1, - b2, - b1Inv, - b2Inv, - invBc1, - invBc2, - eps, - lr, - wd); + continue; } + + StepAdamState( + state, + b1, + b2, + b1Inv, + b2Inv, + invBc1, + invBc2, + eps, + lr, + wd); } } @@ -174,14 +174,13 @@ public void ZeroGrad() { foreach (var state in _states) { - if (ParallelZeroGrad && state.Size >= ParallelElementThreshold) - { - ClearGradParallel(state, state.Size); - } - else + if (!(ParallelZeroGrad && state.Size >= ParallelElementThreshold)) { state.ZeroGrad(); + continue; } + + ClearGradParallel(state, state.Size); } } @@ -637,11 +636,10 @@ public void ZeroGrad() if (_param != null) { _param.ZeroGrad(); + return; } - else - { - _node!.ZeroGrad(); - } + + _node!.ZeroGrad(); } public bool RequiresGrad => diff --git a/Sources/Main/Optimizers/LRScheduler.cs b/Sources/Main/Optimizers/LRScheduler.cs index bdf4464b..b8c03473 100644 --- a/Sources/Main/Optimizers/LRScheduler.cs +++ b/Sources/Main/Optimizers/LRScheduler.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -100,14 +100,19 @@ public void Step(float currentLoss) return; } - if (currentLoss < _bestLoss * (1f - _minDelta)) + // Capture the verdict BEFORE the improved branch mutates _bestLoss — a later `if (!improved)` + // reading the updated _bestLoss would wrongly also bump _badEpochs. + var improved = currentLoss < _bestLoss * (1f - _minDelta); + + if (improved) { _bestLoss = currentLoss; _badEpochs = 0; SaveCheckpoint(); } - else + + if (!improved) { _badEpochs++; } diff --git a/Sources/Main/Optimizers/SGD.cs b/Sources/Main/Optimizers/SGD.cs index 93fa5248..9fd2d848 100644 --- a/Sources/Main/Optimizers/SGD.cs +++ b/Sources/Main/Optimizers/SGD.cs @@ -113,16 +113,15 @@ public void Step() negativeLearningRate, p.Data.AsReadOnlySpan(), p.Data.AsSpan()); + continue; } - else - { - var n = _nodes[i]!; - ElementwiseKernels.MultiplyAdd( - n.GradView.AsReadOnlySpan(), - negativeLearningRate, - n.DataView.AsReadOnlySpan(), - n.DataView.AsSpan()); - } + + var n = _nodes[i]!; + ElementwiseKernels.MultiplyAdd( + n.GradView.AsReadOnlySpan(), + negativeLearningRate, + n.DataView.AsReadOnlySpan(), + n.DataView.AsSpan()); } } @@ -137,11 +136,10 @@ public void ZeroGrad() if (_params[i] != null) { _params[i]!.ZeroGrad(); + continue; } - else - { - _nodes[i]!.ZeroGrad(); - } + + _nodes[i]!.ZeroGrad(); } } } From 9a0bb61a0f4466f8403f8011e18dfefd2c287386 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Tue, 21 Jul 2026 12:19:02 +0200 Subject: [PATCH 03/37] else --- .editorconfig | 2 +- Sources/Main/Audio/MelSpectrogram.cs | 5 +- Sources/Main/Audio/Mp3/ChannelMode.cs | 2 +- Sources/Main/Audio/Mp3/Mp3BitReader.cs | 2 +- Sources/Main/Audio/Mp3/Mp3Decoder.cs | 139 +++++++++--------- Sources/Main/Audio/Mp3/Mp3FrameHeader.cs | 2 +- Sources/Main/Audio/Mp3/Mp3Huffman.cs | 12 +- Sources/Main/Audio/Mp3/Mp3HuffmanData.cs | 2 +- Sources/Main/Audio/Mp3/Mp3Info.cs | 2 +- Sources/Main/Audio/Mp3/Mp3Reader.cs | 2 +- Sources/Main/Audio/Mp3/Mp3SynthWindowData.cs | 2 +- Sources/Main/Audio/Mp3/Mp3Tables.cs | 2 +- Sources/Main/Audio/Mp3/MpegVersion.cs | 2 +- Sources/Main/Audio/Mp3AudioDecoder.cs | 2 +- Sources/Main/Audio/Tts/AudioPostProcessing.cs | 2 +- Sources/Main/Audio/Tts/AudioSegmenter.cs | 2 +- .../Main/Audio/Tts/EnglishNumberToWords.cs | 4 +- Sources/Main/Audio/Tts/IAudioSink.cs | 2 +- Sources/Main/Audio/Tts/ITextToSpeechEngine.cs | 2 +- .../Main/Audio/Tts/Orpheus/OrpheusPrompt.cs | 2 +- .../Audio/Tts/Orpheus/OrpheusSnacBridge.cs | 2 +- .../Tts/Orpheus/OrpheusTrainingExample.cs | 2 +- .../Tts/Orpheus/OrpheusTrainingSequence.cs | 2 +- .../Audio/Tts/Orpheus/OrpheusVoiceEngine.cs | 2 +- .../Tts/Orpheus/VoiceCloneDatasetBuilder.cs | 2 +- .../Audio/Tts/Orpheus/VoiceCloneTrainer.cs | 8 +- .../Main/Audio/Tts/PlaceholderTtsEngine.cs | 4 +- Sources/Main/Audio/Tts/SentenceSplitter.cs | 2 +- Sources/Main/Audio/Tts/Snac/Snac.cs | 2 +- .../Main/Audio/Tts/Snac/SnacActivations.cs | 2 +- Sources/Main/Audio/Tts/Snac/SnacBlocks.cs | 2 +- Sources/Main/Audio/Tts/Snac/SnacConfig.cs | 2 +- Sources/Main/Audio/Tts/Snac/SnacConv.cs | 2 +- Sources/Main/Audio/Tts/Snac/SnacDecoder.cs | 5 +- Sources/Main/Audio/Tts/Snac/SnacEncoder.cs | 12 +- Sources/Main/Audio/Tts/Snac/SnacResidualVq.cs | 2 +- Sources/Main/Audio/Tts/Snac/SnacWeights.cs | 2 +- .../Main/Audio/Tts/SyntheticSpeechMetadata.cs | 2 +- Sources/Main/Audio/Tts/TtsOptions.cs | 2 +- Sources/Main/Audio/Tts/TtsTextNormalizer.cs | 31 ++-- Sources/Main/Audio/Tts/VoiceProfile.cs | 2 +- Sources/Main/Audio/Tts/VoiceProfileStore.cs | 6 +- Sources/Main/Audio/Tts/WavAudioSink.cs | 2 +- Sources/Main/Audio/WavAudioDecoder.cs | 2 +- Sources/Main/Audio/WavReader.cs | 38 +++-- Sources/Main/Audio/WavSampleFormat.cs | 2 +- Sources/Main/Audio/WavWriter.cs | 5 +- Sources/Main/DeepLearning/ConvLayer.cs | 3 +- Sources/Main/DeepLearning/FeedForwardLayer.cs | 56 ++----- Sources/Main/DeepLearning/GPT1Model.cs | 46 ++---- Sources/Main/DeepLearning/LSTMLayer.cs | 7 +- .../DeepLearning/MultiHeadAttentionLayer.cs | 9 +- Sources/Main/Intrinsics/Simd.cs | 21 ++- Sources/Main/Kernels/Conv2DGemmKernels.cs | 9 +- Sources/Main/Kernels/Conv2DKernels.cs | 4 +- Sources/Main/Kernels/LinearKernels.cs | 4 +- Sources/Main/Kernels/PoolingKernels.cs | 49 +++--- Sources/Main/Ops/CtcDecoder.cs | 2 +- Sources/Main/Ops/CtcLoss.cs | 2 +- Sources/Main/Ops/ICtcLanguageModel.cs | 2 +- Sources/Main/Ops/NGramCtcLanguageModel.cs | 2 +- Sources/Main/Ops/TensorMath.Activations.cs | 6 +- Sources/Main/Ops/TensorMath.Algebra.cs | 15 +- Sources/Main/Ops/TensorMath.Attention.cs | 9 +- Sources/Main/Ops/TensorMath.Convolution.cs | 17 ++- Sources/Main/Ops/TensorMath.DepthwiseConv.cs | 6 +- Sources/Main/Ops/TensorMath.Gelu.cs | 6 +- Sources/Main/Ops/TensorMath.LayerNorm.cs | 6 +- Sources/Main/Ops/TensorMath.Normalization.cs | 16 +- Sources/Main/Ops/TensorMath.Pooling.cs | 6 +- Sources/Main/Ops/TensorMath.RmsNorm.cs | 6 +- Sources/Main/Ops/TensorMath.Rope.cs | 6 +- Sources/Main/Ops/TensorMath.Sequence.cs | 9 +- Sources/Main/Ops/TensorMath.Shape.cs | 20 +-- Sources/Main/Ops/TensorMath.SiLU.cs | 6 +- Sources/Main/Ops/TensorMath.cs | 2 +- 76 files changed, 356 insertions(+), 335 deletions(-) diff --git a/.editorconfig b/.editorconfig index 12c95bf6..00d03af5 100644 --- a/.editorconfig +++ b/.editorconfig @@ -240,7 +240,7 @@ dotnet_diagnostic.RS0030.severity = none # 3. The rule ID must NOT appear in in Directory.Build.props. That maps to # -warnaserror-:ID, which reverts the diagnostic to warning EVEN WHERE .editorconfig sets error — # defeating this whole section. (This is why OVERFIT008/015 are absent from that list.) -[Sources/Main/{Anomalies,Autograd,Core,Data,Evolutionary,Exceptions,Inference,Optimizers,Parameters,Randomization,Serving,Statistical,Tensors}/**.cs] +[Sources/Main/{Anomalies,Audio,Autograd,Core,Data,DeepLearning,Evolutionary,Exceptions,Inference,Intrinsics,Kernels,Ops,Optimizers,Parameters,Randomization,Serving,Statistical,Tensors}/**.cs] dotnet_diagnostic.OVERFIT021.severity = error # OVERFIT022 (recursion) — ERROR on the paths that parse untrusted, externally-authored input, where an diff --git a/Sources/Main/Audio/MelSpectrogram.cs b/Sources/Main/Audio/MelSpectrogram.cs index 90a1c12f..8544fe94 100644 --- a/Sources/Main/Audio/MelSpectrogram.cs +++ b/Sources/Main/Audio/MelSpectrogram.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -66,7 +66,8 @@ public MelSpectrogram(int nMels = DefaultMelCount, ReadOnlySpan melFilter { _melFilters = BuildSlaneyMelFilters(nMels, _nFreqs, SampleRate, NFft); } - else + + if (!(melFilters.IsEmpty)) { if (melFilters.Length != nMels * _nFreqs) { diff --git a/Sources/Main/Audio/Mp3/ChannelMode.cs b/Sources/Main/Audio/Mp3/ChannelMode.cs index a0b8c1d5..f06318ce 100644 --- a/Sources/Main/Audio/Mp3/ChannelMode.cs +++ b/Sources/Main/Audio/Mp3/ChannelMode.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Mp3/Mp3BitReader.cs b/Sources/Main/Audio/Mp3/Mp3BitReader.cs index c60913fb..3f503123 100644 --- a/Sources/Main/Audio/Mp3/Mp3BitReader.cs +++ b/Sources/Main/Audio/Mp3/Mp3BitReader.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Mp3/Mp3Decoder.cs b/Sources/Main/Audio/Mp3/Mp3Decoder.cs index 4c1d9245..9ca938d4 100644 --- a/Sources/Main/Audio/Mp3/Mp3Decoder.cs +++ b/Sources/Main/Audio/Mp3/Mp3Decoder.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -230,7 +230,8 @@ private void ParseSideInfo(ref Mp3BitReader br, int nGran) } } } - else + + if (!(_isMpeg1)) { _mainDataBegin = (int)br.ReadBits(8); br.ReadBits(_nch == 1 ? 1 : 2); // private bits @@ -262,7 +263,8 @@ private void ParseSideInfo(ref Mp3BitReader br, int nGran) _region0[g] = _blockType[g] == 2 && _mixedBlock[g] == 0 ? 8 : 7; _region1[g] = 20 - _region0[g]; } - else + + if (!(_winSwitch[g] == 1)) { for (var r = 0; r < 3; r++) { @@ -293,7 +295,8 @@ private void ReadMainData(ref Mp3BitReader br, int nGran) { ReadScaleFactorsMpeg1(ref br, gr, ch); } - else + + if (!(_isMpeg1)) { ReadScaleFactorsLsf(ref br, gr, ch); } @@ -326,7 +329,8 @@ private void ReadScaleFactorsMpeg1(ref Mp3BitReader br, int gr, int ch) } } } - else + + if (!(_mixedBlock[g] != 0)) { for (var sfb = 0; sfb < 12; sfb++) { @@ -338,7 +342,8 @@ private void ReadScaleFactorsMpeg1(ref Mp3BitReader br, int gr, int ch) } } } - else + + if (!(_winSwitch[g] != 0 && _blockType[g] == 2)) { // long blocks, with scfsi sharing between granule 0 and 1 ReadLongBand(ref br, gr, ch, 0, 6, slen1, 0); @@ -358,7 +363,8 @@ private void ReadLongBand(ref Mp3BitReader br, int gr, int ch, int from, int to, _scalefacL[g * 23 + sfb] = (int)br.ReadBits(slen); } } - else + + if (!(_scfsi[ch * 4 + scfsiBand] == 0 || gr == 0)) { var g0 = GC(0, ch); for (var sfb = from; sfb < to; sfb++) @@ -374,32 +380,39 @@ private void ReadScaleFactorsLsf(ref Mp3BitReader br, int gr, int ch) var sfc = _scfCompress[g]; int slen0, slen1, slen2, slen3, tindex; - if (sfc < 400) - { - slen0 = (sfc >> 4) / 5; - slen1 = (sfc >> 4) % 5; - slen2 = (sfc & 0xf) >> 2; - slen3 = sfc & 0x3; - tindex = 0; - } - else if (sfc < 500) - { - sfc -= 400; - slen0 = (sfc >> 2) / 5; - slen1 = (sfc >> 2) % 5; - slen2 = sfc & 0x3; - slen3 = 0; - tindex = 1; - } - else + // Classify BEFORE the bodies run: they subtract from sfc, so a second `sfc < 500` test would + // see the already-adjusted value. switch (with default) also proves definite assignment of + // slen0..3 / tindex to the compiler, which split ifs could not. + var scfBand = sfc < 400 ? 0 : sfc < 500 ? 1 : 2; + + switch (scfBand) { - sfc -= 500; - slen0 = sfc / 3; - slen1 = sfc % 3; - slen2 = 0; - slen3 = 0; - tindex = 2; - _preflag[g] = 1; + case 0: + slen0 = (sfc >> 4) / 5; + slen1 = (sfc >> 4) % 5; + slen2 = (sfc & 0xf) >> 2; + slen3 = sfc & 0x3; + tindex = 0; + break; + + case 1: + sfc -= 400; + slen0 = (sfc >> 2) / 5; + slen1 = (sfc >> 2) % 5; + slen2 = sfc & 0x3; + slen3 = 0; + tindex = 1; + break; + + default: + sfc -= 500; + slen0 = sfc / 3; + slen1 = sfc % 3; + slen2 = 0; + slen3 = 0; + tindex = 2; + _preflag[g] = 1; + break; } var blockClass = _blockType[g] == 2 ? (_mixedBlock[g] != 0 ? 2 : 1) : 0; @@ -429,7 +442,8 @@ private void ReadScaleFactorsLsf(ref Mp3BitReader br, int gr, int ch) } } } - else + + if (!(_blockType[g] == 2)) { // long — fill scalefac_l[0..] sequentially. var sfb = 0; @@ -464,18 +478,12 @@ private void ReadHuffman(ref Mp3BitReader br, int part2Start, int gr, int ch) } var bitPosEnd = part2Start + _part23[g] - 1; - int region1Start, region2Start; - if (_winSwitch[g] == 1 && _blockType[g] == 2) - { - region1Start = 36; - region2Start = 576; - } - else - { - var bl = Mp3Tables.SfBandLong[_sfIndex]; - region1Start = bl[_region0[g] + 1]; - region2Start = bl[_region0[g] + _region1[g] + 2]; - } + // Ternaries: both are read below, so the compiler must see them assigned on every path. + // The band-table lookups stay lazy — only the taken branch of a ternary is evaluated. + var shortBlock = _winSwitch[g] == 1 && _blockType[g] == 2; + var bandsLong = Mp3Tables.SfBandLong[_sfIndex]; + var region1Start = shortBlock ? 36 : bandsLong[_region0[g] + 1]; + var region2Start = shortBlock ? 576 : bandsLong[_region0[g] + _region1[g] + 2]; var pos = 0; var bigEnd = _bigValues[g] * 2; @@ -568,7 +576,8 @@ private void Requantize(int gr, int ch) } } } - else + + if (!(_mixedBlock[g] != 0)) { var sfb = 0; var next = bs[1] * 3; @@ -593,7 +602,8 @@ private void Requantize(int gr, int ch) } } } - else + + if (!(_winSwitch[g] == 1 && _blockType[g] == 2)) { var sfb = 0; var next = bl[1]; @@ -726,7 +736,8 @@ private void Stereo(int gr) } } } - else + + if (!(_mixedBlock[g0] != 0)) { for (var sfb = 0; sfb < 12; sfb++) { @@ -737,7 +748,8 @@ private void Stereo(int gr) } } } - else + + if (!(_winSwitch[g0] == 1 && _blockType[g0] == 2)) { for (var sfb = 0; sfb < 21; sfb++) { @@ -759,17 +771,8 @@ private void IntensityLong(int gr, int sfb) return; } var bl = Mp3Tables.SfBandLong[_sfIndex]; - float ratioL, ratioR; - if (isPos == 6) - { - ratioL = 1f; - ratioR = 0f; - } - else - { - ratioL = IsRatios[isPos] / (1f + IsRatios[isPos]); - ratioR = 1f / (1f + IsRatios[isPos]); - } + var ratioL = isPos == 6 ? 1f : IsRatios[isPos] / (1f + IsRatios[isPos]); + var ratioR = isPos == 6 ? 0f : 1f / (1f + IsRatios[isPos]); var b0 = GC(gr, 0) * 576; var b1 = GC(gr, 1) * 576; for (var i = bl[sfb]; i < bl[sfb + 1]; i++) @@ -794,17 +797,8 @@ private void IntensityShort(int gr, int sfb) { continue; } - float ratioL, ratioR; - if (isPos == 6) - { - ratioL = 1f; - ratioR = 0f; - } - else - { - ratioL = IsRatios[isPos] / (1f + IsRatios[isPos]); - ratioR = 1f / (1f + IsRatios[isPos]); - } + var ratioL = isPos == 6 ? 1f : IsRatios[isPos] / (1f + IsRatios[isPos]); + var ratioR = isPos == 6 ? 0f : 1f / (1f + IsRatios[isPos]); var start = bs[sfb] * 3 + winLen * win; for (var i = start; i < start + winLen; i++) { @@ -862,7 +856,8 @@ private void ImdctWin(ReadOnlySpan input, int blockType) } } } - else + + if (!(blockType == 2)) { var cos = Mp3Tables.CosN36; // [18 × 36] var wb = blockType * 36; diff --git a/Sources/Main/Audio/Mp3/Mp3FrameHeader.cs b/Sources/Main/Audio/Mp3/Mp3FrameHeader.cs index 3010993d..5655e6fb 100644 --- a/Sources/Main/Audio/Mp3/Mp3FrameHeader.cs +++ b/Sources/Main/Audio/Mp3/Mp3FrameHeader.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Mp3/Mp3Huffman.cs b/Sources/Main/Audio/Mp3/Mp3Huffman.cs index 04a79579..834a2efb 100644 --- a/Sources/Main/Audio/Mp3/Mp3Huffman.cs +++ b/Sources/Main/Audio/Mp3/Mp3Huffman.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -135,7 +135,12 @@ private static bool Walk(ref Mp3BitReader br, int tableSelect, out int x, out in y = e & 0xf; return true; } - if (br.ReadBit()) + // ReadBit CONSUMES a bit, so it must be called exactly once per tree step — capture it. + // Two separate `if (br.ReadBit())` / `if (!br.ReadBit())` tests would advance the bitstream + // twice per node and silently desynchronise the whole Huffman decode. + var bit = br.ReadBit(); + + if (bit) { while ((ht[off + point] & 0xff) >= 250) { @@ -143,7 +148,8 @@ private static bool Walk(ref Mp3BitReader br, int tableSelect, out int x, out in } point += ht[off + point] & 0xff; } - else + + if (!bit) { while ((ht[off + point] >> 8) >= 250) { diff --git a/Sources/Main/Audio/Mp3/Mp3HuffmanData.cs b/Sources/Main/Audio/Mp3/Mp3HuffmanData.cs index a09b6765..6f94a85b 100644 --- a/Sources/Main/Audio/Mp3/Mp3HuffmanData.cs +++ b/Sources/Main/Audio/Mp3/Mp3HuffmanData.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Mp3/Mp3Info.cs b/Sources/Main/Audio/Mp3/Mp3Info.cs index 847705f8..4762a553 100644 --- a/Sources/Main/Audio/Mp3/Mp3Info.cs +++ b/Sources/Main/Audio/Mp3/Mp3Info.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Mp3/Mp3Reader.cs b/Sources/Main/Audio/Mp3/Mp3Reader.cs index de19a935..3f04e738 100644 --- a/Sources/Main/Audio/Mp3/Mp3Reader.cs +++ b/Sources/Main/Audio/Mp3/Mp3Reader.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Mp3/Mp3SynthWindowData.cs b/Sources/Main/Audio/Mp3/Mp3SynthWindowData.cs index f1dfbad2..6bedba02 100644 --- a/Sources/Main/Audio/Mp3/Mp3SynthWindowData.cs +++ b/Sources/Main/Audio/Mp3/Mp3SynthWindowData.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Mp3/Mp3Tables.cs b/Sources/Main/Audio/Mp3/Mp3Tables.cs index 77405b37..a5a9e0bf 100644 --- a/Sources/Main/Audio/Mp3/Mp3Tables.cs +++ b/Sources/Main/Audio/Mp3/Mp3Tables.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Mp3/MpegVersion.cs b/Sources/Main/Audio/Mp3/MpegVersion.cs index 72f99807..a983b798 100644 --- a/Sources/Main/Audio/Mp3/MpegVersion.cs +++ b/Sources/Main/Audio/Mp3/MpegVersion.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Mp3AudioDecoder.cs b/Sources/Main/Audio/Mp3AudioDecoder.cs index 78577706..4564024b 100644 --- a/Sources/Main/Audio/Mp3AudioDecoder.cs +++ b/Sources/Main/Audio/Mp3AudioDecoder.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/AudioPostProcessing.cs b/Sources/Main/Audio/Tts/AudioPostProcessing.cs index ae402488..2793661a 100644 --- a/Sources/Main/Audio/Tts/AudioPostProcessing.cs +++ b/Sources/Main/Audio/Tts/AudioPostProcessing.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/AudioSegmenter.cs b/Sources/Main/Audio/Tts/AudioSegmenter.cs index 0251c4d4..3257c573 100644 --- a/Sources/Main/Audio/Tts/AudioSegmenter.cs +++ b/Sources/Main/Audio/Tts/AudioSegmenter.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/EnglishNumberToWords.cs b/Sources/Main/Audio/Tts/EnglishNumberToWords.cs index fec3cc14..6cd32877 100644 --- a/Sources/Main/Audio/Tts/EnglishNumberToWords.cs +++ b/Sources/Main/Audio/Tts/EnglishNumberToWords.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -91,7 +91,7 @@ private static void AppendBelowThousand(StringBuilder sb, int n) sb.Append(' ').Append(Ones[n % 10]); } } - else if (n > 0) + if (n is > 0 and < 20) { if (wrote) { diff --git a/Sources/Main/Audio/Tts/IAudioSink.cs b/Sources/Main/Audio/Tts/IAudioSink.cs index c2f30f1f..5629e75f 100644 --- a/Sources/Main/Audio/Tts/IAudioSink.cs +++ b/Sources/Main/Audio/Tts/IAudioSink.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/ITextToSpeechEngine.cs b/Sources/Main/Audio/Tts/ITextToSpeechEngine.cs index 81725313..2a6fb571 100644 --- a/Sources/Main/Audio/Tts/ITextToSpeechEngine.cs +++ b/Sources/Main/Audio/Tts/ITextToSpeechEngine.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/Orpheus/OrpheusPrompt.cs b/Sources/Main/Audio/Tts/Orpheus/OrpheusPrompt.cs index 6f2010d9..3d3c18a0 100644 --- a/Sources/Main/Audio/Tts/Orpheus/OrpheusPrompt.cs +++ b/Sources/Main/Audio/Tts/Orpheus/OrpheusPrompt.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/Orpheus/OrpheusSnacBridge.cs b/Sources/Main/Audio/Tts/Orpheus/OrpheusSnacBridge.cs index 70124ffd..ef351e6e 100644 --- a/Sources/Main/Audio/Tts/Orpheus/OrpheusSnacBridge.cs +++ b/Sources/Main/Audio/Tts/Orpheus/OrpheusSnacBridge.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/Orpheus/OrpheusTrainingExample.cs b/Sources/Main/Audio/Tts/Orpheus/OrpheusTrainingExample.cs index 14e63b46..cf7cc09f 100644 --- a/Sources/Main/Audio/Tts/Orpheus/OrpheusTrainingExample.cs +++ b/Sources/Main/Audio/Tts/Orpheus/OrpheusTrainingExample.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/Orpheus/OrpheusTrainingSequence.cs b/Sources/Main/Audio/Tts/Orpheus/OrpheusTrainingSequence.cs index 6e7c4d63..d5ee2043 100644 --- a/Sources/Main/Audio/Tts/Orpheus/OrpheusTrainingSequence.cs +++ b/Sources/Main/Audio/Tts/Orpheus/OrpheusTrainingSequence.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/Orpheus/OrpheusVoiceEngine.cs b/Sources/Main/Audio/Tts/Orpheus/OrpheusVoiceEngine.cs index 51e9ac87..6b89d2fc 100644 --- a/Sources/Main/Audio/Tts/Orpheus/OrpheusVoiceEngine.cs +++ b/Sources/Main/Audio/Tts/Orpheus/OrpheusVoiceEngine.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/Orpheus/VoiceCloneDatasetBuilder.cs b/Sources/Main/Audio/Tts/Orpheus/VoiceCloneDatasetBuilder.cs index 14fc60e0..18199fee 100644 --- a/Sources/Main/Audio/Tts/Orpheus/VoiceCloneDatasetBuilder.cs +++ b/Sources/Main/Audio/Tts/Orpheus/VoiceCloneDatasetBuilder.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/Orpheus/VoiceCloneTrainer.cs b/Sources/Main/Audio/Tts/Orpheus/VoiceCloneTrainer.cs index f50dd19b..c5e1445a 100644 --- a/Sources/Main/Audio/Tts/Orpheus/VoiceCloneTrainer.cs +++ b/Sources/Main/Audio/Tts/Orpheus/VoiceCloneTrainer.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -64,7 +64,8 @@ public VoiceCloneTrainer(string orpheusGgufPath, int maxSeqLen = 768, QLoRAOptio _outputStart = Math.Min(Tokenizer.EndOfTextTokenId, audioBase); _vocab = fullVocab - _outputStart; } - else + + if (!(restrictToAudioVocab)) { _outputStart = 0; _vocab = fullVocab; @@ -137,7 +138,8 @@ public IReadOnlyList Train(IReadOnlyList examples { targets[i] = IgnoreIndex; } - else + + if (!(i < ex.PromptLength - 1)) { targets[i] -= _outputStart; } diff --git a/Sources/Main/Audio/Tts/PlaceholderTtsEngine.cs b/Sources/Main/Audio/Tts/PlaceholderTtsEngine.cs index c09b18d9..744546d3 100644 --- a/Sources/Main/Audio/Tts/PlaceholderTtsEngine.cs +++ b/Sources/Main/Audio/Tts/PlaceholderTtsEngine.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -84,7 +84,7 @@ private void EmitTone(IAudioSink output, int wordLen, int hash, float amplitude, { envelope = 0.5f * (1f - MathF.Cos(MathF.PI * n / fade)); } - else if (n >= count - fade) + if (n >= fade && n >= count - fade) { envelope = 0.5f * (1f - MathF.Cos(MathF.PI * (count - 1 - n) / fade)); } diff --git a/Sources/Main/Audio/Tts/SentenceSplitter.cs b/Sources/Main/Audio/Tts/SentenceSplitter.cs index 71f63228..b8ad98e5 100644 --- a/Sources/Main/Audio/Tts/SentenceSplitter.cs +++ b/Sources/Main/Audio/Tts/SentenceSplitter.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/Snac/Snac.cs b/Sources/Main/Audio/Tts/Snac/Snac.cs index a0af0717..f2a1c8f3 100644 --- a/Sources/Main/Audio/Tts/Snac/Snac.cs +++ b/Sources/Main/Audio/Tts/Snac/Snac.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/Snac/SnacActivations.cs b/Sources/Main/Audio/Tts/Snac/SnacActivations.cs index 3a03265e..d74e0a65 100644 --- a/Sources/Main/Audio/Tts/Snac/SnacActivations.cs +++ b/Sources/Main/Audio/Tts/Snac/SnacActivations.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/Snac/SnacBlocks.cs b/Sources/Main/Audio/Tts/Snac/SnacBlocks.cs index 2a350e71..7ce0ba8e 100644 --- a/Sources/Main/Audio/Tts/Snac/SnacBlocks.cs +++ b/Sources/Main/Audio/Tts/Snac/SnacBlocks.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/Snac/SnacConfig.cs b/Sources/Main/Audio/Tts/Snac/SnacConfig.cs index 4c44a92c..5543940e 100644 --- a/Sources/Main/Audio/Tts/Snac/SnacConfig.cs +++ b/Sources/Main/Audio/Tts/Snac/SnacConfig.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/Snac/SnacConv.cs b/Sources/Main/Audio/Tts/Snac/SnacConv.cs index 3403c65a..c5fdc70a 100644 --- a/Sources/Main/Audio/Tts/Snac/SnacConv.cs +++ b/Sources/Main/Audio/Tts/Snac/SnacConv.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/Snac/SnacDecoder.cs b/Sources/Main/Audio/Tts/Snac/SnacDecoder.cs index f029b2af..29564715 100644 --- a/Sources/Main/Audio/Tts/Snac/SnacDecoder.cs +++ b/Sources/Main/Audio/Tts/Snac/SnacDecoder.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -80,7 +80,8 @@ private float[] FromCodes(int[][] codes, out int frames) zq[j] += proj[j]; } } - else + + if (!(stride == 1)) { var up = new float[latent * frames]; SnacResidualVq.RepeatInterleaveTime(proj, up, latent, ti, stride); diff --git a/Sources/Main/Audio/Tts/Snac/SnacEncoder.cs b/Sources/Main/Audio/Tts/Snac/SnacEncoder.cs index 28f99a4b..d1554ad2 100644 --- a/Sources/Main/Audio/Tts/Snac/SnacEncoder.cs +++ b/Sources/Main/Audio/Tts/Snac/SnacEncoder.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -102,16 +102,13 @@ private int[][] Quantize(float[] z, int frames) var ti = frames / stride; // avg_pool1d(stride) when the level is coarser than the latent rate - float[] pooled; + float[] pooled = residual; + if (stride > 1) { pooled = new float[latent * ti]; SnacResidualVq.AveragePoolTime(residual, pooled, latent, frames, stride); } - else - { - pooled = residual; - } // in_proj: latent → codebook_dim (1×1) var zE = new float[cbDim * ti]; @@ -141,7 +138,8 @@ private int[][] Quantize(float[] z, int frames) residual[j] -= up[j]; } } - else + + if (!(stride > 1)) { for (var j = 0; j < residual.Length; j++) { diff --git a/Sources/Main/Audio/Tts/Snac/SnacResidualVq.cs b/Sources/Main/Audio/Tts/Snac/SnacResidualVq.cs index e157c5e6..594d5adb 100644 --- a/Sources/Main/Audio/Tts/Snac/SnacResidualVq.cs +++ b/Sources/Main/Audio/Tts/Snac/SnacResidualVq.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/Snac/SnacWeights.cs b/Sources/Main/Audio/Tts/Snac/SnacWeights.cs index e611a436..7b43da57 100644 --- a/Sources/Main/Audio/Tts/Snac/SnacWeights.cs +++ b/Sources/Main/Audio/Tts/Snac/SnacWeights.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/SyntheticSpeechMetadata.cs b/Sources/Main/Audio/Tts/SyntheticSpeechMetadata.cs index 83a2a485..64b22db9 100644 --- a/Sources/Main/Audio/Tts/SyntheticSpeechMetadata.cs +++ b/Sources/Main/Audio/Tts/SyntheticSpeechMetadata.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/TtsOptions.cs b/Sources/Main/Audio/Tts/TtsOptions.cs index 5f3dffb1..b7952884 100644 --- a/Sources/Main/Audio/Tts/TtsOptions.cs +++ b/Sources/Main/Audio/Tts/TtsOptions.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/TtsTextNormalizer.cs b/Sources/Main/Audio/Tts/TtsTextNormalizer.cs index ad594070..8d3e4407 100644 --- a/Sources/Main/Audio/Tts/TtsTextNormalizer.cs +++ b/Sources/Main/Audio/Tts/TtsTextNormalizer.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -111,19 +111,26 @@ public string Normalize(string text) while (i < expanded.Length) { var c = expanded[i]; - if (char.IsDigit(c) || (c == '-' && i + 1 < expanded.Length && char.IsDigit(expanded[i + 1]))) + + // Decide BEFORE dispatching: AppendNumber/AppendWord advance i, so a later test that + // re-read expanded[i] would be looking at a different character. + var isNumber = char.IsDigit(c) + || (c == '-' && i + 1 < expanded.Length && char.IsDigit(expanded[i + 1])); + + if (isNumber) { i = AppendNumber(sb, expanded, i); + continue; } - else if (IsWordChar(c)) + + if (IsWordChar(c)) { i = AppendWord(sb, expanded, i); + continue; } - else - { - sb.Append(c); - i++; - } + + sb.Append(c); + i++; } return CollapseWhitespace(sb.ToString()); @@ -170,11 +177,15 @@ private static int AppendNumber(StringBuilder sb, string text, int i) } var intPart = text[start..(fracStart < 0 ? i : fracStart - 1)].Replace(",", string.Empty); - if (long.TryParse(intPart, out var intValue)) + // Capture: the second test must not re-run TryParse (it would redeclare `intValue` and parse twice). + var parsed = long.TryParse(intPart, out var intValue); + + if (parsed) { sb.Append(EnglishNumberToWords.Convert(intValue)); } - else + + if (!parsed) { // Too long for a long, or malformed → speak the digits individually. AppendDigits(sb, intPart); diff --git a/Sources/Main/Audio/Tts/VoiceProfile.cs b/Sources/Main/Audio/Tts/VoiceProfile.cs index 435e7878..d8ad7f2c 100644 --- a/Sources/Main/Audio/Tts/VoiceProfile.cs +++ b/Sources/Main/Audio/Tts/VoiceProfile.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/Tts/VoiceProfileStore.cs b/Sources/Main/Audio/Tts/VoiceProfileStore.cs index f26659c6..ab7f384e 100644 --- a/Sources/Main/Audio/Tts/VoiceProfileStore.cs +++ b/Sources/Main/Audio/Tts/VoiceProfileStore.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -44,8 +44,10 @@ public static void Save(VoiceProfile profile, string directory) bw.Write(embedding![k]); } } - else if (File.Exists(binPath)) + if (embeddingDim <= 0 && File.Exists(binPath)) { + // Gated on embeddingDim: the branch above CREATES binPath, so an ungated File.Exists + // here would delete the file that was just written. File.Delete(binPath); // a preset voice has no embedding — drop a stale one } } diff --git a/Sources/Main/Audio/Tts/WavAudioSink.cs b/Sources/Main/Audio/Tts/WavAudioSink.cs index cf9c968e..8406fce5 100644 --- a/Sources/Main/Audio/Tts/WavAudioSink.cs +++ b/Sources/Main/Audio/Tts/WavAudioSink.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/WavAudioDecoder.cs b/Sources/Main/Audio/WavAudioDecoder.cs index 4c99c79d..7932520a 100644 --- a/Sources/Main/Audio/WavAudioDecoder.cs +++ b/Sources/Main/Audio/WavAudioDecoder.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/WavReader.cs b/Sources/Main/Audio/WavReader.cs index 8765afec..3a4ce466 100644 --- a/Sources/Main/Audio/WavReader.cs +++ b/Sources/Main/Audio/WavReader.cs @@ -1,4 +1,4 @@ - + // Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. @@ -59,11 +59,12 @@ public static float[] ReadMono(Stream stream, out int sampleRate) br.ReadBytes(chunkSize - consumed); } // skip extension } - else if (chunkId == "data") + if (chunkId == "data") { data = br.ReadBytes(chunkSize); } - else + + if (chunkId != "fmt " && chunkId != "data") { br.ReadBytes(chunkSize); // skip unknown chunk if ((chunkSize & 1) == 1) @@ -87,29 +88,34 @@ public static float[] ReadMono(Stream stream, out int sampleRate) private static float[] Decode(byte[] data, int audioFormat, int channels, int bitsPerSample) { var span = data.AsSpan(); - float[] interleaved; - if (audioFormat == 1 && bitsPerSample == 16) + + var is16BitPcm = audioFormat == 1 && bitsPerSample == 16; + var is32BitFloat = audioFormat == 3 && bitsPerSample == 32; + + // Reject up front, so the two supported paths below leave `interleaved` definitely assigned. + if (!is16BitPcm && !is32BitFloat) { - var count = data.Length / 2; - interleaved = new float[count]; - for (var i = 0; i < count; i++) + throw new OverfitRuntimeException($"Unsupported WAV format (audioFormat={audioFormat}, bits={bitsPerSample}). Use 16-bit PCM or 32-bit float."); + } + + var bytesPerSample = is16BitPcm ? 2 : 4; + var interleaved = new float[data.Length / bytesPerSample]; + + if (is16BitPcm) + { + for (var i = 0; i < interleaved.Length; i++) { interleaved[i] = BinaryPrimitives.ReadInt16LittleEndian(span.Slice(i * 2, 2)) / 32768f; } } - else if (audioFormat == 3 && bitsPerSample == 32) + + if (is32BitFloat) { - var count = data.Length / 4; - interleaved = new float[count]; - for (var i = 0; i < count; i++) + for (var i = 0; i < interleaved.Length; i++) { interleaved[i] = BinaryPrimitives.ReadSingleLittleEndian(span.Slice(i * 4, 4)); } } - else - { - throw new OverfitRuntimeException($"Unsupported WAV format (audioFormat={audioFormat}, bits={bitsPerSample}). Use 16-bit PCM or 32-bit float."); - } if (channels == 1) { diff --git a/Sources/Main/Audio/WavSampleFormat.cs b/Sources/Main/Audio/WavSampleFormat.cs index d1fd7c04..dbf0bc1e 100644 --- a/Sources/Main/Audio/WavSampleFormat.cs +++ b/Sources/Main/Audio/WavSampleFormat.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Audio/WavWriter.cs b/Sources/Main/Audio/WavWriter.cs index 66893c24..222ef78f 100644 --- a/Sources/Main/Audio/WavWriter.cs +++ b/Sources/Main/Audio/WavWriter.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -72,7 +72,8 @@ public static void WriteMono(Stream stream, ReadOnlySpan samples, int sam bw.Write(samples[i]); } } - else + + if (!(format == WavSampleFormat.Float32)) { for (var i = 0; i < samples.Length; i++) { diff --git a/Sources/Main/DeepLearning/ConvLayer.cs b/Sources/Main/DeepLearning/ConvLayer.cs index bec489a5..5ba9ef79 100644 --- a/Sources/Main/DeepLearning/ConvLayer.cs +++ b/Sources/Main/DeepLearning/ConvLayer.cs @@ -283,7 +283,8 @@ public void ForwardInference(ReadOnlySpan input, Span output) input, Kernels.DataReadOnlySpan, output, _inC, _outC, _h, _w, _k); } - else + + if (_padding != 0 || _stride != 1) { Conv2DKernels.ForwardNchw( input, Kernels.DataReadOnlySpan, output, diff --git a/Sources/Main/DeepLearning/FeedForwardLayer.cs b/Sources/Main/DeepLearning/FeedForwardLayer.cs index 887e96a8..d3789743 100644 --- a/Sources/Main/DeepLearning/FeedForwardLayer.cs +++ b/Sources/Main/DeepLearning/FeedForwardLayer.cs @@ -137,51 +137,27 @@ public AutogradNode Forward(ComputationGraph? graph, AutogradNode input) // [B*T, dModel] @ W1 + b1 → [B*T, dFF]. QLoRA output hook owns the whole projection; // else weight-level LoRA (W_eff = W1 + A@B) or the plain cached parameter node. - AutogradNode h1; - if (W1OutputProvider is not null) - { - h1 = W1OutputProvider(graph, flat); - } - else - { - AutogradNode w1Node; - if (W1WeightProvider is not null) - { - w1Node = W1WeightProvider(graph); - } - else - { - _w1Node ??= W1.AsNode(); - w1Node = _w1Node; - } - - h1 = graph.Linear(flat, w1Node, _b1Node); - } + // Nested ternaries, not if/else: C# evaluates only the taken branch, so a hook that is NOT set + // is never invoked. Hoisting the weight resolution out would call W1WeightProvider even when the + // output hook owns the projection — that delegate records nodes on the graph, so it would be a + // behaviour change, not a refactor. + var h1 = W1OutputProvider is not null + ? W1OutputProvider(graph, flat) + : graph.Linear( + flat, + W1WeightProvider is not null ? W1WeightProvider(graph) : (_w1Node ??= W1.AsNode()), + _b1Node); // GELU([B*T, dFF]) var act = TensorMath.Gelu(graph, h1); // [B*T, dFF] @ W2 + b2 → [B*T, dModel] - AutogradNode h2; - if (W2OutputProvider is not null) - { - h2 = W2OutputProvider(graph, act); - } - else - { - AutogradNode w2Node; - if (W2WeightProvider is not null) - { - w2Node = W2WeightProvider(graph); - } - else - { - _w2Node ??= W2.AsNode(); - w2Node = _w2Node; - } - - h2 = graph.Linear(act, w2Node, _b2Node); - } + var h2 = W2OutputProvider is not null + ? W2OutputProvider(graph, act) + : graph.Linear( + act, + W2WeightProvider is not null ? W2WeightProvider(graph) : (_w2Node ??= W2.AsNode()), + _b2Node); // Reshape back to [B, T, dModel] return graph.Reshape(h2, b, t, _dModel); diff --git a/Sources/Main/DeepLearning/GPT1Model.cs b/Sources/Main/DeepLearning/GPT1Model.cs index 80e2ad4b..457b7048 100644 --- a/Sources/Main/DeepLearning/GPT1Model.cs +++ b/Sources/Main/DeepLearning/GPT1Model.cs @@ -87,7 +87,8 @@ public GPT1Model(GPT1Config config, bool checkpointBlocks = false) config.VocabSize, config.DModel); } - else + + if (!config.TieWeights) { var scale = MathF.Sqrt(2f / config.DModel); var s = LMHead.DataSpan; @@ -257,7 +258,8 @@ public AutogradNode Forward( x = graph.Checkpoint((g, hidden) => blk.Forward(g, hidden), x, subArena); } } - else + + if (!_checkpointBlocks || !graph.IsRecording) { foreach (var block in Blocks) { @@ -416,18 +418,17 @@ public void Load(BinaryReader reader) FinalNorm.Load(reader); - if (!_config.TieWeights) - { - LMHead.Load(reader); - } - else + if (_config.TieWeights) { TransposeInto( TokenEmbedding.Weight.DataReadOnlySpan, LMHead.DataSpan, _config.VocabSize, _config.DModel); + return; } + + LMHead.Load(reader); } public void InvalidateAllCaches() @@ -566,32 +567,15 @@ private AutogradNode LMHeadForward( new TensorShape(vocabSize), clearMemory: true); - AutogradNode flatLogits; - - if (LMHeadOutputProvider is not null) - { - // QLoRA: the hook owns the whole head (FrozenQuantizedLinear(flat) + LoRA(flat)). - flatLogits = LMHeadOutputProvider(graph, flat); - } - else - { - AutogradNode headWeight; - - if (LMHeadWeightProvider is not null) - { - headWeight = LMHeadWeightProvider(graph); - } - else - { - _lmHeadNode ??= LMHead.AsNode(); - headWeight = _lmHeadNode; - } - - flatLogits = graph.Linear( + // QLoRA: the output hook owns the whole head (FrozenQuantizedLinear(flat) + LoRA(flat)). + // Nested ternaries keep this lazy — resolving the head weight eagerly would invoke + // LMHeadWeightProvider (which records graph nodes) even when the output hook is in charge. + var flatLogits = LMHeadOutputProvider is not null + ? LMHeadOutputProvider(graph, flat) + : graph.Linear( flat, - headWeight, + LMHeadWeightProvider is not null ? LMHeadWeightProvider(graph) : (_lmHeadNode ??= LMHead.AsNode()), bias); - } return graph.Reshape( flatLogits, diff --git a/Sources/Main/DeepLearning/LSTMLayer.cs b/Sources/Main/DeepLearning/LSTMLayer.cs index bd443b24..d8dbbd98 100644 --- a/Sources/Main/DeepLearning/LSTMLayer.cs +++ b/Sources/Main/DeepLearning/LSTMLayer.cs @@ -83,11 +83,10 @@ public void ForwardInference(int batchSize, int seqLen, ReadOnlySpan inpu var hNextOut = output.Slice(t * batchSize * hSize, batchSize * hSize); _cell.ForwardInference(batchSize, x_t, hBuf.Span, cBuf.Span, hNextOut, cBuf.Span); hNextOut.CopyTo(hBuf.Span); + continue; } - else - { - _cell.ForwardInference(batchSize, x_t, hBuf.Span, cBuf.Span, hBuf.Span, cBuf.Span); - } + + _cell.ForwardInference(batchSize, x_t, hBuf.Span, cBuf.Span, hBuf.Span, cBuf.Span); } if (!_returnSequences) diff --git a/Sources/Main/DeepLearning/MultiHeadAttentionLayer.cs b/Sources/Main/DeepLearning/MultiHeadAttentionLayer.cs index a7c44e69..4929da3f 100644 --- a/Sources/Main/DeepLearning/MultiHeadAttentionLayer.cs +++ b/Sources/Main/DeepLearning/MultiHeadAttentionLayer.cs @@ -392,11 +392,16 @@ public void Load(BinaryReader reader) _wqHeads[0].Load(reader); - if (IsNewQkvBiasCheckpointFormat(reader)) + // Capture: IsNewQkvBiasCheckpointFormat consumes bytes from the reader, so calling it a second + // time in a negated `if` would advance the stream twice and desynchronise the load. + var isNewFormat = IsNewQkvBiasCheckpointFormat(reader); + + if (isNewFormat) { LoadNewFormatAfterFirstWq(reader); } - else + + if (!isNewFormat) { LoadLegacyFormatAfterFirstWq(reader); } diff --git a/Sources/Main/Intrinsics/Simd.cs b/Sources/Main/Intrinsics/Simd.cs index 8823e95e..64fab056 100644 --- a/Sources/Main/Intrinsics/Simd.cs +++ b/Sources/Main/Intrinsics/Simd.cs @@ -43,8 +43,10 @@ public static void Add(ReadOnlySpan a, ReadOnlySpan b, Span } // Fall through to scalar for remainder } - // AVX2 path (original code, unchanged) - else if (CpuFeatures.HasAvx) + + // AVX2 path (original code, unchanged). CpuFeatures.HasXxx are static readonly bools the JIT + // constant-folds, so restating the AVX-512 guard here costs nothing at runtime. + if (!(CpuFeatures.HasAvx512 && len >= Avx512Threshold) && CpuFeatures.HasAvx) { var simdCount = Vector256.Count; for (; i <= len - simdCount; i += simdCount) @@ -92,8 +94,10 @@ public static void MulAdd(ReadOnlySpan a, float scalar, Span dst) } // Fall through to scalar for remainder } - // AVX2 path (original code, unchanged) - else if (CpuFeatures.HasAvx) + + // AVX2 path (original code, unchanged). CpuFeatures.HasXxx are static readonly bools the JIT + // constant-folds, so restating the AVX-512 guard here costs nothing at runtime. + if (!(CpuFeatures.HasAvx512 && len >= Avx512Threshold) && CpuFeatures.HasAvx) { var simdCount = Vector256.Count; var vs = Vector256.Create(scalar); @@ -156,7 +160,8 @@ public static float Dot(ReadOnlySpan a, ReadOnlySpan b) sum128 = Sse3.HorizontalAdd(sum128, sum128); sum128 = Sse3.HorizontalAdd(sum128, sum128); } - else + + if (!CpuFeatures.HasSse3) { sum128 = Sse.Add(sum128, Sse.Shuffle(sum128, sum128, 0b10_11_00_01)); sum128 = Sse.Add(sum128, Sse.Shuffle(sum128, sum128, 0b00_01_10_11)); @@ -236,8 +241,10 @@ public static void Relu(ReadOnlySpan input, Span output) } // Fall through to scalar for remainder } - // AVX2 path (original code, unchanged) - else if (CpuFeatures.HasAvx) + + // AVX2 path (original code, unchanged). CpuFeatures.HasXxx are static readonly bools the JIT + // constant-folds, so restating the AVX-512 guard here costs nothing at runtime. + if (!(CpuFeatures.HasAvx512 && len >= Avx512Threshold) && CpuFeatures.HasAvx) { var simdCount = Vector256.Count; var zero = Vector256.Zero; diff --git a/Sources/Main/Kernels/Conv2DGemmKernels.cs b/Sources/Main/Kernels/Conv2DGemmKernels.cs index 54aeb3f7..d4249805 100644 --- a/Sources/Main/Kernels/Conv2DGemmKernels.cs +++ b/Sources/Main/Kernels/Conv2DGemmKernels.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -183,11 +183,10 @@ private static unsafe void GemmNPanelWorker(int npStart, int npEnd, void* ctxPtr if (mrEff == Mr) { MicroKernel8x8(c.A, m0, k, pPackB, c.C, n, n0, nrEff); + continue; } - else - { - MicroKernelTail(c.A, m0, mrEff, k, pPackB, c.C, n, n0, nrEff); - } + + MicroKernelTail(c.A, m0, mrEff, k, pPackB, c.C, n, n0, nrEff); } } } diff --git a/Sources/Main/Kernels/Conv2DKernels.cs b/Sources/Main/Kernels/Conv2DKernels.cs index f90c02e6..aabf02d8 100644 --- a/Sources/Main/Kernels/Conv2DKernels.cs +++ b/Sources/Main/Kernels/Conv2DKernels.cs @@ -98,8 +98,10 @@ public static void ForwardValidNchw( inputW, outH, outW); + + continue; } - else + { // Valid conv (no padding, unit stride) is the padded path with padding=0, stride=1 — reuse the // parallel ForwardNchwSingleBatch so 1x1 / generic convs (e.g. ResNet's bottleneck 1x1 layers, diff --git a/Sources/Main/Kernels/LinearKernels.cs b/Sources/Main/Kernels/LinearKernels.cs index 0bcf2ac7..54102615 100644 --- a/Sources/Main/Kernels/LinearKernels.cs +++ b/Sources/Main/Kernels/LinearKernels.cs @@ -105,8 +105,10 @@ public static void Forward( outSlice, inputSize, outputSize); + + continue; } - else + { ForwardOutputMajorDot( inSlice, diff --git a/Sources/Main/Kernels/PoolingKernels.cs b/Sources/Main/Kernels/PoolingKernels.cs index 8d144c01..1f3ee622 100644 --- a/Sources/Main/Kernels/PoolingKernels.cs +++ b/Sources/Main/Kernels/PoolingKernels.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -177,13 +177,13 @@ public static void MaxPool2DForwardWithIndicesNchw( MaxPool2DForwardWithIndicesPool2( input, output, maxIndices, channels, inputH, inputW, outH, outW, batchOffset); + + return; } - else - { - MaxPool2DForwardWithIndicesGeneric( - input, output, maxIndices, - channels, inputH, inputW, pool, outH, outW, batchOffset); - } + + MaxPool2DForwardWithIndicesGeneric( + input, output, maxIndices, + channels, inputH, inputW, pool, outH, outW, batchOffset); } // ───────────────────────────────────────────────────────────────────── @@ -406,25 +406,15 @@ internal static void MaxPool2DForwardWithIndicesPool2Scalar( var a = pairMax[ow * 2]; var b = pairMax[ow * 2 + 1]; - float maxVal; - int maxIdx; - - if (a >= b) - { - // Horizontal winner is left column (ow*2). - // Vertical winner: whichever row had the larger value. - var idxInRow0 = row0Start + ow * 2; - var idxInRow1 = row1Start + ow * 2; - maxVal = a; - maxIdx = input[idxInRow0] >= input[idxInRow1] ? idxInRow0 : idxInRow1; - } - else - { - var idxInRow0 = row0Start + ow * 2 + 1; - var idxInRow1 = row1Start + ow * 2 + 1; - maxVal = b; - maxIdx = input[idxInRow0] >= input[idxInRow1] ? idxInRow0 : idxInRow1; - } + // Horizontal winner picks the column; vertical winner is whichever row held the + // larger value. Ternaries, not two ifs: both outputs are assigned on every path, and + // split ifs would not prove definite assignment to the compiler. + var takeLeft = a >= b; + var maxVal = takeLeft ? a : b; + var col = takeLeft ? ow * 2 : (ow * 2) + 1; + var idxInRow0 = row0Start + col; + var idxInRow1 = row1Start + col; + var maxIdx = input[idxInRow0] >= input[idxInRow1] ? idxInRow0 : idxInRow1; output[outRowBase + ow] = maxVal; maxIndices[outRowBase + ow] = batchOffset + maxIdx; @@ -702,12 +692,15 @@ private static void AveragePool2DForwardSingleBatchNchw( var ix = inputXBase + kx; var inBoundsX = (uint)ix < (uint)inputW; - if (inBoundsY && inBoundsX) + var inBounds = inBoundsY && inBoundsX; + + if (inBounds) { sum += input[inputChanBase + iy * inputW + ix]; count++; } - else if (countIncludePad) + + if (!inBounds && countIncludePad) { // Zero-pad contributes 0 to sum but 1 to count. count++; diff --git a/Sources/Main/Ops/CtcDecoder.cs b/Sources/Main/Ops/CtcDecoder.cs index be31410c..4ac5453f 100644 --- a/Sources/Main/Ops/CtcDecoder.cs +++ b/Sources/Main/Ops/CtcDecoder.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Ops/CtcLoss.cs b/Sources/Main/Ops/CtcLoss.cs index 8bad4b86..ffe834c3 100644 --- a/Sources/Main/Ops/CtcLoss.cs +++ b/Sources/Main/Ops/CtcLoss.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Ops/ICtcLanguageModel.cs b/Sources/Main/Ops/ICtcLanguageModel.cs index f9aabe89..64ea4847 100644 --- a/Sources/Main/Ops/ICtcLanguageModel.cs +++ b/Sources/Main/Ops/ICtcLanguageModel.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Ops/NGramCtcLanguageModel.cs b/Sources/Main/Ops/NGramCtcLanguageModel.cs index 1ff69f17..bbd77aef 100644 --- a/Sources/Main/Ops/NGramCtcLanguageModel.cs +++ b/Sources/Main/Ops/NGramCtcLanguageModel.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com diff --git a/Sources/Main/Ops/TensorMath.Activations.cs b/Sources/Main/Ops/TensorMath.Activations.cs index db560da8..98a59b7f 100644 --- a/Sources/Main/Ops/TensorMath.Activations.cs +++ b/Sources/Main/Ops/TensorMath.Activations.cs @@ -175,7 +175,8 @@ public static AutogradNode Dropout(ComputationGraph? graph, AutogradNode input, { graph?.Record(OpCode.Dropout, output, input, mask); // Mask jako node context B } - else + + if (!(output.RequiresGrad)) { mask.Dispose(); } @@ -232,7 +233,8 @@ public static AutogradNode Dropout2D(ComputationGraph? graph, AutogradNode input { graph?.Record(OpCode.Dropout, output, input, mask); // reuse elementwise dropout backward } - else + + if (!(output.RequiresGrad)) { mask.Dispose(); } diff --git a/Sources/Main/Ops/TensorMath.Algebra.cs b/Sources/Main/Ops/TensorMath.Algebra.cs index 7fdaf400..b1848a7a 100644 --- a/Sources/Main/Ops/TensorMath.Algebra.cs +++ b/Sources/Main/Ops/TensorMath.Algebra.cs @@ -106,7 +106,8 @@ public static AutogradNode AddBias(ComputationGraph? graph, AutogradNode input, Simd.Add(inS.Slice(i * C, C), bS, outS.Slice(i * C, C)); } } - else + + if (!(N < BatchSequentialThreshold)) { var inSpan = input.DataView.AsReadOnlySpan(); var bSpan = bias.DataView.AsReadOnlySpan(); @@ -209,7 +210,8 @@ public static AutogradNode MatMulRaw(ComputationGraph? graph, AutogradNode A, Au { MatMulRawSeq(A.DataView.AsReadOnlySpan(), B.DataView.AsReadOnlySpan(), aR, aC, bC, C.DataView.AsSpan()); } - else + + if (!((long)aR * aC * bC < ParallelThreshold)) { var aSpan = A.DataView.AsReadOnlySpan(); var bSpan = B.DataView.AsReadOnlySpan(); @@ -306,7 +308,8 @@ public static void MatMulAdd_A_BT_Raw(AutogradNode A, bool aGrad, AutogradNode B bGrad ? B.GradView.AsReadOnlySpan() : B.DataView.AsReadOnlySpan(), C.GradView.AsSpan(), N, K, M); } - else + + if (!((long)N * K * M < ParallelThreshold)) { var aSpan = aGrad ? A.GradView.AsReadOnlySpan() : A.DataView.AsReadOnlySpan(); var bSpan = bGrad ? B.GradView.AsReadOnlySpan() : B.DataView.AsReadOnlySpan(); @@ -377,7 +380,8 @@ public static void MatMulAdd_AT_B_Raw(AutogradNode A, bool aGrad, AutogradNode B bGrad ? B.GradView.AsReadOnlySpan() : B.DataView.AsReadOnlySpan(), C.GradView.AsSpan(), K, N, M); } - else + + if (!((long)K * N * M < ParallelThreshold)) { var aSpan = aGrad ? A.GradView.AsReadOnlySpan() : A.DataView.AsReadOnlySpan(); var bSpan = bGrad ? B.GradView.AsReadOnlySpan() : B.DataView.AsReadOnlySpan(); @@ -523,7 +527,8 @@ public static void LinearBackward(AutogradNode input, AutogradNode weights, Auto batchSize, inputSize, outputSize); } } - else + + if (!(ops < LinearBackwardSequentialThreshold)) { // Large matrix: existing parallel MatMul. MatMulBackward(input, weights, output); diff --git a/Sources/Main/Ops/TensorMath.Attention.cs b/Sources/Main/Ops/TensorMath.Attention.cs index df5cab62..08ab4539 100644 --- a/Sources/Main/Ops/TensorMath.Attention.cs +++ b/Sources/Main/Ops/TensorMath.Attention.cs @@ -184,7 +184,8 @@ private static AutogradNode ScaledDotProductAttentionCore( } } } - else + + if (!(batchSize > 1 && work >= AttentionParallelWorkThreshold)) { for (var b = 0; b < batchSize; b++) { @@ -205,7 +206,8 @@ private static AutogradNode ScaledDotProductAttentionCore( i1: dk, i2: causalMask ? 1 : 0); } - else + + if (!(requiresGrad)) { attnWeights.Dispose(); } @@ -408,7 +410,8 @@ private static void ScaledDotProductAttentionBackwardBatch( { dARow[j] = 0f; } - else + + if (!(causalMask && j > i)) { dARow[j] = TensorPrimitives.Dot( dORow, diff --git a/Sources/Main/Ops/TensorMath.Convolution.cs b/Sources/Main/Ops/TensorMath.Convolution.cs index ef7a9c54..38381ee3 100644 --- a/Sources/Main/Ops/TensorMath.Convolution.cs +++ b/Sources/Main/Ops/TensorMath.Convolution.cs @@ -52,13 +52,8 @@ public static AutogradNode Conv2D( // duration of the call. The local path allocates but isn't on the training // hot path; it's the cost of decoupling inference from graph state. Conv2DWorkspace? localWorkspace = null; - Conv2DWorkspace workspace; - if (graph is not null) - { - workspace = graph.GetConv2DWorkspace(batchSize, inC, outC, h, w, k, padding, stride); - } - else + if (graph is null) { localWorkspace = new Conv2DWorkspace(); var localWorkers = Math.Max(1, Math.Min(OverfitParallel.MaxDegreeOfParallelism, Math.Max(1, batchSize))); @@ -66,9 +61,14 @@ public static AutogradNode Conv2D( localWorkers, colLength: kSqInC * spatialOut, partialWeightGradientLength: outC * kSqInC); - workspace = localWorkspace; } + // Single definitely-assigned expression: the graph owns the workspace when there is one, + // otherwise the local we just built. Split ifs could not prove assignment to the compiler. + var workspace = graph is not null + ? graph.GetConv2DWorkspace(batchSize, inC, outC, h, w, k, padding, stride) + : localWorkspace!; + try { var workerCount = workspace.WorkerCount; @@ -457,7 +457,8 @@ public static void Im2Col( : 0f; } } - else + + if (!(inputY >= 0 && inputY < h)) { output .Slice(rowOffset + y * oW, oW) diff --git a/Sources/Main/Ops/TensorMath.DepthwiseConv.cs b/Sources/Main/Ops/TensorMath.DepthwiseConv.cs index 8265ad66..7978d665 100644 --- a/Sources/Main/Ops/TensorMath.DepthwiseConv.cs +++ b/Sources/Main/Ops/TensorMath.DepthwiseConv.cs @@ -84,7 +84,8 @@ public static AutogradNode DepthwiseConv2D( Simd.MulAdd(inRow.Slice(lo - padding + kx, hi - lo), kVal, outRow.Slice(lo, hi - lo)); } } - else + + if (!(stride == 1)) { for (var ox = 0; ox < outW; ox++) { @@ -199,7 +200,8 @@ public static void DepthwiseConv2DBackward( Simd.MulAdd(ogRow.Slice(lo, len), kVal, inGradPlane.Slice(inOff, len)); } } - else + + if (!(stride == 1)) { for (var ox = 0; ox < outW; ox++) { diff --git a/Sources/Main/Ops/TensorMath.Gelu.cs b/Sources/Main/Ops/TensorMath.Gelu.cs index 79e84919..66263e1a 100644 --- a/Sources/Main/Ops/TensorMath.Gelu.cs +++ b/Sources/Main/Ops/TensorMath.Gelu.cs @@ -60,7 +60,8 @@ public static AutogradNode Gelu(ComputationGraph? graph, AutogradNode input) { GeluForwardSimd(inS, outS); } - else + + if (!(inS.Length < ParallelThreshold)) { unsafe { @@ -103,7 +104,8 @@ public static void GeluBackward(AutogradNode input, AutogradNode output) { GeluBackwardSimd(inS, dOut, dIn); } - else + + if (!(inS.Length < ParallelThreshold)) { unsafe { diff --git a/Sources/Main/Ops/TensorMath.LayerNorm.cs b/Sources/Main/Ops/TensorMath.LayerNorm.cs index 3824dbd8..2aa75ebf 100644 --- a/Sources/Main/Ops/TensorMath.LayerNorm.cs +++ b/Sources/Main/Ops/TensorMath.LayerNorm.cs @@ -65,7 +65,8 @@ public static AutogradNode LayerNorm( { LayerNormForwardSeq(inS, outS, gammaS, betaS, meanS, invStdS, numRows, C, eps); } - else + + if (!((long)numRows * C < ParallelThreshold)) { unsafe { @@ -96,7 +97,8 @@ public static AutogradNode LayerNorm( c0: gamma, c1: beta, c2: mean, c3: invStd, contextCount: 4); } - else + + if (!(requiresGrad)) { mean.Dispose(); invStd.Dispose(); diff --git a/Sources/Main/Ops/TensorMath.Normalization.cs b/Sources/Main/Ops/TensorMath.Normalization.cs index 3d63cded..4a6e5c33 100644 --- a/Sources/Main/Ops/TensorMath.Normalization.cs +++ b/Sources/Main/Ops/TensorMath.Normalization.cs @@ -61,7 +61,8 @@ public static AutogradNode BatchNorm1D(ComputationGraph? graph, AutogradNode inp TensorPrimitives.Add(vB.Span, eps, invStdS); TensorPrimitives.ReciprocalSqrt(invStdS, invStdS); } - else + + if (!(isTraining)) { runningMean.AsReadOnlySpan().CopyTo(meanS); TensorPrimitives.Add(runningVar.AsReadOnlySpan(), eps, invStdS); @@ -85,7 +86,10 @@ public static AutogradNode BatchNorm1D(ComputationGraph? graph, AutogradNode inp { graph?.Record(OpCode.BatchNorm1D, output, input, c0: gamma, c1: beta, c2: mean, c3: invStd, contextCount: 4); } - else if (!isTraining) + + // The original else-if guard was `!(RequiresGrad && isTraining) && !isTraining`, which reduces to + // plain `!isTraining` — !isTraining already implies the first condition is false. + if (!isTraining) { mean.Dispose(); invStd.Dispose(); @@ -230,7 +234,8 @@ public static AutogradNode BatchNorm2D( invStdS[c] = 1f / MathF.Sqrt(varS[c] + eps); } } - else + + if (!(isTraining)) { runningMean.AsReadOnlySpan().CopyTo(meanS); var rv = runningVar.AsReadOnlySpan(); @@ -261,7 +266,10 @@ public static AutogradNode BatchNorm2D( { graph?.Record(OpCode.BatchNorm2D, output, input, c0: gamma, c1: beta, c2: mean, c3: invStd, contextCount: 4); } - else if (!isTraining) + + // The original else-if guard was `!(RequiresGrad && isTraining) && !isTraining`, which reduces to + // plain `!isTraining` — !isTraining already implies the first condition is false. + if (!isTraining) { mean.Dispose(); invStd.Dispose(); diff --git a/Sources/Main/Ops/TensorMath.Pooling.cs b/Sources/Main/Ops/TensorMath.Pooling.cs index 7596886b..b0ca93e2 100644 --- a/Sources/Main/Ops/TensorMath.Pooling.cs +++ b/Sources/Main/Ops/TensorMath.Pooling.cs @@ -71,7 +71,8 @@ public static AutogradNode MaxPool2D( batchOffset: n * inputSize); } } - else + + if (!(batchSize < BatchSequentialThreshold)) { var inputSpan = input.DataView.AsReadOnlySpan(); var outputSpan = output.DataView.AsSpan(); @@ -189,7 +190,8 @@ public static AutogradNode GlobalAveragePool2D( } } } - else + + if (!(batchSize < BatchSequentialThreshold)) { unsafe { diff --git a/Sources/Main/Ops/TensorMath.RmsNorm.cs b/Sources/Main/Ops/TensorMath.RmsNorm.cs index 5a94536a..0ef77e6d 100644 --- a/Sources/Main/Ops/TensorMath.RmsNorm.cs +++ b/Sources/Main/Ops/TensorMath.RmsNorm.cs @@ -55,7 +55,8 @@ public static AutogradNode RmsNorm( { RmsNormForwardSeq(inS, outS, gammaS, invRmsS, numRows, C, eps); } - else + + if (!((long)numRows * C < ParallelThreshold)) { unsafe { @@ -79,7 +80,8 @@ public static AutogradNode RmsNorm( { graph?.Record(OpCode.RmsNorm, output, input, c0: gamma, c1: invRms, contextCount: 2); } - else + + if (!(requiresGrad)) { invRms.Dispose(); } diff --git a/Sources/Main/Ops/TensorMath.Rope.cs b/Sources/Main/Ops/TensorMath.Rope.cs index 5e95d624..26884ee9 100644 --- a/Sources/Main/Ops/TensorMath.Rope.cs +++ b/Sources/Main/Ops/TensorMath.Rope.cs @@ -73,7 +73,8 @@ public static AutogradNode Rope( RopeForwardRow(inS, outS, cosS, sinS, r, D, headDim, halfDim, headsPerRow, splitHalf); } } - else + + if (!((long)rows * D < ParallelThreshold)) { unsafe { @@ -132,7 +133,8 @@ public static void RopeBackward( RopeBackwardRow(dOutS, dInS, cosS, sinS, r, D, headDim, halfDim, headsPerRow, splitHalf); } } - else + + if (!((long)rows * D < ParallelThreshold)) { unsafe { diff --git a/Sources/Main/Ops/TensorMath.Sequence.cs b/Sources/Main/Ops/TensorMath.Sequence.cs index 1d510ac7..41950a9c 100644 --- a/Sources/Main/Ops/TensorMath.Sequence.cs +++ b/Sources/Main/Ops/TensorMath.Sequence.cs @@ -43,7 +43,8 @@ public static (AutogradNode hNew, AutogradNode cNew) FusedLSTMStep(ComputationGr ExecuteLSTMInner(b, hS, gDS, uhS, bS, cPrevS, cnDS, hnDS); } } - else + + if (!(batchSize < BatchSequentialThreshold)) { OverfitParallel.For(0, batchSize, b => ExecuteLSTMInner(b, hS, gD.DataView.AsSpan(), @@ -60,7 +61,8 @@ public static (AutogradNode hNew, AutogradNode cNew) FusedLSTMStep(ComputationGr graph.Record(OpCode.FusedLSTMStep, hNode, x, hPrev, nodeContext: [cPrev, W, U, B, cNode, gD]); #pragma warning restore OVERFIT001 } - else + + if (!(graph != null && graph.IsRecording && req)) { if (!req) { @@ -163,7 +165,8 @@ public static void FusedLSTMStepBackward(AutogradNode x, AutogradNode hPrev, Aut processBatch(b, scratch); } } - else + + if (!(OverfitParallel.SuppressParallelismOnCurrentThread)) { #pragma warning disable OVERFIT008 // stateful localInit/localFinally overload (thread-local TensorStorage scratch) — no OverfitParallel equivalent; the suppress case is handled by the inline branch above Parallel.For(0, batchSize, diff --git a/Sources/Main/Ops/TensorMath.Shape.cs b/Sources/Main/Ops/TensorMath.Shape.cs index 61866540..f19aae91 100644 --- a/Sources/Main/Ops/TensorMath.Shape.cs +++ b/Sources/Main/Ops/TensorMath.Shape.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -147,18 +147,12 @@ public static void TransposeLastTwoBackward(AutogradNode input, AutogradNode out private static void GetDims(TensorShape shape, out int b, out int x, out int y) { - if (shape.Rank == 2) - { - b = 1; - x = shape.D0; - y = shape.D1; - } - else - { - b = shape.D0; - x = shape.D1; - y = shape.D2; - } + // Ternaries, not split ifs: all three are `out` parameters, so the compiler must see them + // assigned on every path — two ifs it cannot prove exhaustive would fail CS0177. + var rank2 = shape.Rank == 2; + b = rank2 ? 1 : shape.D0; + x = rank2 ? shape.D0 : shape.D1; + y = rank2 ? shape.D1 : shape.D2; } } } \ No newline at end of file diff --git a/Sources/Main/Ops/TensorMath.SiLU.cs b/Sources/Main/Ops/TensorMath.SiLU.cs index 27cf18d2..a219871f 100644 --- a/Sources/Main/Ops/TensorMath.SiLU.cs +++ b/Sources/Main/Ops/TensorMath.SiLU.cs @@ -41,7 +41,8 @@ public static AutogradNode SiLU(ComputationGraph? graph, AutogradNode input) { SiLUForwardSimd(inS, outS); } - else + + if (!(inS.Length < ParallelThreshold)) { unsafe { @@ -81,7 +82,8 @@ public static void SiLUBackward(AutogradNode input, AutogradNode output) { SiLUBackwardSimd(inS, dOut, dIn); } - else + + if (!(inS.Length < ParallelThreshold)) { unsafe { diff --git a/Sources/Main/Ops/TensorMath.cs b/Sources/Main/Ops/TensorMath.cs index 7ab01c18..f0a1a84b 100644 --- a/Sources/Main/Ops/TensorMath.cs +++ b/Sources/Main/Ops/TensorMath.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com From b0f2f287acbc7ed97e4ee023c246d3f2f0fefeac Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Tue, 21 Jul 2026 14:26:07 +0200 Subject: [PATCH 04/37] else --- .editorconfig | 47 ++++++--- Demo/AnomalyConsoleDemo/Program.cs | 14 ++- .../Infrastructure/ApiKeyAuthMiddleware.cs | 5 +- Demo/LocalAgentAspNetDemo/Rag/RagService.cs | 5 +- Demo/OverfitChatApp/MainActivity.cs | 21 ++-- Demo/OverfitChatApp/VoiceRecorder.cs | 4 +- Demo/QLoRAFineTuneDemo/Program.cs | 5 +- Demo/VoiceClone/Program.cs | 20 ++-- Demo/VoiceLoop/Program.cs | 17 +++- Directory.Build.props | 37 +++++++ .../Analyzers/ArrayParameterToSpanAnalyzer.cs | 5 +- Sources/Analyzers/OverfitPerfAnalysis.cs | 5 +- .../Benchmark/ConcurrentInferenceBenchmark.cs | 5 +- Sources/Benchmark/ElseRefactorBenchmark.cs | 5 + Sources/Benchmark/OverfitParallelLegacy.cs | 5 +- .../SlmCachedGptStackGpt1Benchmark.cs | 5 +- .../SlmCachedGptStackSmallBenchmark.cs | 5 +- Sources/Cli/Commands.cs | 20 ++-- Sources/Cli/HfDownloader.cs | 25 ++--- Sources/Extensions.AI/OverfitChatClient.cs | 13 +-- .../Main/LanguageModels/Chat/ChatSession.cs | 14 ++- .../Main/LanguageModels/Chat/ChatTemplate.cs | 14 +-- .../Constraints/JsonSchemaConstraint.cs | 5 +- .../Constraints/JsonStateMachine.cs | 5 +- .../Constraints/Regex/RegexDfa.cs | 19 ++-- .../Constraints/RegexConstraint.cs | 5 +- .../Constraints/Schema/JsonSchemaCompiler.cs | 3 +- .../Constraints/Schema/JsonSchemaTracker.cs | 14 ++- .../Embeddings/BertConfigReader.cs | 30 ++++-- .../LanguageModels/LoRA/Gpt1LoRAFineTuner.cs | 45 ++++++--- .../LanguageModels/Loading/GgufLlamaLoader.cs | 99 ++++++++++++------- .../Main/LanguageModels/Loading/GgufReader.cs | 3 +- .../Loading/HuggingFaceChatTemplate.cs | 7 +- .../Loading/LlamaConfigReader.cs | 64 ++++++++---- .../Loading/SafetensorsLlamaLoader.cs | 11 ++- .../Memory/ChatHistoryCompactor.cs | 5 +- .../Runtime/BatchedProjectionKernel.cs | 5 +- .../Runtime/BatchedQuantProjection.cs | 23 +++-- .../Runtime/CachedFeedForwardBlock.cs | 30 ++++-- .../LanguageModels/Runtime/CachedGptStack.cs | 31 +++--- .../Runtime/CachedLlamaInferenceEngine.cs | 5 +- .../Runtime/CachedLlamaSession.cs | 21 ++-- .../Runtime/CachedMultiHeadAttention.cs | 45 +++++---- .../Runtime/CachedSingleHeadAttention.cs | 28 +++--- .../Runtime/CachedTransformerBlock.cs | 24 +++-- .../Runtime/DraftModelSpeculativeDrafter.cs | 5 +- .../LanguageModels/Runtime/KeyValueCache.cs | 14 ++- .../Main/LanguageModels/Runtime/MoeRouter.cs | 5 +- .../LanguageModels/Runtime/Q4KDotKernel.cs | 5 +- .../Main/LanguageModels/Runtime/Q8KvQuant.cs | 11 +-- .../Runtime/SingleTokenProjectionKernel.cs | 11 ++- .../Main/LanguageModels/Runtime/SlmSession.cs | 11 ++- .../LanguageModels/Runtime/StackWeights.cs | 5 +- .../Skills/Evaluation/CheckRegistry.cs | 11 ++- .../Skills/Evaluation/SkillEvalReport.cs | 4 +- .../Skills/Optimization/SkillOptimizer.cs | 5 +- .../Tokenizers/GgufTokenizer.cs | 21 ++-- .../Tokenizers/HuggingFaceBpeTokenizer.cs | 13 ++- .../Tokenizers/QwenTokenizer.cs | 11 ++- .../Tokenizers/WordPieceTokenizer.cs | 8 +- .../Tools/ToolCallConstraint.cs | 8 +- .../Whisper/WhisperGgmlLoader.cs | 7 +- .../LanguageModels/Whisper/WhisperKernels.cs | 5 +- .../Whisper/WhisperTranscriber.cs | 16 +-- Sources/Main/Main.csproj | 12 --- Sources/Main/Onnx/OnnxGraphImporter.cs | 10 +- Sources/Main/Onnx/OnnxProtoParser.cs | 12 ++- .../Main/Onnx/Operators/ReduceMeanOperator.cs | 21 ++-- Sources/Server/OpenAi/OpenAiChatMapping.cs | 20 +--- Sources/Server/OverfitOpenAiServer.cs | 35 +++---- format-code.ps1 | 50 +++++++--- 71 files changed, 729 insertions(+), 430 deletions(-) diff --git a/.editorconfig b/.editorconfig index 00d03af5..235ed419 100644 --- a/.editorconfig +++ b/.editorconfig @@ -71,8 +71,8 @@ dotnet_diagnostic.OVERFIT007.severity = suggestion # PROMOTED TO ERROR (criteria: zero existing sites + unconditional rule + trivial fix/pragma + high miss-cost): # OVERFIT008 — a suppress leak silently breaks data-parallel training (measured 1.8x); like RS0030. # OVERFIT015 — CpuFeatures gating is a hard convention; the fix is a one-token change. -dotnet_diagnostic.OVERFIT008.severity = error -dotnet_diagnostic.OVERFIT015.severity = error +# (moved to the [Sources/Main/**.cs] section at the end of this file) +# (moved to the [Sources/Main/**.cs] section at the end of this file) dotnet_diagnostic.OVERFIT009.severity = suggestion # Tier A, added 2026-06-13 — start on the bottom rung (suggestion everywhere). Escalate a directory # once it is swept, exactly like 001-009. 010 (growable collections) and 014 (ToLower compare) are @@ -84,10 +84,13 @@ dotnet_diagnostic.OVERFIT010.severity = suggestion # or a switch expression. Introduced 2026-07-18 against 321 existing occurrences in 124 files, so a # blanket `error` was not an option — it would have forced one mechanical rewrite across the whole # library including hot inference paths, which is precisely the unmeasured behaviour-preserving -# refactor this repo forbids. So: SUGGESTION everywhere (IDE-visible, non-blocking), ERROR in directories -# that are already at zero. Sweep a directory -> add it below. Never remove a directory from the list. +# refactor this repo forbids. The sweep is now COMPLETE (322 -> 0 in Sources/Main, and every other project +# except Tests), so the rule is ERROR repo-wide. Tests are exempted at the END of this file: test code is +# throwaway and local, and the argument for the ban (shipped library, read by strangers, hot paths) does not +# apply there. Sources/Benchmark/ElseRefactorBenchmark.cs keeps its `else` behind a #pragma — those forms are +# the measurement subject that justifies the rule. # Backlog at introduction: LanguageModels 163, Audio 35, Ops 34, Data 13, Onnx 14, DeepLearning 12. -dotnet_diagnostic.OVERFIT021.severity = suggestion +dotnet_diagnostic.OVERFIT021.severity = error # OVERFIT022 (direct recursion) — NASA Power of 10 rule 1. ERROR EVERYWHERE, deliberately: a .NET # StackOverflowException cannot be caught, so unbounded recursion is an uncatchable kill of the HOST @@ -95,13 +98,13 @@ dotnet_diagnostic.OVERFIT021.severity = suggestion # in Sources/Main therefore carries an explicit `#pragma warning disable OVERFIT022` whose comment states # the BOUND that makes it safe (a checked depth cap, or a structural log2(n) argument). If you cannot write # that sentence, the recursion is not safe and needs an explicit stack/worklist instead. -dotnet_diagnostic.OVERFIT022.severity = error +# (moved to the [Sources/Main/**.cs] section at the end of this file) # OVERFIT023 (loop with no exit condition in its header) — NASA Power of 10 rule 2. ERROR EVERYWHERE, same # contract as OVERFIT022: an unbounded loop hangs the HOST process with no exception, no stack trace and no # log line, which is the hardest failure of all to diagnose. Every `while (true)` in Sources/Main carries a # `#pragma warning disable OVERFIT023` whose comment starts with "BOUND:" and names what terminates it. -dotnet_diagnostic.OVERFIT023.severity = error +# (moved to the [Sources/Main/**.cs] section at the end of this file) dotnet_diagnostic.OVERFIT011.severity = suggestion @@ -125,7 +128,7 @@ dotnet_diagnostic.OVERFIT020.severity = suggestion # fires inside a member or type marked [DevOnBike.Overfit.Diagnostics.OverfitHotPath] is reported as # this hard error INSTEAD of its directory-configured severity — the per-member ratchet. Always error; # never loosen it (that would defeat the marker). Suppress a justified site with #pragma + a reason. -dotnet_diagnostic.OVERFIT900.severity = error +# (moved to the [Sources/Main/**.cs] section at the end of this file) [Sources/Main/Kernels/**.cs] # Allocation rules are ERRORS in pure kernels — per-call allocation is never right here; @@ -228,8 +231,10 @@ dotnet_diagnostic.OVERFIT001.severity = suggestion [**/obj/**] dotnet_diagnostic.RS0030.severity = none -# OVERFIT021 (else) — directories already at zero occurrences, locked in as errors. Sweep a directory -> -# add it here. Never remove one. +# OVERFIT021 (else) — Sources/Main is FULLY swept (322 -> 0), so the whole tree is the error scope. +# This replaced a hand-maintained directory list: ten already-clean directories (Diagnostics, Extensions, +# Licensing, Maths, Properties, Redaction, Runtime, Tokenization, Training, Trees) had been swept but never +# added to that list, so the ratchet silently did not hold for them. A whole-tree glob cannot drift. # # THREE gotchas, each of which silently produced a rule that looked configured but did nothing (all three # were hit while wiring this up on 2026-07-18): @@ -240,11 +245,29 @@ dotnet_diagnostic.RS0030.severity = none # 3. The rule ID must NOT appear in in Directory.Build.props. That maps to # -warnaserror-:ID, which reverts the diagnostic to warning EVEN WHERE .editorconfig sets error — # defeating this whole section. (This is why OVERFIT008/015 are absent from that list.) -[Sources/Main/{Anomalies,Audio,Autograd,Core,Data,DeepLearning,Evolutionary,Exceptions,Inference,Intrinsics,Kernels,Ops,Optimizers,Parameters,Randomization,Serving,Statistical,Tensors}/**.cs] -dotnet_diagnostic.OVERFIT021.severity = error # OVERFIT022 (recursion) — ERROR on the paths that parse untrusted, externally-authored input, where an # uncatchable stack overflow would kill the embedding host. Existing bounded sites carry a #pragma naming # their depth cap; this stops NEW unbounded recursion from entering these directories. [Sources/Main/LanguageModels/{Loading,Constraints}/**.cs] dotnet_diagnostic.OVERFIT022.severity = error + +# OVERFIT021 (else) — Tests are the one exemption: test code is local and disposable, so the ban's rationale +# (shipped library, read by strangers, hot paths) does not apply. Kept at suggestion so it still shows in the +# IDE. MUST stay at the END of this file — a [section] header scopes every setting below it. +[Tests/**.cs] +dotnet_diagnostic.OVERFIT021.severity = suggestion + +# ── Library-only OVERFIT rules ───────────────────────────────────────────────────────────── +# These were designed for the SHIPPED library: zero-alloc hot paths (OVERFIT008/900), the CpuFeatures +# convention (015), and the NASA bounded-execution pair (022/023, whose rationale is "Overfit runs inside +# someone else's process"). Once the analyzer was wired into every project they also started firing in +# benchmarks and demos, where they are wrong by construction — Sources/Benchmark deliberately calls raw +# Parallel.For to A/B it against OverfitParallel, and OverfitParallelLegacy is a kept-for-comparison copy. +# So they are scoped to the library; OVERFIT021 (else) stays repo-wide because readability is not scoped. +[Sources/Main/**.cs] +dotnet_diagnostic.OVERFIT008.severity = error +dotnet_diagnostic.OVERFIT015.severity = error +dotnet_diagnostic.OVERFIT022.severity = error +dotnet_diagnostic.OVERFIT023.severity = error +dotnet_diagnostic.OVERFIT900.severity = error diff --git a/Demo/AnomalyConsoleDemo/Program.cs b/Demo/AnomalyConsoleDemo/Program.cs index 7d5f540d..04179cec 100644 --- a/Demo/AnomalyConsoleDemo/Program.cs +++ b/Demo/AnomalyConsoleDemo/Program.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -54,7 +54,8 @@ private static async Task Main(string[] args) { RunMultiPodAdaptiveScenario(model); } - else + + if (!(HasFlag(args, "--multipod"))) { RunScenario(model, config); } @@ -73,7 +74,11 @@ private static async Task Main(string[] args) private static async Task<(GPT1Model model, GptTrainingConfig config)> GetModelAsync( string? csv, string checkpoint, GptTrainingConfig config) { - if (!File.Exists(checkpoint)) + // Capture BEFORE training: RunAsync WRITES the checkpoint, so a second File.Exists test would + // see the file it just produced and announce "Loading checkpoint" right after training one. + var hadCheckpoint = File.Exists(checkpoint); + + if (!hadCheckpoint) { if (csv is null || !File.Exists(csv)) { @@ -95,7 +100,8 @@ private static async Task Main(string[] args) Console.WriteLine($"Trained: {result.SnapshotsLoaded:N0} snapshots, " + $"val loss {result.InitialLoss:F2} → {result.FinalValLoss:F2}, {result.TrainingTime:mm\\:ss}."); } - else + + if (hadCheckpoint) { Console.WriteLine($"Loading {config.DModel}d/{config.NLayers}L checkpoint {checkpoint} ..."); } diff --git a/Demo/LocalAgentAspNetDemo/Infrastructure/ApiKeyAuthMiddleware.cs b/Demo/LocalAgentAspNetDemo/Infrastructure/ApiKeyAuthMiddleware.cs index 560982dd..c83ef55b 100644 --- a/Demo/LocalAgentAspNetDemo/Infrastructure/ApiKeyAuthMiddleware.cs +++ b/Demo/LocalAgentAspNetDemo/Infrastructure/ApiKeyAuthMiddleware.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -36,7 +36,8 @@ public ApiKeyAuthMiddleware(RequestDelegate next, IConfiguration configuration, logger.LogWarning( "API-key auth is OFF — any caller can reach the agent. Set 'ApiKey' (config or env) before exposing it."); } - else + + if (!(string.IsNullOrWhiteSpace(key))) { _keyHash = SHA256.HashData(Encoding.UTF8.GetBytes(key.Trim())); logger.LogInformation("API-key auth is ON — callers must present 'X-API-Key' or 'Authorization: Bearer'."); diff --git a/Demo/LocalAgentAspNetDemo/Rag/RagService.cs b/Demo/LocalAgentAspNetDemo/Rag/RagService.cs index 85e0b479..ec43f2c0 100644 --- a/Demo/LocalAgentAspNetDemo/Rag/RagService.cs +++ b/Demo/LocalAgentAspNetDemo/Rag/RagService.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -260,7 +260,8 @@ private void TrySaveCache(PersistentVectorStore store, string cachePath, float[] File.Delete(meanPath); } } - else + + if (!(mean is null)) { using var stream = new FileStream(meanPath, FileMode.Create, FileAccess.Write); using var writer = new BinaryWriter(stream); diff --git a/Demo/OverfitChatApp/MainActivity.cs b/Demo/OverfitChatApp/MainActivity.cs index a0715ef0..db6c1418 100644 --- a/Demo/OverfitChatApp/MainActivity.cs +++ b/Demo/OverfitChatApp/MainActivity.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -938,20 +938,10 @@ private void SetWelcomeError(string message) private void UpdateSelectFieldText() { - string label; var last = Prefs.GetString("last_model_path", null); - if (last != null && _modelPaths.Contains(last)) - { - label = DisplayName(last); - } - else if (_modelPaths.Count > 0) - { - label = DisplayName(_modelPaths[0]); - } - else - { - label = "No models"; - } + var label = last != null && _modelPaths.Contains(last) ? DisplayName(last) + : _modelPaths.Count > 0 ? DisplayName(_modelPaths[0]) + : "No models"; _modelSelectLabel.Text = label; } @@ -1328,7 +1318,8 @@ public override void OnRequestPermissionsResult( { StartRecording(); } - else + + if (!(grantResults.Length > 0 && grantResults[0] == Permission.Granted)) { Toast.MakeText(this, "Microphone permission is needed for voice input.", ToastLength.Short)!.Show(); diff --git a/Demo/OverfitChatApp/VoiceRecorder.cs b/Demo/OverfitChatApp/VoiceRecorder.cs index cc3d1822..b732af85 100644 --- a/Demo/OverfitChatApp/VoiceRecorder.cs +++ b/Demo/OverfitChatApp/VoiceRecorder.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -142,7 +142,7 @@ private void ReadLoop() } } } - else if (n < 0) + if (n < 0) { break; // read error } diff --git a/Demo/QLoRAFineTuneDemo/Program.cs b/Demo/QLoRAFineTuneDemo/Program.cs index 33067f00..037fa534 100644 --- a/Demo/QLoRAFineTuneDemo/Program.cs +++ b/Demo/QLoRAFineTuneDemo/Program.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -50,7 +50,8 @@ private static int Main(string[] args) tuner.LoadAdapter(adapterPath); Console.WriteLine(" ready.\n"); } - else + + if (!(chatOnly)) { var textPath = args[1]; if (!File.Exists(textPath)) diff --git a/Demo/VoiceClone/Program.cs b/Demo/VoiceClone/Program.cs index ced06270..d8f124bd 100644 --- a/Demo/VoiceClone/Program.cs +++ b/Demo/VoiceClone/Program.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -82,7 +82,7 @@ private static int Main(string[] args) return 1; } } - else if (!synthOnly) + if (!synthOnly && folder is null) { if (recording is null || !PathExists(recording)) { @@ -236,7 +236,9 @@ void Synthesize(VoiceCloneTrainer t, Snac s, string text, string vc, string outW var audioBase = ResolveAudioBase(t.Tokenizer); // Stop at end_of_speech (audioBase+2 = 128258) as well as the text-eos — the canonical prompt makes // the model end the audio with end_of_speech, else it babbles to --max-new after the sentence. - int[] generated; + // Exhaustive across the fast / non-fast pair below; the compiler cannot prove that across + // two separate ifs, and neither generator can run speculatively. + int[] generated = []; var genSw = System.Diagnostics.Stopwatch.StartNew(); if (a.Has("fast")) { @@ -246,7 +248,8 @@ void Synthesize(VoiceCloneTrainer t, Snac s, string text, string vc, string outW mergedEngine ??= t.BuildMergedEngine(mergeLora: !a.Has("no-lora-merge")); generated = GenerateViaMerged(mergedEngine, promptIds, audioBase, t.EndOfTextTokenId); } - else + + if (!(a.Has("fast"))) { generated = t.Generate(promptIds, maxNew, t.EndOfTextTokenId, temperature: temperature, topP: topP, repeatPenalty: repeatPenalty, seed: seed, @@ -393,11 +396,16 @@ private static Args ParseArgs(string[] argv) continue; } var key = argv[i][2..]; - if (i + 1 < argv.Length && !argv[i + 1].StartsWith("--", StringComparison.Ordinal)) + // Capture BEFORE consuming the value: `argv[++i]` advances i, so re-testing would look at + // the NEXT argument and could null out the value that was just parsed. + var hasValue = i + 1 < argv.Length && !argv[i + 1].StartsWith("--", StringComparison.Ordinal); + + if (hasValue) { map[key] = argv[++i]; } - else + + if (!hasValue) { map[key] = null; } diff --git a/Demo/VoiceLoop/Program.cs b/Demo/VoiceLoop/Program.cs index cd35f977..219a5336 100644 --- a/Demo/VoiceLoop/Program.cs +++ b/Demo/VoiceLoop/Program.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -56,14 +56,16 @@ do { - float[] micSamples; + // Exhaustive across the wav-input / microphone pair below (see VoiceClone for the same note). + float[] micSamples = []; if (wavInput is not null) { var raw = AudioFile.ReadMono(wavInput, out var rate); micSamples = rate == MicCapture.SampleRate ? raw : AudioResampler.Resample(raw, rate, MicCapture.SampleRate); Console.WriteLine($"[input] {Path.GetFileName(wavInput)} ({micSamples.Length / (double)MicCapture.SampleRate:F1}s)"); } - else + + if (!(wavInput is not null)) { Console.Write($"Press Enter to record {seconds}s (or type 'q' to quit): "); if (string.Equals(Console.ReadLine()?.Trim(), "q", StringComparison.OrdinalIgnoreCase)) @@ -126,11 +128,16 @@ static Options ParseArgs(string[] argv) continue; } var key = argv[i][2..]; - if (i + 1 < argv.Length && !argv[i + 1].StartsWith("--", StringComparison.Ordinal)) + // Capture BEFORE consuming the value: `argv[++i]` advances i, so re-testing would look at the + // NEXT argument and could null out the value that was just parsed. + var hasValue = i + 1 < argv.Length && !argv[i + 1].StartsWith("--", StringComparison.Ordinal); + + if (hasValue) { map[key] = argv[++i]; } - else + + if (!hasValue) { map[key] = null; // flag } diff --git a/Directory.Build.props b/Directory.Build.props index 1316f09c..ca5e0d01 100644 --- a/Directory.Build.props +++ b/Directory.Build.props @@ -32,6 +32,43 @@ inert for this reason; removing them from this list cost 0 new errors. --> $(WarningsNotAsErrors);RS0030;OVERFIT003;OVERFIT004;OVERFIT005;OVERFIT006;OVERFIT007;OVERFIT010;OVERFIT011;OVERFIT012;OVERFIT013;OVERFIT014;OVERFIT016;OVERFIT017;OVERFIT018;OVERFIT019;OVERFIT020;IDISP001;IDISP002;IDISP003;IDISP004;IDISP005;IDISP006;IDISP007;IDISP008;IDISP009;IDISP010;IDISP011;IDISP012;IDISP013;IDISP014;IDISP015;IDISP016;IDISP017 + + + + + + + + true true diff --git a/Sources/Analyzers/ArrayParameterToSpanAnalyzer.cs b/Sources/Analyzers/ArrayParameterToSpanAnalyzer.cs index 0e74f358..dfecabcb 100644 --- a/Sources/Analyzers/ArrayParameterToSpanAnalyzer.cs +++ b/Sources/Analyzers/ArrayParameterToSpanAnalyzer.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -131,7 +131,8 @@ private static Usage ClassifyParameter(IParameterSymbol parameter, ImmutableArra { result = Usage.Writes; } - else if (result == Usage.None) + + if (usage != Usage.Writes && result == Usage.None) { result = Usage.ReadOnly; } diff --git a/Sources/Analyzers/OverfitPerfAnalysis.cs b/Sources/Analyzers/OverfitPerfAnalysis.cs index 2eea1790..86402f7d 100644 --- a/Sources/Analyzers/OverfitPerfAnalysis.cs +++ b/Sources/Analyzers/OverfitPerfAnalysis.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -36,7 +36,8 @@ internal static void Report(OperationAnalysisContext context, DiagnosticDescript { context.ReportDiagnostic(Diagnostic.Create(HotPathRule, location, rule.Id)); } - else + + if (!(IsInHotPath(context.ContainingSymbol))) { context.ReportDiagnostic(Diagnostic.Create(rule, location, messageArgs)); } diff --git a/Sources/Benchmark/ConcurrentInferenceBenchmark.cs b/Sources/Benchmark/ConcurrentInferenceBenchmark.cs index 7448cfaf..c7f7ef01 100644 --- a/Sources/Benchmark/ConcurrentInferenceBenchmark.cs +++ b/Sources/Benchmark/ConcurrentInferenceBenchmark.cs @@ -482,11 +482,12 @@ private void WorkerLoop( { context.RunOverfitLoop(_innerIterations); } - else if (mode == ModeOnnx) + if (mode == ModeOnnx) { context.RunOnnxLoop(_innerIterations); } - else if (mode == ModeNone) + + if (mode == ModeNone) { context.Checksum = 0.0; } diff --git a/Sources/Benchmark/ElseRefactorBenchmark.cs b/Sources/Benchmark/ElseRefactorBenchmark.cs index 59519a62..bcc2f85b 100644 --- a/Sources/Benchmark/ElseRefactorBenchmark.cs +++ b/Sources/Benchmark/ElseRefactorBenchmark.cs @@ -7,6 +7,11 @@ using BenchmarkDotNet.Attributes; using BenchmarkDotNet.Configs; +// The `else` forms in this file are the SUBJECT of the measurement, not an oversight: OVERFIT021's whole +// justification is the measured cost table below, and it was produced by benchmarking these exact shapes +// against their rewrites. Removing them would delete the evidence for the rule. +// (Sources/Benchmark is excluded from the in-repo analyzers in Directory.Build.props for exactly this reason.) + namespace Benchmarks { /// diff --git a/Sources/Benchmark/OverfitParallelLegacy.cs b/Sources/Benchmark/OverfitParallelLegacy.cs index e464ea96..3b6962cb 100644 --- a/Sources/Benchmark/OverfitParallelLegacy.cs +++ b/Sources/Benchmark/OverfitParallelLegacy.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -148,7 +148,8 @@ private static void WorkerLoop() _completion.Signal(); } } - else + + if (!(index < _chunkCount)) { Debug.Fail($"OverfitParallelLegacy: claim index {index} >= chunkCount {_chunkCount}."); } diff --git a/Sources/Benchmark/SlmCachedGptStackGpt1Benchmark.cs b/Sources/Benchmark/SlmCachedGptStackGpt1Benchmark.cs index 28de2df1..c1074892 100644 --- a/Sources/Benchmark/SlmCachedGptStackGpt1Benchmark.cs +++ b/Sources/Benchmark/SlmCachedGptStackGpt1Benchmark.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -221,7 +221,8 @@ private static float[] CreateVector(int length, bool fillOnes = false, int seed { FillOnes(v); } - else + + if (!(fillOnes)) { FillDeterministic(v, seed + seedBase); } diff --git a/Sources/Benchmark/SlmCachedGptStackSmallBenchmark.cs b/Sources/Benchmark/SlmCachedGptStackSmallBenchmark.cs index eb079fef..1ccdf759 100644 --- a/Sources/Benchmark/SlmCachedGptStackSmallBenchmark.cs +++ b/Sources/Benchmark/SlmCachedGptStackSmallBenchmark.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -221,7 +221,8 @@ private static float[] CreateVector(int length, bool fillOnes = false, int seed { FillOnes(v); } - else + + if (!(fillOnes)) { FillDeterministic(v, seed + seedBase); } diff --git a/Sources/Cli/Commands.cs b/Sources/Cli/Commands.cs index 681cac2c..cdf44321 100644 --- a/Sources/Cli/Commands.cs +++ b/Sources/Cli/Commands.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -463,8 +463,10 @@ public static int Gateway(string? upstream, string keyEnv, string host, int port // via /proc//environ, inherited by child processes, or dumped from the env later. Environment.SetEnvironmentVariable(keyEnv, null); - Redactor redactor; - RedactionPolicy policy; + // The config / built-in pair below is exhaustive, but the compiler cannot prove that across two + // separate ifs; the null! initialisers only state the invariant. + Redactor redactor = null!; + RedactionPolicy policy = null!; string? configUpstream = null; IReadOnlyList configClientKeys = []; var configScanResponses = false; @@ -487,7 +489,8 @@ public static int Gateway(string? upstream, string keyEnv, string host, int port } Console.WriteLine($"config: {Path.GetFullPath(configPath)}"); } - else + + if (!(!string.IsNullOrEmpty(configPath))) { // Built-in: international + Polish (checksum-validated PESEL/NIP/REGON/IBAN) detectors, default policy. var intl = DefaultRedactionRules.All(); @@ -524,7 +527,8 @@ public static int Gateway(string? upstream, string keyEnv, string host, int port { Console.WriteLine($"client auth: ON ({clientKeys.Count} gateway key(s)) — callers must send 'Authorization: Bearer '"); } - else + + if (!(clientKeys.Count > 0)) { Console.WriteLine($"client auth: OFF — set ${clientKeysEnv} (or \"clientKeys\" in --config) before exposing the gateway."); } @@ -640,7 +644,8 @@ public static int Score(string modelPath, string inputPath, string? outputPath, model.PredictRawMargins(rows[r], outputs.AsSpan(r * groups, groups)); } } - else + + if (!(margin)) { model.PredictBatchParallel(flat, rows.Count, outputs); } @@ -652,7 +657,8 @@ void Emit(string text) { Console.Out.WriteLine(text); } - else + + if (!(writer is null)) { writer.WriteLine(text); } diff --git a/Sources/Cli/HfDownloader.cs b/Sources/Cli/HfDownloader.cs index a6b6986e..27d56f52 100644 --- a/Sources/Cli/HfDownloader.cs +++ b/Sources/Cli/HfDownloader.cs @@ -240,7 +240,8 @@ private static async Task StreamWithResumeAsync(string url, string file, string await HashExistingAsync(tmp, hasher); Console.WriteLine($" {file} already downloaded ({existing / (1024.0 * 1024):F1} MB) — verifying"); } - else + + if (!(existing > 0 && response.StatusCode == HttpStatusCode.RequestedRangeNotSatisfiable)) { response.EnsureSuccessStatusCode(); @@ -250,19 +251,9 @@ private static async Task StreamWithResumeAsync(string url, string file, string existing = 0; // server ignored the Range (200 OK) → start over from scratch. } - long total; - if (response.Content.Headers.ContentRange?.Length is long full) - { - total = full; - } - else if (resuming) - { - total = existing + (response.Content.Headers.ContentLength ?? 0); - } - else - { - total = response.Content.Headers.ContentLength ?? -1; - } + var total = response.Content.Headers.ContentRange?.Length is long full ? full + : resuming ? existing + (response.Content.Headers.ContentLength ?? 0) + : response.Content.Headers.ContentLength ?? -1; if (resuming) { @@ -313,7 +304,8 @@ private static async Task StreamWithResumeAsync(string url, string file, string Console.WriteLine($" sha256 verified {actual[..16]}..."); } - else + + if (!(expectedSha256 is not null)) { Console.WriteLine($" sha256 {actual[..16]}... (HF metadata unavailable — not verified)"); } @@ -346,7 +338,8 @@ private static void Report(string file, long read, long total, long sessionBytes var totalMb = total / (1024.0 * 1024); Console.Write($"\r {file} {pct,5:F1}% ({mb,8:F1} / {totalMb:F1} MB) {speed,6:F1} MB/s "); } - else + + if (!(total > 0)) { Console.Write($"\r {file} {mb,8:F1} MB {speed,6:F1} MB/s "); } diff --git a/Sources/Extensions.AI/OverfitChatClient.cs b/Sources/Extensions.AI/OverfitChatClient.cs index 5e097869..7b55b149 100644 --- a/Sources/Extensions.AI/OverfitChatClient.cs +++ b/Sources/Extensions.AI/OverfitChatClient.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -199,15 +199,16 @@ private string Replay(IReadOnlyList messages) if (role == ChatRole.System) { _session.AddSystem(text); + continue; } - else if (role == ChatRole.Assistant) + + if (role == ChatRole.Assistant) { _session.AddAssistant(text); + continue; } - else - { - _session.AddUser(text); - } // user / tool / unknown → user turn + + _session.AddUser(text); // user / tool / unknown → user turn } return messages[^1].Text ?? string.Empty; diff --git a/Sources/Main/LanguageModels/Chat/ChatSession.cs b/Sources/Main/LanguageModels/Chat/ChatSession.cs index 0f57a52b..c784bbc5 100644 --- a/Sources/Main/LanguageModels/Chat/ChatSession.cs +++ b/Sources/Main/LanguageModels/Chat/ChatSession.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -253,7 +253,12 @@ bool EmitToken(int token) // verify, sampling-correct, and ~free when drafts don't fire — but it can't mask the draft // against a per-token constraint, so it only runs unconstrained on a speculation-capable // session. Everything else falls back to the exact single-token loop. - if (constraint is null && _session is CachedLlamaSession spec && spec.CanSpeculate) + // Hoisted out of the condition: the speculative session is needed inside the branch, and a + // second (negated) test could not re-introduce a pattern variable in the same scope. + var spec = _session as CachedLlamaSession; + var useSpeculative = constraint is null && spec is not null && spec.CanSpeculate; + + if (useSpeculative) { const int maxDraft = 8; var history = new List(promptTokens.Length + Math.Min(maxNew, 4096)); @@ -266,7 +271,7 @@ bool EmitToken(int token) while (generated.Count < maxNew && (_slidingWindow || _session.CurrentPosition < _session.MaxContextLength)) { - var n = spec.GenerateSpeculative(CollectionsMarshal.AsSpan(history), committed, in sampling, maxDraft); + var n = spec!.GenerateSpeculative(CollectionsMarshal.AsSpan(history), committed, in sampling, maxDraft); var stop = false; for (var c = 0; c < n; c++) { @@ -284,7 +289,8 @@ bool EmitToken(int token) } } } - else + + if (!useSpeculative) { // With sliding-window enabled the cache never overflows (oldest tokens roll off), so we // bound generation by MaxNewTokens only; otherwise we stop when the context fills. diff --git a/Sources/Main/LanguageModels/Chat/ChatTemplate.cs b/Sources/Main/LanguageModels/Chat/ChatTemplate.cs index bc84307e..339b95aa 100644 --- a/Sources/Main/LanguageModels/Chat/ChatTemplate.cs +++ b/Sources/Main/LanguageModels/Chat/ChatTemplate.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -122,17 +122,19 @@ private static void RenderMistral(StringBuilder sb, IReadOnlyList m if (m.Role == "system") { pendingSystem = pendingSystem is null ? m.Content : pendingSystem + "\n\n" + m.Content; + continue; } - else if (m.Role == "user") + + if (m.Role == "user") { var content = pendingSystem is null ? m.Content : pendingSystem + "\n\n" + m.Content; pendingSystem = null; sb.Append("[INST] ").Append(content).Append(" [/INST]"); + continue; } - else // assistant - { - sb.Append(' ').Append(m.Content).Append(""); - } + + // assistant + sb.Append(' ').Append(m.Content).Append(""); } // For Mistral the open "[INST] … [/INST]" already prompts the assistant reply, // so addGenerationPrompt needs no extra marker. diff --git a/Sources/Main/LanguageModels/Constraints/JsonSchemaConstraint.cs b/Sources/Main/LanguageModels/Constraints/JsonSchemaConstraint.cs index ec9b22b7..ee53b7ba 100644 --- a/Sources/Main/LanguageModels/Constraints/JsonSchemaConstraint.cs +++ b/Sources/Main/LanguageModels/Constraints/JsonSchemaConstraint.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -95,7 +95,8 @@ public void ApplyMask(Span logits) { logits[t] = float.NegativeInfinity; } - else + + if (!(!_tracker.IsCharAllowedBySchema(text[0], in _committed) || !Accepts(text))) { anyAllowed = true; } diff --git a/Sources/Main/LanguageModels/Constraints/JsonStateMachine.cs b/Sources/Main/LanguageModels/Constraints/JsonStateMachine.cs index 9c5aecda..c2c7e468 100644 --- a/Sources/Main/LanguageModels/Constraints/JsonStateMachine.cs +++ b/Sources/Main/LanguageModels/Constraints/JsonStateMachine.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -431,7 +431,8 @@ private bool Push(bool isArray) { _stack |= 1UL << _depth; } - else + + if (!(isArray)) { _stack &= ~(1UL << _depth); } diff --git a/Sources/Main/LanguageModels/Constraints/Regex/RegexDfa.cs b/Sources/Main/LanguageModels/Constraints/Regex/RegexDfa.cs index 92c9bf06..393026ab 100644 --- a/Sources/Main/LanguageModels/Constraints/Regex/RegexDfa.cs +++ b/Sources/Main/LanguageModels/Constraints/Regex/RegexDfa.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -274,7 +274,8 @@ private Fragment ParseCounted(Fragment first) { m = -1; } // {n,} = n then star - else + + if (!(Peek() == '}')) { m = ParseInt(); } @@ -297,7 +298,8 @@ private Fragment ParseCounted(Fragment first) { parts.Add(Star(n == 0 ? first : Clone(first), optional: true)); // {n,} → n then star } - else if (hasMax) + + if (hasMax && m >= 0) { for (var i = n; i < m; i++) { @@ -420,7 +422,8 @@ private Fragment ParseClass() var to = Next(); SetRange(ref lo, ref hi, c, to); } - else + + if (!(Peek() == '-' && Peek(1) is not (']' or '\0'))) { SetBit(ref lo, ref hi, c); } @@ -570,8 +573,10 @@ private static void SetBit(ref ulong lo, ref ulong hi, char c) if (c < 64) { lo |= 1UL << c; + return; } - else if (c < 128) + + if (c < 128) { hi |= 1UL << (c - 64); } @@ -582,8 +587,10 @@ private static void ClearBit(ref ulong lo, ref ulong hi, char c) if (c < 64) { lo &= ~(1UL << c); + return; } - else if (c < 128) + + if (c < 128) { hi &= ~(1UL << (c - 64)); } diff --git a/Sources/Main/LanguageModels/Constraints/RegexConstraint.cs b/Sources/Main/LanguageModels/Constraints/RegexConstraint.cs index c2c756a5..af7910c2 100644 --- a/Sources/Main/LanguageModels/Constraints/RegexConstraint.cs +++ b/Sources/Main/LanguageModels/Constraints/RegexConstraint.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -80,7 +80,8 @@ public void ApplyMask(Span logits) { logits[t] = float.NegativeInfinity; } - else + + if (!(_dfa.Next(_state, text[0]) < 0 || !Accepts(text))) { anyAllowed = true; } diff --git a/Sources/Main/LanguageModels/Constraints/Schema/JsonSchemaCompiler.cs b/Sources/Main/LanguageModels/Constraints/Schema/JsonSchemaCompiler.cs index 5d3e790f..e5bc816b 100644 --- a/Sources/Main/LanguageModels/Constraints/Schema/JsonSchemaCompiler.cs +++ b/Sources/Main/LanguageModels/Constraints/Schema/JsonSchemaCompiler.cs @@ -128,7 +128,8 @@ private static int CompileNode(JsonElement el, List nodes, List< { values.Add(e.GetString() ?? string.Empty); } - else + + if (!(e.ValueKind == JsonValueKind.String)) { allStrings = false; } diff --git a/Sources/Main/LanguageModels/Constraints/Schema/JsonSchemaTracker.cs b/Sources/Main/LanguageModels/Constraints/Schema/JsonSchemaTracker.cs index eea77e12..d3b7a298 100644 --- a/Sources/Main/LanguageModels/Constraints/Schema/JsonSchemaTracker.cs +++ b/Sources/Main/LanguageModels/Constraints/Schema/JsonSchemaTracker.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -83,7 +83,8 @@ public void OnCharAdvanced(char c, in JsonStateMachine machine) { StartKeyString(); } - else + + if (!(machine.CurrentStringIsKey)) { StartValueString(); } @@ -435,7 +436,11 @@ private void FinishKeyString() var keyName = new string(((ReadOnlySpan)_keyBuffer).Slice(0, _keyLength)); ref readonly var objectNode = ref _schema.Nodes[_currentNodeIndex]; - if (objectNode.Properties != null && objectNode.Properties.TryGetValue(keyName, out var valueNodeIndex)) + var valueNodeIndex = 0; + var hasProperty = objectNode.Properties != null + && objectNode.Properties.TryGetValue(keyName, out valueNodeIndex); + + if (hasProperty) { if (objectNode.PropertyNames != null) { @@ -447,7 +452,8 @@ private void FinishKeyString() } _currentNodeIndex = valueNodeIndex; } - else + + if (!hasProperty) { _currentNodeIndex = _schema.UnconstrainedNodeIndex; // additional property — value unconstrained } diff --git a/Sources/Main/LanguageModels/Embeddings/BertConfigReader.cs b/Sources/Main/LanguageModels/Embeddings/BertConfigReader.cs index 47d6c767..37e461d7 100644 --- a/Sources/Main/LanguageModels/Embeddings/BertConfigReader.cs +++ b/Sources/Main/LanguageModels/Embeddings/BertConfigReader.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -63,38 +63,52 @@ public static BertConfig Parse(ReadOnlySpan json) { reader.Read(); hidden = reader.GetInt32(); + continue; } - else if (reader.ValueTextEquals("num_hidden_layers")) + + if (reader.ValueTextEquals("num_hidden_layers")) { reader.Read(); layers = reader.GetInt32(); + continue; } - else if (reader.ValueTextEquals("num_attention_heads")) + + if (reader.ValueTextEquals("num_attention_heads")) { reader.Read(); heads = reader.GetInt32(); + continue; } - else if (reader.ValueTextEquals("intermediate_size")) + + if (reader.ValueTextEquals("intermediate_size")) { reader.Read(); ffn = reader.GetInt32(); + continue; } - else if (reader.ValueTextEquals("max_position_embeddings")) + + if (reader.ValueTextEquals("max_position_embeddings")) { reader.Read(); maxPos = reader.GetInt32(); + continue; } - else if (reader.ValueTextEquals("vocab_size")) + + if (reader.ValueTextEquals("vocab_size")) { reader.Read(); vocab = reader.GetInt32(); + continue; } - else if (reader.ValueTextEquals("type_vocab_size")) + + if (reader.ValueTextEquals("type_vocab_size")) { reader.Read(); typeVocab = reader.GetInt32(); + continue; } - else if (reader.ValueTextEquals("layer_norm_eps")) + + if (reader.ValueTextEquals("layer_norm_eps")) { reader.Read(); eps = (float)reader.GetDouble(); diff --git a/Sources/Main/LanguageModels/LoRA/Gpt1LoRAFineTuner.cs b/Sources/Main/LanguageModels/LoRA/Gpt1LoRAFineTuner.cs index 74126f22..bebd84cb 100644 --- a/Sources/Main/LanguageModels/LoRA/Gpt1LoRAFineTuner.cs +++ b/Sources/Main/LanguageModels/LoRA/Gpt1LoRAFineTuner.cs @@ -520,7 +520,8 @@ private ModuleAdapter CreateAdapter( wBase.Dispose(); } } - else + + if (!(_quantizeBase)) { adapter.WBaseNode = wBase.AsNode(); adapter.Provider = graph => BuildEffectiveWeight(graph, adapter); @@ -573,7 +574,8 @@ private void AttachProviders() { _model.LMHeadOutputProvider = adapter.OutputProvider; } - else + + if (!(adapter.OutputProvider is not null)) { _model.LMHeadWeightProvider = adapter.Provider; } @@ -585,7 +587,8 @@ private void AttachProviders() { _model.Blocks[adapter.Layer].FFN.W1OutputProvider = adapter.OutputProvider; } - else + + if (!(adapter.OutputProvider is not null)) { _model.Blocks[adapter.Layer].FFN.W1WeightProvider = adapter.Provider; } @@ -597,7 +600,8 @@ private void AttachProviders() { _model.Blocks[adapter.Layer].FFN.W2OutputProvider = adapter.OutputProvider; } - else + + if (!(adapter.OutputProvider is not null)) { _model.Blocks[adapter.Layer].FFN.W2WeightProvider = adapter.Provider; } @@ -611,7 +615,8 @@ private void AttachProviders() { attn.SetQueryOutputProvider(adapter.HeadIndex, adapter.OutputProvider); } - else + + if (!(adapter.OutputProvider is not null)) { attn.SetQueryProvider(adapter.HeadIndex, adapter.Provider); } @@ -625,7 +630,8 @@ private void AttachProviders() { attn.SetKeyOutputProvider(adapter.HeadIndex, adapter.OutputProvider); } - else + + if (!(adapter.OutputProvider is not null)) { attn.SetKeyProvider(adapter.HeadIndex, adapter.Provider); } @@ -639,7 +645,8 @@ private void AttachProviders() { attn.SetValueOutputProvider(adapter.HeadIndex, adapter.OutputProvider); } - else + + if (!(adapter.OutputProvider is not null)) { attn.SetValueProvider(adapter.HeadIndex, adapter.Provider); } @@ -653,7 +660,8 @@ private void AttachProviders() { attn.SetOutputOutputProvider(adapter.HeadIndex, adapter.OutputProvider); } - else + + if (!(adapter.OutputProvider is not null)) { attn.SetOutputProvider(adapter.HeadIndex, adapter.Provider); } @@ -675,7 +683,8 @@ private void DetachProvider(ModuleAdapter adapter) _model.LMHeadOutputProvider = null; } } - else if (ReferenceEquals(_model.LMHeadWeightProvider, adapter.Provider)) + + if (adapter.OutputProvider is null && ReferenceEquals(_model.LMHeadWeightProvider, adapter.Provider)) { _model.LMHeadWeightProvider = null; } @@ -692,7 +701,8 @@ private void DetachProvider(ModuleAdapter adapter) ffn.W1OutputProvider = null; } } - else if (ReferenceEquals(ffn.W1WeightProvider, adapter.Provider)) + + if (adapter.OutputProvider is null && ReferenceEquals(ffn.W1WeightProvider, adapter.Provider)) { ffn.W1WeightProvider = null; } @@ -710,7 +720,8 @@ private void DetachProvider(ModuleAdapter adapter) ffn.W2OutputProvider = null; } } - else if (ReferenceEquals(ffn.W2WeightProvider, adapter.Provider)) + + if (adapter.OutputProvider is null && ReferenceEquals(ffn.W2WeightProvider, adapter.Provider)) { ffn.W2WeightProvider = null; } @@ -728,7 +739,8 @@ private void DetachProvider(ModuleAdapter adapter) attn.SetQueryOutputProvider(adapter.HeadIndex, null); } } - else if (ReferenceEquals(attn.GetQueryProvider(adapter.HeadIndex), adapter.Provider)) + + if (adapter.OutputProvider is null && ReferenceEquals(attn.GetQueryProvider(adapter.HeadIndex), adapter.Provider)) { attn.SetQueryProvider(adapter.HeadIndex, null); } @@ -746,7 +758,8 @@ private void DetachProvider(ModuleAdapter adapter) attn.SetKeyOutputProvider(adapter.HeadIndex, null); } } - else if (ReferenceEquals(attn.GetKeyProvider(adapter.HeadIndex), adapter.Provider)) + + if (adapter.OutputProvider is null && ReferenceEquals(attn.GetKeyProvider(adapter.HeadIndex), adapter.Provider)) { attn.SetKeyProvider(adapter.HeadIndex, null); } @@ -764,7 +777,8 @@ private void DetachProvider(ModuleAdapter adapter) attn.SetValueOutputProvider(adapter.HeadIndex, null); } } - else if (ReferenceEquals(attn.GetValueProvider(adapter.HeadIndex), adapter.Provider)) + + if (adapter.OutputProvider is null && ReferenceEquals(attn.GetValueProvider(adapter.HeadIndex), adapter.Provider)) { attn.SetValueProvider(adapter.HeadIndex, null); } @@ -782,7 +796,8 @@ private void DetachProvider(ModuleAdapter adapter) attn.SetOutputOutputProvider(adapter.HeadIndex, null); } } - else if (ReferenceEquals(attn.GetOutputProvider(adapter.HeadIndex), adapter.Provider)) + + if (adapter.OutputProvider is null && ReferenceEquals(attn.GetOutputProvider(adapter.HeadIndex), adapter.Provider)) { attn.SetOutputProvider(adapter.HeadIndex, null); } diff --git a/Sources/Main/LanguageModels/Loading/GgufLlamaLoader.cs b/Sources/Main/LanguageModels/Loading/GgufLlamaLoader.cs index 5fa7928c..e714323c 100644 --- a/Sources/Main/LanguageModels/Loading/GgufLlamaLoader.cs +++ b/Sources/Main/LanguageModels/Loading/GgufLlamaLoader.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -336,7 +336,9 @@ internal static CachedLlamaInferenceEngine LoadFromReader( // Q4_K-native / Q8_0-native / F32-fallback independently from // its file format. Wo dispatches separately (Q8_0 OK per-head; // K-quant not — headDim < the 256-element super-block). - DecodeWeight[] wq, wk, wv; + // The fusedQkv / !fusedQkv pair below is exhaustive, but the compiler cannot prove that + // across two separate ifs, and loading eagerly is not an option (each path reads tensors). + DecodeWeight[] wq = null!, wk = null!, wv = null!; if (fusedQkv) { // Phi-3: one fused attn_qkv [out=(nHeads+2·nKvHeads)·headDim, in=dModel], output-major @@ -350,7 +352,8 @@ internal static CachedLlamaInferenceEngine LoadFromReader( wk = SplitKeyValue(qkvFused.Span.Slice(qElems, kvElems), nKvHeads, dModel, headDim, attnQuantizable); wv = SplitKeyValue(qkvFused.Span.Slice(qElems + kvElems, kvElems), nKvHeads, dModel, headDim, attnQuantizable); } - else + + if (!(fusedQkv)) { wq = LoadQkvHeads(reader, $"blk.{l}.attn_q.weight", nHeads, dModel, headDim, qFull.Span, attnQuantizable, mmap); wk = LoadQkvHeads(reader, $"blk.{l}.attn_k.weight", nKvHeads, dModel, headDim, kFull.Span, attnQuantizable, mmap); @@ -390,7 +393,14 @@ internal static CachedLlamaInferenceEngine LoadFromReader( DecodeWeight[]? moeGate = null, moeUp = null, moeDown = null; DecodeWeight moeShGate = default, moeShUp = default, moeShDown = default; - if (isMoe) + // One classification instead of an if/else-if ladder: the bodies each load tensors, so + // exactly one must run. 0 = MoE, 1 = fused gate_up (Phi-3), 2 = Q8-resident, 3 = F32. + var ffnKind = isMoe ? 0 + : fusedGateUp ? 1 + : quantize && dModel % Q8DotKernel.BlockSize == 0 && dFF % Q8DotKernel.BlockSize == 0 ? 2 + : 3; + + if (ffnKind == 0) { // Router + routed experts (3-D tensors). Qwen-MoE additionally has a // sigmoid-gated shared expert; Mixtral does not (hasSharedExpert == false). @@ -401,7 +411,8 @@ internal static CachedLlamaInferenceEngine LoadFromReader( moeUp = LoadExperts(reader, reader.Tensors[$"blk.{l}.ffn_up_exps.weight"]); moeDown = LoadExperts(reader, reader.Tensors[$"blk.{l}.ffn_down_exps.weight"]); } - else + + if (!(mergedExperts)) { // Older Mixtral: one 2-D weight per expert, loaded with the same resident // dispatch as a dense FFN (Q4_K verbatim/mmap, Q5/Q6/Q8/F32). @@ -417,7 +428,7 @@ internal static CachedLlamaInferenceEngine LoadFromReader( moeSharedGateInp = LoadF32Vector(reader, $"blk.{l}.ffn_gate_inp_shexp.weight", dModel); } } - else if (fusedGateUp) + if (ffnKind == 1) { // Phi-3: one fused ffn_up [out=2·dFF, in=dModel], output-major = [gate rows | up rows] // (HF gate_up_proj → chunk(2): first half gate, second half up). Dequant, slice the two @@ -428,13 +439,13 @@ internal static CachedLlamaInferenceEngine LoadFromReader( ffnUp = Q8Weight.QuantizeRows(gateUpFused.Span.Slice(half, half), dFF, dModel); ffnDown = AllocAndLoadResident(reader, $"blk.{l}.ffn_down.weight", dFF, dModel, mmap); } - else if (quantize && dModel % Q8DotKernel.BlockSize == 0 && dFF % Q8DotKernel.BlockSize == 0) + if (ffnKind == 2) { ffnGate = AllocAndLoadResident(reader, $"blk.{l}.ffn_gate.weight", dModel, dFF, mmap, repacked); ffnUp = AllocAndLoadResident(reader, $"blk.{l}.ffn_up.weight", dModel, dFF, mmap, repacked); ffnDown = AllocAndLoadResident(reader, $"blk.{l}.ffn_down.weight", dFF, dModel, mmap, repacked); } - else + if (ffnKind == 3) { ffnGate = AllocAndLoadTransposed(reader, $"blk.{l}.ffn_gate.weight", dModel, dFF); ffnUp = AllocAndLoadTransposed(reader, $"blk.{l}.ffn_up.weight", dModel, dFF); @@ -487,32 +498,39 @@ internal static CachedLlamaInferenceEngine LoadFromReader( // tied → token_embd; untied → output.weight. When quantization is // disabled, or dModel is not a multiple of the Q8 block size, fall // back to an F32 transposed LM head (the kernel's input-major layout). - DecodeWeight lmHead; + // Exhaustive across the quantized / F32-fallback pair below; see the note on wq/wk/wv. + DecodeWeight lmHead = default; if (quantize && dModel % Q8DotKernel.BlockSize == 0) { var lmHeadInfo = reader.Tensors[tieWeights ? "token_embd.weight" : "output.weight"]; - if (lmHeadInfo.Type == GgmlType.Q4_K && dModel % Q4KWeight.SuperBlockElements == 0) + var headKind = lmHeadInfo.Type == GgmlType.Q4_K && dModel % Q4KWeight.SuperBlockElements == 0 ? 0 + : lmHeadInfo.Type == GgmlType.Q6_K && dModel % Q6KWeight.SuperBlockElements == 0 ? 1 + : lmHeadInfo.Type == GgmlType.Q8_0 ? 2 + : tieWeights ? 3 + : 4; + + if (headKind == 0) { // Native Q4_K — read the file's blocks straight in (step 3.2b). lmHead = LoadQ4KNative(reader, lmHeadInfo, dModel, vocab, mmap, repacked); } - else if (lmHeadInfo.Type == GgmlType.Q6_K && dModel % Q6KWeight.SuperBlockElements == 0) + if (headKind == 1) { // Native Q6_K — read the file's blocks straight in (step 3.3c). lmHead = LoadQ6KNative(reader, lmHeadInfo, dModel, vocab, mmap); } - else if (lmHeadInfo.Type == GgmlType.Q8_0) + if (headKind == 2) { // Native Q8_0 — read the file's blocks straight in (step 2.4). lmHead = LoadQ8Native(reader, lmHeadInfo, dModel, vocab); } - else if (tieWeights) + if (headKind == 3) { // Reached only when token_embd is F16/F32/BF16 (the K-quant/Q8 cases are // caught above), so the embedding is F32-backed here — .F32 is valid. lmHead = Q8Weight.QuantizeRows(embedWeights.F32, vocab, dModel); } - else + if (headKind == 4) { var outElems = checked((int)((long)vocab * dModel)); using var outputRaw = new PooledBuffer(outElems, clearMemory: false); @@ -520,7 +538,8 @@ internal static CachedLlamaInferenceEngine LoadFromReader( lmHead = Q8Weight.QuantizeRows(outputRaw.Span, vocab, dModel); } } - else + + if (!(quantize && dModel % Q8DotKernel.BlockSize == 0)) { // F32 fallback — transpose [vocab, dModel] → [dModel, vocab]. var f32LmHead = TensorStorage.Unpooled(checked((int)((long)vocab * dModel))); @@ -537,7 +556,8 @@ internal static CachedLlamaInferenceEngine LoadFromReader( } } } - else + + if (!(tieWeights)) { var outElems = checked((int)((long)vocab * dModel)); using var outputRaw = new PooledBuffer(outElems, clearMemory: false); @@ -656,7 +676,8 @@ internal static DecodeWeight[] LoadExperts(GgufReader reader, GgufTensorInfo inf { reader.LoadTensorQ4_KRaw(info, whole.Span); } - else + + if (!(info.Type == GgmlType.Q4_K)) { reader.LoadTensorQ6_KRaw(info, whole.Span); } @@ -952,24 +973,26 @@ private static Q4KWeight LoadQ4KNative( var blocksPerRow = inDim / Q4KWeight.SuperBlockElements; var totalBytes = checked((int)((long)outDim * blocksPerRow * Q4KWeight.SuperBlockBytes)); - Q4KWeight weight; - if (mmap is not null) - { - // Zero-copy: the file's block bytes ARE Q4KWeight's layout, verbatim. - var slice = mmap.Slice(reader.DataStart + (long)info.Offset, totalBytes); - weight = new Q4KWeight(slice, inDim, outDim); - } - else - { - var bytes = new byte[totalBytes]; - reader.LoadTensorQ4_KRaw(info, bytes); - weight = new Q4KWeight(bytes, inDim, outDim); - } + // Zero-copy when mmapped (the file's block bytes ARE Q4KWeight's layout, verbatim); otherwise + // read into a heap buffer. A ternary keeps `weight` definitely assigned without an else. + var weight = mmap is not null + ? new Q4KWeight(mmap.Slice(reader.DataStart + (long)info.Offset, totalBytes), inDim, outDim) + : LoadQ4KRawCopy(reader, info, totalBytes, inDim, outDim); AttachPrepacked(weight, info.Name, repacked); return weight; } + /// Non-mmap fallback for : read the raw Q4_K bytes into a heap + /// buffer. Split out only so the caller can stay a single definitely-assigned expression. + private static Q4KWeight LoadQ4KRawCopy( + GgufReader reader, GgufTensorInfo info, int totalBytes, int inDim, int outDim) + { + var bytes = new byte[totalBytes]; + reader.LoadTensorQ4_KRaw(info, bytes); + return new Q4KWeight(bytes, inDim, outDim); + } + // Attaches the offline-repacked (block_q4_Kx8) mmap slice for this tensor when a sidecar carries it, so // EnsureRepacked hands it out zero-copy instead of building a heap copy. No-op when there is no sidecar, // no matching entry, the shape can't repack, or the dims disagree — always safe to call. @@ -1270,11 +1293,14 @@ private static void LoadTensor(GgufReader reader, string name, Span dst) private static void LoadTensorOrZeros(GgufReader reader, string name, Span dst) { - if (reader.Tensors.TryGetValue(name, out var info)) + var found = reader.Tensors.TryGetValue(name, out var info); + + if (found) { reader.LoadTensorAsF32(info, dst); } - else + + if (!found) { dst.Clear(); } @@ -1300,7 +1326,8 @@ private static DecodeWeight[] SplitQuery( wq[h] = Q8Weight.QuantizeRows( qFull.Slice(h * headDim * dModel, headDim * dModel), headDim, dModel); } - else + + if (!(quantize)) { var storage = TensorStorage.Unpooled(checked((int)((long)dModel * headDim))); var dst = storage.AsSpan(); @@ -1329,7 +1356,8 @@ private static DecodeWeight[] SplitKeyValue( wkv[kv] = Q8Weight.QuantizeRows( kvFull.Slice(kv * headDim * dModel, headDim * dModel), headDim, dModel); } - else + + if (!(quantize)) { var storage = TensorStorage.Unpooled(checked((int)((long)dModel * headDim))); var dst = storage.AsSpan(); @@ -1375,7 +1403,8 @@ private static DecodeWeight[] SplitOutput( } wo[h] = Q8Weight.QuantizeRows(gather.Span, dModel, headDim); } - else + + if (!(quantize)) { var storage = TensorStorage.Unpooled(checked((int)((long)headDim * dModel))); var dst = storage.AsSpan(); diff --git a/Sources/Main/LanguageModels/Loading/GgufReader.cs b/Sources/Main/LanguageModels/Loading/GgufReader.cs index 4255e7f9..270018ea 100644 --- a/Sources/Main/LanguageModels/Loading/GgufReader.cs +++ b/Sources/Main/LanguageModels/Loading/GgufReader.cs @@ -750,7 +750,8 @@ internal void LoadQ5RegionAsF32(GgufTensorInfo info, long elementOffset, Span tokenizerConfigJson) { name = reader.GetString(); } - else if (isTemplate && reader.TokenType == JsonTokenType.String) + if (isTemplate && reader.TokenType == JsonTokenType.String) { template = reader.GetString(); } - else if (reader.TokenType is JsonTokenType.StartObject or JsonTokenType.StartArray) + + if (reader.TokenType is JsonTokenType.StartObject or JsonTokenType.StartArray) { reader.Skip(); } diff --git a/Sources/Main/LanguageModels/Loading/LlamaConfigReader.cs b/Sources/Main/LanguageModels/Loading/LlamaConfigReader.cs index 2583874c..238d1f0f 100644 --- a/Sources/Main/LanguageModels/Loading/LlamaConfigReader.cs +++ b/Sources/Main/LanguageModels/Loading/LlamaConfigReader.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -91,58 +91,79 @@ public static GPT1Config Parse(ReadOnlySpan json) { reader.Read(); layers = reader.GetInt32(); + continue; } - else if (reader.ValueTextEquals("hidden_size")) + + if (reader.ValueTextEquals("hidden_size")) { reader.Read(); dModel = reader.GetInt32(); + continue; } - else if (reader.ValueTextEquals("num_attention_heads")) + + if (reader.ValueTextEquals("num_attention_heads")) { reader.Read(); nHeads = reader.GetInt32(); + continue; } - else if (reader.ValueTextEquals("num_key_value_heads")) + + if (reader.ValueTextEquals("num_key_value_heads")) { reader.Read(); nKvHeads = reader.GetInt32(); + continue; } - else if (reader.ValueTextEquals("intermediate_size")) + + if (reader.ValueTextEquals("intermediate_size")) { reader.Read(); dFF = reader.GetInt32(); + continue; } - else if (reader.ValueTextEquals("max_position_embeddings")) + + if (reader.ValueTextEquals("max_position_embeddings")) { reader.Read(); maxPos = reader.GetInt32(); + continue; } - else if (reader.ValueTextEquals("vocab_size")) + + if (reader.ValueTextEquals("vocab_size")) { reader.Read(); vocab = reader.GetInt32(); + continue; } - else if (reader.ValueTextEquals("head_dim")) + + if (reader.ValueTextEquals("head_dim")) { reader.Read(); headDim = reader.GetInt32(); + continue; } - else if (reader.ValueTextEquals("rope_theta")) + + if (reader.ValueTextEquals("rope_theta")) { reader.Read(); ropeTheta = (float)reader.GetDouble(); + continue; } - else if (reader.ValueTextEquals("tie_word_embeddings")) + + if (reader.ValueTextEquals("tie_word_embeddings")) { reader.Read(); tie = reader.TokenType == JsonTokenType.True; + continue; } - else if (reader.ValueTextEquals("rope_scaling")) + + if (reader.ValueTextEquals("rope_scaling")) { reader.Read(); scaling = ReadRopeScaling(ref reader); + continue; } - else + { reader.Read(); if (reader.TokenType is JsonTokenType.StartObject or JsonTokenType.StartArray) @@ -237,28 +258,37 @@ public static GPT1Config Parse(ReadOnlySpan json) { reader.Read(); type = reader.GetString(); + continue; } - else if (reader.ValueTextEquals("factor")) + + if (reader.ValueTextEquals("factor")) { reader.Read(); factor = (float)reader.GetDouble(); + continue; } - else if (reader.ValueTextEquals("low_freq_factor")) + + if (reader.ValueTextEquals("low_freq_factor")) { reader.Read(); lowFreq = (float)reader.GetDouble(); + continue; } - else if (reader.ValueTextEquals("high_freq_factor")) + + if (reader.ValueTextEquals("high_freq_factor")) { reader.Read(); highFreq = (float)reader.GetDouble(); + continue; } - else if (reader.ValueTextEquals("original_max_position_embeddings")) + + if (reader.ValueTextEquals("original_max_position_embeddings")) { reader.Read(); origCtx = reader.GetInt32(); + continue; } - else + { reader.Read(); if (reader.TokenType is JsonTokenType.StartObject or JsonTokenType.StartArray) diff --git a/Sources/Main/LanguageModels/Loading/SafetensorsLlamaLoader.cs b/Sources/Main/LanguageModels/Loading/SafetensorsLlamaLoader.cs index d0829ef9..0f5a48a4 100644 --- a/Sources/Main/LanguageModels/Loading/SafetensorsLlamaLoader.cs +++ b/Sources/Main/LanguageModels/Loading/SafetensorsLlamaLoader.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -185,7 +185,8 @@ private static DecodeWeight[] LoadHeads( { heads[h] = Q8Weight.QuantizeRows(rowMajor, headDim, dModel); } - else + + if (!(quantize)) { var storage = TensorStorage.Unpooled(dModel * headDim); var dst = storage.AsSpan(); @@ -246,7 +247,8 @@ private static DecodeWeight[] LoadOutputHeads( } heads[h] = Q8Weight.QuantizeRows(gather.Span, dModel, headDim); } - else + + if (!(quantize)) { var storage = TensorStorage.Unpooled(headDim * dModel); var dst = storage.AsSpan(); @@ -363,7 +365,8 @@ private static TensorStorage[] LoadBias( { PermuteRopeRows(src, dst, headDim, width: 1); } - else + + if (!(ropePermute)) { src.CopyTo(dst); } diff --git a/Sources/Main/LanguageModels/Memory/ChatHistoryCompactor.cs b/Sources/Main/LanguageModels/Memory/ChatHistoryCompactor.cs index 60ad6c5c..4cf6dd44 100644 --- a/Sources/Main/LanguageModels/Memory/ChatHistoryCompactor.cs +++ b/Sources/Main/LanguageModels/Memory/ChatHistoryCompactor.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -42,7 +42,8 @@ public static CompactionPlan Plan( { systemMessages.Add(m); } - else + + if (!(string.Equals(m.Role, "system", StringComparison.Ordinal))) { nonSystem.Add(m); nonSystemChars += m.Content.Length; diff --git a/Sources/Main/LanguageModels/Runtime/BatchedProjectionKernel.cs b/Sources/Main/LanguageModels/Runtime/BatchedProjectionKernel.cs index d038c694..3060437f 100644 --- a/Sources/Main/LanguageModels/Runtime/BatchedProjectionKernel.cs +++ b/Sources/Main/LanguageModels/Runtime/BatchedProjectionKernel.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -149,7 +149,8 @@ private static void ProjectTile( { outRow.Clear(); } - else + + if (!(bias.IsEmpty)) { bias.Slice(tile, len).CopyTo(outRow); } diff --git a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs index df344927..ecba8bcf 100644 --- a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs +++ b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -39,7 +39,10 @@ public static void Dispatch( int inputSize, int outputSize) { - if (weight.IsQ6K) + // Resident-format dispatch, classified once so the original first-match order is explicit. + var kind = weight.IsQ6K ? 0 : weight.IsQ4K ? 1 : weight.IsQuantized ? 2 : 3; + + if (kind == 0) { var w = weight.Quantized6K; var spr = w.SuperBlocksPerRow; @@ -52,7 +55,7 @@ public static void Dispatch( qBytes.Span.Slice(0, rows * inputSize), scales.Span.Slice(0, rows * spr), sums.Span.Slice(0, groups)); } - else if (weight.IsQ4K) + if (kind == 1) { var w = weight.Quantized4K; var spr = w.SuperBlocksPerRow; @@ -68,8 +71,10 @@ public static void Dispatch( // No-bias only (GemmTiled applies none). AVX2/FMA required — the kernel is x86-only, so on ARM // (e.g. the Android app) this falls through to the weight-stationary path even if a sidecar // mmap'd a prepacked layout (IsPrepacked would otherwise bypass the env flag's AVX2 gate). - if ((w.IsPrepacked || UseTiledPrefillQ4K) && bias.IsEmpty && w.CanRepack - && CpuFeatures.HasAvx2 && CpuFeatures.HasFma) + var tiled = (w.IsPrepacked || UseTiledPrefillQ4K) && bias.IsEmpty && w.CanRepack + && CpuFeatures.HasAvx2 && CpuFeatures.HasFma; + + if (tiled) { DispatchTiledQ4K( input, rows, w, output, @@ -78,14 +83,14 @@ public static void Dispatch( } // Weight-stationary: decode each Q4_K super-block once and reuse across the row tile (bit-identical // to ProjectBatched, measured ~1.3–1.7× on the prefill / speculative-verify batched matmul). - else if (UseWeightStationaryQ4K) + if (!tiled && UseWeightStationaryQ4K) { Q4KDotKernel.ProjectBatchedWeightStationary( input, rows, w, bias, output, qBytes.Span.Slice(0, rows * inputSize), scales.Span.Slice(0, rows * spr), sums.Span.Slice(0, groups)); } - else + if (!tiled && !UseWeightStationaryQ4K) { Q4KDotKernel.ProjectBatched( input, rows, w, bias, output, @@ -93,7 +98,7 @@ public static void Dispatch( sums.Span.Slice(0, groups)); } } - else if (weight.IsQuantized) + if (kind == 2) { var w = weight.Quantized; var bpr = inputSize / Q8DotKernel.BlockSize; @@ -103,7 +108,7 @@ public static void Dispatch( input, rows, w, bias, output, qBytes.Span.Slice(0, rows * inputSize), scales.Span.Slice(0, rows * bpr)); } - else + if (kind == 3) { BatchedProjectionKernel.Project(input, rows, weight.F32, bias, output, inputSize, outputSize); } diff --git a/Sources/Main/LanguageModels/Runtime/CachedFeedForwardBlock.cs b/Sources/Main/LanguageModels/Runtime/CachedFeedForwardBlock.cs index 6456ff2f..095bf1ca 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedFeedForwardBlock.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedFeedForwardBlock.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -269,8 +269,14 @@ internal void DecodeSwiGluDispatched( // gate + up through the repacked kernel (8 rows/lane, no per-row hsum — ~2× the // per-core throughput of the 1-row kernel). Q4_K gate/up only; down stays Q6_K. var sGateUp = DecodeProfiler.Start(); - if (Q4KGemvKernel.Enabled && wGate.IsQ4K && wUp.IsQ4K - && wGate.Quantized4K.CanRepack && wUp.Quantized4K.CanRepack) + // Classified once so the original first-match order stays explicit: + // 0 = repacked 8x8 GEMV, 1 = fused Q4_K gate+up, 2 = generic per-weight dispatch. + var gateUpKind = Q4KGemvKernel.Enabled && wGate.IsQ4K && wUp.IsQ4K + && wGate.Quantized4K.CanRepack && wUp.Quantized4K.CanRepack ? 0 + : wGate.IsQ4K && wUp.IsQ4K ? 1 + : 2; + + if (gateUpKind == 0) { Q4KDotKernel.QuantizeActivationQ8K( hidden.Slice(0, DModel), _q8kInputQuants, _q8kInputScales, _q8kInputBsums); @@ -285,14 +291,14 @@ internal void DecodeSwiGluDispatched( // gate and up project the SAME hidden. When both are Q4_K (the Q4_K_M FFN // case), fuse them: quantize hidden once + one dispatch for both halves // (decode FFN is dispatch-overhead bound). Otherwise keep the two-call path. - else if (wGate.IsQ4K && wUp.IsQ4K) + if (gateUpKind == 1) { Q4KDotKernel.ProjectGateUpParallel( hidden, wGate.Quantized4K, wUp.Quantized4K, _gate, _intermediate, _q8kInputQuants, _q8kInputScales, _q8kInputBsums); ApplyGate(_gate, Activation); } - else + if (gateUpKind == 2) { // gate = SiLU(hidden @ Wgate) ProjectParallelDispatched(hidden, in wGate, [], _gate, DModel, DFF); @@ -360,25 +366,28 @@ private void ProjectParallelDispatched( int inputSize, int outputSize) { - if (weight.IsQ6K) + // Resident-format dispatch, classified once (see BatchedQuantProjection). + var kind = weight.IsQ6K ? 0 : weight.IsQ4K ? 1 : weight.IsQuantized ? 2 : 3; + + if (kind == 0) { Q6KDotKernel.ProjectParallel( input, weight.Quantized6K, bias, output, _q8kInputQuants, _q8kInputScales, _q8kInputBsums); } - else if (weight.IsQ4K) + if (kind == 1) { Q4KDotKernel.ProjectParallel( input, weight.Quantized4K, bias, output, _q8kInputQuants, _q8kInputScales, _q8kInputBsums); } - else if (weight.IsQuantized) + if (kind == 2) { Q8DotKernel.ProjectParallel( input, weight.Quantized, bias, output, _q8InputQuants, _q8InputScales); } - else + if (kind == 3) { SingleTokenProjectionKernel.ProjectParallel( input, weight.F32, bias, output, inputSize, outputSize); @@ -478,7 +487,8 @@ private static void ApplyGate(Span values, FeedForwardActivation activati { ApplyGeLU(values); } - else + + if (!(activation == FeedForwardActivation.GeGLU)) { ApplySiLU(values); } diff --git a/Sources/Main/LanguageModels/Runtime/CachedGptStack.cs b/Sources/Main/LanguageModels/Runtime/CachedGptStack.cs index 9de784dc..167874ed 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedGptStack.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedGptStack.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -270,7 +270,8 @@ private void ApplyFinalNorm(ReadOnlySpan input, StackWeights weights, Spa output[i] = input[i] * scale; } } - else + + if (!(weights.FinalNormGamma.IsEmpty)) { for (var i = 0; i < DModel; i++) { @@ -278,7 +279,8 @@ private void ApplyFinalNorm(ReadOnlySpan input, StackWeights weights, Spa } } } - else + + if (!(weights.FinalNormBeta.IsEmpty)) { SingleTokenLayerNormKernel.Normalize(input, weights.FinalNormGamma, weights.FinalNormBeta, output, DModel, LayerNormEpsilon); } @@ -348,7 +350,8 @@ internal void PrefillBatched( _finalHidden[i] = lastRow[i] * scale; } } - else + + if (!(weights.FinalNormGamma.IsEmpty)) { for (var i = 0; i < DModel; i++) { @@ -356,7 +359,8 @@ internal void PrefillBatched( } } } - else + + if (!(weights.FinalNormBeta.IsEmpty)) { SingleTokenLayerNormKernel.Normalize( lastRow, weights.FinalNormGamma, weights.FinalNormBeta, _finalHidden, DModel, LayerNormEpsilon); @@ -468,7 +472,8 @@ private void FinalNorm(ReadOnlySpan row, StackWeights weights, Span row, StackWeights weights, Span logits) internal void ProjectLogitsFrom(ReadOnlySpan finalNorm, StackWeights weights, Span logits) { var lmHead = weights.LmHeadWeights; - if (lmHead.IsQ6K) + // Resident-format dispatch, classified once (see BatchedQuantProjection). + var kind = lmHead.IsQ6K ? 0 : lmHead.IsQ4K ? 1 : lmHead.IsQuantized ? 2 : 3; + + if (kind == 0) { Q6KDotKernel.ProjectParallel( finalNorm, lmHead.Quantized6K, [], logits, _lmHeadQ8KQuants, _lmHeadQ8KScales, _lmHeadQ8KBsums); } - else if (lmHead.IsQ4K) + if (kind == 1) { Q4KDotKernel.ProjectParallel( finalNorm, lmHead.Quantized4K, [], logits, _lmHeadQ8KQuants, _lmHeadQ8KScales, _lmHeadQ8KBsums); } - else if (lmHead.IsQuantized) + if (kind == 2) { Q8DotKernel.ProjectParallel( finalNorm, lmHead.Quantized, [], logits, _lmHeadInputQuants, _lmHeadInputScales); } - else + if (kind == 3) { SingleTokenProjectionKernel.ProjectParallel( finalNorm, lmHead.F32, [], logits, DModel, VocabSize); diff --git a/Sources/Main/LanguageModels/Runtime/CachedLlamaInferenceEngine.cs b/Sources/Main/LanguageModels/Runtime/CachedLlamaInferenceEngine.cs index 9228d694..b7c890f2 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedLlamaInferenceEngine.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedLlamaInferenceEngine.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -494,7 +494,8 @@ private StackWeights BuildStackWeights() heads[h] = new SingleHeadWeights( wq: layer.Wq[h], bq: layer.Bq[h], wo: layer.Wo[h]); } - else + + if (!(useGqa)) { // MHA: all weights in SingleHeadWeights var kv = h % kvCount; diff --git a/Sources/Main/LanguageModels/Runtime/CachedLlamaSession.cs b/Sources/Main/LanguageModels/Runtime/CachedLlamaSession.cs index 8f9741cc..47a325d1 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedLlamaSession.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedLlamaSession.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -214,7 +214,8 @@ public void Prefill(ReadOnlySpan promptTokens) { DecodeTokenWithoutLogits(promptTokens[i]); } - else + + if (!(i < lastIndex)) { DecodeToken(promptTokens[i]); } @@ -471,11 +472,16 @@ private int GenerateSpeculativeCore( var probe = false; if (gated) { - if (_specProbeCountdown > 0) + // Capture BEFORE decrementing: a second `_specProbeCountdown > 0` test would read the + // already-decremented value, so countdown == 1 would both decrement AND probe. + var countingDown = _specProbeCountdown > 0; + + if (countingDown) { _specProbeCountdown--; } - else + + if (!countingDown) { probe = true; _specProbeCountdown = SpecProbeInterval; @@ -493,7 +499,7 @@ private int GenerateSpeculativeCore( dn = drafter.Draft(t0, draft); } } - else if (canSpeculate && (!gated || probe)) + if (drafter is null && canSpeculate && (!gated || probe)) { #pragma warning disable OVERFIT001 // exact-length contract: PromptLookupDrafter.Draft reads anchor.Length; tiny per-step array var anchor = new int[history.Length + 1]; @@ -557,7 +563,8 @@ private int GenerateSpeculativeCore( committed[1 + j] = draft[j]; accepted++; } - else + + if (!(token == draft[j])) { correction = token; break; @@ -767,7 +774,7 @@ public void Embed( dst[j] += h[j]; } } - else if (i == tokens.Length - 1) + if (pooling != EmbeddingPooling.Mean && i == tokens.Length - 1) { h[..d].CopyTo(dst); } diff --git a/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs b/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs index 32894378..895e0fc5 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -121,7 +121,8 @@ public CachedMultiHeadAttention( _attnScales = new float[(wholeSize + Q4KDotKernel.SuperBlockElements - 1) / Q4KDotKernel.SuperBlockElements]; _attnBsums = new short[(wholeSize + Q4KDotKernel.GroupSize - 1) / Q4KDotKernel.GroupSize]; } - else + + if (!(Q4KGemvKernel.AttnEnabled)) { _qWhole = []; _attnBands = []; @@ -188,7 +189,8 @@ internal void Decode( { output.Slice(0, DModel).Clear(); } - else + + if (!(bo.IsEmpty)) { bo.Slice(0, DModel).CopyTo(output); } @@ -279,11 +281,11 @@ internal void Decode( // heads via the decode dispatch (capped / spin-pool per config). OverfitParallel.ForDecode(0, HeadCount, &DecodeHeadQao, contextPtr); } - else if (KvHeadCount > 1 && OverfitParallel.WorkerCount > 1) + if (!useHeadParallel && KvHeadCount > 1 && OverfitParallel.WorkerCount > 1) { OverfitParallel.For(0, KvHeadCount, &DecodeKvGroup, contextPtr); } - else + if (!useHeadParallel && !(KvHeadCount > 1 && OverfitParallel.WorkerCount > 1)) { DecodeKvGroup(0, KvHeadCount, contextPtr); } @@ -374,7 +376,8 @@ private bool TryDecodeWholeMatrix( { OverfitParallel.ForDecode(0, HeadCount, &DecodeHeadWhole, ctxPtr); } - else + + if (!(OverfitParallel.WorkerCount > 1)) { DecodeHeadWhole(0, HeadCount, ctxPtr); } @@ -494,7 +497,8 @@ internal void DecodeBatched( { outRow.Clear(); } - else + + if (!(bias.IsEmpty)) { bias.Slice(0, dModel).CopyTo(outRow); } @@ -547,7 +551,8 @@ internal void DecodeBatched( { OverfitParallel.For(0, HeadCount, &ProcessHeadRangeBatched, contextPtr); } - else + + if (!(HeadCount > 1 && OverfitParallel.WorkerCount > 1)) { ProcessHeadRangeBatched(0, HeadCount, contextPtr); } @@ -607,7 +612,8 @@ internal void DecodeBatchedQuant( { outRow.Clear(); } - else + + if (!(attnBias.IsEmpty)) { attnBias.Slice(0, dModel).CopyTo(outRow); } @@ -629,8 +635,10 @@ internal void DecodeBatchedQuant( for (var group = 0; group < KvHeadCount; group++) { // K/V weights: GQA shares one KV head per group; MHA uses the head's own. - DecodeWeight wk, wv; - ReadOnlySpan bk, bv; + // Both branches below assign these; `= default` only proves it to the compiler (value types, + // immediately overwritten, so the JIT elides the init). + DecodeWeight wk = default, wv = default; + ReadOnlySpan bk = default, bv = default; if (weights.HasGqa) { ref readonly var kv = ref weights.KvHead(group); @@ -639,7 +647,8 @@ internal void DecodeBatchedQuant( bk = kv.Bk; bv = kv.Bv; } - else + + if (!(weights.HasGqa)) { ref readonly var h0 = ref weights.Head(group); wk = h0.Wk; @@ -665,7 +674,7 @@ internal void DecodeBatchedQuant( cache.WriteValue(layerIndex, group, basePosition + n, vg.Span.Slice(n * headDim, headDim)); } - ReadOnlySpan keys, values; + ReadOnlySpan keys = default, values = default; if (cache.IsQuantized) { cache.DequantizeKeyRange(layerIndex, group, fromPosition: 0, length: cacheLength, kf.Span.Slice(0, cacheLength * headDim)); @@ -673,7 +682,8 @@ internal void DecodeBatchedQuant( keys = kf.Span.Slice(0, cacheLength * headDim); values = vf.Span.Slice(0, cacheLength * headDim); } - else + + if (!(cache.IsQuantized)) { keys = cache.GetKeyReadSpan(layerIndex, group, fromPosition: 0, length: cacheLength); values = cache.GetValueReadSpan(layerIndex, group, fromPosition: 0, length: cacheLength); @@ -810,8 +820,8 @@ private static void DecodeKvGroup(int groupStart, int groupEnd, void* context) var h = group * groupSize + headInGroup; ref readonly var hw = ref ctx.Weights.Head(h); - DecodeWeight wk, wv; - ReadOnlySpan bk, bv; + DecodeWeight wk = default, wv = default; + ReadOnlySpan bk = default, bv = default; if (ctx.UseGqa) { // GQA: every Q head in the group shares one KV head. @@ -821,7 +831,8 @@ private static void DecodeKvGroup(int groupStart, int groupEnd, void* context) bk = kv.Bk; bv = kv.Bv; } - else + + if (!(ctx.UseGqa)) { // Standard MHA: each Q head has its own K/V weights. wk = hw.Wk; diff --git a/Sources/Main/LanguageModels/Runtime/CachedSingleHeadAttention.cs b/Sources/Main/LanguageModels/Runtime/CachedSingleHeadAttention.cs index 9936ef31..e004751f 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedSingleHeadAttention.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedSingleHeadAttention.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -401,40 +401,44 @@ private void ProjectHiddenDispatched( ReadOnlySpan sharedBsums, bool sharedValid) { - if (weight.IsQ6K) + var kind = weight.IsQ6K ? 0 : weight.IsQ4K ? 1 : weight.IsQuantized ? 2 : 3; + + if (kind == 0) { if (sharedValid) { Q6KDotKernel.ProjectPreQuantized( weight.Quantized6K, bias, output, sharedQuants, sharedScales, sharedBsums); } - else + + if (!(sharedValid)) { Q6KDotKernel.Project( hidden, weight.Quantized6K, bias, output, _q8kInputQuants, _q8kInputScales, _q8kInputBsums); } } - else if (weight.IsQ4K) + if (kind == 1) { if (sharedValid) { Q4KDotKernel.ProjectPreQuantized( weight.Quantized4K, bias, output, sharedQuants, sharedScales, sharedBsums); } - else + + if (!(sharedValid)) { Q4KDotKernel.Project( hidden, weight.Quantized4K, bias, output, _q8kInputQuants, _q8kInputScales, _q8kInputBsums); } } - else if (weight.IsQuantized) + if (kind == 2) { Q8DotKernel.Project( hidden, weight.Quantized, bias, output, _q8InputQuants, _q8InputScales); } - else + if (kind == 3) { SingleTokenProjectionKernel.Project( hidden, weight.F32, bias, output, DModel, HeadDimension); @@ -455,25 +459,27 @@ private void ProjectSequentialDispatched( int inputSize, int outputSize) { - if (weight.IsQ6K) + var kind = weight.IsQ6K ? 0 : weight.IsQ4K ? 1 : weight.IsQuantized ? 2 : 3; + + if (kind == 0) { Q6KDotKernel.Project( input, weight.Quantized6K, bias, output, _q8kInputQuants, _q8kInputScales, _q8kInputBsums); } - else if (weight.IsQ4K) + if (kind == 1) { Q4KDotKernel.Project( input, weight.Quantized4K, bias, output, _q8kInputQuants, _q8kInputScales, _q8kInputBsums); } - else if (weight.IsQuantized) + if (kind == 2) { Q8DotKernel.Project( input, weight.Quantized, bias, output, _q8InputQuants, _q8InputScales); } - else + if (kind == 3) { SingleTokenProjectionKernel.Project( input, weight.F32, bias, output, inputSize, outputSize); diff --git a/Sources/Main/LanguageModels/Runtime/CachedTransformerBlock.cs b/Sources/Main/LanguageModels/Runtime/CachedTransformerBlock.cs index 776289e7..d477bc08 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedTransformerBlock.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedTransformerBlock.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -204,7 +204,8 @@ internal void Decode( { RmsNormalize(input, weights.Ln1Gamma, _ln1Output, DModel, LayerNormEpsilon); } - else + + if (!(weights.Ln1Beta.IsEmpty)) { SingleTokenLayerNormKernel.Normalize( input, weights.Ln1Gamma, weights.Ln1Beta, _ln1Output, DModel, LayerNormEpsilon); @@ -237,7 +238,8 @@ internal void Decode( { RmsNormalize(_afterAttentionResidual, weights.Ln2Gamma, _ln2Output, DModel, LayerNormEpsilon); } - else + + if (!(weights.Ln2Beta.IsEmpty)) { SingleTokenLayerNormKernel.Normalize( _afterAttentionResidual, weights.Ln2Gamma, weights.Ln2Beta, _ln2Output, DModel, LayerNormEpsilon); @@ -261,7 +263,7 @@ internal void Decode( } // SwiGLU (Llama/Mistral/Qwen): FfnGate is present. // GeLU/ReLU (GPT-1/GPT-2): FfnGate is empty. - else if (!weights.FfnGate.IsEmpty) + if (!weights.IsMoe && !weights.FfnGate.IsEmpty) { // SwiGLU (Llama/Mistral/Qwen): per-weight dispatch — each of // gate/up/down picks its kernel from its resident format @@ -273,7 +275,7 @@ internal void Decode( weights.FfnW2, _feedForwardOutput); } - else + if (!weights.IsMoe && weights.FfnGate.IsEmpty) { _feedForward.Decode( _ln2Output, @@ -416,7 +418,8 @@ internal void DecodeBatchedQuant( { RmsNormalize(inRow, weights.Ln1Gamma, dst, dModel, LayerNormEpsilon); } - else + + if (!(weights.Ln1Beta.IsEmpty)) { SingleTokenLayerNormKernel.Normalize(inRow, weights.Ln1Gamma, weights.Ln1Beta, dst, dModel, LayerNormEpsilon); } @@ -444,7 +447,8 @@ internal void DecodeBatchedQuant( { RmsNormalize(aRow, weights.Ln2Gamma, dst, dModel, LayerNormEpsilon); } - else + + if (!(weights.Ln2Beta.IsEmpty)) { SingleTokenLayerNormKernel.Normalize(aRow, weights.Ln2Gamma, weights.Ln2Beta, dst, dModel, LayerNormEpsilon); } @@ -458,7 +462,8 @@ internal void DecodeBatchedQuant( weights.MoeSharedGateInp, weights.MoeSharedGate, weights.MoeSharedUp, weights.MoeSharedDown, ffnOut.Span); } - else + + if (!(weights.IsMoe)) { _feedForward.DecodeSwiGluBatchedDispatched( ln2.Span, rows, weights.FfnGate, weights.FfnW1, weights.FfnW2, ffnOut.Span); @@ -542,7 +547,8 @@ private static void RmsNormalize( output[i] = input[i] * scale; } } - else + + if (!(gamma.IsEmpty)) { for (var i = 0; i < dModel; i++) { diff --git a/Sources/Main/LanguageModels/Runtime/DraftModelSpeculativeDrafter.cs b/Sources/Main/LanguageModels/Runtime/DraftModelSpeculativeDrafter.cs index f284246c..cfce7fe7 100644 --- a/Sources/Main/LanguageModels/Runtime/DraftModelSpeculativeDrafter.cs +++ b/Sources/Main/LanguageModels/Runtime/DraftModelSpeculativeDrafter.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -65,7 +65,8 @@ public void Sync(ReadOnlySpan committedThisStep) // append every committed token from the last synced point. ApplyTail(_syncedLen, committedThisStep, 0); } - else + + if (!(_draftBase < 0)) { // The draft fed [seed, d0, d1, …]; committedThisStep = [seed, d0..d(a-1), correction]. // The accepted drafts already sit in the draft KV, so keep [base, base + keep) and re-feed diff --git a/Sources/Main/LanguageModels/Runtime/KeyValueCache.cs b/Sources/Main/LanguageModels/Runtime/KeyValueCache.cs index 00f1a4b8..b5f354ed 100644 --- a/Sources/Main/LanguageModels/Runtime/KeyValueCache.cs +++ b/Sources/Main/LanguageModels/Runtime/KeyValueCache.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -79,7 +79,8 @@ public KeyValueCache(KeyValueCacheShape shape, KvCacheDType dtype) _keys = []; _values = []; } - else + + if (!(dtype == KvCacheDType.Q8)) { _keys = new float[elems]; _values = new float[elems]; @@ -160,7 +161,8 @@ public void Reset() Array.Clear(_keyScales); Array.Clear(_valueScales); } - else + + if (!(IsQuantized)) { Array.Clear(_keys); Array.Clear(_values); @@ -208,7 +210,8 @@ public void Evict(int count) _keyScales.AsSpan(scaleBase + count, keep).CopyTo(_keyScales.AsSpan(scaleBase, keep)); _valueScales.AsSpan(scaleBase + count, keep).CopyTo(_valueScales.AsSpan(scaleBase, keep)); } - else + + if (!(IsQuantized)) { _keys.AsSpan(baseOffset + shift, keepElems).CopyTo(_keys.AsSpan(baseOffset, keepElems)); _values.AsSpan(baseOffset + shift, keepElems).CopyTo(_values.AsSpan(baseOffset, keepElems)); @@ -465,7 +468,8 @@ private void WriteVector( { scales[ScaleOffset(layerIndex, headIndex, position)] = Q8KvQuant.Quantize(vector.Slice(0, hd), q8.AsSpan(offset, hd)); } - else + + if (!(IsQuantized)) { vector.Slice(0, hd).CopyTo(f32.AsSpan(offset, hd)); } diff --git a/Sources/Main/LanguageModels/Runtime/MoeRouter.cs b/Sources/Main/LanguageModels/Runtime/MoeRouter.cs index c1ba06c5..d20892b6 100644 --- a/Sources/Main/LanguageModels/Runtime/MoeRouter.cs +++ b/Sources/Main/LanguageModels/Runtime/MoeRouter.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -109,7 +109,8 @@ public static int SelectTopK( expertWeights[i] *= inv; } } - else + + if (!(normalize)) { // Full softmax over ALL experts; keep the top-k probabilities un-renormalised. var total = 0f; diff --git a/Sources/Main/LanguageModels/Runtime/Q4KDotKernel.cs b/Sources/Main/LanguageModels/Runtime/Q4KDotKernel.cs index da83753c..7c36aa78 100644 --- a/Sources/Main/LanguageModels/Runtime/Q4KDotKernel.cs +++ b/Sources/Main/LanguageModels/Runtime/Q4KDotKernel.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -637,7 +637,8 @@ private static void GateUpChunk(int chunkStart, int chunkEnd, void* context) { ctx.UpOutput[row] = sum; } - else + + if (!(isUp)) { ctx.GateOutput[row] = sum; } diff --git a/Sources/Main/LanguageModels/Runtime/Q8KvQuant.cs b/Sources/Main/LanguageModels/Runtime/Q8KvQuant.cs index 49b7e50a..9d586c2b 100644 --- a/Sources/Main/LanguageModels/Runtime/Q8KvQuant.cs +++ b/Sources/Main/LanguageModels/Runtime/Q8KvQuant.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -62,14 +62,7 @@ public static float Quantize(ReadOnlySpan src, Span dst) for (var d = 0; d < n; d++) { var q = MathF.Round(src[d] * inv); - if (q > 127f) - { - q = 127f; - } - else if (q < -127f) - { - q = -127f; - } + q = Math.Clamp(q, -127f, 127f); dst[d] = (sbyte)q; } return scale; diff --git a/Sources/Main/LanguageModels/Runtime/SingleTokenProjectionKernel.cs b/Sources/Main/LanguageModels/Runtime/SingleTokenProjectionKernel.cs index 4547e495..e2493029 100644 --- a/Sources/Main/LanguageModels/Runtime/SingleTokenProjectionKernel.cs +++ b/Sources/Main/LanguageModels/Runtime/SingleTokenProjectionKernel.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -67,7 +67,8 @@ public static void Project( { output.Slice(0, outputSize).Clear(); } - else + + if (!(bias.IsEmpty)) { bias.Slice(0, outputSize).CopyTo(output); } @@ -240,7 +241,8 @@ private static void ProjectChunk(int chunkStart, int chunkEnd, void* context) { outputBand.Clear(); } - else + + if (!(ctx.BiasLength == 0)) { new ReadOnlySpan(ctx.Bias + chunkStart, count).CopyTo(outputBand); } @@ -310,7 +312,8 @@ public static void ProjectSlice( { outputSlice.Clear(); } - else + + if (!(bias.IsEmpty)) { bias.Slice(outputOffset, outputCount).CopyTo(outputSlice); } diff --git a/Sources/Main/LanguageModels/Runtime/SlmSession.cs b/Sources/Main/LanguageModels/Runtime/SlmSession.cs index 8775e8ae..0134c9b7 100644 --- a/Sources/Main/LanguageModels/Runtime/SlmSession.cs +++ b/Sources/Main/LanguageModels/Runtime/SlmSession.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -196,12 +196,17 @@ public void Dispose() private void AppendToken(int token) { - if (_contextLength < MaxContextLength) + // Capture BEFORE the append increments _contextLength: a second test on the updated value + // would ALSO take the shift path on the very last free slot, writing the token twice. + var hasRoom = _contextLength < MaxContextLength; + + if (hasRoom) { _contextTokens[_contextLength] = token; _contextLength++; } - else + + if (!hasRoom) { // Overlapping in-place left-shift — Span.CopyTo has memmove semantics. _contextTokens.AsSpan(1, MaxContextLength - 1) diff --git a/Sources/Main/LanguageModels/Runtime/StackWeights.cs b/Sources/Main/LanguageModels/Runtime/StackWeights.cs index e268333b..554af262 100644 --- a/Sources/Main/LanguageModels/Runtime/StackWeights.cs +++ b/Sources/Main/LanguageModels/Runtime/StackWeights.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -77,7 +77,8 @@ internal StackWeights(GPT1Model model) cfg.VocabSize, cfg.DModel); } - else + + if (!(cfg.TieWeights)) { _lmHead = model.LMHead.Data; } diff --git a/Sources/Main/LanguageModels/Skills/Evaluation/CheckRegistry.cs b/Sources/Main/LanguageModels/Skills/Evaluation/CheckRegistry.cs index ec020b73..5e7988d5 100644 --- a/Sources/Main/LanguageModels/Skills/Evaluation/CheckRegistry.cs +++ b/Sources/Main/LanguageModels/Skills/Evaluation/CheckRegistry.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -33,11 +33,14 @@ public IReadOnlyList Grade(SkillEvalCase testCase, SkillRunResult re foreach (var id in testCase.ExpectedChecks) { - if (_graders.TryGetValue(id, out var grader)) + var hasGrader = _graders.TryGetValue(id, out var grader); + + if (hasGrader) { - checks.Add(grader.Grade(testCase, result)); + checks.Add(grader!.Grade(testCase, result)); } - else + + if (!hasGrader) { checks.Add(new GradeCheck(id, false, "no grader registered for this check id")); } diff --git a/Sources/Main/LanguageModels/Skills/Evaluation/SkillEvalReport.cs b/Sources/Main/LanguageModels/Skills/Evaluation/SkillEvalReport.cs index e79924bc..ba9c11ca 100644 --- a/Sources/Main/LanguageModels/Skills/Evaluation/SkillEvalReport.cs +++ b/Sources/Main/LanguageModels/Skills/Evaluation/SkillEvalReport.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -96,7 +96,7 @@ public SkillEvalReport(IReadOnlyList cases) { helped++; } - else if (!c.OnPass && c.OffPass) + if (!c.OnPass && c.OffPass) { hurt++; } diff --git a/Sources/Main/LanguageModels/Skills/Optimization/SkillOptimizer.cs b/Sources/Main/LanguageModels/Skills/Optimization/SkillOptimizer.cs index 87ca7d9e..1b1bd398 100644 --- a/Sources/Main/LanguageModels/Skills/Optimization/SkillOptimizer.cs +++ b/Sources/Main/LanguageModels/Skills/Optimization/SkillOptimizer.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -79,7 +79,8 @@ public static SkillOptResult Optimize( currentVal = candidateVal; steps.Add(new SkillOptResult.Step(round, candidate!, candidateVal, true, "accepted (val improved)")); } - else + + if (!(candidateVal > currentVal)) { rejected.Add(candidate!); steps.Add(new SkillOptResult.Step(round, candidate!, candidateVal, false, "rejected (no val improvement)")); diff --git a/Sources/Main/LanguageModels/Tokenizers/GgufTokenizer.cs b/Sources/Main/LanguageModels/Tokenizers/GgufTokenizer.cs index bf10b1c7..ea044f42 100644 --- a/Sources/Main/LanguageModels/Tokenizers/GgufTokenizer.cs +++ b/Sources/Main/LanguageModels/Tokenizers/GgufTokenizer.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -204,7 +204,8 @@ public int[] Encode(string text, bool? addBos = null) { BpeEncode(text, output); } - else + + if (!(_model == "gpt2")) { SpmEncode(text, output); } @@ -254,18 +255,20 @@ void Flush() { bytes.Add(_charToByte![piece[c]]); } + + continue; } - else if (_idToByte[id] >= 0) + + if (_idToByte[id] >= 0) { bytes.Add((byte)_idToByte[id]); + continue; } - else + + var pieceBytes = Encoding.UTF8.GetBytes(_tokens[id].Replace(SpaceMarker, ' ')); + for (var b = 0; b < pieceBytes.Length; b++) { - var pieceBytes = Encoding.UTF8.GetBytes(_tokens[id].Replace(SpaceMarker, ' ')); - for (var b = 0; b < pieceBytes.Length; b++) - { - bytes.Add(pieceBytes[b]); - } + bytes.Add(pieceBytes[b]); } } diff --git a/Sources/Main/LanguageModels/Tokenizers/HuggingFaceBpeTokenizer.cs b/Sources/Main/LanguageModels/Tokenizers/HuggingFaceBpeTokenizer.cs index c293779b..13145459 100644 --- a/Sources/Main/LanguageModels/Tokenizers/HuggingFaceBpeTokenizer.cs +++ b/Sources/Main/LanguageModels/Tokenizers/HuggingFaceBpeTokenizer.cs @@ -154,7 +154,8 @@ public int[] Encode(string text) { tokens.Add(_specialTokens[piece]); } - else + + if (!(isSpecial)) { foreach (Match m in _splitPattern.Matches(piece)) { @@ -187,7 +188,8 @@ public string DecodeToString(ReadOnlySpan tokens) } sb.Append(piece); } - else + + if (!(_specialTokenIds.Contains(id))) { foreach (var ch in piece) { @@ -314,13 +316,16 @@ private static Dictionary ReadVocab(JsonElement vocabJson, out stri var rank = 0; foreach (var merge in mergesJson.EnumerateArray()) { - string? left, right; + string? left = null; + string? right = null; + if (merge.ValueKind == JsonValueKind.Array) { left = merge[0].GetString(); right = merge[1].GetString(); } - else + + if (merge.ValueKind != JsonValueKind.Array) { var parts = merge.GetString()!.Split(' '); left = parts.Length == 2 ? parts[0] : null; diff --git a/Sources/Main/LanguageModels/Tokenizers/QwenTokenizer.cs b/Sources/Main/LanguageModels/Tokenizers/QwenTokenizer.cs index 8a6b6d79..2b201420 100644 --- a/Sources/Main/LanguageModels/Tokenizers/QwenTokenizer.cs +++ b/Sources/Main/LanguageModels/Tokenizers/QwenTokenizer.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -167,7 +167,8 @@ public int[] Encode(string text, bool addBos = false) { tokens.Add(_specialTokens[piece]); } - else + + if (!(isSpecial)) { foreach (Match m in _splitPattern.Matches(piece)) { @@ -204,7 +205,8 @@ public string Decode(ReadOnlySpan tokens) } sb.Append(piece); } - else + + if (!(_specialTokenIds.Contains(id))) { // Decode byte-level piece → raw bytes foreach (var ch in piece) @@ -383,7 +385,8 @@ private static char[] BuildByteToChar() { map[b] = (char)b; } - else + + if (!((b >= '!' && b <= '~') || (b >= '¡' && b <= '¬') || (b >= '®' && b <= 'ÿ'))) { map[b] = (char)0; // placeholder } diff --git a/Sources/Main/LanguageModels/Tokenizers/WordPieceTokenizer.cs b/Sources/Main/LanguageModels/Tokenizers/WordPieceTokenizer.cs index cafa3d47..ecfef6bb 100644 --- a/Sources/Main/LanguageModels/Tokenizers/WordPieceTokenizer.cs +++ b/Sources/Main/LanguageModels/Tokenizers/WordPieceTokenizer.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -200,7 +200,8 @@ public string DecodeToString(ReadOnlySpan tokens) { sb.Append(tok, ContinuationPrefix.Length, tok.Length - ContinuationPrefix.Length); } - else + + if (!(tok.StartsWith(ContinuationPrefix, StringComparison.Ordinal))) { if (sb.Length > 0) { @@ -319,7 +320,8 @@ private void WordPiece(string word, List ids) { ids.Add(_unkId); } - else + + if (!(isBad)) { ids.AddRange(pieces); } diff --git a/Sources/Main/LanguageModels/Tools/ToolCallConstraint.cs b/Sources/Main/LanguageModels/Tools/ToolCallConstraint.cs index dd692680..41f097cc 100644 --- a/Sources/Main/LanguageModels/Tools/ToolCallConstraint.cs +++ b/Sources/Main/LanguageModels/Tools/ToolCallConstraint.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -234,7 +234,8 @@ public bool TryAdvance(char c, string[] names, CompiledSchema?[] schemas) _stage = Stage.Args; _args = default; } - else + + if (!(schemas[_toolIndex] is null)) { _stage = Stage.ArgsSchema; _segIndex = 0; @@ -327,7 +328,8 @@ private bool TryAdvanceSchema(char c, CompiledSchema schema) _valueStarted = false; _valueKind = schema.Kinds[_segIndex]; } - else + + if (!(_segIndex < schema.Kinds.Length)) { // Matched the final "}" that closes the arguments object; one more "}" (the // envelope's own close) follows. diff --git a/Sources/Main/LanguageModels/Whisper/WhisperGgmlLoader.cs b/Sources/Main/LanguageModels/Whisper/WhisperGgmlLoader.cs index cd142956..dcfc966f 100644 --- a/Sources/Main/LanguageModels/Whisper/WhisperGgmlLoader.cs +++ b/Sources/Main/LanguageModels/Whisper/WhisperGgmlLoader.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -103,14 +103,15 @@ public static WhisperModel Load(Stream stream) data[i] = br.ReadSingle(); } } - else if (ftype == 1) + if (ftype == 1) { for (var i = 0L; i < count; i++) { data[i] = (float)BitConverter.UInt16BitsToHalf(br.ReadUInt16()); } } - else + + if (ftype is not (0 or 1)) { throw new OverfitRuntimeException($"Tensor '{name}' has unsupported ftype {ftype} (only F32/F16 supported so far)."); } diff --git a/Sources/Main/LanguageModels/Whisper/WhisperKernels.cs b/Sources/Main/LanguageModels/Whisper/WhisperKernels.cs index dd062355..7a756ad0 100644 --- a/Sources/Main/LanguageModels/Whisper/WhisperKernels.cs +++ b/Sources/Main/LanguageModels/Whisper/WhisperKernels.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -263,7 +263,8 @@ public static void MultiHeadAttention( MhaHead(h, q, k, v, attnOut, scores, tq, tkv, dModel, dHead, scale, causal); } } - else + + if (!((long)nHeads * tq * tkv * dHead < ParallelThreshold)) { fixed (float* qp = q, kp = k, vp = v, ap = attnOut) { diff --git a/Sources/Main/LanguageModels/Whisper/WhisperTranscriber.cs b/Sources/Main/LanguageModels/Whisper/WhisperTranscriber.cs index 0500e2ac..38cf10a4 100644 --- a/Sources/Main/LanguageModels/Whisper/WhisperTranscriber.cs +++ b/Sources/Main/LanguageModels/Whisper/WhisperTranscriber.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -85,16 +85,10 @@ public string Transcribe(ReadOnlySpan samples, string language = "en", in /// internal string Transcribe(ReadOnlySpan samples, string language, int maxNewTokens, bool padToFullWindow) { - int windowLen; - if (padToFullWindow) - { - windowLen = SamplesPerWindow; - } - else - { - var present = Math.Min(samples.Length, SamplesPerWindow); - windowLen = Math.Min(SamplesPerWindow, Math.Max(present + TrailingSamples, MinSamples)); - } + var present = Math.Min(samples.Length, SamplesPerWindow); + var windowLen = padToFullWindow + ? SamplesPerWindow + : Math.Min(SamplesPerWindow, Math.Max(present + TrailingSamples, MinSamples)); // The 30 s buffer is reused across calls; we only clear + fill (and run the mel/encoder over) the // first windowLen samples — the rest is left untouched and never read. diff --git a/Sources/Main/Main.csproj b/Sources/Main/Main.csproj index ac205c67..8b1cf163 100644 --- a/Sources/Main/Main.csproj +++ b/Sources/Main/Main.csproj @@ -59,18 +59,6 @@ - - - - - diff --git a/Sources/Main/Onnx/OnnxGraphImporter.cs b/Sources/Main/Onnx/OnnxGraphImporter.cs index d70672e4..45948e48 100644 --- a/Sources/Main/Onnx/OnnxGraphImporter.cs +++ b/Sources/Main/Onnx/OnnxGraphImporter.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -83,12 +83,16 @@ public static OnnxGraphModel LoadFromBytes( var inName = onnxNode.Inputs[0]; var outName = onnxNode.Outputs[0]; - if (slotMap.TryGetValue(inName, out var slot)) + var hasSlot = slotMap.TryGetValue(inName, out var slot); + var hasInitializer = initializers.TryGetValue(inName, out var initTensor); + + if (hasSlot) { // Relabels an activation tensor: output reads from the same slot. slotMap[outName] = slot; } - else if (initializers.TryGetValue(inName, out var initTensor)) + + if (!hasSlot && hasInitializer) { // Relabels a CONSTANT: a folded/deduplicated weight or bias routed to its // consumer under a new name (e.g. torch's constant-folding aliases equal biases diff --git a/Sources/Main/Onnx/OnnxProtoParser.cs b/Sources/Main/Onnx/OnnxProtoParser.cs index 82a57503..0faca01f 100644 --- a/Sources/Main/Onnx/OnnxProtoParser.cs +++ b/Sources/Main/Onnx/OnnxProtoParser.cs @@ -379,7 +379,8 @@ private static OnnxTensor ParseTensor(ref ProtoReader reader) { floatData = reader.ReadPackedFloat(); } - else + + if (wireType != WireType.LengthDelimited) { // unpacked floatData ??= []; @@ -394,7 +395,8 @@ private static OnnxTensor ParseTensor(ref ProtoReader reader) { int64Data = reader.ReadPackedInt64().ToArray(); } - else + + if (wireType != WireType.LengthDelimited) { int64Data ??= []; var newArr = new long[int64Data.Length + 1]; @@ -572,7 +574,8 @@ private static (OnnxDataType, long?[]) ParseTypeProto(ref ProtoReader reader) } } } - else + + if (fieldNum != 1) { reader.SkipField(wireType); } @@ -619,7 +622,8 @@ private static (OnnxDataType, long?[]) ParseTypeProto(ref ProtoReader reader) dims.Add(dim); } - else + + if (fieldNum != 1) { reader.SkipField(wireType); } diff --git a/Sources/Main/Onnx/Operators/ReduceMeanOperator.cs b/Sources/Main/Onnx/Operators/ReduceMeanOperator.cs index bf5f6781..4a44fbc7 100644 --- a/Sources/Main/Onnx/Operators/ReduceMeanOperator.cs +++ b/Sources/Main/Onnx/Operators/ReduceMeanOperator.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -32,20 +32,27 @@ public static IModule Build( // Opset < 18: axes is an attribute. long[]? axes = null; - if (node.Attributes.TryGetValue("axes", out var axesAttr)) + var hasAxesAttribute = node.Attributes.TryGetValue("axes", out var axesAttr); + + if (hasAxesAttribute) { axes = axesAttr.IntArray; } - else if (node.Inputs.Count >= 2 && - !string.IsNullOrEmpty(node.Inputs[1]) && - initializers.TryGetValue(node.Inputs[1], out var axesTensor)) + + if (!hasAxesAttribute && + node.Inputs.Count >= 2 && + !string.IsNullOrEmpty(node.Inputs[1]) && + initializers.TryGetValue(node.Inputs[1], out var axesTensor)) { // Axes are stored as Int64 tensor — read directly without float conversion. - if (axesTensor.Int64Data != null && axesTensor.Int64Data.Length > 0) + var hasInt64Data = axesTensor.Int64Data != null && axesTensor.Int64Data.Length > 0; + + if (hasInt64Data) { axes = axesTensor.Int64Data; } - else if (axesTensor.RawData.Length > 0) + + if (!hasInt64Data && axesTensor.RawData.Length > 0) { // Raw little-endian int64 bytes. var count = axesTensor.RawData.Length / sizeof(long); diff --git a/Sources/Server/OpenAi/OpenAiChatMapping.cs b/Sources/Server/OpenAi/OpenAiChatMapping.cs index b3acd053..58bd8333 100644 --- a/Sources/Server/OpenAi/OpenAiChatMapping.cs +++ b/Sources/Server/OpenAi/OpenAiChatMapping.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -30,19 +30,9 @@ public static (SamplingOptions Sampling, int MaxTokens) BuildSampling(ChatComple } var temperature = req.Temperature ?? 1.0f; - SamplingOptions sampling; - if (temperature <= 0.0001f) - { - sampling = SamplingOptions.Greedy; - } - else if (req.MinP is > 0f and < 1f) - { - sampling = SamplingOptions.WithMinP(req.MinP.Value, temperature); - } - else - { - sampling = new SamplingOptions(SamplingStrategy.TopP, temperature, topK: 0, topP: req.TopP ?? 1.0f, seed: 0); - } + var sampling = temperature <= 0.0001f ? SamplingOptions.Greedy + : req.MinP is > 0f and < 1f ? SamplingOptions.WithMinP(req.MinP.Value, temperature) + : new SamplingOptions(SamplingStrategy.TopP, temperature, topK: 0, topP: req.TopP ?? 1.0f, seed: 0); return (sampling, maxTokens); } @@ -116,7 +106,7 @@ public static List ParseInputs(JsonElement input) { list.Add(input.GetString() ?? string.Empty); } - else if (input.ValueKind == JsonValueKind.Array) + if (input.ValueKind == JsonValueKind.Array) { foreach (var e in input.EnumerateArray()) { diff --git a/Sources/Server/OverfitOpenAiServer.cs b/Sources/Server/OverfitOpenAiServer.cs index 82260db2..e8fe549f 100644 --- a/Sources/Server/OverfitOpenAiServer.cs +++ b/Sources/Server/OverfitOpenAiServer.cs @@ -1,4 +1,4 @@ -// Copyright (c) 2026 DevOnBike. +// Copyright (c) 2026 DevOnBike. // This file is part of DevonBike Overfit. // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com @@ -345,21 +345,11 @@ private static void HandleAudioSpeech(HttpListenerContext ctx, OrpheusVoiceEngin var voice = string.IsNullOrWhiteSpace(req.Voice) ? OrpheusPrompt.DefaultVoice : req.Voice!; var audio = tts.Synthesize(req.Input!, voice); - byte[] bytes; - string contentType; - if (format == "pcm") - { - bytes = ToPcm16Bytes(audio); - contentType = "audio/pcm"; - } - else - { - using var ms = new MemoryStream(); - WavWriter.WriteMono(ms, audio, tts.SampleRate, WavSampleFormat.Pcm16, - SyntheticSpeechMetadata.ForNow(voice).ToInfoComment()); - bytes = ms.ToArray(); - contentType = "audio/wav"; - } + // Both outputs are read below, so they must be definitely assigned; split ifs the compiler + // cannot prove exhaustive would not do that. The WAV branch keeps its `using` scope in a block. + var isPcm = format == "pcm"; + var contentType = isPcm ? "audio/pcm" : "audio/wav"; + var bytes = isPcm ? ToPcm16Bytes(audio) : ToWavBytes(audio, tts.SampleRate, voice); ctx.Response.StatusCode = (int)HttpStatusCode.OK; ctx.Response.ContentType = contentType; @@ -367,6 +357,16 @@ private static void HandleAudioSpeech(HttpListenerContext ctx, OrpheusVoiceEngin ctx.Response.OutputStream.Write(bytes, 0, bytes.Length); } + /// WAV-encodes the synthesized audio. Split out of the caller so the `using MemoryStream` + /// keeps a scope of its own while the caller stays a single definitely-assigned expression. + private static byte[] ToWavBytes(float[] audio, int sampleRate, string voice) + { + using var ms = new MemoryStream(); + WavWriter.WriteMono(ms, audio, sampleRate, WavSampleFormat.Pcm16, + SyntheticSpeechMetadata.ForNow(voice).ToInfoComment()); + return ms.ToArray(); + } + private static byte[] ToPcm16Bytes(float[] samples) { var bytes = new byte[samples.Length * 2]; @@ -588,7 +588,8 @@ private static string OpenApiYaml() { _openApiYaml = "openapi: 3.0.3\ninfo:\n title: Overfit\n version: '1.0.0'\npaths: {}\n"; } - else + + if (!(stream is null)) { using var reader = new StreamReader(stream, Encoding.UTF8); _openApiYaml = reader.ReadToEnd(); diff --git a/format-code.ps1 b/format-code.ps1 index f76a4f6c..0adf8cde 100644 --- a/format-code.ps1 +++ b/format-code.ps1 @@ -1,10 +1,11 @@ param ( - [string]$SolutionPath = "" + [string]$SolutionPath = "", + + # CI mode: report formatting drift instead of rewriting files (non-zero exit if anything would change). + [switch]$Check ) -# Terminal configuration (runs after the parameters are declared) -Set-ExecutionPolicy Unrestricted -Scope Process -Force -cls +$ErrorActionPreference = 'Stop' if ([string]::IsNullOrWhiteSpace($SolutionPath)) { $slnFiles = Get-ChildItem -Filter *.sln -File @@ -15,20 +16,39 @@ if ([string]::IsNullOrWhiteSpace($SolutionPath)) { $SolutionPath = $slnFiles[0].FullName } -Write-Host "Starting code formatting..." -ForegroundColor Cyan +# --verify-no-changes turns each step into a check instead of a rewrite. +$extraArgs = @() +if ($Check) { $extraArgs += '--verify-no-changes' } -Write-Host "Step 1/3: Whitespace correction..." -ForegroundColor Yellow -dotnet format whitespace $SolutionPath +$mode = if ($Check) { "Checking" } else { "Formatting" } +Write-Host "$mode $SolutionPath" -ForegroundColor Cyan -Write-Host "Step 2/3: Applying general style rules..." -ForegroundColor Yellow -dotnet format style $SolutionPath +# Each step is verified on its own: $LASTEXITCODE only ever reflects the LAST command, so a single +# check at the end would report success even when an earlier step had failed. +function Invoke-FormatStep { + param([string]$Label, [string]$Subcommand) -Write-Host "Step 3/3: Verifying file headers (IDE0073)..." -ForegroundColor Yellow -dotnet format style $SolutionPath --diagnostics IDE0073 + Write-Host " $Label..." -ForegroundColor Yellow + dotnet format $Subcommand $SolutionPath @extraArgs -if ($LASTEXITCODE -eq 0) { - Write-Host "`nSuccess! The entire solution has been formatted." -ForegroundColor Green + if ($LASTEXITCODE -ne 0) { + $why = if ($Check) { "would be reformatted" } else { "failed" } + Write-Host "`n$Label $why (exit $LASTEXITCODE)." -ForegroundColor Red + exit $LASTEXITCODE + } +} + +# whitespace = indentation / spacing only. +# style = the .editorconfig IDE rules, INCLUDING the IDE0073 file header (it is severity=warning in +# .editorconfig and `dotnet format style` fixes warn-and-above by default), so the header +# needs no separate pass. +# `dotnet format analyzers` is deliberately NOT run here: it applies analyzer code fixes, which would mix +# behavioural rewrites into what should be a formatting-only change. Run it by hand if you want it. +Invoke-FormatStep -Label "Step 1/2: whitespace" -Subcommand "whitespace" +Invoke-FormatStep -Label "Step 2/2: style rules (incl. IDE0073 headers)" -Subcommand "style" + +if ($Check) { + Write-Host "`nSuccess! Formatting is already correct." -ForegroundColor Green } else { - Write-Host "`nErrors occurred during formatting." -ForegroundColor Red - exit $LASTEXITCODE + Write-Host "`nSuccess! The entire solution has been formatted." -ForegroundColor Green } From c8e88d1fc540fbe48237ac240e90f544d9f2e846 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Tue, 21 Jul 2026 15:31:21 +0200 Subject: [PATCH 05/37] else --- CLAUDE.md | 15 +- .../LanguageModels/Runtime/DecodeProfiler.cs | 12 +- .../Diagnostics/PrefillPathAbTests.cs | 146 ++++++++++++++++++ .../Diagnostics/PrefillProfileTests.cs | 116 ++++++++++++++ 4 files changed, 285 insertions(+), 4 deletions(-) create mode 100644 Tests/LanguageModels/Diagnostics/PrefillPathAbTests.cs create mode 100644 Tests/LanguageModels/Diagnostics/PrefillProfileTests.cs diff --git a/CLAUDE.md b/CLAUDE.md index 0e5ca5c0..f0d935a1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -234,13 +234,24 @@ Every perf change is a **hypothesis until measured**. Benchmark before/after wit **negative results** honestly — they are the most valuable output: in this codebase register-blocking (direct-conv), K-blocking + A-packing (im2col GEMM), Winograd F(2,3) for 3x3 stride-1 convs (parity-correct cos 1.0 but +79% slower on deepcnn, 119.7→214.4 ms — sequential scalar transforms + -16 small GEMMs + 16x U/V/M blow-up beat the 2.25x FLOP cut), the AVX-512 decode port, and -`OverfitPool` all **regressed or tied and were reverted**; the wins were the *opposite* of the +16 small GEMMs + 16x U/V/M blow-up beat the 2.25x FLOP cut), the AVX-512 decode port, bias support in +the Q4_K tiled prefill GEMM (`GemmTiled` — a path census showed `bias.IsEmpty` barred 88% of prefill +dispatches, i.e. all attention Q/K/V, from the tiled kernel; lifting it measured **0.999x, an exact +tie**, because `ProjectBatchedWeightStationary` already amortises weight decode across the row tile — +the same thing the tiling does; the "~3x" in the kernel docs is against re-decode-per-row, not against +weight-stationary), and `OverfitPool` all **regressed or tied and were reverted**; the wins were the *opposite* of the "obvious" move (`TensorPrimitives` bulk-SIMD beat a hand micro-kernel; the simple register-blocked GEMM beat the cache-blocked one — structure of the data around the technique decides, not the technique). Mind the **measurement environment**: a thermally-throttled or loaded box invalidates A/B (detect it with a *canary* — re-measure an unchanged code path; if it shifted, the box did, not your change). The decode spin-pool assumes dedicated cores, so it is sensitive to background load. +Two corollaries this repo has paid for. **Cross-process before/after does not work here**: a prefill +change read as +5% while the untouched decode path in the same run moved +32% — interleave the +configurations run-by-run in ONE process (ABAB…, not all-A-then-all-B) and time a canary path in +every sample. **Verify the flag you are A/B-ing is actually live**: `OVERFIT_TILED_PREFILL` is a dead +flag whenever a `*.gguf.repack` sidecar sits next to the model, because `IsPrepacked` short-circuits +it — both arms ran an identical mix and the "measurement" was noise. Count the paths taken (a +temporary counter in the dispatcher) before believing any kernel A/B. Never ship, claim, or commit a perf "win" you have not measured on a stable box — and prefer measuring over reasoning even when the reasoning feels airtight. diff --git a/Sources/Main/LanguageModels/Runtime/DecodeProfiler.cs b/Sources/Main/LanguageModels/Runtime/DecodeProfiler.cs index 6d142197..3df655ce 100644 --- a/Sources/Main/LanguageModels/Runtime/DecodeProfiler.cs +++ b/Sources/Main/LanguageModels/Runtime/DecodeProfiler.cs @@ -129,14 +129,22 @@ public static string Report() sb.AppendLine( $" total / token : {tokenMs,9:F3} ms ({(tokenMs > 0 ? 1000.0 / tokenMs : 0),6:F2} tok/s)"); + + // Only the TOP-LEVEL components count toward "accounted": ffn_gateup / ffn_down / ffn_multiply + // are a breakdown of `ffn`, so summing them too would subtract the FFN twice and drive `other` + // negative (it read -67% before this was fixed, which made the whole report untrustworthy). long accounted = 0; for (var i = 0; i < ComponentCount; i++) { var ms = _ticks[i] * toMs / tokens; - accounted += _ticks[i]; + if (i <= (int)Component.Sampler) + { + accounted += _ticks[i]; + } var pct = _tokenTicks > 0 ? 100.0 * _ticks[i] / _tokenTicks : 0; var perTok = (double)_calls[i] / tokens; - sb.AppendLine($" {ComponentName(i),-13} : {ms,9:F3} ms {pct,5:F1}% ({perTok,5:F0}/tok)"); + var indent = i > (int)Component.Sampler ? " " : string.Empty; + sb.AppendLine($" {indent + ComponentName(i),-13} : {ms,9:F3} ms {pct,5:F1}% ({perTok,5:F0}/tok)"); } var otherTicks = _tokenTicks - accounted; diff --git a/Tests/LanguageModels/Diagnostics/PrefillPathAbTests.cs b/Tests/LanguageModels/Diagnostics/PrefillPathAbTests.cs new file mode 100644 index 00000000..cac9b4a8 --- /dev/null +++ b/Tests/LanguageModels/Diagnostics/PrefillPathAbTests.cs @@ -0,0 +1,146 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Diagnostics; +using DevOnBike.Overfit.LanguageModels.Contracts; +using DevOnBike.Overfit.LanguageModels.Runtime; +using DevOnBike.Overfit.LanguageModels.Tokenizers; +using DevOnBike.Overfit.Tests.TestSupport; +using Xunit.Abstractions; + +namespace DevOnBike.Overfit.Tests.LanguageModels.Diagnostics +{ + /// + /// A/B of the Q4_K batched-prefill kernels that can actually switch, + /// on ONE loaded model in ONE process. Prefill is 99.1% of time-to-first-token + /// (), so this is the path worth measuring. + /// + /// Method — two traps this measurement already fell into. (1) A cross-process before/after + /// showed a +5% "regression" while the untouched decode path moved +32%: the box had drifted, not the + /// code. So configurations are interleaved run-by-run and every sample also times a first-token + /// decode as a canary — decode uses a different kernel, so a material canary drift invalidates the + /// prefill numbers. (2) An attempt to A/B UseTiledPrefillQ4K measured nothing at all, because a + /// *.gguf.repack sidecar sets IsPrepacked, which short-circuits that flag in the tiled + /// gate — both arms silently ran the identical mix. + /// + /// Recorded negative — do not re-try without new evidence. The tiled GEMM is barred from + /// biased projections by the bias.IsEmpty term in its gate, and a path census showed that excludes + /// 88% of Q4_K prefill dispatches (only 540 of 4644 were bias-free; AVX2/FMA and CanRepack held + /// for 100%). Adding an optional bias to GemmTiled to lift that restriction was implemented, + /// pinned bit-identical, and measured at 0.999× — an exact tie (raw samples fully interleaved), so + /// it was reverted. The reason it ties: ProjectBatchedWeightStationary already decodes each + /// super-block once and reuses it across the row tile, i.e. it amortises exactly what the tiling + /// amortises. The "~3×" in the kernel docs is against ProjectBatched (re-decode per row), not + /// against weight-stationary — which is the ratio this test measures. + /// + public sealed class PrefillPathAbTests + { + private readonly ITestOutputHelper _out; + + public PrefillPathAbTests(ITestOutputHelper output) => _out = output; + + [LongFact] + public void Prefill_WeightStationaryVsReDecode() + { + var path = TestModelPaths.Qwen3B.Q4KmGgufPath; + if (!File.Exists(path)) + { + _out.WriteLine($"missing {path}"); + return; + } + + var original = BatchedQuantProjection.UseWeightStationaryQ4K; + + using var engine = CachedLlamaInferenceEngine.LoadGguf(path); + var tok = GgufTokenizer.Load(path); + + var paragraph = string.Join(" ", + Enumerable.Repeat( + "The history of computing began with mechanical calculators and evolved through vacuum tubes, " + + "transistors, integrated circuits and finally the microprocessor era.", 24)); + var ids = tok.Encode(paragraph); + + const int Runs = 7; + var onPrefill = new double[Runs]; + var offPrefill = new double[Runs]; + var onDecode = new double[Runs]; + var offDecode = new double[Runs]; + var onToken = 0; + var offToken = 0; + + try + { + Warm(engine, ids, weightStationary: true); + Warm(engine, ids, weightStationary: false); + + // Interleaved so any monotonic drift (thermal, background load) hits both arms equally. + for (var r = 0; r < Runs; r++) + { + (onPrefill[r], onDecode[r], onToken) = Sample(engine, ids, weightStationary: true); + (offPrefill[r], offDecode[r], offToken) = Sample(engine, ids, weightStationary: false); + } + + var onP = Median(onPrefill); + var offP = Median(offPrefill); + var onD = Median(onDecode); + var offD = Median(offDecode); + + _out.WriteLine($"=== Q4_K prefill kernels ({ids.Length} tokens, median of {Runs}, interleaved) ==="); + _out.WriteLine($" weight-stationary : {onP,9:F1} ms {ids.Length / (onP / 1000.0),7:F0} tok/s (default)"); + _out.WriteLine($" re-decode-per-row : {offP,9:F1} ms {ids.Length / (offP / 1000.0),7:F0} tok/s"); + _out.WriteLine($" speedup : {offP / onP,9:F3}x"); + _out.WriteLine(string.Empty); + _out.WriteLine($" CANARY decode : {onD,6:F1} ms vs {offD,6:F1} ms drift {100.0 * (onD - offD) / offD,5:F1}%"); + _out.WriteLine(" (decode shares no kernel with these — material drift invalidates the comparison)"); + _out.WriteLine($" first token id : {onToken} / {offToken}"); + _out.WriteLine($" repack sidecar : {File.Exists(path + ".repack")} " + + "(when true, bias-free projections take the tiled GEMM regardless of OVERFIT_TILED_PREFILL)"); + + // Swapping kernels must not change what the model produces. + Assert.Equal(onToken, offToken); + } + finally + { + BatchedQuantProjection.UseWeightStationaryQ4K = original; + } + } + + private static void Warm(CachedLlamaInferenceEngine engine, int[] ids, bool weightStationary) + { + BatchedQuantProjection.UseWeightStationaryQ4K = weightStationary; + var sampling = SamplingOptions.Greedy; + using var warm = engine.CreateSession(1024); + warm.Reset(ids); + warm.GenerateNextToken(in sampling); + } + + private static (double Prefill, double Decode, int Token) Sample( + CachedLlamaInferenceEngine engine, int[] ids, bool weightStationary) + { + BatchedQuantProjection.UseWeightStationaryQ4K = weightStationary; + var sampling = SamplingOptions.Greedy; + + using var session = engine.CreateSession(1024); + + var sw = Stopwatch.StartNew(); + session.Reset(ids); + sw.Stop(); + var prefill = sw.Elapsed.TotalMilliseconds; + + sw.Restart(); + var token = session.GenerateNextToken(in sampling); + sw.Stop(); + + return (prefill, sw.Elapsed.TotalMilliseconds, token); + } + + private static double Median(double[] values) + { + var copy = (double[])values.Clone(); + Array.Sort(copy); + return copy[copy.Length / 2]; + } + } +} diff --git a/Tests/LanguageModels/Diagnostics/PrefillProfileTests.cs b/Tests/LanguageModels/Diagnostics/PrefillProfileTests.cs new file mode 100644 index 00000000..004a640a --- /dev/null +++ b/Tests/LanguageModels/Diagnostics/PrefillProfileTests.cs @@ -0,0 +1,116 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Diagnostics; +using DevOnBike.Overfit.LanguageModels.Contracts; +using DevOnBike.Overfit.LanguageModels.Runtime; +using DevOnBike.Overfit.LanguageModels.Tokenizers; +using DevOnBike.Overfit.Tests.TestSupport; +using Xunit.Abstractions; + +namespace DevOnBike.Overfit.Tests.LanguageModels.Diagnostics +{ + /// + /// Sizes the PREFILL path (time-to-first-token), which DecodeProfiler does not cover — its hooks + /// live in the single-token decode path only. + /// + /// The question this answers: how much of TTFT is tokenization? That is the only thing that + /// decides whether tokenizer-level work (SearchValues, FrozenDictionary vocab, span-based scanning) can + /// pay for itself. The decode profile already showed the answer for decode — tokenization does not appear + /// there at all, because it runs once, before the first token. + /// + /// Deliberately coarse: three stopwatches around tokenize / prefill / first-decoded-token. A finer + /// split would need permanent profiler hooks in the batched-prefill path, which is not worth adding to the + /// library for a one-off sizing. + /// + public sealed class PrefillProfileTests + { + private readonly ITestOutputHelper _out; + + public PrefillProfileTests(ITestOutputHelper output) => _out = output; + + [LongFact] + public void Prefill_TokenizeVsForward_Shares() + { + var path = TestModelPaths.Qwen3B.Q4KmGgufPath; + if (!File.Exists(path)) + { + _out.WriteLine($"missing {path}"); + return; + } + + using var engine = CachedLlamaInferenceEngine.LoadGguf(path); + var tok = GgufTokenizer.Load(path); + var sampling = SamplingOptions.Greedy; + + // A prompt long enough to take the BATCHED prefill path (the short path decodes token-by-token + // and would measure something else entirely). + var paragraph = string.Join(" ", + Enumerable.Repeat( + "The history of computing began with mechanical calculators and evolved through vacuum tubes, " + + "transistors, integrated circuits and finally the microprocessor era.", 24)); + + // Warm-up: JIT, page-in the weights, prime the tokenizer tables. + { + using var warm = engine.CreateSession(1024); + var warmIds = tok.Encode(paragraph); + warm.Reset(warmIds.AsSpan(0, Math.Min(64, warmIds.Length))); + warm.GenerateNextToken(in sampling); + } + + const int Runs = 5; + var tokenizeMs = new double[Runs]; + var prefillMs = new double[Runs]; + var firstTokenMs = new double[Runs]; + var promptLength = 0; + + for (var r = 0; r < Runs; r++) + { + using var session = engine.CreateSession(1024); + + var sw = Stopwatch.StartNew(); + var ids = tok.Encode(paragraph); + sw.Stop(); + tokenizeMs[r] = sw.Elapsed.TotalMilliseconds; + promptLength = ids.Length; + + sw.Restart(); + session.Reset(ids); + sw.Stop(); + prefillMs[r] = sw.Elapsed.TotalMilliseconds; + + sw.Restart(); + session.GenerateNextToken(in sampling); + sw.Stop(); + firstTokenMs[r] = sw.Elapsed.TotalMilliseconds; + } + + // Median, not mean: one page-fault or a background task skews a 5-run mean badly. + static double Median(double[] values) + { + var copy = (double[])values.Clone(); + Array.Sort(copy); + return copy[copy.Length / 2]; + } + + var t = Median(tokenizeMs); + var p = Median(prefillMs); + var f = Median(firstTokenMs); + var ttft = t + p + f; + + _out.WriteLine($"=== Prefill profile ({promptLength} prompt tokens, median of {Runs}) ==="); + _out.WriteLine($" tokenize : {t,9:F3} ms {100.0 * t / ttft,5:F1}%"); + _out.WriteLine($" prefill fwd : {p,9:F3} ms {100.0 * p / ttft,5:F1}%"); + _out.WriteLine($" first token : {f,9:F3} ms {100.0 * f / ttft,5:F1}%"); + _out.WriteLine($" TTFT total : {ttft,9:F3} ms"); + _out.WriteLine($" prefill rate : {promptLength / (p / 1000.0),9:F0} tok/s"); + _out.WriteLine($" tokenize rate : {promptLength / (t / 1000.0),9:F0} tok/s"); + _out.WriteLine($" raw tokenize : {string.Join(" / ", tokenizeMs.Select(x => x.ToString("F3")))}"); + _out.WriteLine($" raw prefill : {string.Join(" / ", prefillMs.Select(x => x.ToString("F3")))}"); + + Assert.True(promptLength > 128, $"prompt too short to exercise batched prefill ({promptLength} tokens)"); + } + } +} From b1d36dad0b4c032a946f969bbefbe225d771d52f Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Tue, 21 Jul 2026 15:36:38 +0200 Subject: [PATCH 06/37] else --- README.md | 9 +++++++++ ROADMAP.md | 50 ++++++++++++++++++++++++++++++++++++++------------ 2 files changed, 47 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index fce945b4..3b570adf 100644 --- a/README.md +++ b/README.md @@ -177,6 +177,15 @@ build *errors* in the kernels, and `[OverfitHotPath]` escalates every per-call r error inside a marked method) with a CI tripwire that proves the analyzer itself is alive ([`Sources/Analyzers/README.md`](Sources/Analyzers/README.md)). +Readability is held to the same standard: `else` and `else if` are build errors everywhere +outside the test project (`OVERFIT021`), and the sweep that introduced the rule removed all +322 occurrences from the library. That was done only after measuring what the refactor costs, +because a style rule that quietly slows the hot path is not worth having: guard clauses, +ternaries and loop `continue` are **free** (1.00–1.01×), and the one shape that does cost — +extracting a branch into a method the JIT then declines to inline — was measured at 2.25× and +is avoided rather than assumed away ([`Sources/Benchmark/ElseRefactorBenchmark.cs`](Sources/Benchmark/ElseRefactorBenchmark.cs), +the sole intentional exemption since the `else` forms are its measurement subject). + ### 5. Bounded parsing of untrusted model files Overfit runs **inside your process**, so a malformed model file must not be able to take diff --git a/ROADMAP.md b/ROADMAP.md index 37ffc20c..b4f07562 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -35,21 +35,21 @@ Zero-allocation, pure C# deep-learning framework targeting high-performance CPU --- -## ▶ NEXT UP AFTER RELEASE — finish the `else` sweep (OVERFIT021) +## ✅ DONE — the `else` sweep (OVERFIT021), 322 → 0 -**Status: 21 of 322 done, ~301 left.** `else` / `else if` is banned in `Sources/Main` by the in-repo Roslyn -analyzer **OVERFIT021** (`Sources/Analyzers/ElseClauseAnalyzer.cs`). It is *not* an MSBuild task and *not* a -`BannedSymbols.txt` entry — that file bans **API symbols**, and `else` is a language keyword, so it cannot be -expressed there. An MSBuild-task variant with an `ElseDebt.txt` ledger was built and then deleted in favour of -the analyzer (real syntax tree, IDE squiggles, per-directory severity). +**Status: COMPLETE (2026-07-21).** `else` / `else if` is banned by the in-repo Roslyn analyzer **OVERFIT021** +(`Sources/Analyzers/ElseClauseAnalyzer.cs`). It is *not* an MSBuild task and *not* a `BannedSymbols.txt` entry — +that file bans **API symbols**, and `else` is a language keyword, so it cannot be expressed there. An +MSBuild-task variant with an `ElseDebt.txt` ledger was built and then deleted in favour of the analyzer (real +syntax tree, IDE squiggles, per-directory severity). -**Rollout is a ratchet:** `suggestion` repo-wide, `error` for directories already at zero — the scoped section -at the **end** of `.editorconfig`. Clean a directory, add it to that list, and the ban locks in for it. +The ratchet is finished and retired: the rule is now **`error` across every project except `Tests`**, wired +centrally in `Directory.Build.props` rather than per-csproj, so the per-directory allow-list in `.editorconfig` +is gone. `Tests` stays at `suggestion` (test code is local and disposable). The **only** remaining `else` sites +in the repo are 6 in `Sources/Benchmark/ElseRefactorBenchmark.cs` — intentional, since the `else` forms are that +benchmark's measurement subject, and the project is excluded from the analyzer. -- ✅ **Done (9 dirs, 21 sites):** `Anomalies, Core, Diagnostics, Exceptions, Inference, Licensing, Maths, - Parameters, Randomization, Redaction, Runtime, Serving, Statistical, Tensors, Tokenization, Training, Trees` -- ⬜ **Left:** `LanguageModels` 163, `Audio` 35, `Ops` 34, `Onnx` 14, `Data` 13, `DeepLearning` 12, - `Evolutionary` 10, `Kernels` 6, `Intrinsics` 4, `Autograd` 4, `Optimizers` 6, rest small +Verified semantically rather than by grep: the solution builds clean with the rule at `error` globally. ### Cost is measured, not assumed — `Sources/Benchmark/ElseRefactorBenchmark.cs` @@ -77,6 +77,32 @@ So in-place rewrites are free and **the only real risk is extracting a method**. --- +## ▶ NEXT UP — the cheap CPU-perf levers are exhausted; the open item is a product decision + +**Three candidate levers were sized and all three died on measurement (2026-07-21). Do not re-open without +new evidence.** + +1. **`SearchValues` / tokenizer-level work — CLOSED.** A prefill profile (Qwen-3B Q4_K_M, 672-token prompt, + median of 5) puts tokenization at **0.04% of time-to-first-token** — 1.8 ms against 4731 ms of prefill + forward (366 000 tok/s vs 142 tok/s). Infinite tokenizer speedup buys 0.04%. + `Tests/LanguageModels/Diagnostics/PrefillProfileTests.cs`. +2. **Struct-operator (static-abstract interface) dispatch — CLOSED without building.** The premise does not + hold here: `ElementwiseKernels` contains **no delegates** (14 hand-written span loops), the hot parallel + paths already use `delegate*`, and the whole elementwise slice is **0.5% of decode**. + The scalar operator shape also cannot express the `TensorPrimitives` fast path, which is itself built on + this pattern inside the BCL. +3. **Bias support in the Q4_K tiled prefill GEMM — BUILT, MEASURED 0.999×, REVERTED.** A path census showed + `bias.IsEmpty` barred **88% of Q4_K prefill dispatches** (all attention Q/K/V) from `GemmTiled`. Lifting + it was an exact tie, because `ProjectBatchedWeightStationary` already amortises weight decode across the + row tile — the same thing the tiling does. The "~3×" in the kernel docs is measured against + `ProjectBatched` (re-decode per row), **not** against weight-stationary. Recorded in `CLAUDE.md`. + +Decode is ~88% quantized GEMV sitting at the DRAM floor (`ffn 69.3% · attention 19.3% · lm_head 10.3%`), so +there is no cheap kernel win left. What remains open is **not technical**: the product direction (perf course +vs. the on-prem commercial track) has been deferred across several sessions and is the actual blocker. + +--- + ## Agentic / interop / vision backlog (2026-06-21) Deferred ideas captured while shipping the XGBoost tabular predictor; ranked, on-moat, all build on existing From 1b442f2143a3c00de61b469829198079b9a902a6 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Tue, 21 Jul 2026 18:42:17 +0200 Subject: [PATCH 07/37] hybrid --- .../LanguageModels/Retrieval/Bm25Index.cs | 308 ++++++++++++++++++ .../Retrieval/Evaluation/RagEvaluator.cs | 71 +++- .../Retrieval/HybridRetriever.cs | 109 +++++++ .../Retrieval/ReciprocalRankFusion.cs | 102 ++++++ .../Retrieval/TopKMatchSelector.cs | 51 +++ Sources/Mcp/McpRagIndex.cs | 28 +- .../Retrieval/Bm25IndexTests.cs | 231 +++++++++++++ .../Retrieval/HybridRetrieverTests.cs | 155 +++++++++ .../HybridVsDenseOnDocsCorpusTests.cs | 302 +++++++++++++++++ .../Retrieval/HybridVsDenseRecallTests.cs | 219 +++++++++++++ .../Retrieval/ReciprocalRankFusionTests.cs | 126 +++++++ 11 files changed, 1682 insertions(+), 20 deletions(-) create mode 100644 Sources/Main/LanguageModels/Retrieval/Bm25Index.cs create mode 100644 Sources/Main/LanguageModels/Retrieval/HybridRetriever.cs create mode 100644 Sources/Main/LanguageModels/Retrieval/ReciprocalRankFusion.cs create mode 100644 Sources/Main/LanguageModels/Retrieval/TopKMatchSelector.cs create mode 100644 Tests/LanguageModels/Retrieval/Bm25IndexTests.cs create mode 100644 Tests/LanguageModels/Retrieval/HybridRetrieverTests.cs create mode 100644 Tests/LanguageModels/Retrieval/HybridVsDenseOnDocsCorpusTests.cs create mode 100644 Tests/LanguageModels/Retrieval/HybridVsDenseRecallTests.cs create mode 100644 Tests/LanguageModels/Retrieval/ReciprocalRankFusionTests.cs diff --git a/Sources/Main/LanguageModels/Retrieval/Bm25Index.cs b/Sources/Main/LanguageModels/Retrieval/Bm25Index.cs new file mode 100644 index 00000000..b758bd7c --- /dev/null +++ b/Sources/Main/LanguageModels/Retrieval/Bm25Index.cs @@ -0,0 +1,308 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.Tensors; + +namespace DevOnBike.Overfit.LanguageModels.Retrieval +{ + /// + /// Okapi BM25 lexical index — the keyword half of hybrid retrieval, and the half a + /// is structurally bad at. + /// + /// Why this exists. Dense embeddings match on meaning, which is exactly wrong for the + /// tokens enterprise corpora are full of: case numbers, policy numbers, part numbers, ICD codes, tax ids, + /// error codes. "II CSK 345/21" and "II CSK 346/21" are near-identical directions in embedding space and + /// completely different documents in reality. BM25 matches on the literal term, so it retrieves them + /// exactly. Fuse the two with and you get both. + /// + /// Pure algorithm: no second model, no extra weights, no GPU, no network — it is a dictionary and + /// some arithmetic, so it holds the engine's on-prem / Native-AOT / no-native-dependency identity. Search + /// accumulates into a pooled score buffer and does a top-K insertion pass rather than sorting the corpus. + /// + /// Linear scan over the postings of the query's terms — built for the thousands-to-low-millions of + /// chunks one document set produces, matching 's scale. Not thread-safe for + /// concurrent ; concurrent reads are fine once populated. + /// + public sealed class Bm25Index + { + /// Term-frequency saturation. Standard Okapi default; higher = term repetition keeps mattering longer. + public const float DefaultK1 = 1.2f; + + /// Document-length normalisation strength, in [0,1]. 0 = ignore length, 1 = fully normalise. + public const float DefaultB = 0.75f; + + private readonly Dictionary _termIds = new(StringComparer.Ordinal); + private readonly List> _postings = []; + private readonly List _ids = []; + private readonly List _payloads = []; + private readonly List _documentLengths = []; + private long _totalTokens; + + public Bm25Index(float k1 = DefaultK1, float b = DefaultB) + { + ArgumentOutOfRangeException.ThrowIfNegative(k1); + ArgumentOutOfRangeException.ThrowIfLessThan(b, 0f); + ArgumentOutOfRangeException.ThrowIfGreaterThan(b, 1f); + + K1 = k1; + B = b; + } + + /// Number of indexed documents. + public int Count => _ids.Count; + + /// Number of distinct terms seen across the corpus. + public int TermCount => _postings.Count; + + public float K1 + { + get; + } + + public float B + { + get; + } + + /// + /// Indexes under with an optional + /// (usually the source text itself, so a hit can be handed straight to a + /// prompt). Only term frequencies are retained — the text is not stored unless it is the payload. + /// + public void Add(string id, string text, string? payload = null) + { + ArgumentNullException.ThrowIfNull(id); + ArgumentNullException.ThrowIfNull(text); + + var documentIndex = _ids.Count; + var tokens = Tokenize(text); + + // Collapse to per-term counts first: one posting per (document, term), not per occurrence. + var counts = new Dictionary(); + for (var i = 0; i < tokens.Count; i++) + { + var termId = InternTerm(tokens[i]); + counts.TryGetValue(termId, out var current); + counts[termId] = current + 1; + } + + foreach (var pair in counts) + { + _postings[pair.Key].Add(new Posting(documentIndex, pair.Value)); + } + + _ids.Add(id); + _payloads.Add(payload); + _documentLengths.Add(tokens.Count); + _totalTokens += tokens.Count; + } + + /// + /// Fills with the best BM25 matches for , best + /// first, and returns how many were written. Documents scoring zero (no query term present) are never + /// returned, so this can write fewer than results.Length even on a large corpus. + /// + /// carries the BM25 score here, not a cosine — BM25 is + /// unbounded and corpus-relative, so it is meaningful for ranking and meaningless as an absolute + /// threshold. This is precisely why fuses ranks rather than + /// scores: the two arms never need their scales reconciled. + /// + public int Search(string query, Span results) + { + ArgumentNullException.ThrowIfNull(query); + + var k = results.Length; + if (k == 0 || Count == 0) + { + return 0; + } + + var queryTokens = Tokenize(query); + if (queryTokens.Count == 0) + { + return 0; + } + + using var scoreBuffer = new PooledBuffer(Count, clearMemory: true); + var scores = scoreBuffer.Span[..Count]; + + var averageLength = (float)((double)_totalTokens / Count); + var scored = new HashSet(); + + for (var t = 0; t < queryTokens.Count; t++) + { + if (!_termIds.TryGetValue(queryTokens[t], out var termId)) + { + continue; // term absent from the corpus — contributes nothing + } + + // A term repeated in the query must not count twice: BM25 saturates term frequency in the + // DOCUMENT, and query-side repetition is not evidence about the document. + if (!scored.Add(termId)) + { + continue; + } + + var postings = _postings[termId]; + var documentFrequency = postings.Count; + var idf = MathF.Log(1f + (Count - documentFrequency + 0.5f) / (documentFrequency + 0.5f)); + + for (var p = 0; p < postings.Count; p++) + { + var posting = postings[p]; + var termFrequency = (float)posting.Frequency; + var lengthNorm = 1f - B + (B * _documentLengths[posting.DocumentIndex] / averageLength); + scores[posting.DocumentIndex] += + idf * (termFrequency * (K1 + 1f)) / (termFrequency + (K1 * lengthNorm)); + } + } + + var found = 0; + for (var i = 0; i < Count; i++) + { + if (scores[i] <= 0f) + { + continue; // untouched by any query term — not a match at all + } + + TopKMatchSelector.InsertDescending( + results, ref found, k, new VectorMatch(_ids[i], scores[i], _payloads[i])); + } + + return found; + } + + /// Convenience overload: allocates and returns up to matches. + public VectorMatch[] Search(string query, int topK) + { + ArgumentOutOfRangeException.ThrowIfNegativeOrZero(topK); + + var capacity = Math.Min(topK, Count); + if (capacity == 0) + { + return []; + } + + var buffer = new VectorMatch[capacity]; + var written = Search(query, buffer); + return written == buffer.Length ? buffer : buffer[..written]; + } + + /// + /// Splits into lower-cased alphanumeric terms — the shared tokenisation used + /// for both indexing and querying, so the two can never drift apart. + /// + /// Deliberately Unicode-aware via rather than an ASCII + /// range: Polish (ą, ć, ę, ł, ń, ó, ś, ź, ż) and every other non-ASCII alphabet must survive + /// tokenisation, and an ASCII-only split would shred them into fragments. No stemming and no + /// stop-word list — both are language-specific, and getting them wrong costs more recall than the + /// index size they save. + /// + /// Connector-joined runs emit the whole form as an extra term. OVERFIT_DECODE_WORKERS + /// yields overfit, decode, workers AND overfit_decode_workers. Splitting + /// alone was measured to lose exactly this case: the three parts are among the commonest words in a + /// .NET corpus, so their IDF is near zero and the document defining the variable was outranked, while + /// block_q4_Kx8 survived only because q4/kx8 happen to stay rare. The joined form + /// is maximally rare, so it carries the IDF the parts cannot. Keeping the parts too means a query for + /// one component still matches. + /// + /// The joined term is emitted only when the run really has two or more parts — emitting it + /// for ordinary words would double their term frequency and corrupt both the TF saturation and the + /// document-length normalisation. + /// + public static List Tokenize(string text) + { + ArgumentNullException.ThrowIfNull(text); + + var tokens = new List(); + var i = 0; + + // Bounded by text.Length; every path through the body advances `i` at least once. + while (i < text.Length) + { + if (!char.IsLetterOrDigit(text[i])) + { + i++; + continue; + } + + var runStart = i; + var partStart = i; + var parts = 0; + + while (i < text.Length) + { + if (char.IsLetterOrDigit(text[i])) + { + i++; + continue; + } + + // A connector only continues the run when it sits BETWEEN two alphanumerics, so a trailing + // hyphen or an em-dash between words still terminates it. + var isConnector = (text[i] == '_' || text[i] == '-') + && i + 1 < text.Length + && char.IsLetterOrDigit(text[i + 1]); + + if (!isConnector) + { + break; + } + + tokens.Add(Lower(text, partStart, i - partStart)); + parts++; + i++; + partStart = i; + } + + tokens.Add(Lower(text, partStart, i - partStart)); + parts++; + + if (parts > 1) + { + tokens.Add(Lower(text, runStart, i - runStart)); + } + } + + return tokens; + } + + private static string Lower(string text, int start, int length) + => text.AsSpan(start, length).ToString().ToLowerInvariant(); + + private int InternTerm(string term) + { + if (_termIds.TryGetValue(term, out var existing)) + { + return existing; + } + + var termId = _postings.Count; + _termIds[term] = termId; + _postings.Add([]); + return termId; + } + + /// One (document, term-frequency) entry in a term's postings list. + private readonly struct Posting + { + public Posting(int documentIndex, int frequency) + { + DocumentIndex = documentIndex; + Frequency = frequency; + } + + public int DocumentIndex + { + get; + } + + public int Frequency + { + get; + } + } + } +} diff --git a/Sources/Main/LanguageModels/Retrieval/Evaluation/RagEvaluator.cs b/Sources/Main/LanguageModels/Retrieval/Evaluation/RagEvaluator.cs index e204b18d..3b978bac 100644 --- a/Sources/Main/LanguageModels/Retrieval/Evaluation/RagEvaluator.cs +++ b/Sources/Main/LanguageModels/Retrieval/Evaluation/RagEvaluator.cs @@ -22,17 +22,14 @@ namespace DevOnBike.Overfit.LanguageModels.Retrieval.Evaluation /// public sealed class RagEvaluator { - private readonly VectorStore _store; - private readonly Func _embedQuery; + private readonly Func _retrieve; + private readonly bool _scoresAreCosineSimilarity; /// Creates an evaluator over an indexed store and a query-embedding delegate (e.g. /// embedder.EmbedQuery, or a deterministic fake in a unit test). public RagEvaluator(VectorStore store, Func embedQuery) + : this(BuildDenseRetrieve(store, embedQuery), scoresAreCosineSimilarity: true) { - ArgumentNullException.ThrowIfNull(store); - ArgumentNullException.ThrowIfNull(embedQuery); - _store = store; - _embedQuery = embedQuery; } /// Convenience factory over a — uses EmbedQuery so the @@ -43,6 +40,46 @@ public static RagEvaluator ForEmbedder(VectorStore store, SentenceEmbedder embed return new RagEvaluator(store, embedder.EmbedQuery); } + /// + /// Evaluator over a — the same recall / paraphrase-stability checks, run + /// against dense+lexical fusion instead of dense alone. This is what makes "did hybrid actually help on + /// MY corpus?" a measurement rather than an assumption: build both evaluators over the same cases and + /// compare the reports. + /// + /// is not available on this path — see its remarks. + /// + public static RagEvaluator ForHybrid(HybridRetriever retriever, Func embedQuery) + { + ArgumentNullException.ThrowIfNull(retriever); + ArgumentNullException.ThrowIfNull(embedQuery); + + return new RagEvaluator( + (query, topK) => retriever.Search(embedQuery(query), query, topK), + scoresAreCosineSimilarity: false); + } + + /// Convenience factory pairing a with a . + public static RagEvaluator ForHybrid(HybridRetriever retriever, SentenceEmbedder embedder) + { + ArgumentNullException.ThrowIfNull(embedder); + return ForHybrid(retriever, embedder.EmbedQuery); + } + + private RagEvaluator(Func retrieve, bool scoresAreCosineSimilarity) + { + _retrieve = retrieve; + _scoresAreCosineSimilarity = scoresAreCosineSimilarity; + } + + // Null-checked here rather than in the constructor body, because `: this(...)` runs first. + private static Func BuildDenseRetrieve( + VectorStore store, Func embedQuery) + { + ArgumentNullException.ThrowIfNull(store); + ArgumentNullException.ThrowIfNull(embedQuery); + return (query, topK) => store.Search(embedQuery(query), topK); + } + /// Runs every and reports recall@K + MRR + per-case ranks. public RetrievalReport EvaluateRetrieval(IEnumerable cases, int topK = 5) { @@ -117,15 +154,31 @@ public ParaphraseStabilityReport EvaluateParaphraseStability( /// Runs every and flags those whose top match clears /// (a sprung trap — the corpus offered a confident source for an - /// un-grounded question). + /// un-grounded question). + /// + /// Requires a dense evaluator: the threshold is compared against a cosine similarity, which + /// has a fixed, corpus-independent scale. Fused hybrid scores are derived from ranks, so every query + /// produces a top score of roughly the same magnitude whether or not the corpus actually contains an + /// answer — exactly the signal this check depends on. Calling it on a + /// evaluator throws rather than + /// returning a number that looks fine and means nothing. + /// public FalsePremiseReport EvaluateFalsePremise(IEnumerable cases, double groundedThreshold = 0.5) { ArgumentNullException.ThrowIfNull(cases); + if (!_scoresAreCosineSimilarity) + { + throw new OverfitRuntimeException( + "False-premise evaluation compares the top retrieval score against a cosine threshold, so it " + + "requires a dense evaluator. Reciprocal-rank-fused scores have no absolute scale — build a " + + "RagEvaluator over the HybridRetriever's Vectors arm for this check."); + } + var results = new List(); foreach (var c in cases) { - var matches = _store.Search(_embedQuery(c.Query), 1); + var matches = _retrieve(c.Query, 1); string? topId = null; var topScore = 0f; if (matches.Length > 0) @@ -142,7 +195,7 @@ public FalsePremiseReport EvaluateFalsePremise(IEnumerable cas private string[] RetrieveIds(string query, int topK) { - var matches = _store.Search(_embedQuery(query), topK); + var matches = _retrieve(query, topK); var ids = new string[matches.Length]; for (var i = 0; i < matches.Length; i++) { diff --git a/Sources/Main/LanguageModels/Retrieval/HybridRetriever.cs b/Sources/Main/LanguageModels/Retrieval/HybridRetriever.cs new file mode 100644 index 00000000..c1d2e688 --- /dev/null +++ b/Sources/Main/LanguageModels/Retrieval/HybridRetriever.cs @@ -0,0 +1,109 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +namespace DevOnBike.Overfit.LanguageModels.Retrieval +{ + /// + /// Hybrid retrieval: a (semantic) and a (lexical) kept + /// in lock-step over one corpus, queried together and merged with . + /// + /// The two arms fail in opposite directions, which is the entire point. Dense search finds + /// "how do I cancel my policy" in a chunk that says "termination of cover" and never uses the word + /// cancel. Lexical search finds policy number PL-88-40021, which dense search cannot distinguish + /// from PL-88-40022. Neither is a superset of the other, so a corpus containing both prose and + /// identifiers — i.e. essentially every real enterprise corpus — needs both. + /// + /// Both indexes are populated through this type's , so they cannot drift out of + /// sync. If you need one arm alone, and expose them + /// directly — useful for measuring what hybrid actually bought you on your own corpus rather than + /// assuming it helped. + /// + public sealed class HybridRetriever + { + private readonly VectorStore _vectors; + private readonly Bm25Index _lexical; + + public HybridRetriever(int dimension, int initialCapacity = 16) + { + _vectors = new VectorStore(dimension, initialCapacity); + _lexical = new Bm25Index(); + } + + /// Wraps an existing pair of indexes (e.g. a reloaded from disk). + /// The caller is responsible for them describing the same corpus. + public HybridRetriever(VectorStore vectors, Bm25Index lexical) + { + ArgumentNullException.ThrowIfNull(vectors); + ArgumentNullException.ThrowIfNull(lexical); + + _vectors = vectors; + _lexical = lexical; + } + + /// The semantic arm. + public VectorStore Vectors => _vectors; + + /// The lexical arm. + public Bm25Index Lexical => _lexical; + + /// Number of indexed chunks. + public int Count => _vectors.Count; + + /// + /// Indexes one chunk into both arms: into the vector store and + /// into the BM25 index. defaults to + /// , since a retrieved chunk almost always needs its text to build the prompt. + /// + public void Add(string id, ReadOnlySpan vector, string text, string? payload = null) + { + ArgumentNullException.ThrowIfNull(id); + ArgumentNullException.ThrowIfNull(text); + + var effectivePayload = payload ?? text; + _vectors.Add(id, vector, effectivePayload); + _lexical.Add(id, text, effectivePayload); + } + + /// + /// Retrieves the top- chunks for a query given both its embedding and its raw + /// text. + /// + /// is the depth each arm is asked for before fusion; it + /// defaults to max(4·topK, 20). Fusion needs headroom to work — if each arm returns only + /// topK, a document ranked just outside both lists can never be promoted by agreement, which + /// is the effect hybrid retrieval exists to capture. Deeper costs almost nothing here because both + /// arms already scan the corpus; only the merge grows. + /// + public VectorMatch[] Search( + ReadOnlySpan queryVector, + string queryText, + int topK, + int candidatesPerArm = 0) + { + ArgumentNullException.ThrowIfNull(queryText); + ArgumentOutOfRangeException.ThrowIfNegativeOrZero(topK); + + if (Count == 0) + { + return []; + } + + var depth = candidatesPerArm > 0 ? candidatesPerArm : Math.Max(4 * topK, 20); + depth = Math.Min(depth, Count); + + var dense = new VectorMatch[depth]; + var denseCount = _vectors.Search(queryVector, dense); + + var lexical = new VectorMatch[depth]; + var lexicalCount = _lexical.Search(queryText, lexical); + + var results = new VectorMatch[Math.Min(topK, Count)]; + var written = ReciprocalRankFusion.Fuse( + dense.AsSpan(0, denseCount), lexical.AsSpan(0, lexicalCount), results); + + return written == results.Length ? results : results[..written]; + } + } +} diff --git a/Sources/Main/LanguageModels/Retrieval/ReciprocalRankFusion.cs b/Sources/Main/LanguageModels/Retrieval/ReciprocalRankFusion.cs new file mode 100644 index 00000000..689ebe84 --- /dev/null +++ b/Sources/Main/LanguageModels/Retrieval/ReciprocalRankFusion.cs @@ -0,0 +1,102 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +namespace DevOnBike.Overfit.LanguageModels.Retrieval +{ + /// + /// Reciprocal Rank Fusion — merges several ranked result lists into one by summing 1 / (k + rank) + /// per document. + /// + /// Why rank fusion and not score fusion. The two retrieval arms produce incomparable numbers: + /// returns a cosine in [-1,1], returns an unbounded, + /// corpus-relative BM25 score. Normalising them onto a common scale requires knowing each arm's score + /// distribution, which shifts with the corpus and the query — a tuning knob that silently rots. Ranks have + /// no such problem: position 1 means the same thing in both lists, forever. That is the whole reason RRF + /// is the default fusion in practice despite being almost trivially simple. + /// + /// The constant damps the top of each list: without it the rank-1 document of + /// a single arm would dominate every fusion, so one confident-but-wrong arm could not be outvoted. At + /// k=60 the gap between rank 1 and rank 2 is small enough that agreement across arms outranks depth + /// within one arm — which is the behaviour hybrid retrieval is bought for. + /// + public static class ReciprocalRankFusion + { + /// Rank-damping constant. 60 is the value from the original RRF paper and the de-facto default. + public const float DefaultK = 60f; + + /// + /// Fuses two ranked lists (each already best-first) into , best first, and + /// returns how many were written. A document present in both lists accumulates both contributions, + /// which is what lets agreement beat depth. + /// + /// on the output is the fused RRF score — a small positive + /// number with no meaning outside this comparison. Payloads are carried over from whichever input + /// supplied a non-null one. + /// + public static int Fuse( + ReadOnlySpan first, + ReadOnlySpan second, + Span results, + float k = DefaultK) + { + ArgumentOutOfRangeException.ThrowIfNegativeOrZero(k); + + if (results.Length == 0) + { + return 0; + } + + var scores = new Dictionary(StringComparer.Ordinal); + var payloads = new Dictionary(StringComparer.Ordinal); + + Accumulate(first, scores, payloads, k); + Accumulate(second, scores, payloads, k); + + var found = 0; + foreach (var pair in scores) + { + payloads.TryGetValue(pair.Key, out var payload); + TopKMatchSelector.InsertDescending( + results, ref found, results.Length, new VectorMatch(pair.Key, pair.Value, payload)); + } + + return found; + } + + /// Convenience overload: allocates and returns up to fused matches. + public static VectorMatch[] Fuse( + ReadOnlySpan first, + ReadOnlySpan second, + int topK, + float k = DefaultK) + { + ArgumentOutOfRangeException.ThrowIfNegativeOrZero(topK); + + var buffer = new VectorMatch[topK]; + var written = Fuse(first, second, buffer, k); + return written == buffer.Length ? buffer : buffer[..written]; + } + + private static void Accumulate( + ReadOnlySpan ranked, + Dictionary scores, + Dictionary payloads, + float k) + { + for (var rank = 0; rank < ranked.Length; rank++) + { + var id = ranked[rank].Id; + + scores.TryGetValue(id, out var current); + scores[id] = current + (1f / (k + rank + 1f)); // rank is 0-based here, 1-based in the formula + + if (ranked[rank].Payload is not null) + { + payloads[id] = ranked[rank].Payload; + } + } + } + } +} diff --git a/Sources/Main/LanguageModels/Retrieval/TopKMatchSelector.cs b/Sources/Main/LanguageModels/Retrieval/TopKMatchSelector.cs new file mode 100644 index 00000000..5b5cb9dd --- /dev/null +++ b/Sources/Main/LanguageModels/Retrieval/TopKMatchSelector.cs @@ -0,0 +1,51 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +namespace DevOnBike.Overfit.LanguageModels.Retrieval +{ + /// + /// Shared top-K insertion for retrieval result spans: keeps the best k candidates in descending + /// score order without sorting the corpus and without allocating. Candidates arrive in arbitrary order + /// and are shifted into place, so a full scan costs O(n·k) worst case but O(n) once the list is warm and + /// most candidates are rejected by the first comparison. + /// + /// keeps its own private copy of this routine — it predates the shared + /// helper and is covered by its own tests, so it was deliberately left untouched rather than refactored + /// for tidiness alone. + /// + internal static class TopKMatchSelector + { + /// + /// Offers to the descending top- list held in + /// , updating (the number of populated slots). + /// + internal static void InsertDescending( + Span results, + ref int found, + int k, + in VectorMatch candidate) + { + // Reject early when the list is full and the candidate cannot beat the current worst. + if (found == k && candidate.Score <= results[k - 1].Score) + { + return; + } + + var position = found < k ? found : k - 1; + while (position > 0 && results[position - 1].Score < candidate.Score) + { + results[position] = results[position - 1]; + position--; + } + + results[position] = candidate; + + if (found < k) + { + found++; + } + } + } +} diff --git a/Sources/Mcp/McpRagIndex.cs b/Sources/Mcp/McpRagIndex.cs index 2cb41879..65d629cb 100644 --- a/Sources/Mcp/McpRagIndex.cs +++ b/Sources/Mcp/McpRagIndex.cs @@ -14,22 +14,28 @@ namespace DevOnBike.Overfit.Mcp /// A self-contained RAG index over a local document folder for the rag_query MCP tool: /// chunks .txt/.md files on paragraph boundaries, embeds every chunk with the chat /// model's OWN embeddings ( — multilingual, no second model - /// needed) into an in-process , then answers questions grounded in the - /// top-K chunks with per-chunk source citations. Everything stays on the machine. + /// needed), then answers questions grounded in the top-K chunks with per-chunk source citations. + /// Everything stays on the machine. + /// + /// Retrieval is hybrid (): semantic search over the embeddings + /// fused with BM25 over the chunk text. Measured on this repository's own docs/ folder (481 chunks, + /// MiniLM, recall@5) that lifted recall from 0.61 to 0.89 and MRR from 0.500 to 0.736 — + /// see HybridVsDenseOnDocsCorpusTests. Local document sets are full of literal tokens (file names, + /// env vars, error codes, API names) that embeddings blur together and BM25 matches exactly. /// public sealed class McpRagIndex { private const int TargetChunkChars = 1200; private readonly OverfitClient _client; - private readonly VectorStore _store; + private readonly HybridRetriever _retriever; - public int ChunkCount => _store.Count; + public int ChunkCount => _retriever.Count; - private McpRagIndex(OverfitClient client, VectorStore store) + private McpRagIndex(OverfitClient client, HybridRetriever retriever) { _client = client; - _store = store; + _retriever = retriever; } /// @@ -45,7 +51,7 @@ public static McpRagIndex Build(OverfitClient client, string directory, TextWrit throw new DirectoryNotFoundException($"RAG document directory not found: {directory}"); } - var store = new VectorStore(client.EmbeddingDimension); + var retriever = new HybridRetriever(client.EmbeddingDimension); var files = new List(); files.AddRange(Directory.GetFiles(directory, "*.txt", SearchOption.AllDirectories)); files.AddRange(Directory.GetFiles(directory, "*.md", SearchOption.AllDirectories)); @@ -59,18 +65,18 @@ public static McpRagIndex Build(OverfitClient client, string directory, TextWrit for (var i = 0; i < chunks.Count; i++) { var vector = client.Embed(chunks[i]); - store.Add($"{name}#{i + 1}", vector, chunks[i]); + retriever.Add($"{name}#{i + 1}", vector, chunks[i]); } log?.WriteLine($"[overfit-mcp] indexed {name}: {chunks.Count} chunk(s)"); } - if (store.Count == 0) + if (retriever.Count == 0) { throw new OverfitRuntimeException($"No indexable .txt/.md content found under: {directory}"); } - return new McpRagIndex(client, store); + return new McpRagIndex(client, retriever); } /// @@ -84,7 +90,7 @@ public string Query(string question, int topK = 4) ArgumentException.ThrowIfNullOrEmpty(question); var queryVector = _client.Embed(question); - var matches = _store.Search(queryVector, Math.Min(topK, _store.Count)); + var matches = _retriever.Search(queryVector, question, Math.Min(topK, _retriever.Count)); var prompt = new StringBuilder(4096); prompt.AppendLine("Answer the question using ONLY the context below. Cite the context entries you used as [1], [2], … . If the context does not contain the answer, say so plainly."); diff --git a/Tests/LanguageModels/Retrieval/Bm25IndexTests.cs b/Tests/LanguageModels/Retrieval/Bm25IndexTests.cs new file mode 100644 index 00000000..4ed9a17d --- /dev/null +++ b/Tests/LanguageModels/Retrieval/Bm25IndexTests.cs @@ -0,0 +1,231 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.LanguageModels.Retrieval; + +namespace DevOnBike.Overfit.Tests.LanguageModels.Retrieval +{ + /// + /// Pins the BM25 lexical arm: the ranking behaviour that makes it worth having next to a + /// (exact-term retrieval, IDF, length normalisation) and the tokenisation it + /// shares between indexing and querying. + /// + public sealed class Bm25IndexTests + { + [Fact] + public void Search_FindsDocumentContainingTheExactTerm() + { + var index = new Bm25Index(); + index.Add("a", "the cat sat on the mat"); + index.Add("b", "a dog barked loudly"); + index.Add("c", "birds are singing"); + + var hits = index.Search("dog", 3); + + Assert.Single(hits); + Assert.Equal("b", hits[0].Id); + } + + [Fact] + public void Search_DoesNotReturnDocumentsWithoutAnyQueryTerm() + { + var index = new Bm25Index(); + index.Add("a", "alpha beta"); + index.Add("b", "gamma delta"); + + // Asking for 10 must not pad the result with zero-scoring documents. + var hits = index.Search("alpha", 10); + + Assert.Single(hits); + Assert.Equal("a", hits[0].Id); + } + + [Fact] + public void Search_UnknownTerm_ReturnsNothing() + { + var index = new Bm25Index(); + index.Add("a", "alpha beta"); + + Assert.Empty(index.Search("omega", 5)); + } + + [Fact] + public void Search_RareTermOutranksCommonTerm() + { + var index = new Bm25Index(); + // "common" appears everywhere (low IDF); "rare" appears once (high IDF). + index.Add("a", "common word here"); + index.Add("b", "common word there"); + index.Add("c", "common rare word"); + index.Add("d", "common word everywhere"); + + var hits = index.Search("common rare", 4); + + Assert.Equal("c", hits[0].Id); + } + + [Fact] + public void Search_ShorterDocumentOutranksLongerOne_ForTheSameTermCount() + { + var index = new Bm25Index(); + index.Add("short", "needle"); + index.Add("long", "needle " + string.Join(' ', Enumerable.Repeat("filler", 200))); + + var hits = index.Search("needle", 2); + + // Both contain "needle" exactly once; length normalisation (b > 0) must favour the short one. + Assert.Equal(2, hits.Length); + Assert.Equal("short", hits[0].Id); + } + + [Fact] + public void Search_LengthNormalisationDisabled_TreatsBothLengthsAlike() + { + var index = new Bm25Index(b: 0f); + index.Add("short", "needle"); + index.Add("long", "needle " + string.Join(' ', Enumerable.Repeat("filler", 200))); + + var hits = index.Search("needle", 2); + + // With b = 0 the two documents differ only by length, which is now ignored → identical scores. + Assert.Equal(hits[0].Score, hits[1].Score, 5); + } + + [Fact] + public void Search_RepeatedQueryTerm_DoesNotDoubleCount() + { + var index = new Bm25Index(); + index.Add("a", "alpha beta gamma"); + index.Add("b", "alpha delta"); + + var once = index.Search("alpha", 2); + var twice = index.Search("alpha alpha alpha", 2); + + Assert.Equal(once.Length, twice.Length); + for (var i = 0; i < once.Length; i++) + { + Assert.Equal(once[i].Id, twice[i].Id); + Assert.Equal(once[i].Score, twice[i].Score, 5); + } + } + + [Fact] + public void Search_CarriesThePayload() + { + var index = new Bm25Index(); + index.Add("a", "alpha beta", payload: "the original text"); + + var hits = index.Search("beta", 1); + + Assert.Equal("the original text", hits[0].Payload); + } + + [Fact] + public void Search_EmptyIndexOrEmptyQuery_ReturnsNothing() + { + var empty = new Bm25Index(); + Assert.Empty(empty.Search("anything", 5)); + + var index = new Bm25Index(); + index.Add("a", "alpha"); + Assert.Empty(index.Search(" ... ", 5)); + } + + [Fact] + public void Search_SpanOverload_ReportsHowManyItWrote() + { + var index = new Bm25Index(); + index.Add("a", "alpha"); + index.Add("b", "beta"); + + Span buffer = new VectorMatch[5]; + var written = index.Search("alpha", buffer); + + Assert.Equal(1, written); + Assert.Equal("a", buffer[0].Id); + } + + [Fact] + public void Tokenize_LowercasesAndSplitsOnPunctuation() + { + Assert.Equal(["hello", "world", "42"], Bm25Index.Tokenize("Hello, WORLD! (42)")); + } + + [Fact] + public void Tokenize_PreservesPolishDiacritics() + { + // An ASCII-range split would shred these into fragments and silently wreck recall on Polish text. + Assert.Equal(["zażółć", "gęślą", "jaźń"], Bm25Index.Tokenize("Zażółć gęślą jaźń!")); + } + + [Fact] + public void Tokenize_EmitsIdentifierPartsAndTheJoinedForm() + { + // Parts keep component queries working; the joined form carries the IDF that common parts cannot. + Assert.Equal(["pl", "88", "40021", "pl-88-40021"], Bm25Index.Tokenize("PL-88-40021")); + Assert.Equal( + ["overfit", "decode", "workers", "overfit_decode_workers"], + Bm25Index.Tokenize("OVERFIT_DECODE_WORKERS")); + } + + [Fact] + public void Tokenize_DoesNotEmitAJoinedFormForOrdinaryWords() + { + // Emitting a duplicate for single-part words would double their term frequency and corrupt both + // TF saturation and length normalisation. + Assert.Equal(["hello", "world"], Bm25Index.Tokenize("hello world")); + } + + [Fact] + public void Tokenize_ConnectorMustSitBetweenTwoAlphanumerics() + { + // Trailing/leading connectors and dashes used as punctuation must not glue terms together. + Assert.Equal(["abc"], Bm25Index.Tokenize("abc-")); + Assert.Equal(["abc"], Bm25Index.Tokenize("-abc")); + Assert.Equal(["one", "two"], Bm25Index.Tokenize("one -- two")); + } + + [Fact] + public void Search_FindsAnIdentifierWhosePartsAreAllCommonWords() + { + // The regression this tokenisation change was built for: every part is common, so only the joined + // form can rank the defining document first. + var index = new Bm25Index(); + index.Add("defines", "Set OVERFIT_DECODE_WORKERS to cap the decode worker count."); + index.Add("noise-1", "The decode path spawns workers for each overfit projection."); + index.Add("noise-2", "Overfit workers decode tokens; decode workers are capped."); + index.Add("noise-3", "Workers decode. Overfit decode workers overfit decode."); + + var hits = index.Search("OVERFIT_DECODE_WORKERS", 4); + + Assert.Equal("defines", hits[0].Id); + } + + [Fact] + public void Tokenize_EmptyAndSymbolOnlyInput_YieldsNoTerms() + { + Assert.Empty(Bm25Index.Tokenize(string.Empty)); + Assert.Empty(Bm25Index.Tokenize("--- !!! ---")); + } + + [Fact] + public void CountsReflectTheIndexedCorpus() + { + var index = new Bm25Index(); + index.Add("a", "alpha beta"); + index.Add("b", "beta gamma"); + + Assert.Equal(2, index.Count); + Assert.Equal(3, index.TermCount); // alpha, beta, gamma + } + + [Fact] + public void Constructor_RejectsOutOfRangeB() + { + Assert.Throws(() => new Bm25Index(b: -0.1f)); + Assert.Throws(() => new Bm25Index(b: 1.1f)); + } + } +} diff --git a/Tests/LanguageModels/Retrieval/HybridRetrieverTests.cs b/Tests/LanguageModels/Retrieval/HybridRetrieverTests.cs new file mode 100644 index 00000000..f9fe0c9b --- /dev/null +++ b/Tests/LanguageModels/Retrieval/HybridRetrieverTests.cs @@ -0,0 +1,155 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.LanguageModels.Retrieval; + +namespace DevOnBike.Overfit.Tests.LanguageModels.Retrieval +{ + /// + /// Pins hybrid retrieval end-to-end, and — in — + /// demonstrates the failure it was built to fix, rather than merely asserting the plumbing works. + /// + /// Embeddings here are hand-written rather than produced by a model, so the test is deterministic, + /// runs in microseconds and needs no fixture. They are shaped to reproduce the real-world case: several + /// chunks that are near-identical in meaning, exactly one of which carries the identifier the user + /// actually asked for. + /// + public sealed class HybridRetrieverTests + { + [Fact] + public void Search_FindsTheExactIdentifier_WhereDenseSearchAlone() + { + var retriever = new HybridRetriever(dimension: 3); + + // Three chunks on the same topic — cosine-wise almost indistinguishable. Only "target" carries the + // policy number, and its vector is deliberately the FURTHEST from the query, so dense search ranks + // it last. This is the everyday enterprise case: the identifier carries the meaning, the prose does not. + retriever.Add("chatter-a", [1f, 0.02f, 0f], "Termination of cover requires written notice."); + retriever.Add("chatter-b", [1f, 0.01f, 0f], "Cover may be terminated by the insurer at any time."); + retriever.Add("target", [1f, 0.00f, 0f], "Policy PL-88-40021 termination schedule and fees."); + + float[] queryVector = [1f, 0.02f, 0f]; + const string queryText = "PL-88-40021"; + + // Dense arm alone: ranks by vector proximity, so the document the user asked for comes LAST. + var denseOnly = retriever.Vectors.Search(queryVector, 3); + Assert.Equal("chatter-a", denseOnly[0].Id); + Assert.Equal("target", denseOnly[2].Id); + + // Lexical arm alone: the identifier is unique, so it is the only hit. + var lexicalOnly = retriever.Lexical.Search(queryText, 3); + Assert.Equal("target", lexicalOnly[0].Id); + + // Hybrid: the lexical rank-1 outweighs the dense rank-3 and the answer surfaces. + var hybrid = retriever.Search(queryVector, queryText, topK: 3); + Assert.Equal("target", hybrid[0].Id); + } + + [Fact] + public void Search_StillAnswersPurelySemanticQueries() + { + var retriever = new HybridRetriever(dimension: 3); + + // The complement of the test above: no shared vocabulary at all between query and answer, so only + // the dense arm can find it. Hybrid must not have broken that. + retriever.Add("cancel", [1f, 0f, 0f], "How to terminate your cover before renewal."); + retriever.Add("claims", [0f, 1f, 0f], "Submitting a claim after an accident."); + retriever.Add("billing", [0f, 0f, 1f], "Monthly instalment schedule."); + + var hybrid = retriever.Search([1f, 0f, 0f], "cancelling my policy", topK: 3); + + Assert.Equal("cancel", hybrid[0].Id); + } + + [Fact] + public void Add_KeepsBothArmsInSync() + { + var retriever = new HybridRetriever(dimension: 2); + retriever.Add("a", [1f, 0f], "alpha"); + retriever.Add("b", [0f, 1f], "beta"); + + Assert.Equal(2, retriever.Count); + Assert.Equal(2, retriever.Vectors.Count); + Assert.Equal(2, retriever.Lexical.Count); + } + + [Fact] + public void Add_DefaultsThePayloadToTheIndexedText() + { + var retriever = new HybridRetriever(dimension: 2); + retriever.Add("a", [1f, 0f], "the chunk body"); + + var hits = retriever.Search([1f, 0f], "chunk", topK: 1); + + Assert.Equal("the chunk body", hits[0].Payload); + } + + [Fact] + public void Add_ExplicitPayloadOverridesTheText() + { + var retriever = new HybridRetriever(dimension: 2); + retriever.Add("a", [1f, 0f], "searchable text", payload: "displayed text"); + + var hits = retriever.Search([1f, 0f], "searchable", topK: 1); + + Assert.Equal("displayed text", hits[0].Payload); + } + + [Fact] + public void Search_EmptyCorpus_ReturnsNothing() + { + var retriever = new HybridRetriever(dimension: 2); + Assert.Empty(retriever.Search([1f, 0f], "anything", topK: 5)); + } + + [Fact] + public void Search_NeverReturnsMoreThanTheCorpusHolds() + { + var retriever = new HybridRetriever(dimension: 2); + retriever.Add("a", [1f, 0f], "alpha"); + + Assert.Single(retriever.Search([1f, 0f], "alpha", topK: 10)); + } + + [Fact] + public void Search_DeeperCandidatePool_CanPromoteADocumentBothArmsRankLow() + { + var retriever = new HybridRetriever(dimension: 2); + + // "agreed" is mid-ranked by BOTH arms; the others are top of exactly one. With a shallow pool it + // is never seen by fusion, with a deep one its agreement wins — the reason the default pool is 4x topK. + retriever.Add("dense-top", [1f, 0f], "unrelated wording entirely"); + retriever.Add("agreed", [0.9f, 0.1f], "shared keyword here"); + retriever.Add("lexical-top", [0f, 1f], "shared keyword shared keyword shared keyword"); + + var deep = retriever.Search([1f, 0f], "shared keyword", topK: 3, candidatesPerArm: 3); + + Assert.Contains(deep, m => m.Id == "agreed"); + } + + [Fact] + public void Constructor_WrappingExistingIndexes_UsesThem() + { + var vectors = new VectorStore(2); + var lexical = new Bm25Index(); + vectors.Add("a", [1f, 0f], "payload"); + lexical.Add("a", "alpha", "payload"); + + var retriever = new HybridRetriever(vectors, lexical); + + Assert.Equal(1, retriever.Count); + Assert.Equal("a", retriever.Search([1f, 0f], "alpha", topK: 1)[0].Id); + } + + [Fact] + public void Search_RejectsInvalidTopK() + { + var retriever = new HybridRetriever(dimension: 2); + retriever.Add("a", [1f, 0f], "alpha"); + + Assert.Throws(() => retriever.Search([1f, 0f], "alpha", topK: 0)); + } + } +} diff --git a/Tests/LanguageModels/Retrieval/HybridVsDenseOnDocsCorpusTests.cs b/Tests/LanguageModels/Retrieval/HybridVsDenseOnDocsCorpusTests.cs new file mode 100644 index 00000000..7c8915c4 --- /dev/null +++ b/Tests/LanguageModels/Retrieval/HybridVsDenseOnDocsCorpusTests.cs @@ -0,0 +1,302 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Text; +using DevOnBike.Overfit.LanguageModels.Embeddings; +using DevOnBike.Overfit.LanguageModels.Retrieval; +using DevOnBike.Overfit.LanguageModels.Retrieval.Evaluation; +using DevOnBike.Overfit.Tests.TestSupport; +using Xunit.Abstractions; + +namespace DevOnBike.Overfit.Tests.LanguageModels.Retrieval +{ + /// + /// The decisive hybrid-vs-dense measurement, on a real corpus: this repository's own + /// docs/ folder (~30 markdown files, ~500 KB), chunked exactly the way McpRagIndex chunks a + /// user's document folder in production. + /// + /// Why this test exists. measured hybrid as a net + /// regression on an 18-chunk synthetic corpus, but that result was not trustworthy: BM25's IDF cannot + /// discriminate on a corpus that small, so function words like "how" earned a high IDF and dragged the + /// wrong document to rank 1. Several hundred chunks is the smallest honest test of the technique — the + /// scaffolding has to be bigger than the effect being measured. + /// + /// Ground truth is file-level and deliberately strict: a case is a hit only if a chunk of the + /// document that actually covers the topic appears in the top-K. The corpus also contains three + /// cross-cutting summary documents (claim-to-test, release-infographic, use-cases-2026) + /// that mention nearly every feature in passing and therefore act as genuine distractors. That makes the + /// absolute numbers pessimistic — but it biases both arms identically, which is all an A/B + /// requires. + /// + /// MEASURED RESULT (2026-07-21, MiniLM, 45 files / 481 chunks, recall@5). This is what put + /// McpRagIndex onto hybrid retrieval: + /// + /// group dense R@K hybrid R@K dense MRR hybrid MRR + /// semantic 0.67 0.83 0.417 0.542 + /// identifier 0.33 0.83 0.333 0.667 + /// mixed 0.83 1.00 0.750 1.000 + /// OVERALL 0.61 0.89 0.500 0.736 + /// + /// + /// Hybrid won every group — including the semantic one, which the small-corpus run had shown it + /// damaging. Two cases still fail and both are informative rather than mysterious: + /// + /// OVERFIT_DECODE_WORKERS is missed by BOTH arms. split it + /// into overfit + decode + workers — three of the commonest words in this corpus. + /// The joined-identifier fix was then built and measured, and it is a mechanism win with zero + /// end-to-end effect. The lexical arm moved exactly as designed — docker.md#2, the document + /// that defines the variable, went from lexical rank 2 (score 8.21) to rank 1 (13.15) — yet every + /// number in the table above stayed bit-identical, because fusion, not tokenisation, is now the binding + /// constraint: a hit present in one arm at rank 1 scores 1/61 ≈ 0.016, while a document present in both + /// arms at ranks 5 and 3 scores 1/65 + 1/63 ≈ 0.031. Agreement outweighing a single confident arm is + /// RRF working as designed, and it is wrong for a unique identifier. The next hypothesis to measure + /// is therefore fusion weighting (or an exact-identifier short-circuit), NOT more tokenisation + /// work. + /// "teach a model my own private data without a graphics card" regressed from dense rank 1 to + /// missed, because the lexical arm confidently returned unrelated chunks on common words. This is the + /// same function-word weakness the small-corpus test isolated; it is now rare rather than + /// systematic. + /// + /// + public sealed class HybridVsDenseOnDocsCorpusTests + { + private readonly ITestOutputHelper _out; + + public HybridVsDenseOnDocsCorpusTests(ITestOutputHelper output) => _out = output; + + private sealed record Case(string Group, string Query, params string[] ExpectedFiles); + + // Literal-token lookups. Every token below was verified to occur in EXACTLY ONE file of the corpus, + // so the expected answer is not a matter of opinion. This is the group BM25 should win. + private static readonly Case[] Identifier = + [ + new("identifier", "vpmaddubsw", "llamacpp-cpu-analysis.md"), + new("identifier", "block_q4_Kx8", "llamacpp-cpu-analysis.md"), + new("identifier", "OVERFIT_DECODE_WORKERS", "docker.md"), + new("identifier", "PESEL", "redaction-gateway-spec.md"), + new("identifier", "AdamEpsilon", "qlora-finetuning.md"), + new("identifier", "Luhn", "redaction-gateway-spec.md"), + ]; + + // Paraphrases that deliberately avoid the target document's own vocabulary, so only the semantic arm + // can reach them. This is the group hybrid can damage. + private static readonly Case[] Semantic = + [ + new("semantic", "how can I teach a model my own private data without a graphics card?", "qlora-finetuning.md"), + new("semantic", "can I replace Ollama in an app I already wrote?", "microsoft-extensions-ai.md"), + new("semantic", "how do I make a synthetic voice that sounds like one particular person?", "voice-cloning.md"), + new("semantic", "which language models is this able to open?", "supported-models.md"), + new("semantic", "how do I stop my document search from silently getting worse?", "rag-testing.md"), + new("semantic", "how do I run this inside a container?", "docker.md"), + ]; + + // Everyday questions with partial lexical overlap — neither arm is obviously right. + private static readonly Case[] Mixed = + [ + new("mixed", "how do I plug local AI into Claude Code?", "mcp.md"), + new("mixed", "why is decode slower than llama.cpp?", "overfit_perf_decode_analysis.md", "llamacpp-cpu-analysis.md"), + new("mixed", "is the Polish model any good?", "bielik.md"), + new("mixed", "how do I decode an MP3 in pure C#?", "mp3-decoding.md"), + new("mixed", "what does the serving benchmark measure?", "serving-benchmark.md"), + new("mixed", "how do I evaluate prompts locally for free?", "skill-eval.md"), + ]; + + [LocalOnlyFact] + public void Hybrid_VsDense_OnRealDocsCorpus() + { + if (!File.Exists(TestModelPaths.MiniLm.SafetensorsPath)) + { + _out.WriteLine($"missing MiniLM fixture at {TestModelPaths.MiniLm.Dir}"); + return; + } + + var docsDirectory = FindDocsDirectory(); + if (docsDirectory is null) + { + _out.WriteLine("could not locate the repository docs/ folder from the test output directory"); + return; + } + + using var embedder = SentenceEmbedder.ForMiniLm(TestModelPaths.MiniLm.Dir); + + var hybrid = new HybridRetriever(embedder.Dimension, 512); + var chunkIdsByFile = new Dictionary>(StringComparer.OrdinalIgnoreCase); + + var files = Directory.GetFiles(docsDirectory, "*.md", SearchOption.AllDirectories); + Array.Sort(files, StringComparer.OrdinalIgnoreCase); + + foreach (var file in files) + { + var name = Path.GetFileName(file); + var chunks = ChunkParagraphs(File.ReadAllText(file)); + var ids = new List(); + + for (var i = 0; i < chunks.Count; i++) + { + var id = $"{name}#{i + 1}"; + hybrid.Add(id, embedder.Embed(chunks[i]), chunks[i]); + ids.Add(id); + } + + chunkIdsByFile[name] = ids; + } + + _out.WriteLine($"=== Hybrid vs dense on docs/ — {files.Length} files, {hybrid.Count} chunks, MiniLM ==="); + + var denseEvaluator = new RagEvaluator(hybrid.Vectors, embedder.EmbedQuery); + var hybridEvaluator = RagEvaluator.ForHybrid(hybrid, embedder); + + const int TopK = 5; + var groups = new (string Name, Case[] Cases)[] + { + ("semantic", Semantic), + ("identifier", Identifier), + ("mixed", Mixed), + }; + + _out.WriteLine($" {"group",-12} {"dense R@K",10} {"hybrid R@K",11} {"dense MRR",10} {"hybrid MRR",11}"); + + var allCases = new List(); + var allRetrievalCases = new List(); + + foreach (var group in groups) + { + var retrievalCases = ToRetrievalCases(group.Cases, chunkIdsByFile); + var dense = denseEvaluator.EvaluateRetrieval(retrievalCases, TopK); + var hyb = hybridEvaluator.EvaluateRetrieval(retrievalCases, TopK); + + allCases.AddRange(group.Cases); + allRetrievalCases.AddRange(retrievalCases); + + _out.WriteLine( + $" {group.Name,-12} {dense.RecallAtK,10:F2} {hyb.RecallAtK,11:F2} " + + $"{dense.MeanReciprocalRank,10:F3} {hyb.MeanReciprocalRank,11:F3}"); + } + + var denseAll = denseEvaluator.EvaluateRetrieval(allRetrievalCases, TopK); + var hybridAll = hybridEvaluator.EvaluateRetrieval(allRetrievalCases, TopK); + + _out.WriteLine( + $" {"OVERALL",-12} {denseAll.RecallAtK,10:F2} {hybridAll.RecallAtK,11:F2} " + + $"{denseAll.MeanReciprocalRank,10:F3} {hybridAll.MeanReciprocalRank,11:F3}"); + + _out.WriteLine(string.Empty); + _out.WriteLine(" per-case rank (0 = missed); lexical arm shown where hybrid did not improve:"); + + for (var i = 0; i < allCases.Count; i++) + { + var denseRank = denseAll.Cases[i].Rank; + var hybridRank = hybridAll.Cases[i].Rank; + + _out.WriteLine( + $" [{allCases[i].Group,-10}] dense {denseRank} hybrid {hybridRank} \"{allCases[i].Query}\""); + + if (hybridRank == 0 || (denseRank > 0 && hybridRank > denseRank)) + { + var lexical = hybrid.Lexical.Search(allCases[i].Query, 3); + var shown = new List(); + for (var j = 0; j < lexical.Length; j++) + { + shown.Add($"{lexical[j].Id}({lexical[j].Score:F2})"); + } + _out.WriteLine($" lexical top-3: {string.Join(", ", shown)}"); + } + } + + // The one claim the lexical arm is bought for. Everything else is reported, not asserted — the + // corpus is real but the case list is hand-written, so pinning the other numbers would pin my + // choice of questions rather than the retriever. + var denseIdentifier = denseEvaluator.EvaluateRetrieval(ToRetrievalCases(Identifier, chunkIdsByFile), TopK); + var hybridIdentifier = hybridEvaluator.EvaluateRetrieval(ToRetrievalCases(Identifier, chunkIdsByFile), TopK); + + Assert.True( + hybridIdentifier.MeanReciprocalRank >= denseIdentifier.MeanReciprocalRank, + $"hybrid identifier MRR {hybridIdentifier.MeanReciprocalRank:F3} fell below dense " + + $"{denseIdentifier.MeanReciprocalRank:F3}"); + } + + private static List ToRetrievalCases( + Case[] cases, Dictionary> chunkIdsByFile) + { + var result = new List(); + + foreach (var c in cases) + { + var expected = new List(); + foreach (var file in c.ExpectedFiles) + { + if (chunkIdsByFile.TryGetValue(file, out var ids)) + { + expected.AddRange(ids); + } + } + + Assert.True(expected.Count > 0, $"no chunks indexed for the expected file(s) of \"{c.Query}\""); + result.Add(new RetrievalCase(c.Query, [.. expected])); + } + + return result; + } + + /// Mirrors McpRagIndex.ChunkParagraphs (internal to the Mcp assembly) so the corpus is + /// split exactly as it would be in production. + private static List ChunkParagraphs(string text) + { + const int TargetChunkChars = 1200; + + var chunks = new List(); + var current = new StringBuilder(TargetChunkChars + 256); + var paragraphs = text.Replace("\r\n", "\n").Split("\n\n", StringSplitOptions.RemoveEmptyEntries); + + foreach (var raw in paragraphs) + { + var paragraph = raw.Trim(); + if (paragraph.Length == 0) + { + continue; + } + + if (current.Length > 0 && current.Length + paragraph.Length > TargetChunkChars) + { + chunks.Add(current.ToString()); + current.Clear(); + } + + if (current.Length > 0) + { + current.Append('\n').Append('\n'); + } + + current.Append(paragraph); + } + + if (current.Length > 0) + { + chunks.Add(current.ToString()); + } + + return chunks; + } + + // Walks up from the test output directory to the repository root (the folder holding Overfit.sln). + private static string? FindDocsDirectory() + { + var directory = new DirectoryInfo(AppContext.BaseDirectory); + + for (var depth = 0; depth < 12 && directory is not null; depth++) + { + if (File.Exists(Path.Combine(directory.FullName, "Overfit.sln"))) + { + var docs = Path.Combine(directory.FullName, "docs"); + return Directory.Exists(docs) ? docs : null; + } + + directory = directory.Parent; + } + + return null; + } + } +} diff --git a/Tests/LanguageModels/Retrieval/HybridVsDenseRecallTests.cs b/Tests/LanguageModels/Retrieval/HybridVsDenseRecallTests.cs new file mode 100644 index 00000000..89738218 --- /dev/null +++ b/Tests/LanguageModels/Retrieval/HybridVsDenseRecallTests.cs @@ -0,0 +1,219 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.LanguageModels.Embeddings; +using DevOnBike.Overfit.LanguageModels.Retrieval; +using DevOnBike.Overfit.LanguageModels.Retrieval.Evaluation; +using DevOnBike.Overfit.Tests.TestSupport; +using Xunit.Abstractions; + +namespace DevOnBike.Overfit.Tests.LanguageModels.Retrieval +{ + /// + /// Measures what hybrid retrieval actually bought, on real MiniLM embeddings, using the same + /// harness a customer would point at their own corpus — rather than asserting + /// that hybrid must be better because the mechanism sounds convincing. + /// + /// The result depends on the query mix, and that is the finding, not a flaw. Cases are split + /// into three deliberately balanced groups — five purely semantic (the target shares no distinctive + /// vocabulary with the question, so only the dense arm can find it), five identifier lookups (error codes, + /// policy numbers, form numbers), and five natural mixed questions. Reporting one blended number would + /// hide the trade-off; reporting the three separately shows exactly what each arm contributes and what + /// hybrid costs on the semantic side. + /// + /// The corpus is synthetic but written to resemble the target domain: an insurance/support knowledge + /// base where prose and hard identifiers sit side by side. That shape is the claim being tested — on a + /// corpus of pure prose with no identifiers, expect hybrid to gain nothing. + /// + /// MEASURED RESULT (2026-07-21, MiniLM, 18 chunks, recall@5) — hybrid was a NET REGRESSION here, + /// which is why McpRagIndex was NOT switched over to it: + /// + /// group dense R@K hybrid R@K dense MRR hybrid MRR + /// semantic 1.00 0.80 0.900 0.700 <- regression + /// identifier 1.00 1.00 0.800 0.900 <- the predicted gain, confirmed + /// mixed 1.00 1.00 1.000 1.000 + /// OVERALL 1.00 0.93 0.900 0.867 + /// + /// + /// Diagnosed cause. "how long until my case is resolved?" fell out of the top-5 entirely + /// because the lexical arm ranked data-retention ("How long we keep records") first, matching + /// on the function words how and long alone. deliberately + /// applies no stop-word list, and on an 18-document corpus IDF cannot compensate: "how" occurs in 2 of 18 + /// documents, so it earns a HIGH idf. On a realistic corpus of hundreds-to-thousands of chunks those words + /// appear nearly everywhere and their idf collapses toward zero, so this failure mode should largely + /// disappear. + /// + /// That prediction was then confirmed and this result superseded. + /// re-ran the same comparison on a real 481-chunk corpus and + /// hybrid won everywhere — recall@5 0.61 → 0.89, MRR 0.500 → 0.736, including the semantic group. So the + /// regression measured here was an artefact of an 18-chunk corpus, not a property of hybrid retrieval. + /// Keep this test as the small-corpus caveat (hybrid genuinely can hurt when the corpus is tiny), + /// but do not treat it as evidence about the technique. + /// + public sealed class HybridVsDenseRecallTests + { + private readonly ITestOutputHelper _out; + + public HybridVsDenseRecallTests(ITestOutputHelper output) => _out = output; + + private static readonly (string Id, string Text)[] Corpus = + [ + ("policy-cancel", "Ending your cover. You may terminate the agreement at any time by giving fourteen days written notice to the address on your schedule. Any premium paid for the remaining period is refunded pro rata."), + ("policy-renew", "Automatic continuation. Unless you tell us otherwise, the agreement rolls over for a further twelve months on its anniversary and the premium is recalculated."), + ("claims-howto", "Reporting an incident. Contact us within seven days of becoming aware of the event. You will be asked for photographs and, where relevant, the police reference."), + ("claims-time", "Settlement timing. Straightforward matters are concluded within thirty days of the last document we request. Complex matters may take longer."), + ("billing-instalments", "Paying monthly. The premium may be spread across twelve instalments collected by direct debit on the fifth working day of each month."), + ("billing-arrears", "Missed payments. If a collection fails we retry once. After two failures the cover is suspended and a notice is issued."), + ("err-4021", "Error E-4021 is returned when the uploaded document exceeds the size limit. Compress the file below 10 MB and retry the upload."), + ("err-4022", "Error E-4022 is returned when the uploaded document has an unsupported type. Convert the file to PDF and retry the upload."), + ("err-5310", "Error E-5310 indicates the signing service was unreachable. The request is retried automatically for one hour before it is abandoned."), + ("policy-40021", "Policy PL-88-40021 covers the commercial fleet of Northwind Logistics. The schedule lists eighteen vehicles and a named-driver restriction."), + ("policy-40022", "Policy PL-88-40022 covers the commercial fleet of Contoso Freight. The schedule lists four vehicles with unrestricted driving."), + ("form-cl17", "Form CL-17 is the claim notification form for goods in transit. Submit it with the consignment note and the carrier's damage report."), + ("form-cl18", "Form CL-18 is the claim notification form for warehouse stock. Submit it with the stock ledger extract."), + ("contact-hours", "When we are open. The service desk answers calls between eight in the morning and six in the evening, Monday to Friday, excluding public holidays."), + ("data-retention", "How long we keep records. Documents relating to an agreement are retained for six years after it ends, after which they are destroyed securely."), + ("excess", "The amount you pay. Each accepted claim carries a fixed contribution deducted from the settlement, shown on your schedule."), + ("no-claims", "Discount for a clean record. Each consecutive year without an accepted claim increases the reduction applied at renewal, up to a maximum after five years."), + ("drivers", "Who may drive. Only individuals listed on the schedule may operate the vehicles, unless the agreement states unrestricted driving."), + ]; + + // Purely semantic: the question deliberately shares no distinctive term with its target, so BM25 + // cannot possibly find it and only the dense arm can. This is the group hybrid could damage. + private static RetrievalCase[] SemanticCases() => + [ + new("how do I stop my insurance?", "policy-cancel"), + new("how long until my case is resolved?", "claims-time"), + new("what happens if my card is declined?", "billing-arrears"), + new("do I get a reward for being safe?", "no-claims"), + new("when can I reach somebody by phone?", "contact-hours"), + ]; + + // Identifier lookups: the exact token IS the query. This is the group dense retrieval is bad at, + // because neighbouring codes are near-identical directions in embedding space. + private static RetrievalCase[] IdentifierCases() => + [ + new("E-4022", "err-4022"), + new("E-5310", "err-5310"), + new("PL-88-40021", "policy-40021"), + new("CL-18", "form-cl18"), + new("what is form CL-17 for?", "form-cl17"), + ]; + + // Natural questions with some lexical overlap — the everyday case, where neither arm is obviously right. + private static RetrievalCase[] MixedCases() => + [ + new("upload fails because the file is too big", "err-4021"), + new("which policy covers Northwind Logistics?", "policy-40021"), + new("how many days do I have to report an incident", "claims-howto"), + new("how long are documents kept", "data-retention"), + new("who is allowed to drive the vehicles", "drivers"), + ]; + + [LocalOnlyFact] + public void Hybrid_VsDense_RecallByQueryKind() + { + if (!File.Exists(TestModelPaths.MiniLm.SafetensorsPath)) + { + _out.WriteLine($"missing MiniLM fixture at {TestModelPaths.MiniLm.Dir}"); + return; + } + + using var embedder = SentenceEmbedder.ForMiniLm(TestModelPaths.MiniLm.Dir); + + // One corpus, indexed once, shared by both retrievers — so the ONLY difference between the two + // measurements is the retrieval strategy. + var hybrid = new HybridRetriever(embedder.Dimension, Corpus.Length); + foreach (var (id, text) in Corpus) + { + hybrid.Add(id, embedder.Embed(text), text); + } + + var denseEvaluator = new RagEvaluator(hybrid.Vectors, embedder.EmbedQuery); + var hybridEvaluator = RagEvaluator.ForHybrid(hybrid, embedder); + + const int TopK = 5; + + var groups = new (string Name, RetrievalCase[] Cases)[] + { + ("semantic", SemanticCases()), + ("identifier", IdentifierCases()), + ("mixed", MixedCases()), + }; + + _out.WriteLine($"=== Hybrid vs dense, MiniLM, {Corpus.Length} chunks, recall@{TopK} ==="); + _out.WriteLine($" {"group",-12} {"dense R@K",10} {"hybrid R@K",11} {"dense MRR",10} {"hybrid MRR",11}"); + + var allCases = new List(); + foreach (var group in groups) + { + var dense = denseEvaluator.EvaluateRetrieval(group.Cases, TopK); + var hyb = hybridEvaluator.EvaluateRetrieval(group.Cases, TopK); + allCases.AddRange(group.Cases); + + _out.WriteLine( + $" {group.Name,-12} {dense.RecallAtK,10:F2} {hyb.RecallAtK,11:F2} " + + $"{dense.MeanReciprocalRank,10:F3} {hyb.MeanReciprocalRank,11:F3}"); + } + + var denseAll = denseEvaluator.EvaluateRetrieval(allCases, TopK); + var hybridAll = hybridEvaluator.EvaluateRetrieval(allCases, TopK); + + _out.WriteLine( + $" {"OVERALL",-12} {denseAll.RecallAtK,10:F2} {hybridAll.RecallAtK,11:F2} " + + $"{denseAll.MeanReciprocalRank,10:F3} {hybridAll.MeanReciprocalRank,11:F3}"); + + _out.WriteLine(string.Empty); + _out.WriteLine(" per-case rank (0 = missed); lexical arm shown where hybrid did not improve:"); + for (var i = 0; i < allCases.Count; i++) + { + var denseRank = denseAll.Cases[i].Rank; + var hybridRank = hybridAll.Cases[i].Rank; + var worse = hybridRank == 0 || (denseRank > 0 && hybridRank > denseRank); + + _out.WriteLine( + $" dense {denseRank} hybrid {hybridRank} \"{allCases[i].Query}\""); + + if (worse) + { + // What the lexical arm dragged in is the whole explanation for a hybrid regression. + var lexical = hybrid.Lexical.Search(allCases[i].Query, 3); + var ids = new List(); + for (var j = 0; j < lexical.Length; j++) + { + ids.Add($"{lexical[j].Id}({lexical[j].Score:F2})"); + } + _out.WriteLine($" lexical top-3: {string.Join(", ", ids)}"); + } + } + + // Assert only the claim the measurement actually supports: hybrid must rank identifier lookups + // at least as well as dense. An assertion on the OVERALL number was tried and removed — it failed, + // for the reason recorded in the class remarks, and pinning it would have pinned the corpus rather + // than the retriever. + var denseIdentifier = denseEvaluator.EvaluateRetrieval(IdentifierCases(), TopK); + var hybridIdentifier = hybridEvaluator.EvaluateRetrieval(IdentifierCases(), TopK); + + Assert.True( + hybridIdentifier.MeanReciprocalRank >= denseIdentifier.MeanReciprocalRank, + $"hybrid identifier MRR {hybridIdentifier.MeanReciprocalRank:F3} fell below dense " + + $"{denseIdentifier.MeanReciprocalRank:F3} — the one thing the lexical arm is bought for"); + } + + [Fact] + public void ForHybridEvaluator_RejectsFalsePremiseChecks() + { + // RRF scores are rank-derived, so a cosine "grounded" threshold is meaningless against them. + // Failing loudly beats returning a confident-looking number. + var retriever = new HybridRetriever(dimension: 2); + retriever.Add("a", [1f, 0f], "alpha"); + + var evaluator = RagEvaluator.ForHybrid(retriever, _ => [1f, 0f]); + + Assert.Throws( + () => evaluator.EvaluateFalsePremise([new FalsePremiseCase("anything")])); + } + } +} diff --git a/Tests/LanguageModels/Retrieval/ReciprocalRankFusionTests.cs b/Tests/LanguageModels/Retrieval/ReciprocalRankFusionTests.cs new file mode 100644 index 00000000..1462d3dc --- /dev/null +++ b/Tests/LanguageModels/Retrieval/ReciprocalRankFusionTests.cs @@ -0,0 +1,126 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.LanguageModels.Retrieval; + +namespace DevOnBike.Overfit.Tests.LanguageModels.Retrieval +{ + /// + /// Pins Reciprocal Rank Fusion. The property that matters is that agreement across arms beats depth + /// within one arm — that is the entire reason hybrid retrieval works, and it is a property of the + /// damping constant, so it needs a test rather than an assumption. + /// + public sealed class ReciprocalRankFusionTests + { + private static VectorMatch M(string id, string? payload = null) => new(id, 0f, payload); + + [Fact] + public void DocumentInBothLists_BeatsDocumentRankedFirstInOnlyOne() + { + ReadOnlySpan dense = new[] { M("a"), M("b"), M("c") }; + ReadOnlySpan lexical = new[] { M("b"), M("d"), M("e") }; + + var fused = ReciprocalRankFusion.Fuse(dense, lexical, 5); + + // "a" is rank 1 in dense but absent from lexical; "b" is rank 2 and rank 1 → agreement wins. + Assert.Equal("b", fused[0].Id); + Assert.Equal("a", fused[1].Id); + } + + [Fact] + public void ScoreIsTheSumOfReciprocalRanks() + { + ReadOnlySpan dense = new[] { M("a") }; + ReadOnlySpan lexical = new[] { M("x"), M("a") }; + + var fused = ReciprocalRankFusion.Fuse(dense, lexical, 5); + var a = fused[0]; + + // rank 1 in dense + rank 2 in lexical, with the 1-based ranks of the RRF formula. + var expected = (1f / (ReciprocalRankFusion.DefaultK + 1f)) + (1f / (ReciprocalRankFusion.DefaultK + 2f)); + Assert.Equal("a", a.Id); + Assert.Equal(expected, a.Score, 6); + } + + [Fact] + public void SmallerK_SharpensTheAdvantageOfTopRanks() + { + ReadOnlySpan first = new[] { M("top"), M("second") }; + ReadOnlySpan second = []; + + var damped = ReciprocalRankFusion.Fuse(first, second, 2, k: 60f); + var sharp = ReciprocalRankFusion.Fuse(first, second, 2, k: 1f); + + var dampedGap = damped[0].Score - damped[1].Score; + var sharpGap = sharp[0].Score - sharp[1].Score; + + // This is exactly the knob that decides whether one confident arm can be outvoted. + Assert.True(sharpGap > dampedGap, $"expected a sharper gap at k=1 ({sharpGap}) than k=60 ({dampedGap})"); + } + + [Fact] + public void CarriesPayloadFromWhicheverArmSuppliedOne() + { + ReadOnlySpan dense = new[] { M("a", payload: null) }; + ReadOnlySpan lexical = new[] { M("a", payload: "chunk text") }; + + var fused = ReciprocalRankFusion.Fuse(dense, lexical, 1); + + Assert.Equal("chunk text", fused[0].Payload); + } + + [Fact] + public void EmptyInputs_ProduceNoResults() + { + ReadOnlySpan none = []; + Assert.Empty(ReciprocalRankFusion.Fuse(none, none, 5)); + } + + [Fact] + public void OneEmptyArm_DegradesToTheOtherArmsOrder() + { + ReadOnlySpan dense = new[] { M("a"), M("b"), M("c") }; + ReadOnlySpan lexical = []; + + var fused = ReciprocalRankFusion.Fuse(dense, lexical, 3); + + Assert.Equal(["a", "b", "c"], fused.Select(m => m.Id)); + } + + [Fact] + public void ResultsAreTruncatedToTheRequestedDepth() + { + ReadOnlySpan dense = new[] { M("a"), M("b"), M("c"), M("d") }; + ReadOnlySpan lexical = []; + + Span buffer = new VectorMatch[2]; + var written = ReciprocalRankFusion.Fuse(dense, lexical, buffer); + + Assert.Equal(2, written); + Assert.Equal("a", buffer[0].Id); + Assert.Equal("b", buffer[1].Id); + } + + [Fact] + public void RejectsNonPositiveK() + { + ReadOnlySpan none = []; + Span buffer = new VectorMatch[1]; + + // Span args cannot cross a lambda boundary, so the throwing call is made directly. + var threw = false; + try + { + ReciprocalRankFusion.Fuse(none, none, buffer, k: 0f); + } + catch (ArgumentOutOfRangeException) + { + threw = true; + } + + Assert.True(threw, "expected ArgumentOutOfRangeException for k = 0"); + } + } +} From 2f972e0bf4132a1fe25b3c0dafe75af9d51d5a4d Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Tue, 21 Jul 2026 23:21:59 +0200 Subject: [PATCH 08/37] hybrid --- .../Retrieval/Evaluation/RagEvaluator.cs | 7 +- .../Retrieval/HybridRetriever.cs | 28 +++- Sources/Mcp/McpRagIndex.cs | 2 +- .../HybridVsDenseOnDocsCorpusTests.cs | 158 ++++++++++++++---- .../Retrieval/HybridVsDenseRecallTests.cs | 9 +- 5 files changed, 158 insertions(+), 46 deletions(-) diff --git a/Sources/Main/LanguageModels/Retrieval/Evaluation/RagEvaluator.cs b/Sources/Main/LanguageModels/Retrieval/Evaluation/RagEvaluator.cs index 3b978bac..8dde7fb8 100644 --- a/Sources/Main/LanguageModels/Retrieval/Evaluation/RagEvaluator.cs +++ b/Sources/Main/LanguageModels/Retrieval/Evaluation/RagEvaluator.cs @@ -48,13 +48,16 @@ public static RagEvaluator ForEmbedder(VectorStore store, SentenceEmbedder embed /// /// is not available on this path — see its remarks. /// - public static RagEvaluator ForHybrid(HybridRetriever retriever, Func embedQuery) + public static RagEvaluator ForHybrid( + HybridRetriever retriever, + Func embedQuery, + float fusionK = HybridRetriever.DefaultFusionK) { ArgumentNullException.ThrowIfNull(retriever); ArgumentNullException.ThrowIfNull(embedQuery); return new RagEvaluator( - (query, topK) => retriever.Search(embedQuery(query), query, topK), + (query, topK) => retriever.Search(embedQuery(query), query, topK, candidatesPerArm: 0, fusionK), scoresAreCosineSimilarity: false); } diff --git a/Sources/Main/LanguageModels/Retrieval/HybridRetriever.cs b/Sources/Main/LanguageModels/Retrieval/HybridRetriever.cs index c1d2e688..5b0327e1 100644 --- a/Sources/Main/LanguageModels/Retrieval/HybridRetriever.cs +++ b/Sources/Main/LanguageModels/Retrieval/HybridRetriever.cs @@ -22,6 +22,22 @@ namespace DevOnBike.Overfit.LanguageModels.Retrieval /// public sealed class HybridRetriever { + /// + /// RRF damping used when fusing these two arms. Deliberately far below + /// (60), which is the constant from the original paper — + /// tuned there for fusing many retrieval systems over deep result lists, where broad agreement + /// really is the best available evidence. Fusing exactly two arms is a different problem: with + /// only two votes, "both arms ranked it mid-list" is weak evidence, and it should not outrank "one arm + /// is certain" — which is what a unique identifier looks like. + /// + /// Measured on this repository's docs/ corpus (481 chunks, MiniLM, recall@5): recall is + /// flat at 0.94 for k in [0.5, 5] and drops to 0.89 from k=10 upward, while MRR peaks at k=5 (0.775 vs + /// 0.736 at k=60). Identifier recall specifically goes 0.83 → 1.00. See + /// HybridVsDenseOnDocsCorpusTests.Fusion_KSweep_OnRealDocsCorpus, which prints the whole curve + /// so this value can be re-derived on any corpus rather than trusted. + /// + public const float DefaultFusionK = 5f; + private readonly VectorStore _vectors; private readonly Bm25Index _lexical; @@ -75,12 +91,20 @@ public void Add(string id, ReadOnlySpan vector, string text, string? payl /// topK, a document ranked just outside both lists can never be promoted by agreement, which /// is the effect hybrid retrieval exists to capture. Deeper costs almost nothing here because both /// arms already scan the corpus; only the merge grows. + /// + /// is the RRF damping constant (see + /// ), exposed because it is the knob that decides whether + /// one confident arm can outvote agreement between two lukewarm ones. Large k flattens the top + /// of each list, so a document found by both arms wins; small k sharpens rank 1, so a unique + /// identifier found by the lexical arm alone can win. Neither is universally right — measure it on + /// your corpus. /// public VectorMatch[] Search( ReadOnlySpan queryVector, string queryText, int topK, - int candidatesPerArm = 0) + int candidatesPerArm = 0, + float fusionK = DefaultFusionK) { ArgumentNullException.ThrowIfNull(queryText); ArgumentOutOfRangeException.ThrowIfNegativeOrZero(topK); @@ -101,7 +125,7 @@ public VectorMatch[] Search( var results = new VectorMatch[Math.Min(topK, Count)]; var written = ReciprocalRankFusion.Fuse( - dense.AsSpan(0, denseCount), lexical.AsSpan(0, lexicalCount), results); + dense.AsSpan(0, denseCount), lexical.AsSpan(0, lexicalCount), results, fusionK); return written == results.Length ? results : results[..written]; } diff --git a/Sources/Mcp/McpRagIndex.cs b/Sources/Mcp/McpRagIndex.cs index 65d629cb..e72bf6c8 100644 --- a/Sources/Mcp/McpRagIndex.cs +++ b/Sources/Mcp/McpRagIndex.cs @@ -19,7 +19,7 @@ namespace DevOnBike.Overfit.Mcp /// /// Retrieval is hybrid (): semantic search over the embeddings /// fused with BM25 over the chunk text. Measured on this repository's own docs/ folder (481 chunks, - /// MiniLM, recall@5) that lifted recall from 0.61 to 0.89 and MRR from 0.500 to 0.736 — + /// MiniLM, recall@5) that lifted recall from 0.61 to 0.94 and MRR from 0.500 to 0.775 — /// see HybridVsDenseOnDocsCorpusTests. Local document sets are full of literal tokens (file names, /// env vars, error codes, API names) that embeddings blur together and BM25 matches exactly. /// diff --git a/Tests/LanguageModels/Retrieval/HybridVsDenseOnDocsCorpusTests.cs b/Tests/LanguageModels/Retrieval/HybridVsDenseOnDocsCorpusTests.cs index 7c8915c4..f6ce8c28 100644 --- a/Tests/LanguageModels/Retrieval/HybridVsDenseOnDocsCorpusTests.cs +++ b/Tests/LanguageModels/Retrieval/HybridVsDenseOnDocsCorpusTests.cs @@ -34,12 +34,16 @@ namespace DevOnBike.Overfit.Tests.LanguageModels.Retrieval /// McpRagIndex onto hybrid retrieval: /// /// group dense R@K hybrid R@K dense MRR hybrid MRR - /// semantic 0.67 0.83 0.417 0.542 - /// identifier 0.33 0.83 0.333 0.667 + /// semantic 0.67 0.83 0.417 0.625 + /// identifier 0.33 1.00 0.333 0.700 /// mixed 0.83 1.00 0.750 1.000 - /// OVERALL 0.61 0.89 0.500 0.736 + /// OVERALL 0.61 0.94 0.500 0.775 /// /// + /// Those hybrid figures are at the final settings. The first run measured 0.89 / 0.736, and the gap + /// between the two is the subject of the notes below — it took BOTH a tokenisation fix and a fusion-constant + /// change, neither of which does anything without the other. + /// /// Hybrid won every group — including the semantic one, which the small-corpus run had shown it /// damaging. Two cases still fail and both are informative rather than mysterious: /// @@ -103,47 +107,19 @@ private sealed record Case(string Group, string Query, params string[] ExpectedF new("mixed", "how do I evaluate prompts locally for free?", "skill-eval.md"), ]; - [LocalOnlyFact] + [LongFact] public void Hybrid_VsDense_OnRealDocsCorpus() { - if (!File.Exists(TestModelPaths.MiniLm.SafetensorsPath)) + var indexed = BuildIndex(); + if (indexed is null) { - _out.WriteLine($"missing MiniLM fixture at {TestModelPaths.MiniLm.Dir}"); return; } - var docsDirectory = FindDocsDirectory(); - if (docsDirectory is null) - { - _out.WriteLine("could not locate the repository docs/ folder from the test output directory"); - return; - } - - using var embedder = SentenceEmbedder.ForMiniLm(TestModelPaths.MiniLm.Dir); - - var hybrid = new HybridRetriever(embedder.Dimension, 512); - var chunkIdsByFile = new Dictionary>(StringComparer.OrdinalIgnoreCase); - - var files = Directory.GetFiles(docsDirectory, "*.md", SearchOption.AllDirectories); - Array.Sort(files, StringComparer.OrdinalIgnoreCase); - - foreach (var file in files) - { - var name = Path.GetFileName(file); - var chunks = ChunkParagraphs(File.ReadAllText(file)); - var ids = new List(); - - for (var i = 0; i < chunks.Count; i++) - { - var id = $"{name}#{i + 1}"; - hybrid.Add(id, embedder.Embed(chunks[i]), chunks[i]); - ids.Add(id); - } + var (embedder, hybrid, chunkIdsByFile, fileCount) = indexed.Value; + using var _ = embedder; - chunkIdsByFile[name] = ids; - } - - _out.WriteLine($"=== Hybrid vs dense on docs/ — {files.Length} files, {hybrid.Count} chunks, MiniLM ==="); + _out.WriteLine($"=== Hybrid vs dense on docs/ — {fileCount} files, {hybrid.Count} chunks, MiniLM ==="); var denseEvaluator = new RagEvaluator(hybrid.Vectors, embedder.EmbedQuery); var hybridEvaluator = RagEvaluator.ForHybrid(hybrid, embedder); @@ -217,6 +193,114 @@ public void Hybrid_VsDense_OnRealDocsCorpus() + $"{denseIdentifier.MeanReciprocalRank:F3}"); } + /// + /// Sweeps the RRF damping constant on the same corpus and the same cases. This is the follow-on the + /// identifier-tokenisation measurement pointed at: the joined-identifier token moved + /// docker.md to lexical rank 1 yet changed no end-to-end metric, because at k=60 a document + /// found by ONE arm at rank 1 (1/61 ≈ 0.016) loses to a document found by BOTH arms at ranks 5 and 3 + /// (1/65 + 1/63 ≈ 0.031). k is exactly the knob that sets that balance. + /// + [LongFact] + public void Fusion_KSweep_OnRealDocsCorpus() + { + var indexed = BuildIndex(); + if (indexed is null) + { + return; + } + + var (embedder, hybrid, chunkIdsByFile, _) = indexed.Value; + using var disposable = embedder; + + const int TopK = 5; + var semantic = ToRetrievalCases(Semantic, chunkIdsByFile); + var identifier = ToRetrievalCases(Identifier, chunkIdsByFile); + var mixed = ToRetrievalCases(Mixed, chunkIdsByFile); + var all = new List(); + all.AddRange(semantic); + all.AddRange(identifier); + all.AddRange(mixed); + + var dense = new RagEvaluator(hybrid.Vectors, embedder.EmbedQuery); + var denseAll = dense.EvaluateRetrieval(all, TopK); + + _out.WriteLine($"=== RRF damping sweep on docs/ — {hybrid.Count} chunks, recall@{TopK} ==="); + _out.WriteLine($" {"k",6} {"sem R",7} {"ident R",8} {"mixed R",8} {"ALL R",7} {"ALL MRR",8}"); + _out.WriteLine( + $" {"dense",6} {dense.EvaluateRetrieval(semantic, TopK).RecallAtK,7:F2} " + + $"{dense.EvaluateRetrieval(identifier, TopK).RecallAtK,8:F2} " + + $"{dense.EvaluateRetrieval(mixed, TopK).RecallAtK,8:F2} " + + $"{denseAll.RecallAtK,7:F2} {denseAll.MeanReciprocalRank,8:F3}"); + + foreach (var k in new[] { 0.5f, 1f, 2f, 5f, 10f, 20f, 40f, 60f, 100f }) + { + var evaluator = RagEvaluator.ForHybrid(hybrid, embedder.EmbedQuery, k); + var allReport = evaluator.EvaluateRetrieval(all, TopK); + + _out.WriteLine( + $" {k,6:F1} {evaluator.EvaluateRetrieval(semantic, TopK).RecallAtK,7:F2} " + + $"{evaluator.EvaluateRetrieval(identifier, TopK).RecallAtK,8:F2} " + + $"{evaluator.EvaluateRetrieval(mixed, TopK).RecallAtK,8:F2} " + + $"{allReport.RecallAtK,7:F2} {allReport.MeanReciprocalRank,8:F3}"); + } + + _out.WriteLine(string.Empty); + _out.WriteLine(" NOTE: 18 hand-written cases. Reading the argmax off this curve would be fitting k"); + _out.WriteLine(" to my own question list, not to the retriever. Prefer a value on a flat stretch."); + + // The default must remain a defensible choice, not silently the worst one on the curve. + var atDefault = RagEvaluator + .ForHybrid(hybrid, embedder.EmbedQuery, HybridRetriever.DefaultFusionK) + .EvaluateRetrieval(all, TopK); + + Assert.True( + atDefault.RecallAtK >= denseAll.RecallAtK, + $"hybrid at the default k fell below dense ({atDefault.RecallAtK:F2} < {denseAll.RecallAtK:F2})"); + } + + // Embeds docs/ once. Returns null (with a written reason) when the fixture or the folder is missing. + private (SentenceEmbedder Embedder, HybridRetriever Hybrid, + Dictionary> ChunkIdsByFile, int FileCount)? BuildIndex() + { + if (!File.Exists(TestModelPaths.MiniLm.SafetensorsPath)) + { + _out.WriteLine($"missing MiniLM fixture at {TestModelPaths.MiniLm.Dir}"); + return null; + } + + var docsDirectory = FindDocsDirectory(); + if (docsDirectory is null) + { + _out.WriteLine("could not locate the repository docs/ folder from the test output directory"); + return null; + } + + var embedder = SentenceEmbedder.ForMiniLm(TestModelPaths.MiniLm.Dir); + var hybrid = new HybridRetriever(embedder.Dimension, 512); + var chunkIdsByFile = new Dictionary>(StringComparer.OrdinalIgnoreCase); + + var files = Directory.GetFiles(docsDirectory, "*.md", SearchOption.AllDirectories); + Array.Sort(files, StringComparer.OrdinalIgnoreCase); + + foreach (var file in files) + { + var name = Path.GetFileName(file); + var chunks = ChunkParagraphs(File.ReadAllText(file)); + var ids = new List(); + + for (var i = 0; i < chunks.Count; i++) + { + var id = $"{name}#{i + 1}"; + hybrid.Add(id, embedder.Embed(chunks[i]), chunks[i]); + ids.Add(id); + } + + chunkIdsByFile[name] = ids; + } + + return (embedder, hybrid, chunkIdsByFile, files.Length); + } + private static List ToRetrievalCases( Case[] cases, Dictionary> chunkIdsByFile) { diff --git a/Tests/LanguageModels/Retrieval/HybridVsDenseRecallTests.cs b/Tests/LanguageModels/Retrieval/HybridVsDenseRecallTests.cs index 89738218..02883b1e 100644 --- a/Tests/LanguageModels/Retrieval/HybridVsDenseRecallTests.cs +++ b/Tests/LanguageModels/Retrieval/HybridVsDenseRecallTests.cs @@ -28,7 +28,7 @@ namespace DevOnBike.Overfit.Tests.LanguageModels.Retrieval /// corpus of pure prose with no identifiers, expect hybrid to gain nothing. /// /// MEASURED RESULT (2026-07-21, MiniLM, 18 chunks, recall@5) — hybrid was a NET REGRESSION here, - /// which is why McpRagIndex was NOT switched over to it: + /// which is why McpRagIndex was initially NOT switched over to it: /// /// group dense R@K hybrid R@K dense MRR hybrid MRR /// semantic 1.00 0.80 0.900 0.700 <- regression @@ -47,8 +47,9 @@ namespace DevOnBike.Overfit.Tests.LanguageModels.Retrieval /// /// That prediction was then confirmed and this result superseded. /// re-ran the same comparison on a real 481-chunk corpus and - /// hybrid won everywhere — recall@5 0.61 → 0.89, MRR 0.500 → 0.736, including the semantic group. So the - /// regression measured here was an artefact of an 18-chunk corpus, not a property of hybrid retrieval. + /// hybrid won everywhere — recall@5 0.61 → 0.94, MRR 0.500 → 0.775, including the semantic group, and + /// McpRagIndex was switched over on that evidence. So the regression measured here was an artefact + /// of an 18-chunk corpus, not a property of hybrid retrieval. /// Keep this test as the small-corpus caveat (hybrid genuinely can hurt when the corpus is tiny), /// but do not treat it as evidence about the technique. /// @@ -112,7 +113,7 @@ private static RetrievalCase[] MixedCases() => new("who is allowed to drive the vehicles", "drivers"), ]; - [LocalOnlyFact] + [LongFact] public void Hybrid_VsDense_RecallByQueryKind() { if (!File.Exists(TestModelPaths.MiniLm.SafetensorsPath)) From 0829bff8051f323c0b6b1124078880379923a16c Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Wed, 22 Jul 2026 15:46:23 +0200 Subject: [PATCH 09/37] llama --- .claude/settings.json | 24 ++- ROADMAP.md | 185 ++++++++++++++++- .../Q4KPrefillProjectionBenchmark.cs | 195 ++++++++++++++++++ .../Runtime/BatchedQuantProjection.cs | 124 ++++++++++- .../Runtime/CachedFeedForwardBlock.cs | 5 + .../Runtime/CachedLlamaSession.cs | 2 + .../Runtime/CachedMultiHeadAttention.cs | 8 + .../Runtime/CachedTransformerBlock.cs | 6 + .../LanguageModels/Runtime/PrefillProfiler.cs | 185 +++++++++++++++++ .../LanguageModels/Runtime/Q6KGemvKernel.cs | 140 +++++++++++++ .../Diagnostics/PrefillProfileTests.cs | 52 +++++ .../Runtime/Q6KTiledGemmParityTests.cs | 142 +++++++++++++ 12 files changed, 1056 insertions(+), 12 deletions(-) create mode 100644 Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs create mode 100644 Sources/Main/LanguageModels/Runtime/PrefillProfiler.cs create mode 100644 Tests/LanguageModels/Runtime/Q6KTiledGemmParityTests.cs diff --git a/.claude/settings.json b/.claude/settings.json index 5dcfcf25..a5e7a63f 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -14,15 +14,33 @@ "PowerShell(dotnet *)", "Bash(git *)", "PowerShell(git *)", - "Bash(python Scripts/*)", - "PowerShell(python Scripts/*)", + "Bash(python *)", + "Bash(python3 *)", + "PowerShell(python *)", "Bash(./update-code-headers.cmd*)", "Bash(./cleanup.cmd*)", "Bash(./Sources/Benchmark/run.cmd*)", "PowerShell(.\\update-code-headers.cmd*)", "PowerShell(.\\cleanup.cmd*)", "PowerShell(.\\Sources\\Benchmark\\run.cmd*)", - "PowerShell(taskkill /F /IM dotnet.exe)" + "PowerShell(taskkill /F /IM dotnet.exe)", + "PowerShell(Select-String *)", + "PowerShell(Select-Object *)", + "PowerShell(Sort-Object *)", + "PowerShell(Where-Object *)", + "PowerShell(ForEach-Object *)", + "PowerShell(Measure-Object *)", + "PowerShell(Get-Command *)", + "PowerShell(Test-Path *)", + "PowerShell(cmake *)", + "Bash(cmake *)", + "PowerShell($cm = *)", + "PowerShell(& $cm *)", + "PowerShell(& $b *)", + "PowerShell(Set-Location *)", + "PowerShell(cd *)", + "PowerShell(Get-ChildItem *)", + "PowerShell(Get-Content *)" ], "deny": [ "Bash(git commit *)", diff --git a/ROADMAP.md b/ROADMAP.md index b4f07562..eacc5192 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -77,10 +77,184 @@ So in-place rewrites are free and **the only real risk is extracting a method**. --- -## ▶ NEXT UP — the cheap CPU-perf levers are exhausted; the open item is a product decision +## ▶ NEXT UP — PREFILL. Measured 3.76× behind llama.cpp, and it is compute-bound -**Three candidate levers were sized and all three died on measurement (2026-07-21). Do not re-open without -new evidence.** +**Measured 2026-07-22, same file (`qwen.q4km.gguf`), same 672-token prompt, best configuration on both sides:** + +| | prefill (pp672) | notes | +|---|---:|---| +| llama.cpp b10088 (built from `D:\llamacpp-tmp`, `/arch:AVX512`, 16 threads) | **541.7 ± 2.6 tok/s** | `llama-bench -p 672 -n 0 -r 3` | +| Overfit (sidecar `.repack` present, 32 workers) | **144 tok/s** | `PrefillProfileTests` | +| | **3.76×** | | + +**Not a thread-configuration artefact.** Worker sweep: 8 → 92, 16 → 122, 24 → 130, 32 (default) → 144 tok/s — +monotonic, default is best. Prefill *scales* with cores, unlike decode (which has a cliff at +`workers == procCount`). The gap is algorithmic. + +**This corrects the previous heading here, which read "the cheap CPU-perf levers are exhausted".** That was +true of **decode** and was wrongly generalised to performance as a whole. The two paths are not alike: + +| path | gap to llama.cpp | why | +|---|---|---| +| decode | **1.13×**, uniform across context | memory-bound, sitting on the DRAM floor | +| **prefill** | **3.76×** | compute-bound — there is no floor here | + +**The reference kernel is NOT tinyBLAS.** `ggml/src/ggml-cpu/llamafile/sgemm.cpp` contains no +`GGML_TYPE_Q4_K` case at all. The Q4_K prefill path is `ggml_gemm_q4_K_8x8_q8_K` in +`ggml/src/ggml-cpu/arch/x86/repack.cpp` (~1450 lines) — the same `block_q4_Kx8` repacked layout Overfit +already uses. So this is not a missing algorithm; it is the same algorithm implemented far better. + +**Per-projection micro-bench (2026-07-22, `Q4KPrefillProjectionBenchmark`, 672 rows, real Qwen-3B shapes) — +this REFUTED the first hypothesis written here, which claimed the tiled kernel "wins nothing":** + +| shape | Tiled | WeightStationary | ReDecodePerRow | Tiled 1-thread | +|---|---:|---:|---:|---:| +| `ffn_gate_up` (2048→11008) | **15.44 ms** | 53.62 ms | 86.55 ms | 169.3 ms | +| `ffn_down` (11008→2048) | **17.27 ms** | 55.55 ms | 85.61 ms | 169.7 ms | +| `attn_qo` (2048→2048) | **4.41 ms** | 11.85 ms | 20.64 ms | 32.2 ms | + +`GemmTiled` is **~3.2–3.4× faster than weight-stationary**, exactly as its own docs claim. It is a real GEMM +and it already carries the FFN in production (a `.repack` sidecar sets `IsPrepacked`, which routes every +bias-free projection through it). + +**So why did the 2026-07-21 end-to-end A/B tie at 0.999×?** Because that A/B only moved the *biased* +projections — attention Q/K/V. Those are **88% of the dispatch count but only ~6% of the FLOPs**: Q is +dispatched per head at 2048→128, while one FFN layer is 3 × 30.3 GFLOP. The tie was real and correctly +measured; it simply measured the small projections. **Dispatch count is not work — always weight a path +census by FLOPs before drawing a conclusion from it.** + +**The real gap is kernel throughput.** At 672 rows a projection is 30.3 GFLOP, so our best kernel runs at +**≈1.9 TFLOP/s** (15.4 ms) against llama.cpp's **≈3.7 TFLOP/s** whole-model rate — a **~1.9× kernel gap**, +not a missing algorithm. The residual beyond that is dispatch overhead in the per-head attention path, where +the same activation matrix is re-quantized once per head. +**The gap decomposes — measured, not assumed.** llama.cpp was rebuilt AVX2-only +(`-DGGML_NATIVE=OFF -DGGML_AVX2=ON -DGGML_AVX512=OFF`, `D:\llamacpp-tmp\build-avx2`) and re-benched on the +same file: + +| build | pp672 | +|---|---:| +| llama.cpp, AVX-512 | 539.9 tok/s | +| llama.cpp, AVX2 only | 336.7 tok/s | +| Overfit, AVX2 | 144 tok/s | + +**3.76× = 2.34× (kernel quality at equal ISA) × 1.60× (AVX-512).** + +This **refutes the ranking first written here**, which called AVX-512 "the most likely source of ~2×". It is +the *smaller* factor. Porting the kernel to AVX-512 caps out at 1.60×; the larger 2.34× is available without +touching the instruction set. Note also that the old "AVX-512 ≈ 0" result stands for **decode** (memory-bound, +where wider SIMD cannot help by construction) — here it is worth 1.60×, so that negative genuinely does not +transfer to compute-bound prefill. + +### Prefill component breakdown — measured 2026-07-22 (`PrefillProfiler`, first time in the project) + +Qwen-3B Q4_K_M, 672-token prompt, median of 3 (`PrefillProfileTests.Prefill_ComponentBreakdown`): + +``` +total/request : 4697.5 ms (143 tok/s) + attention : 1595.7 ms 34.0% (36 calls) + ffn : 3000.8 ms 63.9% (36 calls) + attn_kv : 179.7 ms 3.8% ( 72) + attn_q : 624.1 ms 13.3% (576) <- per head + attn_scores: 263.9 ms 5.6% (576) + attn_out : 427.9 ms 9.1% (576) <- per head + ffn_gateup: 1222.5 ms 26.0% ( 36) + ffn_down : 1778.1 ms 37.9% ( 36) <- biggest single item + other : 101.0 ms 2.1% +``` + +### ▶▶ THE NEXT LEVER: Q6_K has no batched prefill kernel + +`ffn_down` costs **more** than `ffn_gateup` while doing **half** the work (one 30.3 GFLOP projection vs two). +Per layer that is 0.61 TFLOP/s against gate_up's 1.78 — a 2.9× efficiency gap that the micro-bench did *not* +show (Tiled: 17.27 vs 15.44 ms). So production is not taking the same path. Cause, confirmed by dumping the +GGUF tensor types: + +- **`ffn_down` is Q4_K ×18 + Q6_K ×18** (and `attn_v` likewise) — half the layers are Q6_K. +- In `BatchedQuantProjection`, the Q6_K branch has **only `Q6KDotKernel.ProjectBatched`** (re-decode per row). + There is **no `ProjectBatchedWeightStationary` and no `GemmTiled` for Q6_K**, while Q4_K has both. +- Arithmetic checks out: 18 layers × 17.3 ms (tiled) + 18 × X = 1778 ms ⇒ X ≈ 81.5 ms, and the micro-bench + measured `ReDecodePerRow` at 85.6 ms for that shape. + +**The repack layout for Q6_K already exists** (`Q6KRepack`, `RowsInterleaved = 8`, `Q6KGemvKernel.GemvParallel`) +— it is wired for *decode* only. So this is filling a gap in an existing kernel family, not inventing one. + +**Estimated payoff: `ffn_down` 1778 → ~670 ms ≈ 1.1 s of 4.7 s (~23%), i.e. 143 → ~187 tok/s (1.31×).** +An estimate, not a promise — Q6_K does more work per weight (6-bit vs 4-bit) than the Q4_K kernel it is +modelled on. + +**Attack order, by value/risk rather than by ceiling:** + +| lever | ceiling | risk | +|---|---|---| +| **Q6_K batched prefill kernel** | ~1.31× | **low** — layout exists, structure copied from Q4_K | +| AVX-512 port | 1.60× | high — intrinsics rewritten from scratch | +| per-head attention (`attn_q` + `attn_out` = 22.4%, 576 dispatches each) | unknown | medium — dispatch restructuring | + +#### ✗ Q6_K weight-stationary — BUILT, MEASURED +13.5% SLOWER, REVERTED (2026-07-22) + +`Q6KDotKernel.ProjectBatchedWeightStationary` was written on the Q4_K model: unpack each super-block once +into scratch, contract against a 64-row tile. Bit-identical (12/12 parity tests, including tile-boundary and +no-bias cases). Measured on the real model: + +| component | before | after | Δ | +|---|---:|---:|---:| +| `ffn_down` | 1778.1 ms | **2018.0 ms** | **+13.5%** | +| `ffn_gateup` *(canary)* | 1222.5 ms | 1247.5 ms | +2.0% | +| `attention` *(canary)* | 1595.7 ms | 1616.7 ms | +1.3% | + +Canaries drifted 1–2%, `ffn_down` moved 13.5% — a real regression, reverted. + +**Two mistakes in the analogy, both worth remembering.** (1) Q4_K's weight-stationary hoists only the +*scale/min* decode; the 4-bit nibble unpack still happens **in registers, per row**. I hoisted the entire +6-bit unpack into a 256-byte stack buffer, so every row now stores and reloads it through L1 instead of +consuming it from registers. (2) Inverting the loop order made activation reads strided (one 256-byte slice +per row, 11 008 bytes apart) instead of streaming a row contiguously. + +**So the Q6_K gap is not closed by the obvious transform.** The right analogue to Q4_K's 3.3× is the *tiled* +kernel over the repacked `block_q6_Kx8` layout — and `Q6KRepack` already produces that layout for decode. + +#### ✅ Q6_K tiled GEMM — SHIPPED, prefill 143 → 185 tok/s (1.29×) + +`Q6KGemvKernel.GemmTiled` unpacks each weight super-block once and holds it **in registers** across a tile of +up to 16 activation columns — the opposite of the reverted weight-stationary attempt, which pushed the unpack +through a stack buffer. Wired into `BatchedQuantProjection` via `DispatchTiledQ6K` (gate: +`UseTiledPrefillQ6K && bias.IsEmpty && CanRepack && AVX2 && FMA`). + +| component | before | after | Δ | +|---|---:|---:|---:| +| `ffn_down` | 1778.1 ms | **713.6 ms** | **−59.9%** (2.49×) | +| `ffn_gateup` *(canary)* | 1222.5 ms | 1242.3 ms | +1.6% | +| `attention` *(canary)* | 1595.7 ms | 1571.5 ms | −1.5% | +| **prefill total** | **4697.5 ms · 143 tok/s** | **3632.6 ms · 185 tok/s** | **−22.7% · 1.29×** | + +Canaries within ±1.6%, and an independent run of `PrefillPathAbTests` measured 186 tok/s. The estimate that +motivated the work (1778 → ~670 ms, 143 → ~187 tok/s) landed almost exactly. + +**Correctness.** `Q6KTiledGemmParityTests` pins `GemmTiled` bit-identical to `GemvAvx2` per column (6 cases). +End-to-end the first generated token is **576, unchanged** from before the kernel. Note this is *coherence* +evidence, not byte-parity: the old path (`ProjectBatched`, non-repacked) associates the reduction differently +from the repacked kernels, so outputs differ in the low bits — the same standard `OVERFIT_REPACK_ATTN` is held +to. + +**Cost:** `Q6KWeight` has no prepacked-sidecar path, so `EnsureRepacked()` allocates a heap copy of the Q6_K +tensors on first use. Worth revisiting if RAM matters more than TTFT. + +**Gap to llama.cpp: 3.76× → 2.93×.** Remaining, by measured share: `ffn_gateup` 34.2%, `attn_q` + `attn_out` +28.9% (the per-head dispatches, 576 calls each), `attn_scores` 6.8%. AVX-512 (ceiling 1.60×) still last. + +At 3.4 B params × 672 tokens the gap is ≈3.7 TFLOP/s-equivalent for them against ≈1.0 for us. + +**Why this lever is different from the five that were refuted:** it has a measured ceiling, a named cause, and +a working reference implementation to read. The earlier register-/cache-blocking negatives were on +*memory-bound* paths, where blocking cannot help by construction. Prefill is compute-bound. +**Honest expectation: 3.76× is the ceiling, not a promise — 2× would be a good outcome.** Size a single +projection with a micro-bench against `sgemm.cpp` before writing any kernel. + +--- + +### Refuted levers — do not re-open without new evidence + +**Three candidates were sized and all three died on measurement (2026-07-21).** 1. **`SearchValues` / tokenizer-level work — CLOSED.** A prefill profile (Qwen-3B Q4_K_M, 672-token prompt, median of 5) puts tokenization at **0.04% of time-to-first-token** — 1.8 ms against 4731 ms of prefill @@ -98,8 +272,9 @@ new evidence.** `ProjectBatched` (re-decode per row), **not** against weight-stationary. Recorded in `CLAUDE.md`. Decode is ~88% quantized GEMV sitting at the DRAM floor (`ffn 69.3% · attention 19.3% · lm_head 10.3%`), so -there is no cheap kernel win left. What remains open is **not technical**: the product direction (perf course -vs. the on-prem commercial track) has been deferred across several sessions and is the actual blocker. +there is no cheap **decode** kernel win left — see the prefill section above for the path that *is* open. +The product direction (perf course vs. the on-prem commercial track) remains deferred and is a separate, +non-technical decision. --- diff --git a/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs b/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs new file mode 100644 index 00000000..c20939b3 --- /dev/null +++ b/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs @@ -0,0 +1,195 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using BenchmarkDotNet.Attributes; +using Benchmarks.Helpers; +using DevOnBike.Overfit.LanguageModels.Loading; +using DevOnBike.Overfit.LanguageModels.Runtime; + +namespace Benchmarks +{ + /// + /// Sizes ONE Q4_K prefill projection — the unit of work behind the measured 3.76× prefill gap to + /// llama.cpp (541.7 vs 144 tok/s on qwen.q4km.gguf, 672-token prompt, 2026-07-22). + /// + /// What this is for. Before writing a better GEMM, establish which of the two suspected causes + /// actually costs us. Both are visible here: + /// + /// Kernel quality — vs ·(cores) + /// isolates the kernel from the dispatcher. GemmTiled's own doc admits its per-column + /// accumulators live in stackalloc scratch and spill; llama.cpp's + /// ggml_gemm_q4_K_8x8_q8_K holds a 4×16 tile in registers. A tile whose accumulators spill wins + /// nothing — which is exactly the 0.999× tie measured on 2026-07-21. + /// Parallel efficiency — vs shows how + /// much of the machine the dispatcher actually extracts. + /// + /// + /// Shapes are the real ones (Qwen2.5-3B: hidden 2048, intermediate 11008). The decode profile + /// put FFN at 69.3% of the step, so ffn_gate_up and ffn_down are the shapes that decide the + /// outcome; attn_qo is included because attention is dispatched per head and may behave differently. + /// + /// FLOP reference for converting ns → throughput: one projection is + /// 2 · rows · inputSize · outputSize MACs — at rows=672 that is 30.3 GFLOP for the FFN shapes and + /// 5.6 GFLOP for attn_qo. llama.cpp's whole-model 541.7 tok/s works out to ≈3.7 TFLOP/s-equivalent + /// against our ≈1.0, so a kernel here needs to land near 3.5–4 TFLOP/s to close the gap. + /// + /// The shared (InvocationCount=1) is correct here and NOT the trap + /// described in CLAUDE.md: a single projection at these shapes runs for tens of milliseconds, not + /// microseconds, so there is no timer-noise problem to fix with a microbenchmark job. + /// + /// The synthetic weight is deliberately NOT prepacked, so IsPrepacked cannot + /// short-circuit UseTiledPrefillQ4K — the dead-flag trap that invalidated an entire measurement + /// round on 2026-07-21, when a *.gguf.repack sidecar silently made both A/B arms identical. + /// + /// Run: + /// dotnet run -c Release --project Sources/Benchmark -- --filter "*Q4KPrefillProjection*" + /// + [Config(typeof(BenchmarkConfig))] + public class Q4KPrefillProjectionBenchmark + { + /// Prompt length used in the llama.cpp comparison, so the numbers are directly relatable. + [Params(672)] + public int Rows + { + get; set; + } + + [Params("ffn_gate_up", "ffn_down", "attn_qo")] + public string Shape + { + get; set; + } = "ffn_gate_up"; + + private DecodeWeight _weight; + private Q4KWeight _q4k = null!; + private float[] _input = null!; + private float[] _output = null!; + private sbyte[] _quants = null!; + private float[] _scales = null!; + private short[] _bsums = null!; + private int _inputSize; + private int _outputSize; + private bool _originalTiled; + private bool _originalStationary; + + [GlobalSetup] + public void Setup() + { + (_inputSize, _outputSize) = Shape switch + { + "ffn_gate_up" => (2048, 11008), + "ffn_down" => (11008, 2048), + _ => (2048, 2048), + }; + + _originalTiled = BatchedQuantProjection.UseTiledPrefillQ4K; + _originalStationary = BatchedQuantProjection.UseWeightStationaryQ4K; + + var rng = new Random(20260722); + + // Quantize a random F32 matrix into a real Q4_K weight — the same layout the loader produces. + var f32 = new float[(long)_outputSize * _inputSize]; + for (var i = 0; i < f32.Length; i++) + { + f32[i] = (float)(rng.NextDouble() * 2.0 - 1.0); + } + + _q4k = new Q4KWeight(GgmlQuant.QuantizeQ4_K(f32, _inputSize, _outputSize), _inputSize, _outputSize); + _weight = _q4k; + + _input = new float[(long)Rows * _inputSize]; + for (var i = 0; i < _input.Length; i++) + { + _input[i] = (float)(rng.NextDouble() * 2.0 - 1.0); + } + + _output = new float[(long)Rows * _outputSize]; + + // Activation-quantization scratch for the single-thread kernel path (the dispatcher pools its own). + var superBlocksPerRow = _q4k.SuperBlocksPerRow; + _quants = new sbyte[(long)Rows * _inputSize]; + _scales = new float[(long)Rows * superBlocksPerRow]; + _bsums = new short[(long)Rows * superBlocksPerRow * Q4KDotKernel.GroupsPerSuperBlock]; + + // Pay the one-off repack here so it is not attributed to the timed region. + _q4k.EnsureRepacked(); + } + + [GlobalCleanup] + public void Cleanup() + { + BatchedQuantProjection.UseTiledPrefillQ4K = _originalTiled; + BatchedQuantProjection.UseWeightStationaryQ4K = _originalStationary; + _weight.Dispose(); + } + + /// Today's production path for a bias-free projection when the tiled kernel is NOT enabled: + /// decode each super-block once, reuse it across the row tile. The baseline everything else is judged against. + [Benchmark(Baseline = true)] + public void WeightStationary() + { + BatchedQuantProjection.UseTiledPrefillQ4K = false; + BatchedQuantProjection.UseWeightStationaryQ4K = true; + BatchedQuantProjection.Dispatch(_input, Rows, in _weight, [], _output, _inputSize, _outputSize); + } + + /// The register-tiled GEMM over block_q4_Kx8, parallelised across row tiles. + [Benchmark] + public void Tiled() + { + BatchedQuantProjection.UseTiledPrefillQ4K = true; + BatchedQuantProjection.UseWeightStationaryQ4K = false; + BatchedQuantProjection.Dispatch(_input, Rows, in _weight, [], _output, _inputSize, _outputSize); + } + + /// The original re-decode-per-row kernel — kept as the reference the kernel docs' "~3×" claim + /// is actually measured against. + [Benchmark] + public void ReDecodePerRow() + { + BatchedQuantProjection.UseTiledPrefillQ4K = false; + BatchedQuantProjection.UseWeightStationaryQ4K = false; + BatchedQuantProjection.Dispatch(_input, Rows, in _weight, [], _output, _inputSize, _outputSize); + } + + /// + /// GemmTiled over every row tile on ONE thread. Divided into this gives the + /// dispatcher's parallel efficiency; on its own it is the raw kernel throughput to compare against + /// llama.cpp's single-thread rate — the number that says whether the accumulator spill is the problem. + /// + [Benchmark] + public void Tiled_SingleThread() + { + var superBlocksPerRow = _q4k.SuperBlocksPerRow; + var bsumsPerRow = superBlocksPerRow * Q4KDotKernel.GroupsPerSuperBlock; + + for (var n = 0; n < Rows; n++) + { + Q4KDotKernel.QuantizeActivationQ8K( + _input.AsSpan(n * _inputSize, _inputSize), + _quants.AsSpan(n * _inputSize, _inputSize), + _scales.AsSpan(n * superBlocksPerRow, superBlocksPerRow), + _bsums.AsSpan(n * bsumsPerRow, bsumsPerRow)); + } + + var repacked = _q4k.EnsureRepacked(); + const int TileCols = 8; + + for (var start = 0; start < Rows; start += TileCols) + { + var cols = Math.Min(TileCols, Rows - start); + Q4KGemvKernel.GemmTiled( + repacked, + _outputSize, + _inputSize, + cols, + _quants.AsSpan(start * _inputSize, cols * _inputSize), + _scales.AsSpan(start * superBlocksPerRow, cols * superBlocksPerRow), + _bsums.AsSpan(start * bsumsPerRow, cols * bsumsPerRow), + _output.AsSpan(start * _outputSize, cols * _outputSize)); + } + } + } +} diff --git a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs index ecba8bcf..7340fdfe 100644 --- a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs +++ b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs @@ -30,6 +30,14 @@ internal static class BatchedQuantProjection /// the OVERFIT_TILED_PREFILL env flag; mutable so perf/coherence benches can A/B it in one process. internal static bool UseTiledPrefillQ4K = Q4KGemvKernel.TiledPrefillEnabled; + /// Gates the register-tiled Q6_K prefill GEMM (). + /// Mutable so perf tests can A/B it in one process. + /// + /// Costs RAM: unlike Q4_K, has no prepacked-sidecar path, so + /// EnsureRepacked always allocates a heap copy (~the size of the Q6_K tensors) on first + /// use. + internal static bool UseTiledPrefillQ6K = true; + public static void Dispatch( ReadOnlySpan input, int rows, @@ -50,10 +58,28 @@ public static void Dispatch( using var qBytes = new PooledBuffer(rows * inputSize, clearMemory: false); using var scales = new PooledBuffer(rows * spr, clearMemory: false); using var sums = new PooledBuffer(groups, clearMemory: false); - Q6KDotKernel.ProjectBatched( - input, rows, w, bias, output, - qBytes.Span.Slice(0, rows * inputSize), scales.Span.Slice(0, rows * spr), - sums.Span.Slice(0, groups)); + // Register-tiled Q6_K GEMM over the repacked block_q6_Kx8 layout. Under Q4_K_M half of + // ffn_down is Q6_K, and a prefill profile put ffn_down at 37.9% of prefill running at + // 0.61 TFLOP/s — against ffn_gate_up's 1.78 — precisely because Q6_K had only the + // re-decode-per-row kernel below. No-bias only (GemmTiled applies none); AVX2/FMA required. + var tiled6 = UseTiledPrefillQ6K && bias.IsEmpty && w.CanRepack + && CpuFeatures.HasAvx2 && CpuFeatures.HasFma; + + if (tiled6) + { + DispatchTiledQ6K( + input, rows, w, output, + qBytes.Span.Slice(0, rows * inputSize), scales.Span.Slice(0, rows * spr), + sums.Span.Slice(0, groups)); + } + + if (!tiled6) + { + Q6KDotKernel.ProjectBatched( + input, rows, w, bias, output, + qBytes.Span.Slice(0, rows * inputSize), scales.Span.Slice(0, rows * spr), + sums.Span.Slice(0, groups)); + } } if (kind == 1) { @@ -176,6 +202,96 @@ private static unsafe void DispatchTiledQ4K( } } + // Register-tiled Q6_K prefill GEMM: quantize all rows to Q8_K, then run GemmTiled over row-tiles of + // NR columns in parallel. Mirrors DispatchTiledQ4K, minus the bsums — the Q6_K kernel folds the −32 + // bias correction into the maddubs instead of using the activation group sums. + private static unsafe void DispatchTiledQ6K( + ReadOnlySpan input, + int rows, + Q6KWeight w, + Span output, + Span quants, + Span scales, + Span bsums) + { + var inputSize = w.InputSize; + var outputSize = w.OutputSize; + var spr = w.SuperBlocksPerRow; + var bsumsPerRow = spr * Q6KDotKernel.GroupsPerSuperBlock; + + for (var n = 0; n < rows; n++) + { + Q6KDotKernel.QuantizeActivationQ8K( + input.Slice(n * inputSize, inputSize), + quants.Slice(n * inputSize, inputSize), + scales.Slice(n * spr, spr), + bsums.Slice(n * bsumsPerRow, bsumsPerRow)); + } + + var repacked = w.EnsureRepacked(); + + var cores = Environment.ProcessorCount; + var nr = rows / 8 >= cores ? 8 : 4; + if (nr > Q6KGemvKernel.MaxTileCols) + { + nr = Q6KGemvKernel.MaxTileCols; + } + var tiles = (rows + nr - 1) / nr; + + fixed (byte* rp = repacked) + fixed (sbyte* q = quants) + fixed (float* sc = scales) + fixed (float* o = output) + { + var ctx = new TiledQ6KContext + { + Repacked = rp, + RepackedLength = repacked.Length, + Quants = q, + Scales = sc, + Output = o, + InputSize = inputSize, + OutputSize = outputSize, + Spr = spr, + Nr = nr, + Rows = rows, + }; + OverfitParallel.For(0, tiles, &TiledQ6KChunk, &ctx); + } + } + + private unsafe struct TiledQ6KContext + { + public byte* Repacked; + public int RepackedLength; + public sbyte* Quants; + public float* Scales; + public float* Output; + public int InputSize; + public int OutputSize; + public int Spr; + public int Nr; + public int Rows; + } + + private static unsafe void TiledQ6KChunk(int start, int end, void* context) + { + ref var c = ref Unsafe.AsRef(context); + for (var t = start; t < end; t++) + { + var s = t * c.Nr; + var cols = Math.Min(c.Nr, c.Rows - s); + Q6KGemvKernel.GemmTiled( + new ReadOnlySpan(c.Repacked, c.RepackedLength), + c.OutputSize, + c.InputSize, + cols, + new ReadOnlySpan(c.Quants + (long)s * c.InputSize, cols * c.InputSize), + new ReadOnlySpan(c.Scales + (long)s * c.Spr, cols * c.Spr), + new Span(c.Output + (long)s * c.OutputSize, cols * c.OutputSize)); + } + } + private unsafe struct TiledContext { public byte* Repacked; diff --git a/Sources/Main/LanguageModels/Runtime/CachedFeedForwardBlock.cs b/Sources/Main/LanguageModels/Runtime/CachedFeedForwardBlock.cs index 095bf1ca..9572a458 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedFeedForwardBlock.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedFeedForwardBlock.cs @@ -346,11 +346,16 @@ internal void DecodeSwiGluBatchedDispatched( var gate = gateArr.Span; var up = upArr.Span; + var profGateUp = PrefillProfiler.Start(); BatchedQuantProjection.Dispatch(hidden, rows, in wGate, [], gate, DModel, DFF); ApplyGate(gate, Activation); BatchedQuantProjection.Dispatch(hidden, rows, in wUp, [], up, DModel, DFF); TensorPrimitives.Multiply(gate, up, up); + PrefillProfiler.Stop(PrefillProfiler.Component.FfnGateUp, profGateUp); + + var profDown = PrefillProfiler.Start(); BatchedQuantProjection.Dispatch(up, rows, in wDown, [], output.Slice(0, rows * DModel), DFF, DModel); + PrefillProfiler.Stop(PrefillProfiler.Component.FfnDown, profDown); } /// diff --git a/Sources/Main/LanguageModels/Runtime/CachedLlamaSession.cs b/Sources/Main/LanguageModels/Runtime/CachedLlamaSession.cs index 47a325d1..b1f5b5aa 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedLlamaSession.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedLlamaSession.cs @@ -192,7 +192,9 @@ public void Prefill(ReadOnlySpan promptTokens) && _config.FfnActivation is FeedForwardActivation.SwiGLU or FeedForwardActivation.GeGLU && _cache.CurrentLength + promptTokens.Length <= _cache.MaxLength) { + PrefillProfiler.BeginRequest(promptTokens.Length); PrefillBatchedQuant(promptTokens); + PrefillProfiler.EndRequest(); return; } diff --git a/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs b/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs index 895e0fc5..60d58099 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs @@ -658,8 +658,10 @@ internal void DecodeBatchedQuant( } // K/V projected once per group, RoPE-rotated, stored — every Q head reads the cache. + var profKv = PrefillProfiler.Start(); BatchedQuantProjection.Dispatch(hidden, rows, in wk, bk, kg.Span, dModel, headDim); BatchedQuantProjection.Dispatch(hidden, rows, in wv, bv, vg.Span, dModel, headDim); + PrefillProfiler.Stop(PrefillProfiler.Component.AttnKv, profKv); if (weights.HasQkNorm) { QkNormKernel.Apply(kg.Span, weights.QkNormK, rows, headDim); @@ -696,7 +698,9 @@ internal void DecodeBatchedQuant( var wq = hw.Wq; var wo = hw.Wo; + var profQ = PrefillProfiler.Start(); BatchedQuantProjection.Dispatch(hidden, rows, in wq, hw.Bq, qh.Span, dModel, headDim); + PrefillProfiler.Stop(PrefillProfiler.Component.AttnQ, profQ); if (weights.HasQkNorm) { QkNormKernel.Apply(qh.Span, weights.QkNormQ, rows, headDim); @@ -709,9 +713,13 @@ internal void DecodeBatchedQuant( } } + var profScores = PrefillProfiler.Start(); BatchedAttentionKernel.ComputeParallel(qh.Span, keys, values, attn.Span, score.Span, rows, cacheLength, headDim, scale, AttnLogitSoftcap); + PrefillProfiler.Stop(PrefillProfiler.Component.AttnScores, profScores); + var profOut = PrefillProfiler.Start(); BatchedQuantProjection.Dispatch(attn.Span, rows, in wo, [], band.Span, headDim, dModel); + PrefillProfiler.Stop(PrefillProfiler.Component.AttnOut, profOut); for (var n = 0; n < rows; n++) { var outRow = output.Slice(n * dModel, dModel); diff --git a/Sources/Main/LanguageModels/Runtime/CachedTransformerBlock.cs b/Sources/Main/LanguageModels/Runtime/CachedTransformerBlock.cs index d477bc08..665d6fd7 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedTransformerBlock.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedTransformerBlock.cs @@ -425,7 +425,9 @@ internal void DecodeBatchedQuant( } } + var profAttn = PrefillProfiler.Start(); _attention.DecodeBatchedQuant(ln1.Span, rows, in weights, cache, layerIndex, basePosition, attnOut.Span, rope); + PrefillProfiler.Stop(PrefillProfiler.Component.Attention, profAttn); for (var n = 0; n < rows; n++) { @@ -454,6 +456,8 @@ internal void DecodeBatchedQuant( } } + var profFfn = PrefillProfiler.Start(); + if (weights.IsMoe) { _moe!.DecodeBatched( @@ -469,6 +473,8 @@ internal void DecodeBatchedQuant( ln2.Span, rows, weights.FfnGate, weights.FfnW1, weights.FfnW2, ffnOut.Span); } + PrefillProfiler.Stop(PrefillProfiler.Component.Ffn, profFfn); + for (var n = 0; n < rows; n++) { var ffnRow = ffnOut.Span.Slice(n * dModel, dModel); diff --git a/Sources/Main/LanguageModels/Runtime/PrefillProfiler.cs b/Sources/Main/LanguageModels/Runtime/PrefillProfiler.cs new file mode 100644 index 00000000..03436f63 --- /dev/null +++ b/Sources/Main/LanguageModels/Runtime/PrefillProfiler.cs @@ -0,0 +1,185 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Diagnostics; +using System.Runtime.CompilerServices; +using System.Text; + +namespace DevOnBike.Overfit.LanguageModels.Runtime +{ + /// + /// Opt-in per-component profiler for the BATCHED PREFILL path — the counterpart to + /// , which hooks only single-token decode and therefore reports nothing about + /// time-to-first-token. + /// + /// Why this exists. Prefill was measured at 3.76× behind llama.cpp on the same file + /// (144 vs 541.7 tok/s, 672-token prompt), and that gap decomposes as 2.34× kernel quality at equal + /// instruction set × 1.60× AVX-512. Deciding what to fix first requires knowing whether the 2.34× sits in + /// the FFN matmuls or in the attention path — where Q and O are dispatched once per head, each + /// re-quantizing the same loop-invariant activation matrix. Guessing that split has already been wrong + /// three times, so it gets measured. + /// + /// Off by default. When is false every hook is one predicted-false + /// branch — no timestamp, no allocation. Hooks sit at layer and projection granularity (tens per request), + /// never per element, so even the branch is immeasurable. Accumulators are a static singleton: this + /// profiles one prefill at a time and is a diagnostic, not a concurrent-safe meter. + /// + public static class PrefillProfiler + { + /// Prefill components timed independently. + public enum Component + { + /// Whole batched attention block for one layer. TOP-LEVEL. + Attention = 0, + + /// Whole batched FFN for one layer (SwiGLU / MoE). TOP-LEVEL. + Ffn = 1, + + /// K and V projections — once per KV group, not per head. Sub-slice of . + AttnKv = 2, + + /// Q projection — dispatched once PER HEAD over the same activations. Sub-slice of . + AttnQ = 3, + + /// Causal scores + weighted sum over the KV cache. Sub-slice of . + AttnScores = 4, + + /// Output projection — also dispatched per head. Sub-slice of . + AttnOut = 5, + + /// FFN gate+up projections and the gate activation. Sub-slice of . + FfnGateUp = 6, + + /// FFN down projection. Sub-slice of . + FfnDown = 7, + } + + private const int ComponentCount = 8; + private const int LastTopLevel = (int)Component.Ffn; + + private static readonly long[] _ticks = new long[ComponentCount]; + private static readonly long[] _calls = new long[ComponentCount]; + private static long _requestTicks; + private static long _requestStart; + private static long _requests; + private static long _rows; + + /// Master switch. Leave false in production; flip on around a measured prefill. + public static bool Enabled; + + /// Timestamp to pass to a matching . Cheap no-op when off. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static long Start() => Enabled ? Stopwatch.GetTimestamp() : 0L; + + /// Accumulate elapsed ticks (and one call) for . + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void Stop(Component component, long start) + { + if (!Enabled) + { + return; + } + + _ticks[(int)component] += Stopwatch.GetTimestamp() - start; + _calls[(int)component]++; + } + + /// Mark the start of one prefill request over prompt tokens. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void BeginRequest(int rows) + { + if (!Enabled) + { + return; + } + + _requestStart = Stopwatch.GetTimestamp(); + _rows += rows; + } + + /// Close the current request and add it to the totals. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + public static void EndRequest() + { + if (!Enabled) + { + return; + } + + _requestTicks += Stopwatch.GetTimestamp() - _requestStart; + _requests++; + } + + /// Clear all accumulators (call before the measured segment). + public static void Reset() + { + Array.Clear(_ticks); + Array.Clear(_calls); + _requestTicks = 0; + _requestStart = 0; + _requests = 0; + _rows = 0; + } + + /// Prompt tokens prefilled since the last . + public static long Rows => _rows; + + /// + /// Per-request breakdown: ms, % of prefill wall time, and calls. Also prints the prefill rate in + /// tok/s, which is directly comparable to llama-bench -p N -n 0. + /// + public static string Report() + { + var sb = new StringBuilder(); + var toMs = 1000.0 / Stopwatch.Frequency; + var requests = _requests == 0 ? 1 : _requests; + var requestMs = _requestTicks * toMs / requests; + var rowsPerRequest = (double)_rows / requests; + + sb.AppendLine($"=== PrefillProfiler ({_requests} request(s), {rowsPerRequest:F0} tokens each) ==="); + sb.AppendLine( + $" total/request : {requestMs,9:F1} ms ({(requestMs > 0 ? rowsPerRequest * 1000.0 / requestMs : 0),7:F0} tok/s)"); + + // Only TOP-LEVEL components count toward "accounted" — the sub-slices overlap their parent, and + // summing them too would subtract the same work twice and drive `other` negative. DecodeProfiler + // shipped with exactly that bug (it read -67%), so the same mistake is not repeated here. + long accounted = 0; + for (var i = 0; i < ComponentCount; i++) + { + var ms = _ticks[i] * toMs / requests; + if (i <= LastTopLevel) + { + accounted += _ticks[i]; + } + + var pct = _requestTicks > 0 ? 100.0 * _ticks[i] / _requestTicks : 0; + var perRequest = (double)_calls[i] / requests; + var indent = i > LastTopLevel ? " " : string.Empty; + sb.AppendLine( + $" {indent + ComponentName(i),-14} : {ms,9:F1} ms {pct,5:F1}% ({perRequest,6:F0} calls)"); + } + + var otherTicks = _requestTicks - accounted; + var otherMs = otherTicks * toMs / requests; + var otherPct = _requestTicks > 0 ? 100.0 * otherTicks / _requestTicks : 0; + sb.AppendLine( + $" {"other",-14} : {otherMs,9:F1} ms {otherPct,5:F1}% (norms/residual/embed/finalnorm/RoPE)"); + return sb.ToString(); + } + + private static string ComponentName(int i) => i switch + { + 0 => "attention", + 1 => "ffn", + 2 => "attn_kv", + 3 => "attn_q", + 4 => "attn_scores", + 5 => "attn_out", + 6 => "ffn_gateup", + 7 => "ffn_down", + _ => "?", + }; + } +} diff --git a/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs b/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs index fcf789db..1a00144d 100644 --- a/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs +++ b/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs @@ -263,6 +263,146 @@ private static unsafe void ComputeGroupRange( } } + /// Max activation columns per call — the register-tile width. The + /// caller splits a longer prompt into tiles of this many columns. + public const int MaxTileCols = 16; + + /// + /// Register-tiled Q6_K prefill GEMM over the repacked block_q6_Kx8 layout: produces + /// output columns (prompt tokens) at once, unpacking each weight super-block + /// once and reusing it across every column — the loop the decode has + /// nothing to tile. + /// + /// Why this and not the weight-stationary shape. A weight-stationary Q6_K kernel was + /// built first, modelled on the Q4_K one, and measured 13.5% slower: it hoisted the whole 6-bit + /// unpack into a stack buffer, so each row paid a store+reload through L1 instead of consuming the + /// quants from registers, and inverting the loops made activation reads strided. Tiling keeps the + /// unpacked quants in registers and amortises them across columns instead — which is exactly + /// why the Q4_K tiled kernel measures ~3.3× over its own weight-stationary variant. + /// + /// Bit-identical to per column: the per-(row, column) operation + /// sequence and accumulation order are unchanged; only weight decoding moves outward. Layout matches + /// the Q4_K tiled kernel — activations column-contiguous, output column-major + /// (output[c*outputSize + row]). AVX2 + FMA. + /// + public static unsafe void GemmTiled( + ReadOnlySpan repacked, + int outputSize, + int inputSize, + int cols, + ReadOnlySpan actQuants, + ReadOnlySpan actScales, + Span output) + { + if (cols is < 1 or > MaxTileCols) + { + throw new ArgumentOutOfRangeException(nameof(cols), cols, $"cols must be in [1, {MaxTileCols}]."); + } + + var nb = inputSize / 256; + + var m4b = Vector256.Create((byte)0x0F); + var m2 = Vector256.Create((byte)0x03); + var m32 = Vector256.Create((byte)32); + var ones = Vector256.Create((short)1); + var reduce = Vector256.Create(0, 1, 4, 5, 2, 3, 6, 7); + + // Per-column accumulators. cols <= MaxTileCols keeps this a small bounded frame. + Span> sumf = stackalloc Vector256[cols]; + Span> iacc = stackalloc Vector256[cols]; + + fixed (byte* rep = repacked) + fixed (sbyte* aqAll = actQuants) + fixed (float* asc = actScales) + fixed (float* outp = output) + { + for (var x = 0; x < outputSize / 8; x++) + { + var bptr = rep + (long)x * nb * BlockKx8Bytes; + + for (var c = 0; c < cols; c++) + { + sumf[c] = Vector256.Zero; + } + + for (var l = 0; l < nb; l++) + { + var blk = bptr + (long)l * BlockKx8Bytes; + var scales = blk + DstScalesOffset; + var ql = blk + DstQlOffset; + var qh = blk + DstQhOffset; + var dVec = LoadF16x8Int(blk); + + for (var c = 0; c < cols; c++) + { + iacc[c] = Vector256.Zero; + } + + for (var k = 0; k < 16; k++) + { + var baseL = (k / 8) * 128 + (k % 8) * 8; + var baseH = baseL + 64; + var qhShiftL = (byte)(((baseL % 128) / 32) * 2); + var qhShiftH = (byte)(((baseH % 128) / 32) * 2); + var qhHalfL = (baseL / 128) * 32; + var qhHalfH = (baseH / 128) * 32; + var qhBlockL = ((qhHalfL + (baseL % 32)) / 8) * 64; + var qhBlockH = ((qhHalfH + (baseH % 32)) / 8) * 64; + + // ── Weight side: decoded ONCE, reused across every column (the tiling win). ── + var ql03 = Vector256.Load(ql + k * 64); + var ql47 = Vector256.Load(ql + k * 64 + 32); + var qhL03 = Vector256.Load(qh + qhBlockL); + var qhL47 = Vector256.Load(qh + qhBlockL + 32); + var qhH03 = Vector256.Load(qh + qhBlockH); + var qhH47 = Vector256.Load(qh + qhBlockH + 32); + + var qLu03 = Avx2.Or(LoNib(ql03, m4b), QhBits(qhL03, qhShiftL, m2)); + var qLu47 = Avx2.Or(LoNib(ql47, m4b), QhBits(qhL47, qhShiftL, m2)); + var qHu03 = Avx2.Or(HiNib(ql03, m4b), QhBits(qhH03, qhShiftH, m2)); + var qHu47 = Avx2.Or(HiNib(ql47, m4b), QhBits(qhH47, qhShiftH, m2)); + + var scaleL = ScaleVec(scales + (baseL / 16) * 8); + var scaleH = ScaleVec(scales + (baseH / 16) * 8); + + // ── Activation side: per column. ── + for (var c = 0; c < cols; c++) + { + var aqs = aqAll + (long)c * inputSize + l * 256; + var actL = TileAct(aqs + baseL); + var actH = TileAct(aqs + baseH); + + var sumL = ReduceRows( + Avx2.Subtract(Avx2.MultiplyAddAdjacent(qLu03, actL), Avx2.MultiplyAddAdjacent(m32, actL)), + Avx2.Subtract(Avx2.MultiplyAddAdjacent(qLu47, actL), Avx2.MultiplyAddAdjacent(m32, actL)), + ones, reduce); + var sumH = ReduceRows( + Avx2.Subtract(Avx2.MultiplyAddAdjacent(qHu03, actH), Avx2.MultiplyAddAdjacent(m32, actH)), + Avx2.Subtract(Avx2.MultiplyAddAdjacent(qHu47, actH), Avx2.MultiplyAddAdjacent(m32, actH)), + ones, reduce); + + iacc[c] = Avx2.Add(iacc[c], Avx2.Add( + Avx2.MultiplyLow(sumL, scaleL), Avx2.MultiplyLow(sumH, scaleH))); + } + } + + for (var c = 0; c < cols; c++) + { + sumf[c] = Fma.MultiplyAdd( + Avx.ConvertToVector256Single(iacc[c]), + Avx.Multiply(dVec, Vector256.Create(asc[(long)c * nb + l])), + sumf[c]); + } + } + + for (var c = 0; c < cols; c++) + { + sumf[c].Store(outp + (long)c * outputSize + x * 8); + } + } + } + } + [MethodImpl(MethodImplOptions.AggressiveInlining)] private static Vector256 LoNib(Vector256 v, Vector256 m4b) => Avx2.And(v, m4b); diff --git a/Tests/LanguageModels/Diagnostics/PrefillProfileTests.cs b/Tests/LanguageModels/Diagnostics/PrefillProfileTests.cs index 004a640a..89a55e12 100644 --- a/Tests/LanguageModels/Diagnostics/PrefillProfileTests.cs +++ b/Tests/LanguageModels/Diagnostics/PrefillProfileTests.cs @@ -112,5 +112,57 @@ static double Median(double[] values) Assert.True(promptLength > 128, $"prompt too short to exercise batched prefill ({promptLength} tokens)"); } + + /// + /// Splits prefill into attention vs FFN via . This is the measurement + /// that decides where the 2.34×-at-equal-ISA gap to llama.cpp actually sits: in the FFN matmuls, or + /// in the attention path where Q and O are dispatched once per head over the same activations. + /// + [LongFact] + public void Prefill_ComponentBreakdown() + { + var path = TestModelPaths.Qwen3B.Q4KmGgufPath; + if (!File.Exists(path)) + { + _out.WriteLine($"missing {path}"); + return; + } + + using var engine = CachedLlamaInferenceEngine.LoadGguf(path); + var tok = GgufTokenizer.Load(path); + var sampling = SamplingOptions.Greedy; + + var paragraph = string.Join(" ", + Enumerable.Repeat( + "The history of computing began with mechanical calculators and evolved through vacuum tubes, " + + "transistors, integrated circuits and finally the microprocessor era.", 24)); + var ids = tok.Encode(paragraph); + + // Warm up OUTSIDE the profiled region: JIT, page-in, one-off repack. + using (var warm = engine.CreateSession(1024)) + { + warm.Reset(ids); + warm.GenerateNextToken(in sampling); + } + + PrefillProfiler.Reset(); + PrefillProfiler.Enabled = true; + try + { + const int Runs = 3; + for (var r = 0; r < Runs; r++) + { + using var session = engine.CreateSession(1024); + session.Reset(ids); + } + } + finally + { + PrefillProfiler.Enabled = false; + } + + _out.WriteLine(PrefillProfiler.Report()); + Assert.True(PrefillProfiler.Rows > 0, "profiler recorded no prefill rows — hooks not reached"); + } } } diff --git a/Tests/LanguageModels/Runtime/Q6KTiledGemmParityTests.cs b/Tests/LanguageModels/Runtime/Q6KTiledGemmParityTests.cs new file mode 100644 index 00000000..eca3f07d --- /dev/null +++ b/Tests/LanguageModels/Runtime/Q6KTiledGemmParityTests.cs @@ -0,0 +1,142 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Runtime.Intrinsics.X86; +using DevOnBike.Overfit.LanguageModels.Loading; +using DevOnBike.Overfit.LanguageModels.Runtime; + +namespace DevOnBike.Overfit.Tests.LanguageModels.Runtime +{ + /// + /// Pins — the register-tiled Q6_K prefill GEMM over the repacked + /// block_q6_Kx8 layout — as bit-identical to calling the validated decode + /// once per activation column. Same weights, same Q8_K activations, + /// same per-(row, column) reduction order; only the weight unpack moves out of the column loop. + /// + /// This exists because Q6_K carries half of ffn_down under Q4_K_M, which a prefill profile + /// measured at 37.9% of prefill running at 0.61 TFLOP/s. The first attempt at closing that gap — a + /// weight-stationary kernel — was bit-identical but 13.5% slower, so correctness alone is not the + /// bar here; this test is the gate that lets the performance question be asked honestly. + /// + public sealed class Q6KTiledGemmParityTests + { + [Theory] + [InlineData(1)] + [InlineData(2)] + [InlineData(3)] + [InlineData(8)] + [InlineData(16)] + public void GemmTiled_MatchesGemvPerColumn_BitIdentical(int cols) + { + if (!Avx2.IsSupported || !Fma.IsSupported) + { + return; // AVX2/FMA kernel — nothing to verify on this CPU + } + + const int inputSize = 512; // 2 Q6_K super-blocks + const int outputSize = 64; // 8 row-groups + var nb = inputSize / 256; + var bsumsPerRow = nb * Q6KDotKernel.GroupsPerSuperBlock; + + var rng = new Random(6060 + cols); + + // Any byte pattern is a valid Q6_K super-block as long as the fp16 scale is sane — same + // construction the existing Q6KDotKernelTests use, so no quantizer is needed. + var blocks = new byte[outputSize * nb * Q6KWeight.SuperBlockBytes]; + for (var b = 0; b < outputSize * nb; b++) + { + var block = blocks.AsSpan(b * Q6KWeight.SuperBlockBytes, Q6KWeight.SuperBlockBytes); + for (var i = 0; i < block.Length; i++) + { + block[i] = (byte)rng.Next(256); + } + + // Random bits at offset 208 could decode to NaN/Inf — overwrite with a small positive fp16. + var d = (Half)(rng.NextDouble() * 0.05 + 0.001); + BitConverter.GetBytes(BitConverter.HalfToUInt16Bits(d)).CopyTo(block.Slice(208, 2)); + } + + var weight = new Q6KWeight(blocks, inputSize, outputSize); + Assert.True(weight.CanRepack); + var repacked = weight.EnsureRepacked().ToArray(); + + var inputs = new float[cols * inputSize]; + for (var i = 0; i < inputs.Length; i++) + { + inputs[i] = (float)(rng.NextDouble() * 2.0 - 1.0); + } + + var aqAll = new sbyte[cols * inputSize]; + var ascAll = new float[cols * nb]; + var abAll = new short[cols * bsumsPerRow]; + for (var c = 0; c < cols; c++) + { + Q6KDotKernel.QuantizeActivationQ8K( + inputs.AsSpan(c * inputSize, inputSize), + aqAll.AsSpan(c * inputSize, inputSize), + ascAll.AsSpan(c * nb, nb), + abAll.AsSpan(c * bsumsPerRow, bsumsPerRow)); + } + + var tiled = new float[cols * outputSize]; + Q6KGemvKernel.GemmTiled(repacked, outputSize, inputSize, cols, aqAll, ascAll, tiled); + + var reference = new float[outputSize]; + for (var c = 0; c < cols; c++) + { + Q6KGemvKernel.GemvAvx2( + repacked, outputSize, inputSize, + aqAll.AsSpan(c * inputSize, inputSize), + ascAll.AsSpan(c * nb, nb), + reference); + + for (var o = 0; o < outputSize; o++) + { + Assert.Equal(reference[o], tiled[c * outputSize + o]); + } + } + } + + [Fact] + public void GemmTiled_RejectsOutOfRangeColumnCount() + { + if (!Avx2.IsSupported || !Fma.IsSupported) + { + return; + } + + const int inputSize = 256; + const int outputSize = 8; + var repacked = new byte[Q6KRepack.BlockKx8Bytes]; + var aq = new sbyte[inputSize]; + var asc = new float[1]; + var output = new float[outputSize]; + + var threwZero = false; + try + { + Q6KGemvKernel.GemmTiled(repacked, outputSize, inputSize, 0, aq, asc, output); + } + catch (ArgumentOutOfRangeException) + { + threwZero = true; + } + + var threwTooMany = false; + try + { + Q6KGemvKernel.GemmTiled( + repacked, outputSize, inputSize, Q6KGemvKernel.MaxTileCols + 1, aq, asc, output); + } + catch (ArgumentOutOfRangeException) + { + threwTooMany = true; + } + + Assert.True(threwZero, "expected ArgumentOutOfRangeException for cols = 0"); + Assert.True(threwTooMany, $"expected ArgumentOutOfRangeException for cols > {Q6KGemvKernel.MaxTileCols}"); + } + } +} From fd5c136467968a739b8a1cba3209d097754cfac0 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Wed, 22 Jul 2026 16:24:59 +0200 Subject: [PATCH 10/37] cluude --- .claude/settings.json | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/.claude/settings.json b/.claude/settings.json index a5e7a63f..00ab769d 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -32,6 +32,20 @@ "PowerShell(Measure-Object *)", "PowerShell(Get-Command *)", "PowerShell(Test-Path *)", + "Write(D:\\Overfit\\.claude\\run.py)", + "Edit(D:\\Overfit\\.claude\\run.py)", + "Write(.claude/run.py)", + "Edit(.claude/run.py)", + "Bash(python D:\\Overfit\\.claude\\run.py)", + "Bash(python .claude/run.py)", + "PowerShell(python D:\\Overfit\\.claude\\run.py)", + "Write(D:\\Overfit\\.claude\\run.ps1)", + "Edit(D:\\Overfit\\.claude\\run.ps1)", + "Write(.claude/run.ps1)", + "Edit(.claude/run.ps1)", + "PowerShell(pwsh -File D:\\Overfit\\.claude\\run.ps1)", + "PowerShell(pwsh -File .claude/run.ps1)", + "PowerShell(pwsh -File *)", "PowerShell(cmake *)", "Bash(cmake *)", "PowerShell($cm = *)", From 9b666909cf511c5b5e190f74cfbd9ec733576c8a Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Wed, 22 Jul 2026 16:46:57 +0200 Subject: [PATCH 11/37] llama --- .claude/settings.json | 3 + ROADMAP.md | 125 ++++++++++ .../Q4KPrefillProjectionBenchmark.cs | 26 +++ .../Runtime/BatchedQuantProjection.cs | 217 ++++++++++++------ .../Runtime/CachedMultiHeadAttention.cs | 107 ++++++++- .../LanguageModels/Runtime/Q4KDotKernel.cs | 44 ++-- .../LanguageModels/Runtime/Q6KDotKernel.cs | 22 +- .../Parity/BatchedPrefillParityTests.cs | 97 ++++++++ 8 files changed, 540 insertions(+), 101 deletions(-) diff --git a/.claude/settings.json b/.claude/settings.json index 00ab769d..ddf349ef 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -36,6 +36,9 @@ "Edit(D:\\Overfit\\.claude\\run.py)", "Write(.claude/run.py)", "Edit(.claude/run.py)", + "Bash(python D:/Overfit/.claude/run.py)", + "Bash(python3 D:/Overfit/.claude/run.py)", + "PowerShell(python D:/Overfit/.claude/run.py)", "Bash(python D:\\Overfit\\.claude\\run.py)", "Bash(python .claude/run.py)", "PowerShell(python D:\\Overfit\\.claude\\run.py)", diff --git a/ROADMAP.md b/ROADMAP.md index eacc5192..7908a197 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -242,6 +242,131 @@ tensors on first use. Worth revisiting if RAM matters more than TTFT. **Gap to llama.cpp: 3.76× → 2.93×.** Remaining, by measured share: `ffn_gateup` 34.2%, `attn_q` + `attn_out` 28.9% (the per-head dispatches, 576 calls each), `attn_scores` 6.8%. AVX-512 (ceiling 1.60×) still last. +#### ▶▶ NEXT LEVER (sized 2026-07-22): hoist activation quantization out of the per-head loop — ~18.8% + +`Q4KPrefillProjectionBenchmark.QuantizeActivationsOnly` measures Q8_K quantization of `672 × 2048` +activations at **~1.0 ms**. Against the profile: + +| | dispatches over `hidden` | quantization cost | actually needed | +|---|---:|---:|---:| +| `attn_q` (621.7 ms / 576 calls = 1.079 ms) | 576 | ~576 ms | — | +| `attn_kv` (181.4 ms) | 144 | ~144 ms | — | +| **total** | **720** | **~720 ms** | **36** (once per layer) | + +So **~93% of a Q-head dispatch is activation quantization** — the projection itself is 2048→128, roughly +0.08 ms. `hidden` is loop-invariant across heads, so the same matrix is quantized 16× per layer. `attn_out` +is NOT affected: its input is the per-head `attn` band. + +**Recoverable ≈ 684 ms of 3632.6 ms ≈ 18.8% → prefill 185 → ~228 tok/s.** + +Decode already fixed exactly this in 2026-05 (`ProjectPreQuantized`, "hidden was re-quantized per head, now +quantized once per layer"); the batched prefill path never got the equivalent. + +#### ✅ SHIPPED — shared activation quantization: 185 → 194 tok/s (1.05×), but 3.3× short of the estimate + +`BatchedQuantProjection.Dispatch` takes optional pre-quantized Q8_K scratch; +`CachedMultiHeadAttention.DecodeBatchedQuant` quantizes `hidden` once per layer and passes it to every Q/K/V +dispatch. Q4_K and Q6_K share the Q8_K format bit-for-bit, so one buffer serves all three. + +| component | before | after | Δ | +|---|---:|---:|---:| +| `attn_q` | 621.7 ms | **456.5 ms** | −26.6% | +| `attn_kv` | 181.4 ms | **139.3 ms** | −23.2% | +| `ffn_gateup` *(canary)* | 1242.3 ms | 1211.0 ms | −2.5% | +| `attn_out` *(canary)* | 427.4 ms | 427.7 ms | +0.1% | +| **prefill total** | **3632.6 ms · 185 tok/s** | **3462.3 ms · 194 tok/s** | **−4.7% · 1.05×** | + +**The estimate said ~684 ms; the measurement says ~207 ms — 3.3× optimistic.** Cause: the sizing benchmark +timed quantization of a 672×2048 block **in isolation** (~1.0 ms), i.e. reading 5.5 MB cold. In production +the 16 repeats run back-to-back on a cache-resident `hidden`, so the redundant passes were far cheaper than +the isolated measurement implied. **Lesson: an operation benchmarked alone over-states its cost when the +thing you are removing is a repeat on hot data — size the repeat, not the first call.** + +#### ✅ SHIPPED — whole-matrix O projection: 194 → 219 tok/s (1.13×) + +Per head the O projection is `[headDim → dModel]`, and **headDim (128) is not a multiple of the 256-element +Q4_K super-block**, so `CanRepack` is false and all 16 dispatches per layer were stuck on the +weight-stationary kernel. The whole matrix is `[nHeads·headDim → dModel]` = 2048 wide, which *does* repack. +`BlockWeights.WoWhole` was already loaded **zero-copy from the mmap** (and prepacked when a sidecar exists), +so this costs no extra RAM — it only needed the per-head bands concatenated before one dispatch. + +| component | before | after | Δ | +|---|---:|---:|---:| +| `attn_out` | 427.7 ms / 576 calls | **109.6 ms / 36 calls** | **−74.4%** (3.9×) | +| `attn_q` *(canary)* | 456.5 ms | 459.1 ms | +0.6% | +| `ffn_gateup` *(canary)* | 1211.0 ms | 1222.7 ms | +1.0% | +| **prefill total** | **3462.3 ms · 194 tok/s** | **3068.5 ms · 219 tok/s** | **−11.4% · 1.13×** | + +**Gate on `WoWhole.IsQ4K`, NOT `HasWholeAttnQ4K`.** The latter also demands Q/K/V, and under Q4_K_M `attn_v` +is Q6_K in half the layers — so the four-way gate enabled this in only 18 of 36. The measurement caught it: +`attn_out` reported **306 calls** (18 layers × 16 heads + 18 × 1) instead of 36, and fixing the gate roughly +doubled the win. + +Contracting all heads inside one matmul reassociates a sum the per-head path does in head order, so +`useWholeO` also honours `DisableRepackedKernelsForParity` — without that the batched-vs-single-token parity +test can never reach its 1e-2 bound. + +#### ▶ WHAT IS LEFT — profile after the three wins (219 tok/s, gap 2.47×) + +``` +ffn_gateup 1222.7 ms 39.8% (36) <- Q4_K tiled already +ffn_down 720.1 ms 23.5% (36) <- Q6_K tiled already +attn_q 459.1 ms 15.0% (576) <- weight-stationary: blocked by `bias.IsEmpty` +attn_scores 251.1 ms 8.2% (576) +attn_kv 140.3 ms 4.6% (72) +attn_out 109.6 ms 3.6% (36) <- done +other 105.1 ms 3.4% +``` + +**The structural waste is spent.** Every remaining component is already on the best kernel Overfit has, with +two exceptions: + +1. **`attn_q` — 15.0%, and it is blocked by one gate, not by shape.** Per-head Q is `[2048 → 128]`: + `inputSize % 256 == 0` ✓ and `outputSize % 8 == 0` ✓, so **`CanRepack` is TRUE** — the only thing keeping + it off the tiled kernel is `bias.IsEmpty` (Qwen puts a bias on Q/K/V). Micro-bench for that shape class: + tiled 4.41 ms vs weight-stationary 12.95 ms; measured `attn_q` is 12.75 ms/layer. **Ceiling ≈ 300 ms of + 3068 ≈ 9.8% → ~243 tok/s.** + Bias support in `GemmTiled` was built once and reverted on a measured **0.999× tie** — but that tie was + taken when the biased projections were ~6% of FLOPs and the FFN dwarfed them. The composition has changed; + **re-measure before rebuilding, and re-measure with the FLOP-weighted census, not the dispatch count.** + +2. **`attn_scores` — 8.2%, never examined.** `BatchedAttentionKernel.ComputeParallel` has had no profiling + pass at all. + +**Everything else is kernel quality, i.e. writing better SIMD.** The measured headline: llama.cpp built +AVX2-only does 336.7 tok/s against our 219 — so **1.54× of the remaining 2.47× is pure kernel craft at equal +instruction set**, and AVX-512 accounts for the other 1.60×. Both are intrinsics work on `GemmTiled` +(register-blocking the accumulators, 512-bit lanes), not structural fixes. Expect weeks, not evenings, and +size each step against its share before building. + +#### ✅ RESOLVED — `BatchedPrefillParityTests` (was failing since before this work) + +`BatchedPrefill_MatchesSingleToken_OnRealQwen` asserts `maxAbsLogitDiff == 0` between batched prefill and the +single-token path. It now reports `argmax batched=11 single=13, maxAbsLogitDiff ≈ 0.44`. + +**Not caused by the changes above.** Disabling *both* repacked paths (Q6_K tiled off AND the `IsPrepacked` +short-circuit removed from the Q4_K gate) makes it pass 5/5 — with the shared quantization still enabled, +which also proves that change is bit-identical. The trigger is the `*.gguf.repack` sidecar created +2026-07-20: it sets `IsPrepacked`, routing bias-free Q4_K projections through the repacked `GemmTiled`, whose +reduction is associated differently. The Q6_K tiled kernel is the same class of change and breaks it +independently. + +**Nobody noticed because the test is `[LongFact]`** — skipped by default, so a numerics regression sat +unobserved for two days. Decision needed: either make the test explicitly disable the repacked paths (so it +keeps testing the batched-vs-single-token *math* it claims to), or replace the exact-equality gate with a +coherence check, as `OVERFIT_REPACK_ATTN` already is. Do not silently relax it. + +**Plan.** Q4_K and Q6_K share the Q8_K scratch format bit-for-bit (`SuperBlockElements 256`, `GroupSize 16`, +and `Q6KDotKernel.QuantizeActivationQ8K` delegates to Q4_K's), so ONE pre-quantized buffer serves Q, K and V +regardless of whether V is Q4_K or Q6_K. Steps: (1) add `bool preQuantized = false` to the three batched +kernel entry points — `Q4KDotKernel.ProjectBatched` / `ProjectBatchedWeightStationary`, +`Q6KDotKernel.ProjectBatched` — guarding their internal quantize loop (anchor: the +`"Activation quantization scratch is too small for rows."` validation, which occurs exactly at those three); +(2) give `BatchedQuantProjection.Dispatch` optional pre-quantized scratch spans, defaulting to today's +pooled-and-quantize behaviour; (3) quantize `hidden` once at the top of +`CachedMultiHeadAttention.DecodeBatchedQuant` and pass it to the Q/K/V dispatches. Output must stay +bit-identical — quantization is deterministic, so this is a pure de-duplication. + At 3.4 B params × 672 tokens the gap is ≈3.7 TFLOP/s-equivalent for them against ≈1.0 for us. **Why this lever is different from the five that were refuted:** it has a measured ceiling, a named cause, and diff --git a/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs b/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs index c20939b3..9d97dc41 100644 --- a/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs +++ b/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs @@ -144,6 +144,32 @@ public void Tiled() BatchedQuantProjection.Dispatch(_input, Rows, in _weight, [], _output, _inputSize, _outputSize); } + /// + /// Q8_K activation quantization ALONE, for the same rows × inputSize the projections consume. + /// + /// This sizes the next lever. Attention dispatches Q once per head over a + /// loop-invariant hidden, so this cost is paid 16× per layer where once would do — a prefill + /// profile put attn_q at 621.7 ms across 576 calls. Decode already fixed exactly this + /// (ProjectPreQuantized, 2026-05); prefill never got the equivalent. What this benchmark + /// answers is whether the redundant share is worth ~5% or ~13% of prefill — two estimates that + /// differ by enough to change the decision. + /// + [Benchmark] + public void QuantizeActivationsOnly() + { + var superBlocksPerRow = _q4k.SuperBlocksPerRow; + var bsumsPerRow = superBlocksPerRow * Q4KDotKernel.GroupsPerSuperBlock; + + for (var n = 0; n < Rows; n++) + { + Q4KDotKernel.QuantizeActivationQ8K( + _input.AsSpan(n * _inputSize, _inputSize), + _quants.AsSpan(n * _inputSize, _inputSize), + _scales.AsSpan(n * superBlocksPerRow, superBlocksPerRow), + _bsums.AsSpan(n * bsumsPerRow, bsumsPerRow)); + } + } + /// The original re-decode-per-row kernel — kept as the reference the kernel docs' "~3×" claim /// is actually measured against. [Benchmark] diff --git a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs index 7340fdfe..034327b5 100644 --- a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs +++ b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs @@ -38,6 +38,37 @@ internal static class BatchedQuantProjection /// use. internal static bool UseTiledPrefillQ6K = true; + /// + /// Test hook: forces the NON-repacked batched kernels for both Q4_K and Q6_K, overriding even + /// IsPrepacked. Mirrors CachedLlamaSession.DisableBatchedPrefillForParity. + /// + /// Needed because the repacked block_q*_Kx8 GEMMs associate their reduction differently + /// from the per-row kernels, so they are not bit-identical to the single-token path — measured + /// at maxAbsLogitDiff ≈ 0.44 on Qwen-3B, enough to flip an argmax. That is the accepted trade + /// (the same standard OVERFIT_REPACK_ATTN is held to: validated by end-to-end coherence, not + /// byte-parity), but it means a test asserting batched == single-token has to hold the kernel layout + /// constant, or it silently stops testing the thing it claims to. + /// + /// A *.gguf.repack sidecar sets IsPrepacked and therefore turns the repacked path + /// on regardless of the env flag — which is exactly how BatchedPrefillParityTests came to be + /// failing unnoticed for two days, being [LongFact]. + /// + internal static bool DisableRepackedKernelsForParity; + + /// + /// / / let the + /// caller supply activations ALREADY quantized to Q8_K, skipping the internal quantization pass. + /// Empty (the default) keeps the original behaviour: pool the scratch and quantize here. + /// + /// Attention needs this because it dispatches Q once per head and K/V once per group, + /// every one of them over the same loop-invariant hidden — a benchmark measured the Q8_K + /// quantization of a 672×2048 activation block at ~1.0 ms against a 1.079 ms Q-head dispatch, i.e. + /// ~93% of the call. Quantizing once per layer is bit-identical, since the quantization is + /// deterministic. + /// + /// Honoured for the Q6_K and Q4_K paths (everything attention uses); the Q8_0 and F32 paths + /// ignore it and quantize as before. + /// public static void Dispatch( ReadOnlySpan input, int rows, @@ -45,10 +76,32 @@ public static void Dispatch( ReadOnlySpan bias, Span output, int inputSize, - int outputSize) + int outputSize, + Span preQuants = default, + Span preScales = default, + Span preBsums = default) { // Resident-format dispatch, classified once so the original first-match order is explicit. var kind = weight.IsQ6K ? 0 : weight.IsQ4K ? 1 : weight.IsQuantized ? 2 : 3; + var pre = !preQuants.IsEmpty; + + if (kind == 0 && pre) + { + var wp = weight.Quantized6K; + DispatchQ6K( + input, rows, wp, bias, output, inputSize, + preQuants, preScales, preBsums, preQuantized: true); + return; + } + + if (kind == 1 && pre) + { + var wp = weight.Quantized4K; + DispatchQ4K( + input, rows, wp, bias, output, inputSize, + preQuants, preScales, preBsums, preQuantized: true); + return; + } if (kind == 0) { @@ -58,28 +111,10 @@ public static void Dispatch( using var qBytes = new PooledBuffer(rows * inputSize, clearMemory: false); using var scales = new PooledBuffer(rows * spr, clearMemory: false); using var sums = new PooledBuffer(groups, clearMemory: false); - // Register-tiled Q6_K GEMM over the repacked block_q6_Kx8 layout. Under Q4_K_M half of - // ffn_down is Q6_K, and a prefill profile put ffn_down at 37.9% of prefill running at - // 0.61 TFLOP/s — against ffn_gate_up's 1.78 — precisely because Q6_K had only the - // re-decode-per-row kernel below. No-bias only (GemmTiled applies none); AVX2/FMA required. - var tiled6 = UseTiledPrefillQ6K && bias.IsEmpty && w.CanRepack - && CpuFeatures.HasAvx2 && CpuFeatures.HasFma; - - if (tiled6) - { - DispatchTiledQ6K( - input, rows, w, output, - qBytes.Span.Slice(0, rows * inputSize), scales.Span.Slice(0, rows * spr), - sums.Span.Slice(0, groups)); - } - - if (!tiled6) - { - Q6KDotKernel.ProjectBatched( - input, rows, w, bias, output, - qBytes.Span.Slice(0, rows * inputSize), scales.Span.Slice(0, rows * spr), - sums.Span.Slice(0, groups)); - } + DispatchQ6K( + input, rows, w, bias, output, inputSize, + qBytes.Span.Slice(0, rows * inputSize), scales.Span.Slice(0, rows * spr), + sums.Span.Slice(0, groups), preQuantized: false); } if (kind == 1) { @@ -89,40 +124,10 @@ public static void Dispatch( using var qBytes = new PooledBuffer(rows * inputSize, clearMemory: false); using var scales = new PooledBuffer(rows * spr, clearMemory: false); using var sums = new PooledBuffer(groups, clearMemory: false); - - // Register-tiled GEMM: repacked block_q4_Kx8, decode each super-block once and reuse across a - // tile of NR columns — measured ~3× vs weight-stationary under parallelism, 1.61× end-to-end - // prefill. Default-on when the weight is already prepacked (an offline sidecar mmap'd it → zero - // extra RAM); otherwise opt-in via OVERFIT_TILED_PREFILL since repacking copies the weight. - // No-bias only (GemmTiled applies none). AVX2/FMA required — the kernel is x86-only, so on ARM - // (e.g. the Android app) this falls through to the weight-stationary path even if a sidecar - // mmap'd a prepacked layout (IsPrepacked would otherwise bypass the env flag's AVX2 gate). - var tiled = (w.IsPrepacked || UseTiledPrefillQ4K) && bias.IsEmpty && w.CanRepack - && CpuFeatures.HasAvx2 && CpuFeatures.HasFma; - - if (tiled) - { - DispatchTiledQ4K( - input, rows, w, output, - qBytes.Span.Slice(0, rows * inputSize), scales.Span.Slice(0, rows * spr), - sums.Span.Slice(0, groups)); - } - // Weight-stationary: decode each Q4_K super-block once and reuse across the row tile (bit-identical - // to ProjectBatched, measured ~1.3–1.7× on the prefill / speculative-verify batched matmul). - if (!tiled && UseWeightStationaryQ4K) - { - Q4KDotKernel.ProjectBatchedWeightStationary( - input, rows, w, bias, output, - qBytes.Span.Slice(0, rows * inputSize), scales.Span.Slice(0, rows * spr), - sums.Span.Slice(0, groups)); - } - if (!tiled && !UseWeightStationaryQ4K) - { - Q4KDotKernel.ProjectBatched( - input, rows, w, bias, output, - qBytes.Span.Slice(0, rows * inputSize), scales.Span.Slice(0, rows * spr), - sums.Span.Slice(0, groups)); - } + DispatchQ4K( + input, rows, w, bias, output, inputSize, + qBytes.Span.Slice(0, rows * inputSize), scales.Span.Slice(0, rows * spr), + sums.Span.Slice(0, groups), preQuantized: false); } if (kind == 2) { @@ -140,6 +145,68 @@ public static void Dispatch( } } + // Q6_K format path, shared by the pooled and pre-quantized entries so the kernel-selection gates + // exist in exactly one place. + private static void DispatchQ6K( + ReadOnlySpan input, int rows, Q6KWeight w, ReadOnlySpan bias, Span output, + int inputSize, Span quants, Span scales, Span sums, bool preQuantized) + { + // Register-tiled Q6_K GEMM over the repacked block_q6_Kx8 layout. Under Q4_K_M half of ffn_down + // is Q6_K, and a prefill profile put ffn_down at 37.9% of prefill running at 0.61 TFLOP/s - + // against ffn_gate_up's 1.78 - precisely because Q6_K had only the re-decode-per-row kernel. + // No-bias only (GemmTiled applies none); AVX2/FMA required. + var tiled6 = UseTiledPrefillQ6K && !DisableRepackedKernelsForParity + && bias.IsEmpty && w.CanRepack + && CpuFeatures.HasAvx2 && CpuFeatures.HasFma; + + if (tiled6) + { + DispatchTiledQ6K(input, rows, w, output, quants, scales, sums, preQuantized); + } + + if (!tiled6) + { + Q6KDotKernel.ProjectBatched( + input, rows, w, bias, output, quants, scales, sums, preQuantized); + } + } + + // Q4_K format path, shared by the pooled and pre-quantized entries. + private static void DispatchQ4K( + ReadOnlySpan input, int rows, Q4KWeight w, ReadOnlySpan bias, Span output, + int inputSize, Span quants, Span scales, Span sums, bool preQuantized) + { + // Register-tiled GEMM: repacked block_q4_Kx8, decode each super-block once and reuse across a + // tile of NR columns - measured ~3x vs weight-stationary under parallelism, 1.61x end-to-end + // prefill. Default-on when the weight is already prepacked (an offline sidecar mmap'd it -> zero + // extra RAM); otherwise opt-in via OVERFIT_TILED_PREFILL since repacking copies the weight. + // No-bias only (GemmTiled applies none). AVX2/FMA required - the kernel is x86-only, so on ARM + // (e.g. the Android app) this falls through to the weight-stationary path even if a sidecar + // mmap'd a prepacked layout (IsPrepacked would otherwise bypass the env flag's AVX2 gate). + var tiled = (w.IsPrepacked || UseTiledPrefillQ4K) && !DisableRepackedKernelsForParity + && bias.IsEmpty && w.CanRepack + && CpuFeatures.HasAvx2 && CpuFeatures.HasFma; + + if (tiled) + { + DispatchTiledQ4K(input, rows, w, output, quants, scales, sums, preQuantized); + } + + // Weight-stationary: decode each Q4_K super-block once and reuse across the row tile + // (bit-identical to ProjectBatched, measured ~1.3-1.7x on the batched matmul). + if (!tiled && UseWeightStationaryQ4K) + { + Q4KDotKernel.ProjectBatchedWeightStationary( + input, rows, w, bias, output, quants, scales, sums, preQuantized); + } + + if (!tiled && !UseWeightStationaryQ4K) + { + Q4KDotKernel.ProjectBatched( + input, rows, w, bias, output, quants, scales, sums, preQuantized); + } + } + // Register-tiled Q4_K prefill GEMM: quantize all rows to Q8_K, then run GemmTiled over row-tiles of NR // columns in parallel. NR is chosen so the tile count stays >= cores (an under-filled pool regressed // hard in the Phase-3 bench). No-bias only (checked at the call site) — GemmTiled applies no bias. @@ -150,7 +217,8 @@ private static unsafe void DispatchTiledQ4K( Span output, Span quants, Span scales, - Span bsums) + Span bsums, + bool preQuantized) { var inputSize = w.InputSize; var outputSize = w.OutputSize; @@ -158,13 +226,16 @@ private static unsafe void DispatchTiledQ4K( var bsumsPerRow = spr * Q4KDotKernel.GroupsPerSuperBlock; // Q8_K activation quantization — column-contiguous (column c == row c owns inputSize quants). - for (var n = 0; n < rows; n++) + if (!preQuantized) { - Q4KDotKernel.QuantizeActivationQ8K( - input.Slice(n * inputSize, inputSize), - quants.Slice(n * inputSize, inputSize), - scales.Slice(n * spr, spr), - bsums.Slice(n * bsumsPerRow, bsumsPerRow)); + for (var n = 0; n < rows; n++) + { + Q4KDotKernel.QuantizeActivationQ8K( + input.Slice(n * inputSize, inputSize), + quants.Slice(n * inputSize, inputSize), + scales.Slice(n * spr, spr), + bsums.Slice(n * bsumsPerRow, bsumsPerRow)); + } } var repacked = w.EnsureRepacked(); @@ -212,20 +283,24 @@ private static unsafe void DispatchTiledQ6K( Span output, Span quants, Span scales, - Span bsums) + Span bsums, + bool preQuantized) { var inputSize = w.InputSize; var outputSize = w.OutputSize; var spr = w.SuperBlocksPerRow; var bsumsPerRow = spr * Q6KDotKernel.GroupsPerSuperBlock; - for (var n = 0; n < rows; n++) + if (!preQuantized) { - Q6KDotKernel.QuantizeActivationQ8K( - input.Slice(n * inputSize, inputSize), - quants.Slice(n * inputSize, inputSize), - scales.Slice(n * spr, spr), - bsums.Slice(n * bsumsPerRow, bsumsPerRow)); + for (var n = 0; n < rows; n++) + { + Q6KDotKernel.QuantizeActivationQ8K( + input.Slice(n * inputSize, inputSize), + quants.Slice(n * inputSize, inputSize), + scales.Slice(n * spr, spr), + bsums.Slice(n * bsumsPerRow, bsumsPerRow)); + } } var repacked = w.EnsureRepacked(); diff --git a/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs b/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs index 60d58099..b890b335 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs @@ -632,6 +632,59 @@ internal void DecodeBatchedQuant( using var kf = new PooledBuffer(cache.IsQuantized ? cacheLength * headDim : 0, clearMemory: false); using var vf = new PooledBuffer(cache.IsQuantized ? cacheLength * headDim : 0, clearMemory: false); + // Q, K and V all project the SAME loop-invariant `hidden`, but Q is dispatched once per head and + // K/V once per KV group, so the Q8_K quantization of `hidden` was being repeated ~16x per layer. + // A benchmark put that quantization at ~1.0 ms for a 672x2048 block against a 1.079 ms Q-head + // dispatch - i.e. ~93% of the call - so it is hoisted here and shared. Deterministic, therefore + // bit-identical to quantizing inside each dispatch. Decode got this fix in 2026-05 + // (ProjectPreQuantized); the batched prefill path never did. + var hiddenSpr = dModel / Q4KDotKernel.SuperBlockElements; + var shareActivations = hiddenSpr * Q4KDotKernel.SuperBlockElements == dModel; + using var hq = new PooledBuffer(shareActivations ? rows * dModel : 0, clearMemory: false); + using var hs = new PooledBuffer(shareActivations ? rows * hiddenSpr : 0, clearMemory: false); + using var hb = new PooledBuffer( + shareActivations ? rows * hiddenSpr * Q4KDotKernel.GroupsPerSuperBlock : 0, clearMemory: false); + + // Whole-matrix O: per head the O projection is [headDim -> dModel], and headDim (128) is not a + // multiple of the 256-element Q4_K super-block, so `CanRepack` is false and every one of the 16 + // dispatches per layer is stuck on the weight-stationary kernel. The whole matrix is + // [nHeads*headDim -> dModel] = 2048 wide, which DOES repack - micro-bench: 4.41 ms tiled versus + // 11.9 ms measured for the 16 per-head calls. WoWhole is already loaded zero-copy from the mmap + // (and prepacked when a sidecar exists), so this costs no extra RAM. + var totalHeadDim = HeadCount * headDim; + // Gate on the O handle ALONE, not `HasWholeAttnQ4K` (which also demands Q/K/V). Under Q4_K_M + // attn_v is Q6_K in half the layers, so the four-way gate enabled this in only 18 of 36 — visible + // as 306 attn_out dispatches instead of 36 (18 layers x 16 heads + 18 x 1). + // + // Also honours DisableRepackedKernelsForParity: contracting all heads inside one matmul + // reassociates a sum the per-head path performs in ascending head order, so it is the same class + // of numerical change as the repacked GEMMs and must be switchable off by the same test hook — + // otherwise the batched-vs-single-token parity test can never reach its 1e-2 bound. + var useWholeO = weights.WoWhole.IsQ4K && weights.AttentionBias.IsEmpty + && !BatchedQuantProjection.DisableRepackedKernelsForParity; + using var attnAll = new PooledBuffer(useWholeO ? rows * totalHeadDim : 0, clearMemory: false); + + Span hQuants = default; + Span hScales = default; + Span hBsums = default; + + if (shareActivations) + { + var hBsumsPerRow = hiddenSpr * Q4KDotKernel.GroupsPerSuperBlock; + hQuants = hq.Span.Slice(0, rows * dModel); + hScales = hs.Span.Slice(0, rows * hiddenSpr); + hBsums = hb.Span.Slice(0, rows * hBsumsPerRow); + + for (var n = 0; n < rows; n++) + { + Q4KDotKernel.QuantizeActivationQ8K( + hidden.Slice(n * dModel, dModel), + hQuants.Slice(n * dModel, dModel), + hScales.Slice(n * hiddenSpr, hiddenSpr), + hBsums.Slice(n * hBsumsPerRow, hBsumsPerRow)); + } + } + for (var group = 0; group < KvHeadCount; group++) { // K/V weights: GQA shares one KV head per group; MHA uses the head's own. @@ -659,8 +712,10 @@ internal void DecodeBatchedQuant( // K/V projected once per group, RoPE-rotated, stored — every Q head reads the cache. var profKv = PrefillProfiler.Start(); - BatchedQuantProjection.Dispatch(hidden, rows, in wk, bk, kg.Span, dModel, headDim); - BatchedQuantProjection.Dispatch(hidden, rows, in wv, bv, vg.Span, dModel, headDim); + BatchedQuantProjection.Dispatch( + hidden, rows, in wk, bk, kg.Span, dModel, headDim, hQuants, hScales, hBsums); + BatchedQuantProjection.Dispatch( + hidden, rows, in wv, bv, vg.Span, dModel, headDim, hQuants, hScales, hBsums); PrefillProfiler.Stop(PrefillProfiler.Component.AttnKv, profKv); if (weights.HasQkNorm) { @@ -699,7 +754,8 @@ internal void DecodeBatchedQuant( var wo = hw.Wo; var profQ = PrefillProfiler.Start(); - BatchedQuantProjection.Dispatch(hidden, rows, in wq, hw.Bq, qh.Span, dModel, headDim); + BatchedQuantProjection.Dispatch( + hidden, rows, in wq, hw.Bq, qh.Span, dModel, headDim, hQuants, hScales, hBsums); PrefillProfiler.Stop(PrefillProfiler.Component.AttnQ, profQ); if (weights.HasQkNorm) { @@ -717,14 +773,47 @@ internal void DecodeBatchedQuant( BatchedAttentionKernel.ComputeParallel(qh.Span, keys, values, attn.Span, score.Span, rows, cacheLength, headDim, scale, AttnLogitSoftcap); PrefillProfiler.Stop(PrefillProfiler.Component.AttnScores, profScores); - var profOut = PrefillProfiler.Start(); - BatchedQuantProjection.Dispatch(attn.Span, rows, in wo, [], band.Span, headDim, dModel); - PrefillProfiler.Stop(PrefillProfiler.Component.AttnOut, profOut); - for (var n = 0; n < rows; n++) + if (useWholeO) { - var outRow = output.Slice(n * dModel, dModel); - TensorPrimitives.Add(outRow, band.Span.Slice(n * dModel, dModel), outRow); + // Stash this head's band; the single whole-matrix projection runs after all heads. + for (var n = 0; n < rows; n++) + { + attn.Span.Slice(n * headDim, headDim) + .CopyTo(attnAll.Span.Slice(n * totalHeadDim + h * headDim, headDim)); + } } + + if (!useWholeO) + { + var profOut = PrefillProfiler.Start(); + BatchedQuantProjection.Dispatch(attn.Span, rows, in wo, [], band.Span, headDim, dModel); + PrefillProfiler.Stop(PrefillProfiler.Component.AttnOut, profOut); + for (var n = 0; n < rows; n++) + { + var outRow = output.Slice(n * dModel, dModel); + TensorPrimitives.Add(outRow, band.Span.Slice(n * dModel, dModel), outRow); + } + } + } + } + + // One whole-matrix O projection over every head's band at once. Note this REASSOCIATES the sum: + // the per-head path adds 16 partial results in ascending head order, whereas here the contraction + // happens inside the matmul. Not bit-identical to the single-token reference — the same trade the + // repacked kernels already make, covered by RepackedPrefill_AgreesWithNonRepacked_OnArgmax. + if (useWholeO) + { + var wholeO = weights.WoWhole; // property returns by value — needs a local to pass by `in` + var profOut = PrefillProfiler.Start(); + BatchedQuantProjection.Dispatch( + attnAll.Span.Slice(0, rows * totalHeadDim), rows, in wholeO, [], + band.Span, totalHeadDim, dModel); + PrefillProfiler.Stop(PrefillProfiler.Component.AttnOut, profOut); + + for (var n = 0; n < rows; n++) + { + var outRow = output.Slice(n * dModel, dModel); + TensorPrimitives.Add(outRow, band.Span.Slice(n * dModel, dModel), outRow); } } } diff --git a/Sources/Main/LanguageModels/Runtime/Q4KDotKernel.cs b/Sources/Main/LanguageModels/Runtime/Q4KDotKernel.cs index 7c36aa78..b3dd73eb 100644 --- a/Sources/Main/LanguageModels/Runtime/Q4KDotKernel.cs +++ b/Sources/Main/LanguageModels/Runtime/Q4KDotKernel.cs @@ -687,7 +687,8 @@ public static void ProjectBatched( Span output, Span activationQuants, Span activationScales, - Span activationBsums) + Span activationBsums, + bool preQuantized = false) { ArgumentNullException.ThrowIfNull(weight); ArgumentOutOfRangeException.ThrowIfNegativeOrZero(rows); @@ -716,13 +717,20 @@ public static void ProjectBatched( throw new ArgumentException("Activation quantization scratch is too small for rows."); } - for (var n = 0; n < rows; n++) + // `preQuantized` lets the caller hand in activations already in Q8_K form. Attention + // dispatches Q once PER HEAD over a loop-invariant `hidden`, and a benchmark put the + // quantization at ~93% of that dispatch, so quantizing once per layer and reusing it is + // pure de-duplication - bit-identical, because the quantization is deterministic. + if (!preQuantized) { - QuantizeActivationQ8K( - input.Slice(n * inputSize, inputSize), - activationQuants.Slice(n * inputSize, inputSize), - activationScales.Slice(n * superBlocksPerRow, superBlocksPerRow), - activationBsums.Slice(n * bsumsPerRow, bsumsPerRow)); + for (var n = 0; n < rows; n++) + { + QuantizeActivationQ8K( + input.Slice(n * inputSize, inputSize), + activationQuants.Slice(n * inputSize, inputSize), + activationScales.Slice(n * superBlocksPerRow, superBlocksPerRow), + activationBsums.Slice(n * bsumsPerRow, bsumsPerRow)); + } } fixed (byte* blocksPtr = weight.BlockSpan) @@ -825,7 +833,8 @@ public static void ProjectBatchedWeightStationary( Span output, Span activationQuants, Span activationScales, - Span activationBsums) + Span activationBsums, + bool preQuantized = false) { ArgumentNullException.ThrowIfNull(weight); ArgumentOutOfRangeException.ThrowIfNegativeOrZero(rows); @@ -854,13 +863,20 @@ public static void ProjectBatchedWeightStationary( throw new ArgumentException("Activation quantization scratch is too small for rows."); } - for (var n = 0; n < rows; n++) + // `preQuantized` lets the caller hand in activations already in Q8_K form. Attention + // dispatches Q once PER HEAD over a loop-invariant `hidden`, and a benchmark put the + // quantization at ~93% of that dispatch, so quantizing once per layer and reusing it is + // pure de-duplication - bit-identical, because the quantization is deterministic. + if (!preQuantized) { - QuantizeActivationQ8K( - input.Slice(n * inputSize, inputSize), - activationQuants.Slice(n * inputSize, inputSize), - activationScales.Slice(n * superBlocksPerRow, superBlocksPerRow), - activationBsums.Slice(n * bsumsPerRow, bsumsPerRow)); + for (var n = 0; n < rows; n++) + { + QuantizeActivationQ8K( + input.Slice(n * inputSize, inputSize), + activationQuants.Slice(n * inputSize, inputSize), + activationScales.Slice(n * superBlocksPerRow, superBlocksPerRow), + activationBsums.Slice(n * bsumsPerRow, bsumsPerRow)); + } } fixed (byte* blocksPtr = weight.BlockSpan) diff --git a/Sources/Main/LanguageModels/Runtime/Q6KDotKernel.cs b/Sources/Main/LanguageModels/Runtime/Q6KDotKernel.cs index 5982ec41..769caee0 100644 --- a/Sources/Main/LanguageModels/Runtime/Q6KDotKernel.cs +++ b/Sources/Main/LanguageModels/Runtime/Q6KDotKernel.cs @@ -408,7 +408,8 @@ public static void ProjectBatched( Span output, Span activationQuants, Span activationScales, - Span activationBsums) + Span activationBsums, + bool preQuantized = false) { ArgumentNullException.ThrowIfNull(weight); ArgumentOutOfRangeException.ThrowIfNegativeOrZero(rows); @@ -437,13 +438,20 @@ public static void ProjectBatched( throw new ArgumentException("Activation quantization scratch is too small for rows."); } - for (var n = 0; n < rows; n++) + // `preQuantized` lets the caller hand in activations already in Q8_K form. Attention + // dispatches Q once PER HEAD over a loop-invariant `hidden`, and a benchmark put the + // quantization at ~93% of that dispatch, so quantizing once per layer and reusing it is + // pure de-duplication - bit-identical, because the quantization is deterministic. + if (!preQuantized) { - QuantizeActivationQ8K( - input.Slice(n * inputSize, inputSize), - activationQuants.Slice(n * inputSize, inputSize), - activationScales.Slice(n * superBlocksPerRow, superBlocksPerRow), - activationBsums.Slice(n * bsumsPerRow, bsumsPerRow)); + for (var n = 0; n < rows; n++) + { + QuantizeActivationQ8K( + input.Slice(n * inputSize, inputSize), + activationQuants.Slice(n * inputSize, inputSize), + activationScales.Slice(n * superBlocksPerRow, superBlocksPerRow), + activationBsums.Slice(n * bsumsPerRow, bsumsPerRow)); + } } fixed (byte* blocksPtr = weight.BlockSpan) diff --git a/Tests/LanguageModels/Runtime/Parity/BatchedPrefillParityTests.cs b/Tests/LanguageModels/Runtime/Parity/BatchedPrefillParityTests.cs index d68ffa8f..48ddbb4a 100644 --- a/Tests/LanguageModels/Runtime/Parity/BatchedPrefillParityTests.cs +++ b/Tests/LanguageModels/Runtime/Parity/BatchedPrefillParityTests.cs @@ -135,6 +135,7 @@ public void BatchedPrefill_MatchesSingleToken_OnRealQwen() } using var engine = CachedLlamaInferenceEngine.LoadGguf(ModelPath); + using var kernels = UseNonRepackedKernels(); // A ≥16-token prompt to trigger the batched path; arbitrary in-vocab ids. var prompt = new int[40]; @@ -176,6 +177,78 @@ public void BatchedPrefill_MatchesSingleToken_OnRealQwen() Assert.True(maxDiff < 1e-2f, $"batched vs single logit divergence {maxDiff:G4} (> 1e-2)."); } + /// + /// Guards the DEFAULT prefill configuration — the repacked block_q*_Kx8 GEMMs — at the standard + /// they can actually meet: same predicted token, not bit-equality. + /// + /// Those kernels associate their reduction differently from the per-row ones, so they diverge + /// from the single-token reference by ~0.44 in absolute logits on Qwen-3B. That is why + /// pins the layout via + /// — and why the fast path needs its own, looser gate rather than + /// simply being untested. Without this test, turning a repacked kernel on by default would be covered + /// by nothing at all. + /// + /// The tolerance is deliberately not tightened to the point of pinning today's exact numerics: + /// the contract being asserted is "the reassociation does not change what the model says", which is + /// the same bar OVERFIT_REPACK_ATTN is held to. + /// + /// The prompt is real text, not the synthetic id ramp its neighbours use. An arbitrary + /// in-vocab sequence like 100 + 37·i is out-of-distribution, so the top logits come out nearly + /// tied and the argmax flips on any numerical perturbation — this test failed exactly that way with + /// the ramp (argmax 11 vs 13 at maxAbsLogitDiff 0.42) while the same kernels agree on the first + /// generated token for real text. Argmax stability is only a meaningful assertion where the model is + /// actually confident. + /// + [LongFact] + public void RepackedPrefill_AgreesWithNonRepacked_OnArgmax() + { + if (!File.Exists(ModelPath)) + { + _out.WriteLine($"missing {ModelPath}"); + return; + } + + using var engine = CachedLlamaInferenceEngine.LoadGguf(ModelPath); + var tokenizer = GgufTokenizer.Load(ModelPath); + var prompt = tokenizer.Encode( + "The history of computing began with mechanical calculators and evolved through vacuum tubes, " + + "transistors, integrated circuits and finally the microprocessor era. The next paragraph " + + "explains why that progression mattered for modern software."); + + // Default configuration: whatever the repacked gates decide (sidecar / env flag / Q6_K tiled). + using var fast = engine.CreateSession(256); + fast.Reset(prompt); + var fastLogits = fast.LastLogits.ToArray(); + + float[] referenceLogits; + using (var kernels = UseNonRepackedKernels()) + { + using var reference = engine.CreateSession(256); + reference.Reset(prompt); + referenceLogits = reference.LastLogits.ToArray(); + } + + var maxDiff = 0f; + int argFast = 0, argReference = 0; + for (var i = 0; i < referenceLogits.Length; i++) + { + maxDiff = MathF.Max(maxDiff, MathF.Abs(fastLogits[i] - referenceLogits[i])); + if (fastLogits[i] > fastLogits[argFast]) + { + argFast = i; + } + if (referenceLogits[i] > referenceLogits[argReference]) + { + argReference = i; + } + } + + _out.WriteLine( + $"repacked argmax={argFast} non-repacked argmax={argReference} maxAbsLogitDiff={maxDiff:G4}"); + + Assert.Equal(argReference, argFast); + } + [LongFact] public void BatchedPrefill_MatchesSingleToken_OnRealQwenMoE() { @@ -262,5 +335,29 @@ double Time(bool disableBatched) var batched = Time(disableBatched: false); _out.WriteLine($"TTFT {prompt.Length}-token prompt: single={single:F1} ms batched={batched:F1} ms speedup={single / batched:F2}×"); } + /// + /// Forces the NON-repacked batched kernels for the duration of the scope. + /// + /// Without this a batched-vs-single-token parity test silently stops testing what it claims. + /// The repacked block_q*_Kx8 GEMMs associate their reduction differently from the per-row + /// kernels the single-token path uses, so they are NOT bit-identical - measured at + /// maxAbsLogitDiff ~ 0.44 on Qwen-3B, enough to flip an argmax. Worse, a *.gguf.repack + /// sidecar sets IsPrepacked and switches that path on regardless of the env flag, which is + /// how this test came to fail unnoticed for two days (it is [LongFact], so it never ran). + /// The repacked kernels are held to end-to-end coherence instead - see + /// . + /// + /// The flag is process-global, so these tests must not run concurrently with other prefill + /// tests - they are [LongFact] and run one at a time in practice. + /// + private static NonRepackedScope UseNonRepackedKernels() => new(); + + private readonly struct NonRepackedScope : IDisposable + { + public NonRepackedScope() => BatchedQuantProjection.DisableRepackedKernelsForParity = true; + + public void Dispose() => BatchedQuantProjection.DisableRepackedKernelsForParity = false; + } + } } From 24cf0aaeb9c63c86b0dee97447513e80d175aa25 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Wed, 22 Jul 2026 17:51:34 +0200 Subject: [PATCH 12/37] llama --- ROADMAP.md | 49 ++++++++++++++++++- .../Runtime/BatchedQuantProjection.cs | 16 ++++-- .../Runtime/CachedMultiHeadAttention.cs | 44 +++++++++++++++-- .../LanguageModels/Runtime/Q4KGemvKernel.cs | 34 +++++++++++-- 4 files changed, 131 insertions(+), 12 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 7908a197..fe591afa 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -306,7 +306,54 @@ Contracting all heads inside one matmul reassociates a sum the per-head path doe `useWholeO` also honours `DisableRepackedKernelsForParity` — without that the batched-vs-single-token parity test can never reach its 1e-2 bound. -#### ▶ WHAT IS LEFT — profile after the three wins (219 tok/s, gap 2.47×) +#### ✅ MEASURED — biased projections on the tiled kernel: 220 → 249 tok/s (1.13×) + +`GemmTiled` gained the optional bias again (it folds into the final store; the no-bias path keeps two +separate store loops so its bit-identity is untouched), and `bias.IsEmpty` came out of the Q4_K tiled gate. + +**On its own that changed nothing — `attn_q` moved 459.1 → 463.9 ms, a tie for the second time.** The reason +was not the shape and not the bias: per-head Q/K/V weights are *slices* of the tensor the `*.gguf.repack` +sidecar covers, so `IsPrepacked` is false for them, and `OVERFIT_TILED_PREFILL` was unset — the gate +`(IsPrepacked || UseTiledPrefillQ4K)` failed before `bias.IsEmpty` ever mattered. **The same dead-flag trap +as 2026-07-21. Check that the path is taken before concluding the kernel does not help.** + +With `OVERFIT_TILED_PREFILL=1`: + +| component | before | after | Δ | +|---|---:|---:|---:| +| `attn_q` | 463.9 ms | **171.3 ms** | **−63%** (2.7×) | +| `attn_kv` | 140.5 ms | **80.3 ms** | −43% | +| `ffn_gateup` *(canary)* | 1208.5 ms | 1214.2 ms | +0.5% | +| **prefill total** | **3056.4 ms · 220 tok/s** | **2699.6 ms · 249 tok/s** | **1.13×** | + +Parity green in BOTH configurations: reference path `maxAbsLogitDiff = 0`, fast path agrees on the token. + +**Not enabled by default — it costs RAM.** The flag makes `EnsureRepacked()` allocate a heap copy for every +repackable Q4_K weight that the sidecar does not cover, i.e. all ~600 per-head Q/K/V slices (~100 MB on +Qwen-3B). **The zero-RAM version is whole-matrix Q/K/V**: `WqWhole` / `WkWhole` / `WvWhole` are already loaded +zero-copy from the mmap and prepacked by the sidecar, exactly like `WoWhole` — so the same gather/scatter +refactor that landed for O would buy this win without the allocation. That is the next build. + +#### ✅ SHIPPED — whole-matrix Q: 249 tok/s at ZERO extra RAM + +One `[dModel → nHeads·headDim]` projection replaces 16 per-head ones, then each head gathers its columns +(and adds its own bias, since `BlockWeights` keeps the Q bias per head and there is no concatenated form). + +| component | per-head | whole-matrix | Δ | +|---|---:|---:|---:| +| `attn_q` | 463.9 ms / 576 calls | **103.8 ms / 36 calls** | **−78%** (4.5×) | +| **prefill total** | **3056.4 ms · 220 tok/s** | **2695.8 ms · 249 tok/s** | **1.13×** | + +**This is the same 249 tok/s the `OVERFIT_TILED_PREFILL=1` experiment produced, without its ~100 MB** — +`WqWhole` is mmap'd zero-copy and covered by the sidecar, so it is prepacked without allocating anything. +It also beats the flag on the component itself (103.8 vs 171.3 ms): one large matmul wins over sixteen small +ones even on the same kernel. + +**No parity gate needed here, unlike whole-matrix O.** O contracts over `nHeads·headDim` and therefore +reassociates a sum the per-head path performs in head order; Q's contraction is over `dModel` in both +shapes, so every output element is the same dot product either way. + +#### ▶ WHAT IS LEFT — profile at 249 tok/s, gap 2.18× ``` ffn_gateup 1222.7 ms 39.8% (36) <- Q4_K tiled already diff --git a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs index 034327b5..36345b05 100644 --- a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs +++ b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs @@ -183,13 +183,16 @@ private static void DispatchQ4K( // No-bias only (GemmTiled applies none). AVX2/FMA required - the kernel is x86-only, so on ARM // (e.g. the Android app) this falls through to the weight-stationary path even if a sidecar // mmap'd a prepacked layout (IsPrepacked would otherwise bypass the env flag's AVX2 gate). + // `bias.IsEmpty` used to sit here because GemmTiled applied none. With attention Q/K/V biased, + // that kept attn_q (15% of prefill) on the weight-stationary kernel even though its shape + // [2048 -> 128] repacks fine. GemmTiled now folds the bias into its final store. var tiled = (w.IsPrepacked || UseTiledPrefillQ4K) && !DisableRepackedKernelsForParity - && bias.IsEmpty && w.CanRepack + && w.CanRepack && CpuFeatures.HasAvx2 && CpuFeatures.HasFma; if (tiled) { - DispatchTiledQ4K(input, rows, w, output, quants, scales, sums, preQuantized); + DispatchTiledQ4K(input, rows, w, bias, output, quants, scales, sums, preQuantized); } // Weight-stationary: decode each Q4_K super-block once and reuse across the row tile @@ -214,6 +217,7 @@ private static unsafe void DispatchTiledQ4K( ReadOnlySpan input, int rows, Q4KWeight w, + ReadOnlySpan bias, Span output, Span quants, Span scales, @@ -253,6 +257,7 @@ private static unsafe void DispatchTiledQ4K( fixed (float* sc = scales) fixed (short* bs = bsums) fixed (float* o = output) + fixed (float* bi = bias) // null when the projection has no bias { var ctx = new TiledContext { @@ -262,6 +267,8 @@ private static unsafe void DispatchTiledQ4K( Scales = sc, Bsums = bs, Output = o, + Bias = bi, + BiasLength = bias.Length, InputSize = inputSize, OutputSize = outputSize, Spr = spr, @@ -375,6 +382,8 @@ private unsafe struct TiledContext public float* Scales; public short* Bsums; public float* Output; + public float* Bias; + public int BiasLength; public int InputSize; public int OutputSize; public int Spr; @@ -398,7 +407,8 @@ private static unsafe void TiledChunk(int start, int end, void* context) new ReadOnlySpan(c.Quants + (long)s * c.InputSize, cols * c.InputSize), new ReadOnlySpan(c.Scales + (long)s * c.Spr, cols * c.Spr), new ReadOnlySpan(c.Bsums + (long)s * c.BsumsPerRow, cols * c.BsumsPerRow), - new Span(c.Output + (long)s * c.OutputSize, cols * c.OutputSize)); + new Span(c.Output + (long)s * c.OutputSize, cols * c.OutputSize), + new ReadOnlySpan(c.Bias, c.BiasLength)); } } } diff --git a/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs b/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs index b890b335..c7750b65 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs @@ -685,6 +685,24 @@ internal void DecodeBatchedQuant( } } + // Whole-matrix Q: one [dModel -> nHeads*headDim] projection instead of 16 per-head ones. Unlike + // whole-matrix O this reassociates NOTHING - every Q output element is an independent dot over + // dModel either way - so it needs no parity gate. What it buys is kernel eligibility: WqWhole is + // covered by the *.gguf.repack sidecar (per-head weights are slices, which the sidecar cannot + // match by name), so IsPrepacked is true and the tiled GEMM applies at zero extra RAM. + var useWholeQ = weights.WqWhole.IsQ4K; + using var qAll = new PooledBuffer(useWholeQ ? rows * totalHeadDim : 0, clearMemory: false); + + if (useWholeQ) + { + var wholeQ = weights.WqWhole; // property returns by value - needs a local to pass by `in` + var profWholeQ = PrefillProfiler.Start(); + BatchedQuantProjection.Dispatch( + hidden, rows, in wholeQ, [], qAll.Span.Slice(0, rows * totalHeadDim), + dModel, totalHeadDim, hQuants, hScales, hBsums); + PrefillProfiler.Stop(PrefillProfiler.Component.AttnQ, profWholeQ); + } + for (var group = 0; group < KvHeadCount; group++) { // K/V weights: GQA shares one KV head per group; MHA uses the head's own. @@ -753,10 +771,28 @@ internal void DecodeBatchedQuant( var wq = hw.Wq; var wo = hw.Wo; - var profQ = PrefillProfiler.Start(); - BatchedQuantProjection.Dispatch( - hidden, rows, in wq, hw.Bq, qh.Span, dModel, headDim, hQuants, hScales, hBsums); - PrefillProfiler.Stop(PrefillProfiler.Component.AttnQ, profQ); + if (useWholeQ) + { + // Gather this head's columns; the bias is added here because BlockWeights keeps it + // per head and there is no concatenated form to hand the matmul. + for (var n = 0; n < rows; n++) + { + var dst = qh.Span.Slice(n * headDim, headDim); + qAll.Span.Slice(n * totalHeadDim + h * headDim, headDim).CopyTo(dst); + if (!hw.Bq.IsEmpty) + { + TensorPrimitives.Add(dst, hw.Bq.Slice(0, headDim), dst); + } + } + } + + if (!useWholeQ) + { + var profQ = PrefillProfiler.Start(); + BatchedQuantProjection.Dispatch( + hidden, rows, in wq, hw.Bq, qh.Span, dModel, headDim, hQuants, hScales, hBsums); + PrefillProfiler.Stop(PrefillProfiler.Component.AttnQ, profQ); + } if (weights.HasQkNorm) { QkNormKernel.Apply(qh.Span, weights.QkNormQ, rows, headDim); diff --git a/Sources/Main/LanguageModels/Runtime/Q4KGemvKernel.cs b/Sources/Main/LanguageModels/Runtime/Q4KGemvKernel.cs index b7726777..d6c3aee3 100644 --- a/Sources/Main/LanguageModels/Runtime/Q4KGemvKernel.cs +++ b/Sources/Main/LanguageModels/Runtime/Q4KGemvKernel.cs @@ -302,13 +302,20 @@ public static void GemmTiled( ReadOnlySpan actQuants, ReadOnlySpan actScales, ReadOnlySpan actBsums, - Span output) + Span output, + ReadOnlySpan bias = default) { if (cols is < 1 or > MaxTileCols) { throw new ArgumentOutOfRangeException(nameof(cols), cols, $"cols must be in [1, {MaxTileCols}]."); } + if (!bias.IsEmpty && bias.Length < outputSize) + { + throw new ArgumentException( + $"bias length {bias.Length} < outputSize {outputSize}.", nameof(bias)); + } + var nb = inputSize / 256; // Per-column accumulator state, hoisted out of the group loop (stackalloc-in-loop = CA2014). Reset @@ -333,6 +340,7 @@ public static void GemmTiled( fixed (float* asc = actScales) fixed (short* ab = actBsums) fixed (float* o = output) + fixed (float* bs = bias) // null when empty — keeps the no-bias path branch-free per store { for (var x = 0; x < outputSize / 8; x++) { @@ -462,10 +470,28 @@ public static void GemmTiled( } } - for (var c = 0; c < cols; c++) + // Two stores rather than one with a zero vector: `x + 0f` rewrites -0.0 to +0.0, + // which would break the bit-identity the no-bias path is pinned to. The branch is + // per output-group, not per column, and is perfectly predicted. + if (bs is null) + { + for (var c = 0; c < cols; c++) + { + var row = Avx2.PermuteVar8x32(accRow[c], finalpermute); + Avx.Subtract(row, accMin[c]).Store(o + (long)c * outputSize + x * 8); + } + } + + if (bs is not null) { - var row = Avx2.PermuteVar8x32(accRow[c], finalpermute); - Avx.Subtract(row, accMin[c]).Store(o + (long)c * outputSize + x * 8); + // Same 8 bias floats for every column - hoisted out of the column loop. + var biasVec = Vector256.Load(bs + x * 8); + for (var c = 0; c < cols; c++) + { + var row = Avx2.PermuteVar8x32(accRow[c], finalpermute); + Avx.Add(Avx.Subtract(row, accMin[c]), biasVec) + .Store(o + (long)c * outputSize + x * 8); + } } } } From e45f9938fbfb1e8740126b18eb2bd5453764427e Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Wed, 22 Jul 2026 20:45:18 +0200 Subject: [PATCH 13/37] llama --- ROADMAP.md | 89 +++++++++++++++++++ .../Q4KPrefillProjectionBenchmark.cs | 8 +- 2 files changed, 95 insertions(+), 2 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index fe591afa..bf084cbc 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -353,6 +353,95 @@ ones even on the same kernel. reassociates a sum the per-head path performs in head order; Q's contraction is over `dModel` in both shapes, so every output element is the same dot product either way. +#### 📖 READ — how llama.cpp's `ggml_gemm_q4_K_8x8_q8_K` differs from ours + +`D:\llamacpp-tmp\ggml\src\ggml-cpu\arch\x86\repack.cpp:2042`. Four variants: + +| ISA | tile (act rows × out cols) | accumulators | +|---|---|---| +| AVX-512 main | 16 × 16 | `__m512 acc_rows[16]` + `acc_min_rows[16]` = 32 ZMM | +| AVX-512 tail | 4 × 16 | 8 ZMM | +| AVX2 main | 16 × 8 | `__m256 acc_rows[16]` + `[16]` | +| AVX2 tail | 4 × 8 | 8 YMM | +| **ours (`GemmTiled`)** | **`cols` × 8** | **5 `stackalloc` spans of length `cols`** | + +Differences, in order of likely cost: + +1. **Constant vs runtime accumulator index.** Theirs are `acc_rows[0]`…`[15]` with fully unrolled updates + (lines 2789-2792, 3464-3467 are four explicit FMAs, not a loop), so the compiler register-allocates and + spills selectively. Ours are indexed by a runtime `c`, so every access is a stack read/write **and** a + bounds check — register allocation is impossible, not merely unlucky. Note their AVX2 path declares 32 + `__m256` against 16 YMM, so it spills too and is still fast: the win is *selective* spilling. +2. **Five accumulator arrays to their two.** `accRow`, `accMin`, `iaccB`, `iaccMinB`, `q8s` — 40 vectors of + stack traffic per iteration at `cols=8`. +3. **Activations are repacked too** (`block_q8_Kx4`, four rows interleaved), so one load feeds four rows. + That is why their row tile is always a multiple of 4. Ours loads each column separately. +4. AVX-512 is a consequence of (1), not an independent lever: 32 ZMM is what makes the 16×16 tile fit. + +#### ✗ ATTEMPTED — unrolled fixed-tile specialisation: INCONCLUSIVE, reverted + +A `cols == 4` specialisation with named accumulators was written and passed parity — **but was never +executed**: the dispatcher picks `nr = rows/8 >= cores ? 8 : 4`, which is 8 at 672 rows on 32 cores. That is +the **third** unreached-path mistake in one day (after the dead `OVERFIT_TILED_PREFILL` flag and the +`IsPrepacked` gate hiding the bias change). + +Retargeting it to 8 columns by regex-rewriting the existing kernel text produced **incorrect code** — +duplicate unrolled bodies (the generator reported 11 where 8 were expected, and I proceeded anyway), parity +failed at `cols: 8`, and the kernel ran 7-9× slower (70-83× single-threaded). Reverted. + +#### ✗ TESTED AND REFUTED — register pressure is not the bottleneck + +Register accounting first, since it reframes the task: **AVX2 has 16 YMM registers**, and the kernel keeps +**16 decoded weight vectors** live across the column loop plus 3 hot accumulators per column — 40 vectors +wanted at `cols=8`. Naming the accumulators cannot help, because they have nowhere to go. (This also explains +why llama.cpp's own AVX2 path spills: it declares 32 `__m256`.) + +That analysis produced a concrete, small change instead: the low-nibble weight vectors feed only `iacc0` and +the high-nibble ones only `iacc1`, so they are never needed simultaneously. **Splitting the sub-block into +two half-passes over the columns halves peak weight pressure from 16 vectors to 8** — bit-identical (parity +5/5), and the only thing it changes is register lifetime. + +**Measured: a tie.** Single-thread is the low-noise signal (StdDev ~1%) and it did not move — +`ffn_gate_up` 167.8 → 168.9 ms, `attn_qo` 31.7 → 32.1 ms, i.e. marginally *worse*. The parallel column showed +`attn_qo` −12.7%, but that sits inside the run-to-run spread of that measurement (5050 / 5217 / 4408 µs +across runs) and the FFN shapes — 71% of prefill — did not move at all. Reverted. + +**So spilling is not what costs us.** The remaining structural difference to llama.cpp is the one that +reduces *loads*, not register pressure: `block_q8_Kx4` interleaves four activation rows so one load feeds +four of them, where we issue four `BroadcastLo` per column. That is the next thing to size — and it is a +change to the activation-quantization output layout, not to the kernel's register allocation. + +#### ★ LIKE-FOR-LIKE KERNEL COMPARISON — our Q4_K matmul is FASTER than llama.cpp's + +Everything above compared whole-model tok/s and *inferred* the kernel difference. That inference was wrong. +llama.cpp's own `test-backend-ops perf -o MUL_MAT` (AVX2 build, 32 threads — it uses +`std::thread::hardware_concurrency`) reports for `q4_K m=4096 k=14336 n=512`, 60.13 GFLOP/run: + +| | time | TFLOP/s | +|---|---:|---:| +| llama.cpp | 38 559 µs | **1.56** | +| **Overfit `GemmTiled`** (same shape, 32 workers) | **35 308 µs** | **1.70** | + +**Ours is 1.09× faster**, and ~1.91 TFLOP/s with the activation quantization (3 750 µs) excluded. +**So the Q4_K matmul is not where we lose.** Two earlier conclusions are retracted: the "2.34× kernel craft +at equal ISA" attribution, and the register/interleaving hypotheses built on top of it. + +**The unexplained part, restated honestly.** Prefill FLOPs are ≈3.72 TFLOP (36 layers; the LM head runs on +the last position only). Ours: 2.695 s = 1.38 TFLOP/s. Theirs (AVX2): 1.996 s = 1.86 TFLOP/s. Our own split: + +| | time | FLOPs | TFLOP/s | +|---|---:|---:|---:| +| FFN | 1923 ms | 3.27 T | **1.70** | +| attention projections | 662 ms | 0.45 T | **0.68** | +| other | 110 ms | — | — | + +Our FFN already matches the isolated kernel rate. **Attention runs at 0.4× the FFN's efficiency** — that is +where the FLOP throughput collapses, and it is 25% of prefill. + +Also unresolved: their production run (`llama-bench`) chose **16 threads** and beat a 32-thread +`test-backend-ops`, so thread count is worth re-sweeping on our side too. The last worker sweep +(8→92, 16→122, 24→130, 32→144 tok/s) predates every optimisation since and may no longer hold at 249 tok/s. + #### ▶ WHAT IS LEFT — profile at 249 tok/s, gap 2.18× ``` diff --git a/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs b/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs index 9d97dc41..c322ed79 100644 --- a/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs +++ b/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs @@ -50,13 +50,13 @@ namespace Benchmarks public class Q4KPrefillProjectionBenchmark { /// Prompt length used in the llama.cpp comparison, so the numbers are directly relatable. - [Params(672)] + [Params(672, 512)] public int Rows { get; set; } - [Params("ffn_gate_up", "ffn_down", "attn_qo")] + [Params("ffn_gate_up", "ffn_down", "attn_qo", "llama_ref")] public string Shape { get; set; @@ -81,6 +81,10 @@ public void Setup() { "ffn_gate_up" => (2048, 11008), "ffn_down" => (11008, 2048), + // The exact shape llama.cpp's own test-backend-ops reports (m=4096, k=14336): 60.13 GFLOP + // at n=512, where its AVX2 build measured 1.56 TFLOPS for q4_K. Same shape, same thread + // count (32) - the only like-for-like kernel comparison available without editing their tests. + "llama_ref" => (14336, 4096), _ => (2048, 2048), }; From fd53688f0a46aff5a7391fdb9f28b66ec6b85f62 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Wed, 22 Jul 2026 21:05:20 +0200 Subject: [PATCH 14/37] llama --- ROADMAP.md | 40 ++ Sources/Benchmark/Helpers/BenchmarkConfig.cs | 5 + Sources/Benchmark/Helpers/ThroughputColumn.cs | 120 ++++++ Sources/Benchmark/Helpers/WorkAmount.cs | 47 +++ Sources/Benchmark/MachineRooflineBenchmark.cs | 373 ++++++++++++++++++ .../Q4KPrefillProjectionBenchmark.cs | 36 +- 6 files changed, 618 insertions(+), 3 deletions(-) create mode 100644 Sources/Benchmark/Helpers/ThroughputColumn.cs create mode 100644 Sources/Benchmark/Helpers/WorkAmount.cs create mode 100644 Sources/Benchmark/MachineRooflineBenchmark.cs diff --git a/ROADMAP.md b/ROADMAP.md index bf084cbc..02295780 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -442,6 +442,46 @@ Also unresolved: their production run (`llama-bench`) chose **16 threads** and b `test-backend-ops`, so thread count is worth re-sweeping on our side too. The last worker sweep (8→92, 16→122, 24→130, 32→144 tok/s) predates every optimisation since and may no longer hold at 249 tok/s. +#### ★ MACHINE ROOFLINE — measured, in-repo (`MachineRooflineBenchmark`) + +Every kernel figure above was a bare number. These are the ceilings that make them readable +(32 workers, AVX2). Rates are derived by `Helpers/WorkAmount.cs` + `Helpers/ThroughputColumn.cs`, +declared next to each benchmark — *not* in a script, after an out-of-repo script credited a +quantization-only benchmark with the matmul's FLOP count and reported a fictitious 29.6 TFLOP/s. + +| ceiling | measured | +|---|---:| +| peak float FMA | **2.19 TFLOP/s** | +| peak int8 dot (`vpmaddubsw`+`vpmaddwd`) | **11.21 TOPS** | +| DRAM read | **89.9 GB/s** | +| copy | 73.0 GB/s | +| STREAM triad | 47.3 GB/s | + +**Where our Q4_K GEMM (1.70 TFLOP/s) actually sits:** 15% of the integer ceiling, **78% of the float +ceiling**, and 1% of DRAM bandwidth (33 MB of weights in 35.3 ms = 0.94 GB/s). So the kernel is neither +memory-bound nor integer-issue-bound — **it is bound by the float side of dequantization** (scale +multiplication and int32→float conversion of the accumulators). That is also why llama.cpp's AVX-512 +build wins 1.60×: AVX-512 doubles both ceilings. + +**Decode, for the first time with a number under it:** Qwen-3B Q4_K (~1.9 GB) at 24.4 tok/s consumes +≈46 GB/s against an 89.9 GB/s read ceiling. The long-standing "decode is at the DRAM floor" conclusion +was previously reasoning only; it now has a measurement. + +*Benchmark trap paid for here:* the first version put the accumulator chains in a `stackalloc` span and +measured a float peak of **0.79 TFLOP/s** — below the 1.70 our real matmul achieves, which is impossible +for a loop that touches no memory. The span forced an L1 round-trip per accumulator per iteration. +Constant-index named locals are what keep a value in a register. + +#### ▶ NEGATIVE — prefill worker sweep: llama.cpp's 16-thread choice does not transfer + +llama.cpp's `llama-bench` picks 16 threads over the machine's 32 and beats a 32-thread +`test-backend-ops`, so our worker count was re-swept at 249 tok/s (the previous sweep predated every +optimisation in this section). More workers still wins for us; there is nothing to take here. + +| workers | 8 | 12 | 16 | 24 | 31 | 32 (default) | +|---|---:|---:|---:|---:|---:|---:| +| prefill | 151 | 197 | 218 | 209 | 238 | **246** | + #### ▶ WHAT IS LEFT — profile at 249 tok/s, gap 2.18× ``` diff --git a/Sources/Benchmark/Helpers/BenchmarkConfig.cs b/Sources/Benchmark/Helpers/BenchmarkConfig.cs index ff2d9587..036097a3 100644 --- a/Sources/Benchmark/Helpers/BenchmarkConfig.cs +++ b/Sources/Benchmark/Helpers/BenchmarkConfig.cs @@ -25,6 +25,11 @@ public BenchmarkConfig() AddDiagnoser(MemoryDiagnoser.Default); AddColumn(RankColumn.Arabic); + // Rates are derived in-repo from the benchmark's own declared WorkAmount; classes that declare + // none get empty cells. See WorkAmount for why this is not computed in an external script. + AddColumn(ThroughputColumn.Teraflops); + AddColumn(ThroughputColumn.Gigabytes); + WithOrderer(new DefaultOrderer(SummaryOrderPolicy.FastestToSlowest)); } } diff --git a/Sources/Benchmark/Helpers/ThroughputColumn.cs b/Sources/Benchmark/Helpers/ThroughputColumn.cs new file mode 100644 index 00000000..9fe14608 --- /dev/null +++ b/Sources/Benchmark/Helpers/ThroughputColumn.cs @@ -0,0 +1,120 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Globalization; +using System.Reflection; +using BenchmarkDotNet.Columns; +using BenchmarkDotNet.Reports; +using BenchmarkDotNet.Running; + +namespace Benchmarks.Helpers +{ + /// + /// Adds a TFLOP/s and a GB/s column to the summary table, derived from BenchmarkDotNet's own + /// measured mean and from a the benchmark class declares for itself. + /// + /// Opting in. A benchmark class declares a public static method + /// WorkAmount GetWorkAmount(BenchmarkCase). The column finds it by reflection — the Benchmark + /// project is not subject to Sources/Main's reflection ban, and reflection is what lets the lookup + /// work without relying on a static constructor having run in the host process. Classes that do not + /// declare the method simply get empty cells. + /// + /// Cells stay empty when the relevant part of the declared work is zero, so a benchmark that moves + /// memory without doing arithmetic can never be reported as if it had done arithmetic. + /// + internal sealed class ThroughputColumn : IColumn + { + public static readonly IColumn Teraflops = new ThroughputColumn(compute: true); + public static readonly IColumn Gigabytes = new ThroughputColumn(compute: false); + + private readonly bool _compute; + + private ThroughputColumn(bool compute) + { + _compute = compute; + } + + public string Id => nameof(ThroughputColumn) + (_compute ? ".Compute" : ".Memory"); + + public string ColumnName => _compute ? "TFLOP/s" : "GB/s"; + + public string Legend => _compute + ? "Logical multiply-accumulates per second (MAC counted as 2 ops), as declared by the benchmark" + : "Bytes read plus written per second, as declared by the benchmark"; + + public bool AlwaysShow => false; + + public ColumnCategory Category => ColumnCategory.Custom; + + public int PriorityInCategory => _compute ? 0 : 1; + + public bool IsNumeric => true; + + public UnitType UnitType => UnitType.Dimensionless; + + public bool IsAvailable(Summary summary) + { + return true; + } + + public bool IsDefault(Summary summary, BenchmarkCase benchmarkCase) + { + return false; + } + + public string GetValue(Summary summary, BenchmarkCase benchmarkCase) + { + var statistics = summary[benchmarkCase]?.ResultStatistics; + + if (statistics is null || statistics.Mean <= 0.0) + { + return "-"; + } + + if (!TryGetWorkAmount(benchmarkCase, out var work)) + { + return "-"; + } + + // BenchmarkDotNet reports the mean in nanoseconds. + var seconds = statistics.Mean / 1e9; + + if (_compute) + { + return work.Flops <= 0L + ? "-" + : (work.Flops / seconds / 1e12).ToString("F2", CultureInfo.InvariantCulture); + } + + return work.Bytes <= 0L + ? "-" + : (work.Bytes / seconds / 1e9).ToString("F1", CultureInfo.InvariantCulture); + } + + public string GetValue(Summary summary, BenchmarkCase benchmarkCase, SummaryStyle style) + { + return GetValue(summary, benchmarkCase); + } + + private static bool TryGetWorkAmount(BenchmarkCase benchmarkCase, out WorkAmount work) + { + work = default; + + var provider = benchmarkCase.Descriptor.Type.GetMethod( + "GetWorkAmount", + BindingFlags.Public | BindingFlags.Static, + [typeof(BenchmarkCase)]); + + if (provider is null || provider.ReturnType != typeof(WorkAmount)) + { + return false; + } + + work = (WorkAmount)provider.Invoke(null, [benchmarkCase])!; + + return true; + } + } +} diff --git a/Sources/Benchmark/Helpers/WorkAmount.cs b/Sources/Benchmark/Helpers/WorkAmount.cs new file mode 100644 index 00000000..3f2d6775 --- /dev/null +++ b/Sources/Benchmark/Helpers/WorkAmount.cs @@ -0,0 +1,47 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +namespace Benchmarks.Helpers +{ + /// + /// How much work one invocation of a benchmark performs, so BenchmarkDotNet's measured time can be + /// turned into a rate (TFLOP/s, GB/s) in the repository rather than in a throwaway script. + /// + /// Why this type exists. On 2026-07-22 a prefill comparison against llama.cpp was reported in + /// TFLOP/s computed ad hoc outside the repo. The matmul rows were right, but the same + /// 2·rows·k·n formula was also applied to a benchmark that performs no multiply-accumulate at + /// all (activation quantization), producing a nonsense "29.6 TFLOP/s" that was really memory + /// bandwidth wearing a FLOP costume. Encoding the work amount next to the benchmark makes that class of + /// error impossible: a benchmark that does no arithmetic declares = 0 and simply gets + /// no TFLOP/s column. + /// + /// Conventions. counts a multiply-accumulate as 2 operations, which + /// is what llama.cpp's own test-backend-ops does — its printed "60.13 GFLOP/run" for + /// q4_K m=4096 k=14336 n=512 is exactly 2·512·14336·4096, so the two projects' numbers are + /// directly comparable. For quantized kernels the count is the logical MAC count of the matmul, not + /// the number of machine instructions retired — a Q4_K matmul and an F32 matmul of the same shape are + /// credited identically, which is the only way a quantized kernel can be compared to a dense roofline. + /// + /// counts bytes that must cross the memory bus, reads plus writes. Follow the + /// STREAM convention and count the write itself but not the read-for-ownership traffic it implies, so + /// a copy of N bytes is 2N, not 3N. + /// + /// Logical floating-point operations per invocation; 0 when the benchmark does no arithmetic. + /// Bytes moved to/from memory per invocation; 0 when the benchmark is not bandwidth-bound. + public readonly record struct WorkAmount(long Flops, long Bytes) + { + /// A matmul of × by ×. + public static WorkAmount Matmul(long rows, long k, long n) + { + return new WorkAmount(2L * rows * k * n, 0L); + } + + /// Pure memory traffic, no arithmetic worth counting. + public static WorkAmount Memory(long bytes) + { + return new WorkAmount(0L, bytes); + } + } +} diff --git a/Sources/Benchmark/MachineRooflineBenchmark.cs b/Sources/Benchmark/MachineRooflineBenchmark.cs new file mode 100644 index 00000000..ecae0bb7 --- /dev/null +++ b/Sources/Benchmark/MachineRooflineBenchmark.cs @@ -0,0 +1,373 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Runtime.Intrinsics; +using System.Runtime.Intrinsics.X86; +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Running; +using Benchmarks.Helpers; + +namespace Benchmarks +{ + /// + /// Measures this machine's ceilings — sustained memory bandwidth and peak arithmetic rate — so that + /// every other kernel number in this project can be read as a fraction of what the hardware can do, + /// instead of as a bare figure that sounds fast or slow depending on the reader's mood. + /// + /// The question that motivated it. On 2026-07-22 our Q4_K prefill GEMM measured 1.70 TFLOP/s + /// on llama.cpp's own benchmark shape, against their 1.56 — so our kernel is not the reason prefill is + /// behind. But "1.70" only becomes actionable next to a ceiling: at 40% of peak there is a kernel left to + /// write, at 85% there is not, and the only remaining lever is a wider instruction set. Without this + /// benchmark that distinction is guesswork, and CLAUDE.md is explicit that a perf claim without a + /// measurement is a guess however confident the reasoning sounds. + /// + /// Reading the results. + /// + /// / / — the + /// STREAM-style ceiling that bounds decode, which is memory-bound: per token it must stream the + /// whole weight file once, so decode tok/s can never exceed bandwidth ÷ model bytes. Buffers are far + /// larger than any L3, so these are DRAM figures, not cache figures. + /// — the classic dense-FMA roofline, purely register-resident. + /// — the ceiling that actually bounds our quantized kernels, + /// which reach their MACs through vpmaddubsw/vpmaddwd rather than vfmadd. This is + /// the number the Q4_K/Q6_K GEMMs should be judged against; comparing them to the float FMA peak + /// flatters or maligns them depending on how the two instruction paths happen to be provisioned. + /// + /// + /// Counting convention is 's: a multiply-accumulate is 2 operations, + /// matching llama.cpp's test-backend-ops, so the columns are directly comparable across projects. + /// The integer benchmark is credited with the logical int8 MACs it performs, on the same footing a + /// quantized matmul is credited with the MACs of the dense matmul it stands in for. + /// + /// All five saturate every core, because a single-threaded ceiling would not bound anything this + /// project runs — both prefill and decode are parallel over the whole machine. + /// + /// Run: + /// dotnet run -c Release --project Sources/Benchmark -- --filter "*MachineRoofline*" + /// + [Config(typeof(BenchmarkConfig))] + public class MachineRooflineBenchmark + { + /// Per-buffer size. Must comfortably exceed L3 so the bandwidth figures are DRAM, not cache. + public const long BufferBytes = 256L * 1024 * 1024; + + /// Independent accumulator chains per thread — enough to cover FMA latency and saturate the ports. + public const int Chains = 12; + + /// Inner iterations per thread, sized so one invocation lasts several milliseconds. + public const int Iterations = 2_000_000; + + private const int FloatCount = (int)(BufferBytes / sizeof(float)); + + /// Lanes per 256-bit vector of . + private const int FloatLanes = 8; + + /// Lanes per 256-bit vector of — the logical MAC count of one vpmaddubsw. + private const int ByteLanes = 32; + + private float[] _a = null!; + private float[] _b = null!; + private float[] _c = null!; + private int _workers; + + /// Consumed so the JIT cannot eliminate the measured loops. + public float FloatSink; + + /// Consumed so the JIT cannot eliminate the measured loops. + public int IntSink; + + /// + /// Declares to how much work each benchmark performs, so the rate columns + /// are derived from code that sits next to the loop being measured. + /// + public static WorkAmount GetWorkAmount(BenchmarkCase benchmarkCase) + { + var workers = Environment.ProcessorCount; + + // One logical MAC per lane per chain per iteration per thread. + var macs = (long)Chains * Iterations * workers; + + return benchmarkCase.Descriptor.WorkloadMethod.Name switch + { + // One pass over one buffer. + nameof(ReadBandwidth) => WorkAmount.Memory(BufferBytes), + + // Read one buffer, write another. + nameof(CopyBandwidth) => WorkAmount.Memory(2L * BufferBytes), + + // STREAM triad: two reads and one write. + nameof(TriadBandwidth) => WorkAmount.Memory(3L * BufferBytes), + + nameof(PeakFmaFloat) => new WorkAmount(2L * macs * FloatLanes, 0L), + + nameof(PeakIntegerDot) => new WorkAmount(2L * macs * ByteLanes, 0L), + + _ => default, + }; + } + + [GlobalSetup] + public void Setup() + { + _workers = Environment.ProcessorCount; + + _a = new float[FloatCount]; + _b = new float[FloatCount]; + _c = new float[FloatCount]; + + var rng = new Random(20260722); + + for (var i = 0; i < FloatCount; i++) + { + _a[i] = (float)rng.NextDouble(); + _b[i] = (float)rng.NextDouble(); + _c[i] = (float)rng.NextDouble(); + } + } + + /// Sustained read bandwidth: one streaming pass over a buffer far larger than L3. + [Benchmark(Baseline = true)] + public void ReadBandwidth() + { + var source = _a; + var partials = new float[_workers]; + + Parallel.For(0, _workers, new ParallelOptions { MaxDegreeOfParallelism = _workers }, worker => + { + var (start, end) = SliceFor(worker); + var span = source.AsSpan(start, end - start); + var acc = Vector256.Zero; + var i = 0; + + for (; i <= span.Length - FloatLanes; i += FloatLanes) + { + acc += Vector256.Create(span.Slice(i, FloatLanes)); + } + + var sum = Vector256.Sum(acc); + + for (; i < span.Length; i++) + { + sum += span[i]; + } + + partials[worker] = sum; + }); + + var total = 0f; + + for (var i = 0; i < partials.Length; i++) + { + total += partials[i]; + } + + FloatSink = total; + } + + /// Sustained copy bandwidth: one read stream plus one write stream. + [Benchmark] + public void CopyBandwidth() + { + var source = _a; + var destination = _b; + + Parallel.For(0, _workers, new ParallelOptions { MaxDegreeOfParallelism = _workers }, worker => + { + var (start, end) = SliceFor(worker); + source.AsSpan(start, end - start).CopyTo(destination.AsSpan(start, end - start)); + }); + + FloatSink = destination[0]; + } + + /// STREAM triad a = b + s·c: two read streams plus one write stream, with real arithmetic. + [Benchmark] + public void TriadBandwidth() + { + var a = _a; + var b = _b; + var c = _c; + var scalar = Vector256.Create(3.0f); + + Parallel.For(0, _workers, new ParallelOptions { MaxDegreeOfParallelism = _workers }, worker => + { + var (start, end) = SliceFor(worker); + var i = start; + + for (; i <= end - FloatLanes; i += FloatLanes) + { + var vb = Vector256.Create(b.AsSpan(i, FloatLanes)); + var vc = Vector256.Create(c.AsSpan(i, FloatLanes)); + + (vb + (scalar * vc)).CopyTo(a.AsSpan(i, FloatLanes)); + } + + for (; i < end; i++) + { + a[i] = b[i] + (3.0f * c[i]); + } + }); + + FloatSink = a[0]; + } + + /// + /// Peak float FMA rate with no memory traffic at all: independent register-resident accumulator chains, + /// so the result is bounded by issue width and latency rather than by cache or DRAM. + /// + [Benchmark] + public void PeakFmaFloat() + { + if (!Fma.IsSupported) + { + return; + } + + var partials = new float[_workers]; + + Parallel.For(0, _workers, new ParallelOptions { MaxDegreeOfParallelism = _workers }, worker => + { + partials[worker] = FmaChains(); + }); + + var sum = 0f; + + for (var i = 0; i < partials.Length; i++) + { + sum += partials[i]; + } + + FloatSink = sum; + } + + /// + /// Peak int8 dot-product rate through the exact instruction pair our quantized kernels use — + /// vpmaddubsw then vpmaddwd, accumulated into 32-bit lanes. This, not + /// , is the ceiling Q4KGemvKernel.GemmTiled is actually racing. + /// + [Benchmark] + public void PeakIntegerDot() + { + if (!Avx2.IsSupported) + { + return; + } + + var partials = new int[_workers]; + + Parallel.For(0, _workers, new ParallelOptions { MaxDegreeOfParallelism = _workers }, worker => + { + partials[worker] = IntegerDotChains(); + }); + + var sum = 0; + + for (var i = 0; i < partials.Length; i++) + { + sum += partials[i]; + } + + IntSink = sum; + } + + /// + /// independent FMA chains held in named locals, one per accumulator. + /// + /// The accumulators must not live in a stackalloc span. A first version of this benchmark + /// put them there and measured 0.79 TFLOP/s — below the 1.70 TFLOP/s our real Q4_K matmul + /// achieves, which is impossible for a loop that touches no memory. The span forced a load and a store + /// per accumulator per iteration, so it measured L1 round-trips rather than FMA issue rate. Constant + /// indices are what keep a value in a register, and that is precisely the property under test here. + /// + private static float FmaChains() + { + var multiplicand = Vector256.Create(1.000001f); + var addend = Vector256.Create(0.000001f); + + var a0 = Vector256.Create(1f); + var a1 = Vector256.Create(2f); + var a2 = Vector256.Create(3f); + var a3 = Vector256.Create(4f); + var a4 = Vector256.Create(5f); + var a5 = Vector256.Create(6f); + var a6 = Vector256.Create(7f); + var a7 = Vector256.Create(8f); + var a8 = Vector256.Create(9f); + var a9 = Vector256.Create(10f); + var a10 = Vector256.Create(11f); + var a11 = Vector256.Create(12f); + + for (var iteration = 0; iteration < Iterations; iteration++) + { + a0 = Fma.MultiplyAdd(a0, multiplicand, addend); + a1 = Fma.MultiplyAdd(a1, multiplicand, addend); + a2 = Fma.MultiplyAdd(a2, multiplicand, addend); + a3 = Fma.MultiplyAdd(a3, multiplicand, addend); + a4 = Fma.MultiplyAdd(a4, multiplicand, addend); + a5 = Fma.MultiplyAdd(a5, multiplicand, addend); + a6 = Fma.MultiplyAdd(a6, multiplicand, addend); + a7 = Fma.MultiplyAdd(a7, multiplicand, addend); + a8 = Fma.MultiplyAdd(a8, multiplicand, addend); + a9 = Fma.MultiplyAdd(a9, multiplicand, addend); + a10 = Fma.MultiplyAdd(a10, multiplicand, addend); + a11 = Fma.MultiplyAdd(a11, multiplicand, addend); + } + + return Vector256.Sum(a0 + a1 + a2 + a3 + a4 + a5 + a6 + a7 + a8 + a9 + a10 + a11); + } + + /// + /// independent vpmaddubsw+vpmaddwd chains in named locals, for the + /// same register-residency reason as . + /// + private static int IntegerDotChains() + { + var weights = Vector256.Create((byte)3); + var activations = Vector256.Create((sbyte)5); + var ones = Vector256.Create((short)1); + + var a0 = Vector256.Create(1); + var a1 = Vector256.Create(2); + var a2 = Vector256.Create(3); + var a3 = Vector256.Create(4); + var a4 = Vector256.Create(5); + var a5 = Vector256.Create(6); + var a6 = Vector256.Create(7); + var a7 = Vector256.Create(8); + var a8 = Vector256.Create(9); + var a9 = Vector256.Create(10); + var a10 = Vector256.Create(11); + var a11 = Vector256.Create(12); + + for (var iteration = 0; iteration < Iterations; iteration++) + { + a0 = Avx2.Add(a0, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a1 = Avx2.Add(a1, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a2 = Avx2.Add(a2, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a3 = Avx2.Add(a3, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a4 = Avx2.Add(a4, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a5 = Avx2.Add(a5, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a6 = Avx2.Add(a6, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a7 = Avx2.Add(a7, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a8 = Avx2.Add(a8, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a9 = Avx2.Add(a9, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a10 = Avx2.Add(a10, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a11 = Avx2.Add(a11, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + } + + return Vector256.Sum(a0 + a1 + a2 + a3 + a4 + a5 + a6 + a7 + a8 + a9 + a10 + a11); + } + + /// Splits the buffer into one contiguous, vector-aligned slice per worker. + private (int Start, int End) SliceFor(int worker) + { + var perWorker = (FloatCount / _workers / FloatLanes) * FloatLanes; + var start = worker * perWorker; + var end = worker == _workers - 1 ? FloatCount : start + perWorker; + + return (start, end); + } + } +} diff --git a/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs b/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs index c322ed79..f7499a0d 100644 --- a/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs +++ b/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs @@ -4,6 +4,7 @@ // For commercial licensing options, contact: devonbike@gmail.com using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Running; using Benchmarks.Helpers; using DevOnBike.Overfit.LanguageModels.Loading; using DevOnBike.Overfit.LanguageModels.Runtime; @@ -74,19 +75,48 @@ public string Shape private bool _originalTiled; private bool _originalStationary; - [GlobalSetup] - public void Setup() + private static (int InputSize, int OutputSize) ShapeOf(string shape) { - (_inputSize, _outputSize) = Shape switch + return shape switch { "ffn_gate_up" => (2048, 11008), "ffn_down" => (11008, 2048), + // The exact shape llama.cpp's own test-backend-ops reports (m=4096, k=14336): 60.13 GFLOP // at n=512, where its AVX2 build measured 1.56 TFLOPS for q4_K. Same shape, same thread // count (32) - the only like-for-like kernel comparison available without editing their tests. "llama_ref" => (14336, 4096), + _ => (2048, 2048), }; + } + + /// + /// Declares each benchmark's work amount so the TFLOP/s and GB/s columns are computed in-repo. + /// + /// Note that declares bytes, not FLOPs: it performs no + /// multiply-accumulate, so crediting it with the matmul's FLOP count — as an out-of-repo script briefly + /// did on 2026-07-22, yielding a fictitious 29.6 TFLOP/s — describes memory traffic as arithmetic. + /// + public static WorkAmount GetWorkAmount(BenchmarkCase benchmarkCase) + { + var rows = (int)benchmarkCase.Parameters["Rows"]; + var (inputSize, outputSize) = ShapeOf((string)benchmarkCase.Parameters["Shape"]); + + if (benchmarkCase.Descriptor.WorkloadMethod.Name == nameof(QuantizeActivationsOnly)) + { + // Reads rows×inputSize floats, writes the same count of sbyte quants plus per-super-block + // scales and bsums — the scales/bsums are ~1% of the traffic and are not modelled. + return WorkAmount.Memory((long)rows * inputSize * (sizeof(float) + sizeof(sbyte))); + } + + return WorkAmount.Matmul(rows, inputSize, outputSize); + } + + [GlobalSetup] + public void Setup() + { + (_inputSize, _outputSize) = ShapeOf(Shape); _originalTiled = BatchedQuantProjection.UseTiledPrefillQ4K; _originalStationary = BatchedQuantProjection.UseWeightStationaryQ4K; From 915032dc2bd9e9385361fd769c2a78a98ff46aac Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Wed, 22 Jul 2026 21:19:27 +0200 Subject: [PATCH 15/37] llama --- ROADMAP.md | 30 +++++++++++++ .../Runtime/BatchedAttentionKernel.cs | 43 ++++++++++++++++++- 2 files changed, 72 insertions(+), 1 deletion(-) diff --git a/ROADMAP.md b/ROADMAP.md index 02295780..58d04f3e 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -482,6 +482,36 @@ optimisation in this section). More workers still wins for us; there is nothing |---|---:|---:|---:|---:|---:|---:| | prefill | 151 | 197 | 218 | 209 | 238 | **246** | +#### ▶ attn_scores — load-balanced query order: real but 6× smaller than predicted + +`OverfitParallel.For` splits its range into **contiguous** chunks, but under the causal mask query `i` +attends over `basePos+i+1` keys, so work grows linearly with the index. On a 672-token prefill across +32 workers, worker 0 got rows 0-20 (≈231 dot products) and worker 31 rows 651-671 (≈13 902). +`BatchedAttentionKernel.BalancedQueryIndex` now pairs slot `2k`→query `k` with slot `2k+1`→query +`rows-1-k`, so every consecutive pair costs `rows+1` wherever it lands. Bit-identical (queries are +independent; nothing is reduced across them), zero cost, `OVERFIT_BALANCED_ATTN=0` disables it. + +ABAB-interleaved, 3 rounds, best-of-N, untouched FFN as the canary: + +| component | baseline | balanced | ratio | +|---|---:|---:|---:| +| **attn_scores** | 275.5 ms | **244.9 ms** | **1.12×** | +| attn_q (canary) | 105.4 | 105.6 | 1.00× | +| ffn_down (canary) | 725.4 | 722.2 | 1.00× | +| total/request | 2793.4 | 2768.5 | 1.01× | + +Kept — clean separation across all three rounds, canaries flat. But **the prediction was 1.97× and the +measurement was 1.12×**, so the model behind it was wrong: chunk imbalance is a real cost but not what +dominates this kernel. Worth recording as the correction, because the same "longest chunk sets the +duration" reasoning would misprice the next scheduling change too. + +**What the profile actually says about attn_scores.** Per head-layer the causal QK plus softmax·V is +≈115.7 MFLOP; across 16 heads × 36 layers that is **66.6 GFLOP in 244.9 ms = 0.27 TFLOP/s** — **12% of +this machine's 2.19 TFLOP/s float ceiling**, and 6× below our own Q4_K GEMM. Keys per head are 344 KB, +so this fits L2 and is not bandwidth-bound. The kernel itself (`CachedAttentionKernel.ComputeSingleHead`, +reached one query at a time) is the open question — that, and the float side of Q4_K dequantization, are +the two measured candidates left. + #### ▶ WHAT IS LEFT — profile at 249 tok/s, gap 2.18× ``` diff --git a/Sources/Main/LanguageModels/Runtime/BatchedAttentionKernel.cs b/Sources/Main/LanguageModels/Runtime/BatchedAttentionKernel.cs index 005f7268..2c9fbc9c 100644 --- a/Sources/Main/LanguageModels/Runtime/BatchedAttentionKernel.cs +++ b/Sources/Main/LanguageModels/Runtime/BatchedAttentionKernel.cs @@ -34,6 +34,13 @@ namespace DevOnBike.Overfit.LanguageModels.Runtime /// public static unsafe class BatchedAttentionKernel { + /// + /// Whether hands workers an interleaved slot order instead of the raw + /// query order. A/B switch for the measurement below; set OVERFIT_BALANCED_ATTN=0 to disable. + /// + internal static bool UseBalancedQueryOrder = + Environment.GetEnvironmentVariable("OVERFIT_BALANCED_ATTN") != "0"; + /// /// Sequential batched attention. is row-major /// [rows × headDim], / @@ -101,6 +108,7 @@ public static void ComputeParallel( Scale = scale, Softcap = softcap, BasePos = cacheLength - rows, + Balanced = UseBalancedQueryOrder, }; OverfitParallel.For(0, rows, &ComputeQueryRange, &context); @@ -120,12 +128,44 @@ private static void ComputeQueryRange(int start, int end, void* context) var output = new Span(ctx.Output, ctx.Rows * headDim); var scratch = new Span(ctx.Scratch, ctx.Rows * cacheLength); - for (var i = start; i < end; i++) + var balanced = ctx.Balanced; + var rows = ctx.Rows; + + for (var slot = start; slot < end; slot++) { + var i = balanced ? BalancedQueryIndex(slot, rows) : slot; + ComputeQuery(query, keys, values, output, scratch, i, ctx.BasePos, cacheLength, headDim, ctx.Scale, ctx.Softcap); } } + /// + /// Maps a scheduling slot to a query index so that any contiguous run of slots carries the same + /// amount of work, by pairing the cheapest remaining query with the most expensive one. + /// + /// Why this is needed. Under the causal mask query i attends over basePos+i+1 + /// keys, so work grows linearly with the query index — yet OverfitParallel.For splits its range + /// into contiguous chunks (perChunk = ceil(total/chunks)). On a 672-token prefill across + /// 32 workers that gave worker 0 rows 0-20 (≈231 dot products) and worker 31 rows 651-671 (≈13 902). + /// The region's duration is its longest chunk, so it ran 1.97× longer than the balanced ideal + /// of 7 067 while 31 workers sat idle. + /// + /// Slot 2k maps to query k and slot 2k+1 to query rows-1-k, so each + /// consecutive pair costs rows+1 regardless of where it lands. The mapping is a bijection over + /// [0, rows) for both parities of rows. + /// + /// Output is bit-identical: queries are independent — each writes its own disjoint output + /// row and score-scratch row, and no value is reduced across queries — so reordering them changes + /// nothing but which worker runs which. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int BalancedQueryIndex(int slot, int rows) + { + var half = slot >> 1; + + return (slot & 1) == 0 ? half : rows - 1 - half; + } + /// /// One query: causal-visible length basePos + i + 1, delegating to the /// proven single-token kernel against its own output/scratch rows. @@ -209,6 +249,7 @@ private struct AttnContext public float Scale; public float Softcap; public int BasePos; + public bool Balanced; } } } From ccf6d99959e4ae535a778071ab386df1d5fd0413 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Wed, 22 Jul 2026 22:28:49 +0200 Subject: [PATCH 16/37] llama --- ROADMAP.md | 146 ++++ Sources/Benchmark/MachineRooflineBenchmark.cs | 669 ++++++++++++++++++ .../Q4KPrefillProjectionBenchmark.cs | 84 +++ .../Runtime/BatchedQuantProjection.cs | 66 +- .../LanguageModels/Runtime/Q4KGemvKernel.cs | 112 ++- 5 files changed, 1042 insertions(+), 35 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 58d04f3e..df2d75a2 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -512,6 +512,152 @@ so this fits L2 and is not bandwidth-bound. The kernel itself (`CachedAttentionK reached one query at a time) is the open question — that, and the float side of Q4_K dequantization, are the two measured candidates left. +#### ★ AVX-512 GO/NO-GO GATE — PASSED, the port is worth writing + +Our Q4_K GEMM sits at 78% of the 256-bit float ceiling and FFN is 69% of prefill, so the only way to move +the dominant cost is to raise the ceiling. Before writing any kernel, `MachineRooflineBenchmark` was +extended with `Vector512` variants to check whether this silicon actually delivers the wider ceiling. +This was a real risk: Zen 4 double-pumps 512-bit ops through a 256-bit datapath (~1.1×) and many Intel +parts drop clocks under 512-bit load, either of which would have killed the plan. + +Box: **AMD Ryzen 9 9950X3D** (Zen 5), 16 physical / 32 logical, AVX-512 F+BW+CD+DQ+VL + VNNI + VBMI + IFMA. + +| ceiling | 256-bit | 512-bit | gain | +|---|---:|---:|---:| +| float FMA | 2.20 TFLOP/s | **4.15** | **1.89×** | +| int8 dot | 11.16 TOPS | **22.87** | **2.05×** | + +Zen 5 has the full 512-bit datapath and the measurement shows it — essentially the theoretical 2×, with no +visible clock penalty. **Our 1.70 TFLOP/s GEMM is 78% of the 256-bit ceiling but only 41% of the 512-bit +one.** If the port preserves utilisation, FFN 1923 ms → ~1000 ms and prefill 2769 → ~1850 ms ≈ **373 tok/s**, +which would be *above* llama.cpp's AVX2 build (336.7) and 1.45× off their AVX-512 (541.7). + +Two notes. **AVX-512 VNNI is present**: `vpdpbusd` collapses the `vpmaddubsw`+`vpmaddwd`+`add` triple our +kernel issues into one instruction — a second-order lever here since the kernel is float-bound, but it is +what llama.cpp uses. And the existing **"AVX-512 decode port" negative does not transfer**: decode is +memory-bound (measured today at ~46 GB/s against an 89.9 GB/s ceiling), where wider vectors buy nothing; +prefill is compute-bound and pinned against the float ceiling. + +Next: port `Q4KGemvKernel.GemmTiled` (ffn_gate_up, attn_q/o), then `Q6KGemvKernel.GemmTiled` (ffn_down) — +parity test first, then measure. Dispatch at run time through `CpuFeatures.HasAvx512`, never at compile +time: `Cli.csproj` pins `IlcInstructionSet=avx2` and the AOT build must keep running on machines without it. + +#### ★ RETRACTION + the kernel's real ceiling + +**Retracted: "our Q4_K GEMM runs at 78% of the float ceiling."** That divided *logical* MACs by the +*floating-point instruction* ceiling, but the kernel performs one `vpmaddubsw` per **32** MACs and issues +only ~6 float ops per column per block against ~160 integer/shuffle ops. Against the ceiling that actually +applies it sits at **15% of 11.2 TOPS**, not 78%. The AVX-512 recommendation survives the correction, but +its stated reason ("the ceiling is too low") was wrong — the real problem is instructions issued per MAC. + +`MachineRooflineBenchmark` now measures the ceiling **for this kernel's instruction mix** — the eight- +statement `iacc0` block verbatim, `Blend` + two lane shuffles + `vpmaddubsw` + `Add` — rather than an +idealised dot chain: + +| | TFLOP/s | +|---|---:| +| idealised int8 chain (3 instructions / 32 MACs) | 11.2 | +| **kernel's instruction mix, 4 live accumulators** | **4.63** | +| same mix, 512-bit | 7.71 | +| **our real GEMM** | **1.70** | + +So the shuffles cost 2.4× against the idealised chain, and we then reach only **37% of our own mix's +ceiling**. That residual 2.7× is not arithmetic — it is loads, weight decode, the float tail, and spills. + +**NEGATIVE — register pressure is not the explanation.** The suspicion was that `GemmTiled`'s `stackalloc` +accumulator spans spill: at four columns it holds `accRow`+`accMin` (8 vectors, live across the block loop), +`iaccB`+`iaccMinB` (8, across the sub-block loop) and `iacc0`+`iacc1` (8) — 24 vectors before a single +weight, against 16 ymm registers. Probing it with `IntegerDotChains`' body at 12 vs 16 chains (12+3 +constants fit ymm, 16+3 do not; both fit 512-bit's 32 zmm): + +| live accumulators | 256-bit | 512-bit | +|---|---:|---:| +| 12 (fits ymm) | 11.50 | 23.52 | +| 16 (exceeds ymm) | **14.99** | 24.19 | + +Sixteen chains are **30% faster** at 256-bit, not slower — more independent chains cover latency better and +any spill hides behind the surrounding work. **There is no register cliff, and the "de-spill first" plan is +dropped.** + +An earlier version of this probe reported the opposite (−41% at 256-bit, and 512-bit falling *harder* than +256-bit, which cannot be true if the larger file helps at all). It routed each step through a helper taking +five vector parameters; once the statements were written inline the effect vanished entirely. The tell was +the impossible 512-bit ordering — treat that shape of result as a broken benchmark, not a discovery. + +**Still unexplained: 1.70 actual vs 4.64 for its own instruction mix, a 2.7× residual.** The mix benchmark +models only the eight `iacc0` statements. Ablating the three pieces it omits, inside the real kernel +(`Q4KGemvKernel.Ablate*` — measurement-only toggles, default off), on `ffn_gate_up` at 672 rows: + +| ablation | mean | vs `Tiled` | +|---|---:|---:| +| `Tiled` (baseline) | 15.567 ms | — | +| no F16 scale/min decode | 13.701 ms | **−12.0%** | +| no scalar `Unpack` of 6-bit scales | 15.016 ms | −3.5% | +| no nibble `And`/shift | 15.679 ms | +0.7% (tie) | + +Error bars are ±1.0–1.1 ms on ~15 ms (≈7%), so only the F16 result clears the noise, and barely; the other +two sit inside it. **Together they bound at ~15% and do not explain a 2.7× residual (≈63% of runtime).** + +The one thing the mix benchmark did not model at all is **memory traffic** — it ran on register constants. +The real kernel issues 8×32 B weight loads per sub-block (12.7 MB streamed per projection), per-column +activation loads, and span-backed accumulator accesses. That is the remaining suspect, and it is untested. + +**NEGATIVE — the F16 decode's 12% is the scalar conversions, not the memory round-trip.** +`LoadF16x8Rearrange` used to store its shuffled vector to `stackalloc` and immediately re-read it as eight +`ushort`s — textbook store-to-load forwarding stall. Extracting the lanes from the register with `pextrw` +instead measured **15,269 µs vs 15,567 µs, i.e. −1.9% against ±7% noise: a tie**, with the ablation floor +unchanged at 11.9%. The round-trip was free; the eight scalar `Half`→`float` conversions are the cost. + +Capturing it therefore means *eliminating* the conversions, not speeding them up: store the scales as **f32 at +repack time**. `block_q4_Kx8` is our own layout, so this is available — +32 B on a 1152 B block (**+2.8% +weight RAM for 12%**), at the price of a `.gguf.repack` sidecar format change. Note x86 could do this in one +`vcvtph2ps`, but .NET exposes neither an `F16C` intrinsic class nor a `Half` overload of `Vector128.Widen`. + +#### ★★ THE WEIGHT STREAM IS READ 84 TIMES PER PROJECTION — no cache blocking exists + +Applying the standard model (Goto & van de Geijn, *Anatomy of High-Performance Matrix Multiplication* — the +GotoBLAS/BLIS scheme, where block sizes are derived from cache sizes: an `mr×nr` tile of C in registers, a +`kc×nr` panel of B in L1, an `mc×kc` block of A in L2, a `kc×n` panel of B in L3) exposes what the profiling +missed all day: + +`GemmTiled` receives **8 columns** and walks the **entire** weight matrix. The dispatcher splits 672 rows +into tiles of 8, so **84 tiles each stream all 12.68 MB** of `ffn_gate_up`'s weights: + + 84 × 12.68 MB = 1.07 GB per projection, in 15.27 ms = ~70 GB/s + measured DRAM read ceiling = 90 GB/s + +Our tiling is a *register* tile (`MaxTileCols`) only — there is **no L2/L3 blocking level at all**. This is a +candidate for the whole remaining 1.56× residual, and unlike everything else on the list it is a structural +fix with a textbook algorithm behind it. + +**Measured — the traffic argument holds, then breaks on parallel granularity.** `ffn_gate_up`, two runs each, +agreeing on ordering: + +| tile | passes | 672 rows | 1024 rows | +|---|---:|---:|---:| +| NR=4 | rows/4 | 17.8 ms · 1.70 | 27–29 ms · 1.66 | +| NR=8 | rows/8 | **15.2–15.9 ms · 1.95** | 23.7–24.5 ms · 1.92 | +| NR=16 | rows/16 | 16.5–17.0 ms · 1.80 | **22.7–23.0 ms · 2.02** | + +Halving the traffic 4→8 buys **+17%** exactly as predicted. Halving it again 8→16 *loses* **8%** at 672 rows, +because 42 tiles over 32 cores leaves ten workers with two and twenty-two with one — a 1.52× imbalance +against 1.14× at NR=8. At 1024 rows there are enough tiles again and the wider tile wins by 4–7%. + +So `ResolveTileCols` now takes the widest tile that still gives **~2 tiles per core**, not one. That +reproduces NR=8 at 672 (no change to the profiled prompt, prefill stays 249 tok/s) and switches to NR=16 from +~1024 rows — a real gain for long prompts. The ≥1024 branch was measured rather than reasoned, because six +mechanism hypotheses were refuted the same day. + +*The trade-off itself is the finding.* Traffic falls as 1/NR while parallel granularity falls as NR, so tile +width alone cannot buy much. Escaping it needs the blocking level the kernel does not have: **block over +output rows as well**, so a wide column tile and many independent work items stop being alternatives. That is +the Goto/BLIS structure and it is the outstanding work. + +*Invalidated run, kept as a warning:* the first tile sweep ran inside an 11-benchmark class and reported +`Tiled` and `Tiled_Cols8` — **the same configuration** — 21% apart, far outside their ±9% bars. Two identical +arms in one table is the cheapest canary there is; narrowing the filter so the arms sit adjacent in time made +the result reproducible. + #### ▶ WHAT IS LEFT — profile at 249 tok/s, gap 2.18× ``` diff --git a/Sources/Benchmark/MachineRooflineBenchmark.cs b/Sources/Benchmark/MachineRooflineBenchmark.cs index ecae0bb7..ef01ea2c 100644 --- a/Sources/Benchmark/MachineRooflineBenchmark.cs +++ b/Sources/Benchmark/MachineRooflineBenchmark.cs @@ -3,6 +3,9 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com +using System.Diagnostics.CodeAnalysis; +using System.Runtime.CompilerServices; +using System.Runtime.InteropServices; using System.Runtime.Intrinsics; using System.Runtime.Intrinsics.X86; using BenchmarkDotNet.Attributes; @@ -59,6 +62,30 @@ public class MachineRooflineBenchmark /// Inner iterations per thread, sized so one invocation lasts several milliseconds. public const int Iterations = 2_000_000; + /// Inner iterations for the kernel-shaped chains, which do far more work per iteration. + public const int ShapeIterations = 200_000; + + /// Independent chains in the kernel-shaped benchmarks — fewer, because each holds more live vectors. + public const int ShapeChains = 4; + + /// `Mul(Blend(...), Sh32(...))` statements per chain, mirroring the kernel's iacc0 block. + public const int ShapeMuls = 8; + + /// Live accumulator chains in the wide variants — more than the 16 ymm registers AVX2 offers. + public const int WideChains = 16; + + /// Iterations for the wide variants, scaled down by the extra chains so runtime stays comparable. + public const int WideIterations = ShapeIterations / 4; + + /// Backing buffer for the streamed-weight probes. + public const long StreamBytes = 256L * 1024 * 1024; + + /// Per-worker window that stays resident in its own L2. + public const long L2WindowBytes = 512L * 1024; + + /// Per-worker window that overflows L2 but stays within the shared L3. + public const long L3WindowBytes = 4L * 1024 * 1024; + private const int FloatCount = (int)(BufferBytes / sizeof(float)); /// Lanes per 256-bit vector of . @@ -67,9 +94,16 @@ public class MachineRooflineBenchmark /// Lanes per 256-bit vector of — the logical MAC count of one vpmaddubsw. private const int ByteLanes = 32; + /// Lanes per 512-bit vector of . + private const int FloatLanes512 = 16; + + /// Lanes per 512-bit vector of . + private const int ByteLanes512 = 64; + private float[] _a = null!; private float[] _b = null!; private float[] _c = null!; + private byte[] _stream = null!; private int _workers; /// Consumed so the JIT cannot eliminate the measured loops. @@ -104,6 +138,35 @@ public static WorkAmount GetWorkAmount(BenchmarkCase benchmarkCase) nameof(PeakIntegerDot) => new WorkAmount(2L * macs * ByteLanes, 0L), + // Report nothing rather than a fictitious rate when the silicon lacks AVX-512: the body + // returns immediately, so crediting it with work would show an infinite throughput. + nameof(PeakFmaFloat512) => Avx512F.IsSupported + ? new WorkAmount(2L * macs * FloatLanes512, 0L) + : default, + + nameof(PeakIntegerDot512) => Avx512BW.IsSupported + ? new WorkAmount(2L * macs * ByteLanes512, 0L) + : default, + + // Each of the ShapeMuls statements is one vpmaddubsw over a full vector of int8 pairs. + nameof(PeakQ4KShape) => new WorkAmount( + 2L * ShapeChains * ShapeMuls * ShapeIterations * workers * ByteLanes, 0L), + + nameof(PeakQ4KShape512) => Avx512BW.IsSupported + ? new WorkAmount(2L * ShapeChains * ShapeMuls * ShapeIterations * workers * ByteLanes512, 0L) + : default, + + nameof(PeakIntegerDotWide) => new WorkAmount( + 2L * WideChains * Iterations * workers * ByteLanes, 0L), + + // Same instruction mix and MAC count as PeakQ4KShape — only the weight source differs. + nameof(PeakQ4KShapeFromL2) or nameof(PeakQ4KShapeFromL3) or nameof(PeakQ4KShapeFromDram) => + new WorkAmount(2L * ShapeChains * ShapeMuls * ShapeIterations * workers * ByteLanes, 0L), + + nameof(PeakIntegerDotWide512) => Avx512BW.IsSupported + ? new WorkAmount(2L * WideChains * Iterations * workers * ByteLanes512, 0L) + : default, + _ => default, }; } @@ -117,7 +180,9 @@ public void Setup() _b = new float[FloatCount]; _c = new float[FloatCount]; + _stream = new byte[StreamBytes]; var rng = new Random(20260722); + rng.NextBytes(_stream); for (var i = 0; i < FloatCount; i++) { @@ -360,6 +425,610 @@ private static int IntegerDotChains() return Vector256.Sum(a0 + a1 + a2 + a3 + a4 + a5 + a6 + a7 + a8 + a9 + a10 + a11); } + /// + /// The 512-bit counterpart of — the go/no-go gate for porting the quantized + /// GEMMs to AVX-512. + /// + /// What decides it. Our Q4_K GEMM already runs at 78% of the 256-bit float ceiling, so the + /// only way to move FFN — 69% of prefill — is to raise the ceiling. llama.cpp's AVX-512 build measured + /// 1.60× its AVX2 build on this machine, so the instruction set pays there. What is not yet known + /// is whether it pays here: many parts run one 512-bit FMA unit rather than two, or drop clocks + /// under 512-bit load, and then the honest headroom is ~1.1× and the port is not worth writing. If this + /// benchmark does not beat by a clear margin, the plan dies here rather than + /// after a kernel rewrite. + /// + /// Note this measures a burst, not a sustained thermal steady state; a part that downclocks only + /// after seconds of 512-bit work will look better here than in production. + /// + [Benchmark] + public void PeakFmaFloat512() + { + if (!Avx512F.IsSupported) + { + return; + } + + var partials = new float[_workers]; + + Parallel.For(0, _workers, new ParallelOptions { MaxDegreeOfParallelism = _workers }, worker => + { + partials[worker] = FmaChains512(); + }); + + var sum = 0f; + + for (var i = 0; i < partials.Length; i++) + { + sum += partials[i]; + } + + FloatSink = sum; + } + + /// The 512-bit counterpart of — the ceiling the quantized inner + /// loop would race after an AVX-512 port. + [Benchmark] + public void PeakIntegerDot512() + { + if (!Avx512BW.IsSupported) + { + return; + } + + var partials = new int[_workers]; + + Parallel.For(0, _workers, new ParallelOptions { MaxDegreeOfParallelism = _workers }, worker => + { + partials[worker] = IntegerDotChains512(); + }); + + var sum = 0; + + for (var i = 0; i < partials.Length; i++) + { + sum += partials[i]; + } + + IntSink = sum; + } + + /// Named locals for the same register-residency reason as . + private static float FmaChains512() + { + var multiplicand = Vector512.Create(1.000001f); + var addend = Vector512.Create(0.000001f); + + var a0 = Vector512.Create(1f); + var a1 = Vector512.Create(2f); + var a2 = Vector512.Create(3f); + var a3 = Vector512.Create(4f); + var a4 = Vector512.Create(5f); + var a5 = Vector512.Create(6f); + var a6 = Vector512.Create(7f); + var a7 = Vector512.Create(8f); + var a8 = Vector512.Create(9f); + var a9 = Vector512.Create(10f); + var a10 = Vector512.Create(11f); + var a11 = Vector512.Create(12f); + + for (var iteration = 0; iteration < Iterations; iteration++) + { + a0 = Avx512F.FusedMultiplyAdd(a0, multiplicand, addend); + a1 = Avx512F.FusedMultiplyAdd(a1, multiplicand, addend); + a2 = Avx512F.FusedMultiplyAdd(a2, multiplicand, addend); + a3 = Avx512F.FusedMultiplyAdd(a3, multiplicand, addend); + a4 = Avx512F.FusedMultiplyAdd(a4, multiplicand, addend); + a5 = Avx512F.FusedMultiplyAdd(a5, multiplicand, addend); + a6 = Avx512F.FusedMultiplyAdd(a6, multiplicand, addend); + a7 = Avx512F.FusedMultiplyAdd(a7, multiplicand, addend); + a8 = Avx512F.FusedMultiplyAdd(a8, multiplicand, addend); + a9 = Avx512F.FusedMultiplyAdd(a9, multiplicand, addend); + a10 = Avx512F.FusedMultiplyAdd(a10, multiplicand, addend); + a11 = Avx512F.FusedMultiplyAdd(a11, multiplicand, addend); + } + + return Vector512.Sum(a0 + a1 + a2 + a3 + a4 + a5 + a6 + a7 + a8 + a9 + a10 + a11); + } + + /// Named locals for the same register-residency reason as . + private static int IntegerDotChains512() + { + var weights = Vector512.Create((byte)3); + var activations = Vector512.Create((sbyte)5); + var ones = Vector512.Create((short)1); + + var a0 = Vector512.Create(1); + var a1 = Vector512.Create(2); + var a2 = Vector512.Create(3); + var a3 = Vector512.Create(4); + var a4 = Vector512.Create(5); + var a5 = Vector512.Create(6); + var a6 = Vector512.Create(7); + var a7 = Vector512.Create(8); + var a8 = Vector512.Create(9); + var a9 = Vector512.Create(10); + var a10 = Vector512.Create(11); + var a11 = Vector512.Create(12); + + for (var iteration = 0; iteration < Iterations; iteration++) + { + a0 = Avx512F.Add(a0, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a1 = Avx512F.Add(a1, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a2 = Avx512F.Add(a2, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a3 = Avx512F.Add(a3, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a4 = Avx512F.Add(a4, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a5 = Avx512F.Add(a5, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a6 = Avx512F.Add(a6, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a7 = Avx512F.Add(a7, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a8 = Avx512F.Add(a8, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a9 = Avx512F.Add(a9, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a10 = Avx512F.Add(a10, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a11 = Avx512F.Add(a11, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + } + + return Vector512.Sum(a0 + a1 + a2 + a3 + a4 + a5 + a6 + a7 + a8 + a9 + a10 + a11); + } + + /// + /// The ceiling for the instruction mix Q4KGemvKernel.GemmTiled actually issues, rather than for + /// an idealised dot-product chain — the number that predicts what an AVX-512 port of that kernel can buy. + /// + /// Why the plain integer peak is the wrong reference. reaches + /// 32 MACs in three instructions. The real kernel spends five per group — Blend, two lane + /// shuffles, vpmaddubsw, Add — because the repacked block_q4_Kx8 layout must be + /// rearranged into position before it can be multiplied, and three of those five contend for the shuffle + /// port rather than the vector-ALU ports. That mix, not the dot-product itself, is what the kernel is + /// racing, so this benchmark reproduces the eight-statement iacc0 block verbatim. + /// + /// Comparing this against gives the port's headroom directly: + /// whatever ratio the two show is roughly what widening the column tile to 512 bits can deliver, before + /// any memory or scheduling effects. Measuring it costs 40 lines instead of the ~250 an AVX-512 kernel + /// rewrite would, and today has twice punished reasoning about mechanism ahead of measuring it. + /// + [Benchmark] + public void PeakQ4KShape() + { + var partials = new int[_workers]; + + Parallel.For(0, _workers, new ParallelOptions { MaxDegreeOfParallelism = _workers }, worker => + { + partials[worker] = Q4KShapeChains(); + }); + + var sum = 0; + + for (var i = 0; i < partials.Length; i++) + { + sum += partials[i]; + } + + IntSink = sum; + } + + /// The same instruction mix at 512 bits — two activation columns per instruction. + [Benchmark] + public void PeakQ4KShape512() + { + if (!Avx512BW.IsSupported) + { + return; + } + + var partials = new int[_workers]; + + Parallel.For(0, _workers, new ParallelOptions { MaxDegreeOfParallelism = _workers }, worker => + { + partials[worker] = Q4KShapeChains512(); + }); + + var sum = 0; + + for (var i = 0; i < partials.Length; i++) + { + sum += partials[i]; + } + + IntSink = sum; + } + + private static int Q4KShapeChains() + { + // Four decoded weight-nibble vectors and one broadcast activation vector, exactly as the kernel + // holds them across its eight iacc0 statements. + var w0 = Vector256.Create((byte)3); + var w1 = Vector256.Create((byte)5); + var w2 = Vector256.Create((byte)7); + var w3 = Vector256.Create((byte)9); + var act = Vector256.Create((sbyte)2); + + var a0 = Vector256.Zero; + var a1 = Vector256.Zero; + var a2 = Vector256.Zero; + var a3 = Vector256.Zero; + + for (var iteration = 0; iteration < ShapeIterations; iteration++) + { + a0 = Step(a0, w0, w1, act); + a1 = Step(a1, w1, w2, act); + a2 = Step(a2, w2, w3, act); + a3 = Step(a3, w3, w0, act); + } + + return Vector256.Sum(Avx2.Add(Avx2.Add(a0, a1), Avx2.Add(a2, a3)).AsInt16()); + + // Two of the kernel's eight statements, repeated four times: Blend + two shuffles + maddubs + Add. + static Vector256 Step(Vector256 acc, Vector256 lo, Vector256 hi, Vector256 act) + { + for (var repeat = 0; repeat < ShapeMuls / 2; repeat++) + { + acc = Avx2.Add(acc, Mul256(Blend256(lo, Sh256(hi, 177)), Sh32_256(act, 0))); + acc = Avx2.Add(acc, Mul256(Blend256(Sh256(lo, 177), hi), Sh32_256(act, 85))); + } + + return acc; + } + } + + private static int Q4KShapeChains512() + { + var w0 = Vector512.Create((byte)3); + var w1 = Vector512.Create((byte)5); + var w2 = Vector512.Create((byte)7); + var w3 = Vector512.Create((byte)9); + var act = Vector512.Create((sbyte)2); + + // Blend mask: the 256-bit form used Avx2.Blend with imm 170 (odd int32 lanes from the right + // operand); at 512 bits that is the same pattern repeated over 16 lanes. + var blendMask = Vector512.Create(0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1); + + var a0 = Vector512.Zero; + var a1 = Vector512.Zero; + var a2 = Vector512.Zero; + var a3 = Vector512.Zero; + + for (var iteration = 0; iteration < ShapeIterations; iteration++) + { + a0 = Step(a0, w0, w1, act, blendMask); + a1 = Step(a1, w1, w2, act, blendMask); + a2 = Step(a2, w2, w3, act, blendMask); + a3 = Step(a3, w3, w0, act, blendMask); + } + + return Vector512.Sum(Avx512BW.Add(Avx512BW.Add(a0, a1), Avx512BW.Add(a2, a3)).AsInt16()); + + static Vector512 Step( + Vector512 acc, Vector512 lo, Vector512 hi, Vector512 act, Vector512 mask) + { + for (var repeat = 0; repeat < ShapeMuls / 2; repeat++) + { + acc = Avx512BW.Add(acc, Mul512(Blend512(lo, Sh512(hi, 177), mask), Sh32_512(act, 0))); + acc = Avx512BW.Add(acc, Mul512(Blend512(Sh512(lo, 177), hi, mask), Sh32_512(act, 85))); + } + + return acc; + } + } + + /// + /// Register-pressure probe: the body of verbatim, with the chain count + /// raised from 12 to and nothing else changed. + /// + /// What it settles. Our real GEMM reaches 1.70 TFLOP/s against its own instruction mix's + /// 4.63, and the suspect is that its state does not fit the register file: at four activation columns + /// GemmTiled keeps 8 float accumulators live across the block loop, 8 integer accumulators + /// across the sub-block loop and 8 more inside it — 24 vectors before a single weight, against 16 ymm. + /// Counting registers is reasoning about mechanism, so it is measured: 12 chains plus 3 constants fit + /// in 16 ymm, 16 chains do not, and both fit comfortably in 512-bit's 32 zmm. If the cliff is real, + /// 256-bit falls between the two and 512-bit does not. + /// + /// An earlier version of this probe routed each step through a helper taking five vector + /// parameters and reported 512-bit as worse than 256-bit — impossible if zmm's larger file + /// helps at all, and the giveaway that a non-inlined call was making it time the calling convention + /// rather than the register file. Everything here is inline and every accumulator is a named local. + /// + [Benchmark] + public void PeakIntegerDotWide() + { + var partials = new int[_workers]; + + Parallel.For(0, _workers, new ParallelOptions { MaxDegreeOfParallelism = _workers }, worker => + { + partials[worker] = IntegerDotChainsWide(); + }); + + var sum = 0; + + for (var i = 0; i < partials.Length; i++) + { + sum += partials[i]; + } + + IntSink = sum; + } + + /// The same probe at 512 bits, where 16 chains still fit the 32-register file. + [Benchmark] + public void PeakIntegerDotWide512() + { + if (!Avx512BW.IsSupported) + { + return; + } + + var partials = new int[_workers]; + + Parallel.For(0, _workers, new ParallelOptions { MaxDegreeOfParallelism = _workers }, worker => + { + partials[worker] = IntegerDotChainsWide512(); + }); + + var sum = 0; + + for (var i = 0; i < partials.Length; i++) + { + sum += partials[i]; + } + + IntSink = sum; + } + + private static int IntegerDotChainsWide() + { + var weights = Vector256.Create((byte)3); + var activations = Vector256.Create((sbyte)5); + var ones = Vector256.Create((short)1); + + var a0 = Vector256.Create(1); + var a1 = Vector256.Create(2); + var a2 = Vector256.Create(3); + var a3 = Vector256.Create(4); + var a4 = Vector256.Create(5); + var a5 = Vector256.Create(6); + var a6 = Vector256.Create(7); + var a7 = Vector256.Create(8); + var a8 = Vector256.Create(9); + var a9 = Vector256.Create(10); + var a10 = Vector256.Create(11); + var a11 = Vector256.Create(12); + var a12 = Vector256.Create(13); + var a13 = Vector256.Create(14); + var a14 = Vector256.Create(15); + var a15 = Vector256.Create(16); + + for (var iteration = 0; iteration < Iterations; iteration++) + { + { + a0 = Avx2.Add(a0, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a1 = Avx2.Add(a1, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a2 = Avx2.Add(a2, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a3 = Avx2.Add(a3, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a4 = Avx2.Add(a4, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a5 = Avx2.Add(a5, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a6 = Avx2.Add(a6, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a7 = Avx2.Add(a7, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a8 = Avx2.Add(a8, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a9 = Avx2.Add(a9, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a10 = Avx2.Add(a10, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a11 = Avx2.Add(a11, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a12 = Avx2.Add(a12, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a13 = Avx2.Add(a13, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a14 = Avx2.Add(a14, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + a15 = Avx2.Add(a15, Avx2.MultiplyAddAdjacent(Avx2.MultiplyAddAdjacent(weights, activations), ones)); + } + } + + return Vector256.Sum(a0 + a1 + a2 + a3 + a4 + a5 + a6 + a7 + a8 + a9 + a10 + a11 + a12 + a13 + a14 + a15); + } + + private static int IntegerDotChainsWide512() + { + var weights = Vector512.Create((byte)3); + var activations = Vector512.Create((sbyte)5); + var ones = Vector512.Create((short)1); + + var a0 = Vector512.Create(1); + var a1 = Vector512.Create(2); + var a2 = Vector512.Create(3); + var a3 = Vector512.Create(4); + var a4 = Vector512.Create(5); + var a5 = Vector512.Create(6); + var a6 = Vector512.Create(7); + var a7 = Vector512.Create(8); + var a8 = Vector512.Create(9); + var a9 = Vector512.Create(10); + var a10 = Vector512.Create(11); + var a11 = Vector512.Create(12); + var a12 = Vector512.Create(13); + var a13 = Vector512.Create(14); + var a14 = Vector512.Create(15); + var a15 = Vector512.Create(16); + + for (var iteration = 0; iteration < Iterations; iteration++) + { + { + a0 = Avx512F.Add(a0, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a1 = Avx512F.Add(a1, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a2 = Avx512F.Add(a2, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a3 = Avx512F.Add(a3, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a4 = Avx512F.Add(a4, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a5 = Avx512F.Add(a5, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a6 = Avx512F.Add(a6, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a7 = Avx512F.Add(a7, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a8 = Avx512F.Add(a8, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a9 = Avx512F.Add(a9, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a10 = Avx512F.Add(a10, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a11 = Avx512F.Add(a11, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a12 = Avx512F.Add(a12, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a13 = Avx512F.Add(a13, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a14 = Avx512F.Add(a14, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + a15 = Avx512F.Add(a15, Avx512BW.MultiplyAddAdjacent(Avx512BW.MultiplyAddAdjacent(weights, activations), ones)); + } + } + + return Vector512.Sum(a0 + a1 + a2 + a3 + a4 + a5 + a6 + a7 + a8 + a9 + a10 + a11 + a12 + a13 + a14 + a15); + } + + /// + /// 's instruction mix with the weight vectors loaded from memory + /// instead of held in registers — the one thing the register-resident mix benchmark did not model, and + /// the last untested candidate for the kernel's unexplained 2.7× residual. + /// + /// Why it matters before anything else. Ablation showed the kernel's non-arithmetic work + /// (F16 scale decode 12%, scalar scale unpack 3.5%, nibble unpack ~0%) bounds at ~15%, nowhere near the + /// 63% of runtime the residual represents. The real kernel streams 8×32 B of weight per sub-block — + /// 12.7 MB per projection. If these probes land near the kernel's measured 1.70–1.95 TFLOP/s, the + /// kernel is load-bound, and an AVX-512 port would widen compute against a memory wall — the + /// same mistake the reverted AVX-512 decode port already made in this repo. + /// + /// The byte-per-MAC ratio matches the kernel: 128 B loaded per 1024 MACs. Each worker walks its + /// own window so the three variants really do sit in L2, in L3 and in DRAM respectively rather than + /// all sharing one hot region. + /// + [Benchmark] + public void PeakQ4KShapeFromL2() + { + StreamProbe(L2WindowBytes); + } + + /// Same mix and byte-per-MAC ratio, from a window sized to live in L3. + [Benchmark] + public void PeakQ4KShapeFromL3() + { + StreamProbe(L3WindowBytes); + } + + /// Same mix and byte-per-MAC ratio, from a window far larger than any cache. + [Benchmark] + public void PeakQ4KShapeFromDram() + { + StreamProbe(StreamBytes); + } + + private void StreamProbe(long window) + { + var stream = _stream; + var partials = new int[_workers]; + + Parallel.For(0, _workers, new ParallelOptions { MaxDegreeOfParallelism = _workers }, worker => + { + // When the window IS the whole buffer there is no room to give workers distinct bases, so they + // share one stream. (Dividing by the zero-sized remainder is what made this probe throw.) + var slack = StreamBytes - window; + var start = slack <= 0 ? 0L : (long)worker * window % slack; + + partials[worker] = Q4KStreamChains(stream, start, window); + }); + + var sum = 0; + + for (var i = 0; i < partials.Length; i++) + { + sum += partials[i]; + } + + IntSink = sum; + } + + /// + /// The body with the four weight vectors reloaded from + /// every outer iteration. Statements are inline — a helper taking vector + /// parameters is what invalidated the first register-pressure probe. + /// + private static int Q4KStreamChains(byte[] stream, long start, long window) + { + ref var origin = ref MemoryMarshal.GetArrayDataReference(stream); + + var act = Vector256.Create((sbyte)2); + var act0 = Sh32_256(act, 0); + var act1 = Sh32_256(act, 85); + + var a0 = Vector256.Zero; + var a1 = Vector256.Zero; + var a2 = Vector256.Zero; + var a3 = Vector256.Zero; + + long offset = 0; + + for (var iteration = 0; iteration < ShapeIterations; iteration++) + { + ref var p = ref Unsafe.Add(ref origin, (nint)(start + offset)); + + var w0 = Unsafe.ReadUnaligned>(ref p); + var w1 = Unsafe.ReadUnaligned>(ref Unsafe.Add(ref p, 32)); + var w2 = Unsafe.ReadUnaligned>(ref Unsafe.Add(ref p, 64)); + var w3 = Unsafe.ReadUnaligned>(ref Unsafe.Add(ref p, 96)); + + offset += 128; + + if (offset >= window - 128) + { + offset = 0; + } + + for (var repeat = 0; repeat < ShapeMuls / 2; repeat++) + { + a0 = Avx2.Add(a0, Mul256(Blend256(w0, Sh256(w1, 177)), act0)); + a0 = Avx2.Add(a0, Mul256(Blend256(Sh256(w0, 177), w1), act1)); + a1 = Avx2.Add(a1, Mul256(Blend256(w1, Sh256(w2, 177)), act0)); + a1 = Avx2.Add(a1, Mul256(Blend256(Sh256(w1, 177), w2), act1)); + a2 = Avx2.Add(a2, Mul256(Blend256(w2, Sh256(w3, 177)), act0)); + a2 = Avx2.Add(a2, Mul256(Blend256(Sh256(w2, 177), w3), act1)); + a3 = Avx2.Add(a3, Mul256(Blend256(w3, Sh256(w0, 177)), act0)); + a3 = Avx2.Add(a3, Mul256(Blend256(Sh256(w3, 177), w0), act1)); + } + } + + return Vector256.Sum(Avx2.Add(Avx2.Add(a0, a1), Avx2.Add(a2, a3))); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector256 Sh256(Vector256 v, [ConstantExpected] byte imm) + { + return Avx2.Shuffle(v.AsInt32(), imm).AsByte(); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector256 Sh32_256(Vector256 v, [ConstantExpected] byte imm) + { + return Avx2.Shuffle(v.AsInt32(), imm).AsSByte(); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector256 Blend256(Vector256 a, Vector256 b) + { + return Avx2.Blend(a.AsInt32(), b.AsInt32(), 170).AsByte(); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector256 Mul256(Vector256 rhs, Vector256 lhs) + { + return Avx2.MultiplyAddAdjacent(rhs, lhs); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 Sh512(Vector512 v, [ConstantExpected] byte imm) + { + return Avx512F.Shuffle(v.AsInt32(), imm).AsByte(); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 Sh32_512(Vector512 v, [ConstantExpected] byte imm) + { + return Avx512F.Shuffle(v.AsInt32(), imm).AsSByte(); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 Blend512(Vector512 a, Vector512 b, Vector512 mask) + { + return Avx512F.BlendVariable(a.AsInt32(), b.AsInt32(), mask).AsByte(); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 Mul512(Vector512 rhs, Vector512 lhs) + { + return Avx512BW.MultiplyAddAdjacent(rhs, lhs); + } + /// Splits the buffer into one contiguous, vector-aligned slice per worker. private (int Start, int End) SliceFor(int worker) { diff --git a/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs b/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs index f7499a0d..6c63d165 100644 --- a/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs +++ b/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs @@ -156,6 +156,10 @@ public void Cleanup() { BatchedQuantProjection.UseTiledPrefillQ4K = _originalTiled; BatchedQuantProjection.UseWeightStationaryQ4K = _originalStationary; + Q4KGemvKernel.AblateF16Scales = false; + Q4KGemvKernel.AblateScaleUnpack = false; + Q4KGemvKernel.AblateNibbleUnpack = false; + BatchedQuantProjection.TileColsOverride = 0; _weight.Dispose(); } @@ -204,6 +208,86 @@ public void QuantizeActivationsOnly() } } + /// + /// with the column tile forced to 8 — today's dispatcher choice, and the arm the + /// wider tiles below are judged against. + /// + /// A tile of NR columns walks the whole weight matrix, so the matrix is streamed rows/NR + /// times. At NR=8 and 672 rows that is 84 passes over 12.68 MB = 1.07 GB per projection, roughly 70 GB/s + /// against a measured 90 GB/s ceiling. If that traffic is what caps the kernel at 1.98 TFLOP/s against + /// its instruction mix's 4.6, halving it should show here. + /// + [Benchmark] + public void Tiled_Cols8() + { + RunTiledWithCols(8); + } + + /// Half the weight traffic of — 42 passes instead of 84. + [Benchmark] + public void Tiled_Cols16() + { + RunTiledWithCols(16); + } + + /// Twice the traffic of , to confirm the trend runs both ways. + [Benchmark] + public void Tiled_Cols4() + { + RunTiledWithCols(4); + } + + private void RunTiledWithCols(int cols) + { + BatchedQuantProjection.UseTiledPrefillQ4K = true; + BatchedQuantProjection.UseWeightStationaryQ4K = false; + BatchedQuantProjection.TileColsOverride = cols; + BatchedQuantProjection.Dispatch(_input, Rows, in _weight, [], _output, _inputSize, _outputSize); + BatchedQuantProjection.TileColsOverride = 0; + } + + /// + /// with the per-block F16 scale/min decode replaced by constants — i.e. without + /// LoadF16x8Rearrange, which stores a vector to stackalloc and reads it back through + /// eight separate BitConverter.UInt16BitsToHalf calls, plus the second LoadF16x8. + /// + /// This and the two ablations below split the unexplained gap between the kernel's measured + /// 1.70 TFLOP/s and the 4.64 its own arithmetic instruction mix reaches — the residual is work the mix + /// benchmark never modelled, and each of these is a candidate. Ratios against are + /// upper bounds: removing a computation also lets the JIT fold what depended on it. + /// + [Benchmark] + public void Tiled_NoF16Decode() + { + BatchedQuantProjection.UseTiledPrefillQ4K = true; + BatchedQuantProjection.UseWeightStationaryQ4K = false; + Q4KGemvKernel.AblateF16Scales = true; + BatchedQuantProjection.Dispatch(_input, Rows, in _weight, [], _output, _inputSize, _outputSize); + Q4KGemvKernel.AblateF16Scales = false; + } + + /// without the scalar Unpack of the 6-bit sub-block scales. + [Benchmark] + public void Tiled_NoScaleUnpack() + { + BatchedQuantProjection.UseTiledPrefillQ4K = true; + BatchedQuantProjection.UseWeightStationaryQ4K = false; + Q4KGemvKernel.AblateScaleUnpack = true; + BatchedQuantProjection.Dispatch(_input, Rows, in _weight, [], _output, _inputSize, _outputSize); + Q4KGemvKernel.AblateScaleUnpack = false; + } + + /// without the 16 And/shift ops that split bytes into nibbles. + [Benchmark] + public void Tiled_NoNibbleUnpack() + { + BatchedQuantProjection.UseTiledPrefillQ4K = true; + BatchedQuantProjection.UseWeightStationaryQ4K = false; + Q4KGemvKernel.AblateNibbleUnpack = true; + BatchedQuantProjection.Dispatch(_input, Rows, in _weight, [], _output, _inputSize, _outputSize); + Q4KGemvKernel.AblateNibbleUnpack = false; + } + /// The original re-decode-per-row kernel — kept as the reference the kernel docs' "~3×" claim /// is actually measured against. [Benchmark] diff --git a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs index 36345b05..94e1a800 100644 --- a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs +++ b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs @@ -55,6 +55,60 @@ internal static class BatchedQuantProjection /// internal static bool DisableRepackedKernelsForParity; + /// Forces a specific prefill column-tile width; 0 leaves to choose. + internal static int TileColsOverride; + + /// + /// Picks the column-tile width (NR) for one prefill projection: the widest tile that still leaves + /// at least one tile per core. + /// + /// Why width matters more than it looks. A tile of NR columns walks the entire weight + /// matrix, so the matrix is streamed rows/NR times per projection. At NR=8 and 672 rows that is 84 + /// passes over `ffn_gate_up`'s 12.68 MB — 1.07 GB of traffic in 15.3 ms, about 70 GB/s against a measured + /// 90 GB/s read ceiling. Doubling NR halves that traffic outright. This is the only blocking level the + /// kernel has: the tile lives in registers, and there is nothing sized to L2 or L3 between it and memory. + /// + /// + /// And why it cannot simply be maximised. Fewer, fatter tiles are fewer independent work + /// items, and the parallel region costs its longest worker. Measured on 672 rows across 32 cores + /// (`ffn_gate_up`, two runs, agreeing): + /// + /// + /// NR=4168 passes, 17.8 ms, 1.70 TFLOP/s + /// NR=884 passes, 15.2–15.9 ms, 1.95 TFLOP/s + /// NR=1642 passes, 16.5–17.0 ms, 1.80 TFLOP/s + /// + /// + /// Halving the traffic 4→8 buys +17%, exactly as the bandwidth argument predicts. Halving it again + /// 8→16 loses 8%, because 42 tiles over 32 cores leaves ten workers with two tiles and twenty-two + /// with one — a 1.52× imbalance against 1.14× at NR=8. So the rule is not "widest that fits a core" but + /// "widest that still gives every core a couple of tiles"; below that the granularity loss outruns the + /// traffic saving. A longer prompt moves the balance back toward the wider tile. + /// + /// Escaping the trade-off entirely needs the missing blocking level: block over output rows as + /// well, so a wide column tile and a large number of independent work items stop being alternatives. + /// + private static int ResolveTileCols(int rows, int cores, int maxTileCols) + { + if (TileColsOverride > 0) + { + return Math.Min(TileColsOverride, maxTileCols); + } + + // Require ~2 tiles per core, not 1: at exactly one the tail worker doubles the region's duration. + const int TilesPerCore = 2; + + for (var nr = 16; nr > 4; nr >>= 1) + { + if (nr <= maxTileCols && rows / nr >= cores * TilesPerCore) + { + return nr; + } + } + + return 4; + } + /// /// / / let the /// caller supply activations ALREADY quantized to Q8_K, skipping the internal quantization pass. @@ -245,11 +299,7 @@ private static unsafe void DispatchTiledQ4K( var repacked = w.EnsureRepacked(); var cores = Environment.ProcessorCount; - var nr = rows / 8 >= cores ? 8 : 4; - if (nr > Q4KGemvKernel.MaxTileCols) - { - nr = Q4KGemvKernel.MaxTileCols; - } + var nr = ResolveTileCols(rows, cores, Q4KGemvKernel.MaxTileCols); var tiles = (rows + nr - 1) / nr; fixed (byte* rp = repacked) @@ -313,11 +363,7 @@ private static unsafe void DispatchTiledQ6K( var repacked = w.EnsureRepacked(); var cores = Environment.ProcessorCount; - var nr = rows / 8 >= cores ? 8 : 4; - if (nr > Q6KGemvKernel.MaxTileCols) - { - nr = Q6KGemvKernel.MaxTileCols; - } + var nr = ResolveTileCols(rows, cores, Q6KGemvKernel.MaxTileCols); var tiles = (rows + nr - 1) / nr; fixed (byte* rp = repacked) diff --git a/Sources/Main/LanguageModels/Runtime/Q4KGemvKernel.cs b/Sources/Main/LanguageModels/Runtime/Q4KGemvKernel.cs index d6c3aee3..70432adf 100644 --- a/Sources/Main/LanguageModels/Runtime/Q4KGemvKernel.cs +++ b/Sources/Main/LanguageModels/Runtime/Q4KGemvKernel.cs @@ -50,6 +50,27 @@ public static unsafe class Q4KGemvKernel /// public static readonly bool TiledPrefillEnabled = ResolveFlag(OverfitEnvironment.TiledPrefill); + /// + /// Measurement-only ablations for , all default-off. Each replaces one piece of + /// per-block work with a constant so its share of the kernel's runtime can be read off directly. + /// + /// Why ablation rather than micro-benchmarks. The kernel reaches 1.70 TFLOP/s against 4.64 + /// for its own arithmetic instruction mix, and the 2.7× residual is work the mix benchmark does not + /// model. Timing that work in isolation would measure a synthetic harness; toggling it inside the real + /// kernel measures its actual share. Results are wrong while an ablation is on — these are timing + /// probes, never a production path. + /// + /// Caveat when reading the numbers: removing a computation also lets the JIT fold or hoist what + /// depended on it, so an ablation is an upper bound on the removed work's cost. + /// + internal static bool AblateF16Scales; + + /// + internal static bool AblateScaleUnpack; + + /// + internal static bool AblateNibbleUnpack; + private static bool ResolveFlag(string envVar) { if (!CpuFeatures.HasAvx2) @@ -354,8 +375,12 @@ public static void GemmTiled( for (var b = 0; b < nb; b++) { var blk = bptr + (long)b * BlockKx8Bytes; - var colScale = LoadF16x8Rearrange(blk, deltamask); - var colDmin = LoadF16x8(blk + 16); + var colScale = AblateF16Scales + ? Vector256.Create(1f) + : LoadF16x8Rearrange(blk, deltamask); + var colDmin = AblateF16Scales + ? Vector256.Create(0f) + : LoadF16x8(blk + 16); var qsBase = blk + DstQsOffset; var scBase = blk + DstScalesOffset; @@ -382,23 +407,25 @@ public static void GemmTiled( var raw4567_3 = Vector256.Load(qs + 224); // Weight nibbles — decoded ONCE, reused across all cols (the tiling win). - var v0123_00 = Avx2.And(raw0123_0, m4b); - var v4567_00 = Avx2.And(raw4567_0, m4b); - var v0123_01 = Avx2.And(raw0123_1, m4b); - var v4567_01 = Avx2.And(raw4567_1, m4b); - var v0123_02 = Avx2.And(raw0123_2, m4b); - var v4567_02 = Avx2.And(raw4567_2, m4b); - var v0123_03 = Avx2.And(raw0123_3, m4b); - var v4567_03 = Avx2.And(raw4567_3, m4b); - - var v0123_10 = Avx2.And(Hi(raw0123_0), m4b); - var v4567_10 = Avx2.And(Hi(raw4567_0), m4b); - var v0123_11 = Avx2.And(Hi(raw0123_1), m4b); - var v4567_11 = Avx2.And(Hi(raw4567_1), m4b); - var v0123_12 = Avx2.And(Hi(raw0123_2), m4b); - var v4567_12 = Avx2.And(Hi(raw4567_2), m4b); - var v0123_13 = Avx2.And(Hi(raw0123_3), m4b); - var v4567_13 = Avx2.And(Hi(raw4567_3), m4b); + var ablateNibbles = AblateNibbleUnpack; + + var v0123_00 = ablateNibbles ? raw0123_0 : Avx2.And(raw0123_0, m4b); + var v4567_00 = ablateNibbles ? raw4567_0 : Avx2.And(raw4567_0, m4b); + var v0123_01 = ablateNibbles ? raw0123_1 : Avx2.And(raw0123_1, m4b); + var v4567_01 = ablateNibbles ? raw4567_1 : Avx2.And(raw4567_1, m4b); + var v0123_02 = ablateNibbles ? raw0123_2 : Avx2.And(raw0123_2, m4b); + var v4567_02 = ablateNibbles ? raw4567_2 : Avx2.And(raw4567_2, m4b); + var v0123_03 = ablateNibbles ? raw0123_3 : Avx2.And(raw0123_3, m4b); + var v4567_03 = ablateNibbles ? raw4567_3 : Avx2.And(raw4567_3, m4b); + + var v0123_10 = ablateNibbles ? raw0123_0 : Avx2.And(Hi(raw0123_0), m4b); + var v4567_10 = ablateNibbles ? raw4567_0 : Avx2.And(Hi(raw4567_0), m4b); + var v0123_11 = ablateNibbles ? raw0123_1 : Avx2.And(Hi(raw0123_1), m4b); + var v4567_11 = ablateNibbles ? raw4567_1 : Avx2.And(Hi(raw4567_1), m4b); + var v0123_12 = ablateNibbles ? raw0123_2 : Avx2.And(Hi(raw0123_2), m4b); + var v4567_12 = ablateNibbles ? raw4567_2 : Avx2.And(Hi(raw4567_2), m4b); + var v0123_13 = ablateNibbles ? raw0123_3 : Avx2.And(Hi(raw0123_3), m4b); + var v4567_13 = ablateNibbles ? raw4567_3 : Avx2.And(Hi(raw4567_3), m4b); u0[0] = Unsafe.ReadUnaligned(scBase + 24 * sb); u0[1] = Unsafe.ReadUnaligned(scBase + 24 * sb + 4); @@ -406,8 +433,12 @@ public static void GemmTiled( u1[0] = Unsafe.ReadUnaligned(scBase + 12 + sb * 24); u1[1] = Unsafe.ReadUnaligned(scBase + 12 + sb * 24 + 4); u1[2] = Unsafe.ReadUnaligned(scBase + 12 + sb * 24 + 8); - Unpack(u0, kmask1, kmask2, kmask3); - Unpack(u1, kmask1, kmask2, kmask3); + + if (!AblateScaleUnpack) + { + Unpack(u0, kmask1, kmask2, kmask3); + Unpack(u1, kmask1, kmask2, kmask3); + } var ms0 = Vector128.Create(u0[0], u0[1], u0[2], u0[3]).AsByte(); var ms1 = Vector128.Create(u1[0], u1[1], u1[2], u1[3]).AsByte(); @@ -529,6 +560,15 @@ private static void Unpack(uint* u, uint k1, uint k2, uint k3) u[0] &= k1; } + /// + /// Widens eight IEEE half-precision values to . + /// + /// The hardware path is one vcvtph2ps. The scalar fallback below costs eight + /// calls plus a build, and ablating + /// this decode out of GemmTiled measured 12% of the kernel's runtime — it runs once per + /// weight super-block and so is not amortised across the column tile. Half→float is exact in both + /// paths (no rounding is possible when widening), so the two are bit-identical. + /// private static Vector256 LoadF16x8(byte* p) { var u = (ushort*)p; @@ -539,12 +579,34 @@ private static Vector256 LoadF16x8(byte* p) (float)BitConverter.UInt16BitsToHalf(u[6]), (float)BitConverter.UInt16BitsToHalf(u[7])); } + /// + /// The same widening as , after the repacked layout's byte rearrangement. + /// + /// The lanes are extracted straight out of the register with pextrw. The previous version + /// stored the shuffled vector into a stackalloc buffer and immediately re-read it as eight + /// s — a 16-byte store followed by eight narrow loads of the same address, which + /// is the pathological case for store-to-load forwarding: the loads cannot be satisfied from the store + /// buffer and stall until the store retires to L1. + /// + /// x86 has vcvtph2ps, which would widen all eight in one instruction, but .NET exposes + /// neither an F16C intrinsic class nor a overload of + /// , so the conversions stay scalar. Ablating this decode + /// out of the kernel measured 12% of its runtime; removing only the round-trip recovers whatever share + /// of that was the stall rather than the arithmetic. + /// private static Vector256 LoadF16x8Rearrange(byte* p, Vector128 deltamask) { - var bytes = Ssse3.Shuffle(Vector128.Load(p), deltamask); - var tmp = stackalloc byte[16]; - bytes.Store(tmp); - return LoadF16x8(tmp); + var v = Ssse3.Shuffle(Vector128.Load(p), deltamask).AsUInt16(); + + return Vector256.Create( + (float)BitConverter.UInt16BitsToHalf(v.GetElement(0)), + (float)BitConverter.UInt16BitsToHalf(v.GetElement(1)), + (float)BitConverter.UInt16BitsToHalf(v.GetElement(2)), + (float)BitConverter.UInt16BitsToHalf(v.GetElement(3)), + (float)BitConverter.UInt16BitsToHalf(v.GetElement(4)), + (float)BitConverter.UInt16BitsToHalf(v.GetElement(5)), + (float)BitConverter.UInt16BitsToHalf(v.GetElement(6)), + (float)BitConverter.UInt16BitsToHalf(v.GetElement(7))); } } } From f857f12242c0e23f4e58c44997f733ee26c65e42 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Wed, 22 Jul 2026 22:48:57 +0200 Subject: [PATCH 17/37] llama --- ROADMAP.md | 57 +++++++- Sources/Benchmark/Helpers/ThroughputColumn.cs | 12 +- Sources/Benchmark/Helpers/WorkAmount.cs | 4 +- .../Q4KPrefillProjectionBenchmark.cs | 44 +++++- Sources/Main/Diagnostics/Throughput.cs | 82 +++++++++++ .../Runtime/BatchedQuantProjection.cs | 138 +++++++++++++++++- .../LanguageModels/Runtime/Q4KGemvKernel.cs | 78 +++++++++- 7 files changed, 395 insertions(+), 20 deletions(-) create mode 100644 Sources/Main/Diagnostics/Throughput.cs diff --git a/ROADMAP.md b/ROADMAP.md index df2d75a2..6d3d4cc5 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -648,10 +648,59 @@ reproduces NR=8 at 672 (no change to the profiled prompt, prefill stays 249 tok/ ~1024 rows — a real gain for long prompts. The ≥1024 branch was measured rather than reasoned, because six mechanism hypotheses were refuted the same day. -*The trade-off itself is the finding.* Traffic falls as 1/NR while parallel granularity falls as NR, so tile -width alone cannot buy much. Escaping it needs the blocking level the kernel does not have: **block over -output rows as well**, so a wide column tile and many independent work items stop being alternatives. That is -the Goto/BLIS structure and it is the outstanding work. +#### ▶ NEGATIVE — output-row banding (the "missing L2 blocking level") is 20% SLOWER + +Implemented as `BatchedQuantProjection.UseOutputBlocking` (default **off**, kept as the record): parallelise +over bands of output groups instead of column tiles, so each worker owns an L2-sized slice of the weight +matrix and re-reads it from its own cache for every column tile. `Q4KGemvKernel.GemmTiled` gained +`groupStart`/`groupCount` for this. Two runs, agreeing: + +| arm | run 1 | run 2 | TFLOP/s | +|---|---:|---:|---:| +| **Cols8 (today's production)** | **14.91 ms** | **15.06** | **2.02** | +| Banded16 | 16.72 | 16.04 | 1.85 | +| Cols16 | 17.10 | 16.89 | 1.78 | +| **Banded8** | **18.21** | **17.81** | **1.68** | + +**And this refutes the traffic story that motivated it.** Banding removes 84× of the weight re-reads; if that +traffic were the constraint it had to show. It did not — this chip's 128 MB L3 (V-cache) holds the whole +12.7 MB matrix, so those re-reads were never going to DRAM in the first place. + +**Unified explanation that fits every measurement taken today.** The per-block fixed work — F16 scale decode +(ablated at **12%**), scalar `Unpack` (**3.5%**), nibble unpack (~0%) — is amortised across the columns in a +tile. At NR=8 that is ~15% of runtime; at NR=4, ~30%; at NR=16, ~7.5%. Predicted 4→8 gain +`1.30/1.15 = 1.13×` against **1.17× measured**; predicted 8→16 gain `1.15/1.075 = 1.07×`, overwhelmed by the +1.52×/1.14× imbalance shift, against **−8% measured**. No bandwidth term is needed anywhere. + +**So the lever is to delete the fixed work, not to move the data.** + +#### ★ WIN — hoisting the F16 scale decode: prefill 249 → 256 tok/s + +`GemmTiled` decodes each block's F16 scale/min pair inline, which reads as amortised — but the kernel runs +**once per column tile**, 84 times at 672 rows and NR=8, so every pair is widened 84 times over. +`Q4KGemvKernel.DecodeBlockScales` now widens them once per projection into a pooled scratch +(`BatchedQuantProjection.UsePrecomputedScales`), which the tiles share. Bit-identical — same conversions, +fewer of them — and it needs **no format change and no extra weight RAM**, unlike storing f32 in +`block_q4_Kx8` (+2.8% permanently, and every `.gguf.repack` sidecar invalidated). + +| arm (672 rows, `ffn_gate_up`, two runs) | run 1 | run 2 | TFLOP/s | +|---|---:|---:|---:| +| **hoisted, NR=8** | **13.88 ms** | **13.60** | **2.21** | +| ablation floor (decode removed entirely) | 14.45 | 14.31 | 2.11 | +| NR=8 baseline | 15.91 | 15.21 | 1.95 | +| hoisted, NR=16 | 16.66 | 17.05 | 1.80 | +| NR=16 baseline | 18.01 | 17.87 | 1.69 | + +**+13% at NR=8** — faster than the ablation floor, because ablation still built a constant vector and took the +branch, so the full 12% was recovered and a little more. + +**The amortisation theory predicted this before it was measured, twice over.** Fixed per-block work is ~15% of +runtime at NR=8 and ~7.5% at NR=16, so the gain should roughly halve with the wider tile: predicted 2.0×, +measured 13%/6% = 2.2×. End to end it predicted `0.45 × 0.13 = 5.9%` off prefill → 2606 ms; measured +**2622–2632 ms, 255–256 tok/s**, within 0.6%. Gap to llama.cpp's AVX-512 build: 2.18× → **2.12×**. + +Suite 1486/0/229. **Next: the same hoist for `Q6KGemvKernel` — `ffn_down` is 27% of prefill (708 ms) and has +the identical per-block decode**, so the same ~13% there is worth roughly another 9 tok/s. *Invalidated run, kept as a warning:* the first tile sweep ran inside an 11-benchmark class and reported `Tiled` and `Tiled_Cols8` — **the same configuration** — 21% apart, far outside their ±9% bars. Two identical diff --git a/Sources/Benchmark/Helpers/ThroughputColumn.cs b/Sources/Benchmark/Helpers/ThroughputColumn.cs index 9fe14608..d9694fb6 100644 --- a/Sources/Benchmark/Helpers/ThroughputColumn.cs +++ b/Sources/Benchmark/Helpers/ThroughputColumn.cs @@ -8,6 +8,7 @@ using BenchmarkDotNet.Columns; using BenchmarkDotNet.Reports; using BenchmarkDotNet.Running; +using DevOnBike.Overfit.Diagnostics; namespace Benchmarks.Helpers { @@ -78,19 +79,22 @@ public string GetValue(Summary summary, BenchmarkCase benchmarkCase) return "-"; } - // BenchmarkDotNet reports the mean in nanoseconds. - var seconds = statistics.Mean / 1e9; + // BenchmarkDotNet reports the mean in nanoseconds. The rates themselves come from the shared + // Throughput helper so the benchmark table and the runtime profiler cannot drift apart. + var elapsed = TimeSpan.FromTicks((long)(statistics.Mean / 100.0)); if (_compute) { return work.Flops <= 0L ? "-" - : (work.Flops / seconds / 1e12).ToString("F2", CultureInfo.InvariantCulture); + : Throughput.TeraflopsPerSecond(work.Flops, elapsed) + .ToString("F2", CultureInfo.InvariantCulture); } return work.Bytes <= 0L ? "-" - : (work.Bytes / seconds / 1e9).ToString("F1", CultureInfo.InvariantCulture); + : Throughput.GigabytesPerSecond(work.Bytes, elapsed) + .ToString("F1", CultureInfo.InvariantCulture); } public string GetValue(Summary summary, BenchmarkCase benchmarkCase, SummaryStyle style) diff --git a/Sources/Benchmark/Helpers/WorkAmount.cs b/Sources/Benchmark/Helpers/WorkAmount.cs index 3f2d6775..7cd58614 100644 --- a/Sources/Benchmark/Helpers/WorkAmount.cs +++ b/Sources/Benchmark/Helpers/WorkAmount.cs @@ -3,6 +3,8 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com +using DevOnBike.Overfit.Diagnostics; + namespace Benchmarks.Helpers { /// @@ -35,7 +37,7 @@ public readonly record struct WorkAmount(long Flops, long Bytes) /// A matmul of × by ×. public static WorkAmount Matmul(long rows, long k, long n) { - return new WorkAmount(2L * rows * k * n, 0L); + return new WorkAmount(Throughput.MatmulFlops(rows, k, n), 0L); } /// Pure memory traffic, no arithmetic worth counting. diff --git a/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs b/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs index 6c63d165..c9e70e70 100644 --- a/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs +++ b/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs @@ -237,12 +237,54 @@ public void Tiled_Cols4() RunTiledWithCols(4); } - private void RunTiledWithCols(int cols) + /// + /// Output-row banding at today's tile width — isolates the blocking change from the tile width, so the + /// L2-residency effect is measured on its own rather than tangled with a different NR. + /// + [Benchmark] + public void Tiled_Banded8() + { + RunTiledWithCols(8, banded: true); + } + + /// + /// Banding plus the wide tile. Banding decouples work-item count from NR, so the tile width that lost + /// 8% to scheduling imbalance at 672 rows should now keep its halved weight traffic. + /// + [Benchmark] + public void Tiled_Banded16() + { + RunTiledWithCols(16, banded: true); + } + + /// + /// The tile width in production, with the F16 scale decode hoisted to once per projection instead of + /// once per column tile. Ablation put that decode at 12%; this is the arm that says how much of it a + /// legitimate implementation actually recovers. + /// + [Benchmark] + public void Tiled_HoistedScales8() + { + RunTiledWithCols(8, hoistScales: true); + } + + /// Hoisted scales at the wide tile, where the fixed per-block work is already thinner. + [Benchmark] + public void Tiled_HoistedScales16() + { + RunTiledWithCols(16, hoistScales: true); + } + + private void RunTiledWithCols(int cols, bool banded = false, bool hoistScales = false) { BatchedQuantProjection.UseTiledPrefillQ4K = true; BatchedQuantProjection.UseWeightStationaryQ4K = false; BatchedQuantProjection.TileColsOverride = cols; + BatchedQuantProjection.UseOutputBlocking = banded; + BatchedQuantProjection.UsePrecomputedScales = hoistScales; BatchedQuantProjection.Dispatch(_input, Rows, in _weight, [], _output, _inputSize, _outputSize); + BatchedQuantProjection.UsePrecomputedScales = true; + BatchedQuantProjection.UseOutputBlocking = false; BatchedQuantProjection.TileColsOverride = 0; } diff --git a/Sources/Main/Diagnostics/Throughput.cs b/Sources/Main/Diagnostics/Throughput.cs new file mode 100644 index 00000000..3cb40aec --- /dev/null +++ b/Sources/Main/Diagnostics/Throughput.cs @@ -0,0 +1,82 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +namespace DevOnBike.Overfit.Diagnostics +{ + /// + /// Turns "this much work in this much time" into a rate — TFLOP/s, GFLOP/s, GB/s — so throughput is + /// computed the same way everywhere instead of being re-derived per caller. + /// + /// Why this is a type and not a formula at each call site. Rates are trivial arithmetic and + /// exactly for that reason they get written ad hoc, in scripts, next to the numbers they describe. On + /// 2026-07-22 that produced a reported 29.6 TFLOP/s for a routine that performs no multiply-add at + /// all: the matmul FLOP formula had been applied to an activation-quantization pass, so memory bandwidth + /// was published as arithmetic throughput. Naming the work — versus + /// — makes that category error visible at the call site. + /// + /// Counting convention. A multiply-accumulate is two operations, matching llama.cpp's + /// test-backend-ops: its printed "60.13 GFLOP" for m=4096, k=14336, n=512 is exactly + /// 2·512·14336·4096, so figures produced here are directly comparable with that project's. For + /// quantized kernels count the logical MACs of the matmul being performed, not the machine + /// instructions retired — a Q4_K matmul and an F32 matmul of the same shape are credited identically, + /// which is the only way a quantized kernel can be placed against a dense roofline. + /// + /// Reading a rate needs a ceiling. A bare "1.70 TFLOP/s" says nothing; the same kernel was 78% + /// of one ceiling and 15% of another, and only the second was the ceiling that applied. Compare against the + /// instruction mix the code actually issues, not against a peak it could never reach. + /// + public static class Throughput + { + /// Operations charged to one multiply-accumulate. + public const int OperationsPerMultiplyAccumulate = 2; + + /// + /// Logical operations in a matmul of × by + /// ×. + /// + public static long MatmulFlops(long rows, long inner, long columns) + { + return OperationsPerMultiplyAccumulate * rows * inner * columns; + } + + /// Teraflops sustained by operations over . + public static double TeraflopsPerSecond(long flops, TimeSpan elapsed) + { + return RatePerSecond(flops, elapsed) / 1e12; + } + + /// Gigaflops sustained by operations over . + public static double GigaflopsPerSecond(long flops, TimeSpan elapsed) + { + return RatePerSecond(flops, elapsed) / 1e9; + } + + /// + /// Gigabytes per second moved by over . Count reads + /// plus writes, following the STREAM convention: a copy of N bytes is 2N, and the read-for-ownership + /// traffic a write implies is not counted. + /// + public static double GigabytesPerSecond(long bytes, TimeSpan elapsed) + { + return RatePerSecond(bytes, elapsed) / 1e9; + } + + /// + /// What fraction of a measured rate reaches, as a value in [0, ∞). + /// Both arguments must be in the same unit; the point of the helper is to make the pairing explicit. + /// + public static double FractionOfCeiling(double achieved, double ceiling) + { + return ceiling <= 0.0 ? 0.0 : achieved / ceiling; + } + + private static double RatePerSecond(long amount, TimeSpan elapsed) + { + var seconds = elapsed.TotalSeconds; + + return amount <= 0L || seconds <= 0.0 ? 0.0 : amount / seconds; + } + } +} diff --git a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs index 94e1a800..f76e5dd2 100644 --- a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs +++ b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs @@ -58,6 +58,58 @@ internal static class BatchedQuantProjection /// Forces a specific prefill column-tile width; 0 leaves to choose. internal static int TileColsOverride; + /// + /// Parallelise the tiled prefill GEMM over bands of output rows instead of over column tiles. + /// + /// The problem it addresses. Today one work item is one column tile, and a column tile walks + /// the entire weight matrix — so every worker streams all 12.68 MB of `ffn_gate_up`, and the matrix + /// is re-read rows/NR times per projection. Widening NR halves that traffic but also halves the + /// number of work items, and the measured sweep showed the two cancelling: NR=16 lost 8% at 672 rows to + /// scheduling imbalance despite halving the traffic. + /// + /// The change. Give each worker a contiguous band of output groups and let it loop over all + /// column tiles inside that band. Its weight working set is then one band — sized below to fit L2 — + /// which it reads once from L3 and re-reads from its own cache for every remaining column tile. The + /// memory probe measured exactly this distinction: the same instruction mix ran at 4.60 TFLOP/s against + /// an L2-resident window and 3.04 against an L3-resident one, a 1.50× difference. + /// + /// It also decouples the two knobs: work-item count no longer depends on NR, so a wide column tile + /// stops costing parallelism. This is the L2 blocking level of the standard GEMM structure + /// (Goto & van de Geijn), which this kernel has never had. + /// + internal static bool UseOutputBlocking; + + /// + /// Decode every weight block's F16 scale/min pair to once per projection instead of + /// once per column tile. + /// + /// GemmTiled decodes them inline, which reads as amortised — but the kernel runs once per + /// column tile, 84 times for a 672-token prompt at NR=8, so each F16 pair is widened 84 times over. + /// Ablation put that decode at 12% of the kernel, the largest non-arithmetic item measured, and + /// it is fixed work per block, so it is exactly the term the tile-width sweep showed being amortised + /// across columns. Hoisting it divides the work by the tile count. + /// + internal static bool UsePrecomputedScales = true; + + /// + /// Weight bytes one worker's band may occupy. Half of a 1 MB Zen-5 L2, leaving the rest for the + /// activation tile and the output band; the point is residency, not filling the cache exactly. + /// + private const int BandWeightBudgetBytes = 512 * 1024; + + /// + /// Output groups per band: small enough that the band's weights sit in L2, and numerous enough that + /// every core still gets several bands so the tail worker does not set the region's duration. + /// + private static int ResolveGroupsPerBand(int totalGroups, int superBlocksPerRow, int blockBytes, int cores) + { + var bytesPerGroup = Math.Max(1, superBlocksPerRow * blockBytes); + var byCache = Math.Max(1, BandWeightBudgetBytes / bytesPerGroup); + var byParallelism = Math.Max(1, totalGroups / (cores * 2)); + + return Math.Min(byCache, byParallelism); + } + /// /// Picks the column-tile width (NR) for one prefill projection: the widest tile that still leaves /// at least one tile per core. @@ -302,7 +354,22 @@ private static unsafe void DispatchTiledQ4K( var nr = ResolveTileCols(rows, cores, Q4KGemvKernel.MaxTileCols); var tiles = (rows + nr - 1) / nr; + // One decode of the F16 scales for the whole projection, reused by every column tile. Skipped when + // there is only one tile, where hoisting would just move the same work. + var scaleCount = UsePrecomputedScales && tiles > 1 + ? (outputSize / 8) * spr * Q4KGemvKernel.DecodedScalesPerBlock + : 0; + + using var decodedScales = new PooledBuffer(scaleCount, clearMemory: false); + + if (scaleCount > 0) + { + Q4KGemvKernel.DecodeBlockScales( + repacked, outputSize, inputSize, decodedScales.Span.Slice(0, scaleCount)); + } + fixed (byte* rp = repacked) + fixed (float* dsc = decodedScales.Span.Slice(0, scaleCount)) fixed (sbyte* q = quants) fixed (float* sc = scales) fixed (short* bs = bsums) @@ -325,8 +392,24 @@ private static unsafe void DispatchTiledQ4K( BsumsPerRow = bsumsPerRow, Nr = nr, Rows = rows, + Tiles = tiles, + DecodedScales = dsc, + DecodedScalesLength = scaleCount, }; - OverfitParallel.For(0, tiles, &TiledChunk, &ctx); + + if (!UseOutputBlocking) + { + OverfitParallel.For(0, tiles, &TiledChunk, &ctx); + return; + } + + var totalGroups = outputSize / 8; + ctx.GroupsPerBand = ResolveGroupsPerBand( + totalGroups, spr, Q4KRepack.BlockKx8Bytes, cores); + + var bands = (totalGroups + ctx.GroupsPerBand - 1) / ctx.GroupsPerBand; + + OverfitParallel.For(0, bands, &TiledBandChunk, &ctx); } } @@ -436,6 +519,18 @@ private unsafe struct TiledContext public int BsumsPerRow; public int Nr; public int Rows; + + /// Column tiles per projection — the inner loop when banding over output rows. + public int Tiles; + + /// Output groups per band; see . + public int GroupsPerBand; + + /// F16 scales widened once for the whole projection; null when decoded inline. + public float* DecodedScales; + + /// Length of ; 0 when decoded inline. + public int DecodedScalesLength; } private static unsafe void TiledChunk(int start, int end, void* context) @@ -454,7 +549,46 @@ private static unsafe void TiledChunk(int start, int end, void* context) new ReadOnlySpan(c.Scales + (long)s * c.Spr, cols * c.Spr), new ReadOnlySpan(c.Bsums + (long)s * c.BsumsPerRow, cols * c.BsumsPerRow), new Span(c.Output + (long)s * c.OutputSize, cols * c.OutputSize), - new ReadOnlySpan(c.Bias, c.BiasLength)); + new ReadOnlySpan(c.Bias, c.BiasLength), + 0, + 0, + new ReadOnlySpan(c.DecodedScales, c.DecodedScalesLength)); + } + } + + // One band of output groups, swept by every column tile in turn. The band's weights are read from L3 + // once and then re-read from this worker's own L2 for each remaining tile — the whole point of the + // structure. Bands are disjoint in both weights (read-only) and output rows, so no worker writes where + // another reads. + private static unsafe void TiledBandChunk(int start, int end, void* context) + { + ref var c = ref Unsafe.AsRef(context); + var totalGroups = c.OutputSize / 8; + + for (var band = start; band < end; band++) + { + var groupStart = band * c.GroupsPerBand; + var groupCount = Math.Min(c.GroupsPerBand, totalGroups - groupStart); + + for (var t = 0; t < c.Tiles; t++) + { + var s = t * c.Nr; + var cols = Math.Min(c.Nr, c.Rows - s); + + Q4KGemvKernel.GemmTiled( + new ReadOnlySpan(c.Repacked, c.RepackedLength), + c.OutputSize, + c.InputSize, + cols, + new ReadOnlySpan(c.Quants + (long)s * c.InputSize, cols * c.InputSize), + new ReadOnlySpan(c.Scales + (long)s * c.Spr, cols * c.Spr), + new ReadOnlySpan(c.Bsums + (long)s * c.BsumsPerRow, cols * c.BsumsPerRow), + new Span(c.Output + (long)s * c.OutputSize, cols * c.OutputSize), + new ReadOnlySpan(c.Bias, c.BiasLength), + groupStart, + groupCount, + new ReadOnlySpan(c.DecodedScales, c.DecodedScalesLength)); + } } } } diff --git a/Sources/Main/LanguageModels/Runtime/Q4KGemvKernel.cs b/Sources/Main/LanguageModels/Runtime/Q4KGemvKernel.cs index 70432adf..796b4a2d 100644 --- a/Sources/Main/LanguageModels/Runtime/Q4KGemvKernel.cs +++ b/Sources/Main/LanguageModels/Runtime/Q4KGemvKernel.cs @@ -294,6 +294,52 @@ private static void ComputeGroup(byte* w, sbyte* aq, float* asc, short* ab, floa Avx.Subtract(accRow, accMin).Store(output + x * 8); } + /// Floats written per weight block by : 8 scales then 8 mins. + public const int DecodedScalesPerBlock = 16; + + /// + /// Decodes every weight block's F16 scale/min pair to once, into + /// laid out as [(group·nb + block) · 16] — eight rearranged + /// scales followed by eight mins. + /// + /// Why this exists. decodes these inline, once per (group, block). + /// That looks amortised, but the kernel is invoked once per column tile — 84 times for a + /// 672-token prompt at NR=8 — so the same F16 pairs are decoded 84 times over. Ablating the decode out + /// of the kernel measured 12% of its runtime, the largest single non-arithmetic item found. + /// Hoisting it here reduces that work by the tile count instead of removing capability. + /// + /// Bit-identical to the inline path: the same conversions produce the same values, only fewer + /// times. The alternative — widening block_q4_Kx8 to hold f32 scales — costs 2.8% weight RAM + /// permanently and invalidates every .gguf.repack sidecar; this costs a pooled scratch buffer + /// that lives for one projection. + /// + public static void DecodeBlockScales( + ReadOnlySpan repacked, + int outputSize, + int inputSize, + Span destination) + { + var nb = inputSize / 256; + var groups = outputSize / 8; + var deltamask = Vector128.Create((byte)0, 1, 8, 9, 2, 3, 10, 11, 4, 5, 12, 13, 6, 7, 14, 15); + + fixed (byte* w = repacked) + fixed (float* d = destination) + { + for (var x = 0; x < groups; x++) + { + for (var b = 0; b < nb; b++) + { + var index = ((long)x * nb + b) * DecodedScalesPerBlock; + var blk = w + ((long)x * nb + b) * BlockKx8Bytes; + + LoadF16x8Rearrange(blk, deltamask).Store(d + index); + LoadF16x8(blk + 16).Store(d + index + 8); + } + } + } + } + /// Max activation columns per call — the register-tile width (NR). Kept /// small so the per-column accumulator scratch stays a bounded stack allocation; the caller tiles a /// larger prompt into NR-column chunks. @@ -324,7 +370,10 @@ public static void GemmTiled( ReadOnlySpan actScales, ReadOnlySpan actBsums, Span output, - ReadOnlySpan bias = default) + ReadOnlySpan bias = default, + int groupStart = 0, + int groupCount = 0, + ReadOnlySpan decodedScales = default) { if (cols is < 1 or > MaxTileCols) { @@ -362,8 +411,16 @@ public static void GemmTiled( fixed (short* ab = actBsums) fixed (float* o = output) fixed (float* bs = bias) // null when empty — keeps the no-bias path branch-free per store + fixed (float* ds = decodedScales) // null when the caller did not pre-decode; see DecodeBlockScales { - for (var x = 0; x < outputSize / 8; x++) + // Absolute group index throughout, so a caller can hand this kernel one band of output rows + // and every weight/output/bias offset below still lands in the right place. + var totalGroups = outputSize / 8; + var groupEnd = groupCount <= 0 + ? totalGroups + : Math.Min(groupStart + groupCount, totalGroups); + + for (var x = groupStart; x < groupEnd; x++) { var bptr = w + (long)x * nb * BlockKx8Bytes; for (var c = 0; c < cols; c++) @@ -375,12 +432,17 @@ public static void GemmTiled( for (var b = 0; b < nb; b++) { var blk = bptr + (long)b * BlockKx8Bytes; - var colScale = AblateF16Scales - ? Vector256.Create(1f) - : LoadF16x8Rearrange(blk, deltamask); - var colDmin = AblateF16Scales - ? Vector256.Create(0f) - : LoadF16x8(blk + 16); + + // Pre-decoded when the caller hoisted the F16 widening out of the tile loop; the values + // are identical either way, so the two paths are bit-identical. + var decodedAt = ds + (((long)x * nb) + b) * DecodedScalesPerBlock; + var colScale = ds is not null + ? Vector256.Load(decodedAt) + : AblateF16Scales ? Vector256.Create(1f) : LoadF16x8Rearrange(blk, deltamask); + var colDmin = ds is not null + ? Vector256.Load(decodedAt + 8) + : AblateF16Scales ? Vector256.Create(0f) : LoadF16x8(blk + 16); + var qsBase = blk + DstQsOffset; var scBase = blk + DstScalesOffset; From f5a8ef2fe0c5e2340a76bdfdde809a5187d2cf67 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Wed, 22 Jul 2026 22:56:27 +0200 Subject: [PATCH 18/37] llama --- ROADMAP.md | 13 ++++- .../Runtime/BatchedQuantProjection.cs | 26 +++++++++- .../LanguageModels/Runtime/Q6KGemvKernel.cs | 50 ++++++++++++++++++- 3 files changed, 84 insertions(+), 5 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 6d3d4cc5..77c8e599 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -699,8 +699,17 @@ runtime at NR=8 and ~7.5% at NR=16, so the gain should roughly halve with the wi measured 13%/6% = 2.2×. End to end it predicted `0.45 × 0.13 = 5.9%` off prefill → 2606 ms; measured **2622–2632 ms, 255–256 tok/s**, within 0.6%. Gap to llama.cpp's AVX-512 build: 2.18× → **2.12×**. -Suite 1486/0/229. **Next: the same hoist for `Q6KGemvKernel` — `ffn_down` is 27% of prefill (708 ms) and has -the identical per-block decode**, so the same ~13% there is worth roughly another 9 tok/s. +**The same hoist for Q6_K pays 5× less than predicted.** `Q6KGemvKernel.DecodeBlockScales` mirrors the Q4_K +one and carries `ffn_down` (27% of prefill). Predicted ~13% and another ~9 tok/s; measured **ffn_down 708.6 → +690.4 ms, −2.6%**, worth 3 tok/s. The reason was checkable in advance and was not checked: Q6_K widens +**eight** values per block against Q4_K's sixteen (it has no `dmin`), and its block is larger — 1680 B vs +1152 B — with more compute in the 6-bit unpack. The fixed decode is therefore a much smaller fraction of a +bigger block: 12% / ~4.6 ≈ 2.6%, which is what came out. The amortisation model predicts well *within* a +kernel — it called the tile-width scaling and the end-to-end figure correctly — but extrapolating it *across* +kernels without re-reading their inputs was a guess. + +**Both hoists together: prefill 249 → 258–259 tok/s, gap to llama.cpp's AVX-512 build 2.18× → 2.10×.** +Suite 1486/0/229. *Invalidated run, kept as a warning:* the first tile sweep ran inside an 11-benchmark class and reported `Tiled` and `Tiled_Cols8` — **the same configuration** — 21% apart, far outside their ±9% bars. Two identical diff --git a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs index f76e5dd2..02229f8b 100644 --- a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs +++ b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs @@ -449,7 +449,22 @@ private static unsafe void DispatchTiledQ6K( var nr = ResolveTileCols(rows, cores, Q6KGemvKernel.MaxTileCols); var tiles = (rows + nr - 1) / nr; + // Same hoist as the Q4_K path: widen the F16 row scales once per projection rather than once per + // column tile. Q6_K has no dmin, so this is half the scratch. + var scaleCount = UsePrecomputedScales && tiles > 1 + ? (outputSize / 8) * spr * Q6KGemvKernel.DecodedScalesPerBlock + : 0; + + using var decodedScales = new PooledBuffer(scaleCount, clearMemory: false); + + if (scaleCount > 0) + { + Q6KGemvKernel.DecodeBlockScales( + repacked, outputSize, inputSize, decodedScales.Span.Slice(0, scaleCount)); + } + fixed (byte* rp = repacked) + fixed (float* dsc = decodedScales.Span.Slice(0, scaleCount)) fixed (sbyte* q = quants) fixed (float* sc = scales) fixed (float* o = output) @@ -466,6 +481,8 @@ private static unsafe void DispatchTiledQ6K( Spr = spr, Nr = nr, Rows = rows, + DecodedScales = dsc, + DecodedScalesLength = scaleCount, }; OverfitParallel.For(0, tiles, &TiledQ6KChunk, &ctx); } @@ -483,6 +500,12 @@ private unsafe struct TiledQ6KContext public int Spr; public int Nr; public int Rows; + + /// F16 row scales widened once for the whole projection; null when decoded inline. + public float* DecodedScales; + + /// Length of ; 0 when decoded inline. + public int DecodedScalesLength; } private static unsafe void TiledQ6KChunk(int start, int end, void* context) @@ -499,7 +522,8 @@ private static unsafe void TiledQ6KChunk(int start, int end, void* context) cols, new ReadOnlySpan(c.Quants + (long)s * c.InputSize, cols * c.InputSize), new ReadOnlySpan(c.Scales + (long)s * c.Spr, cols * c.Spr), - new Span(c.Output + (long)s * c.OutputSize, cols * c.OutputSize)); + new Span(c.Output + (long)s * c.OutputSize, cols * c.OutputSize), + new ReadOnlySpan(c.DecodedScales, c.DecodedScalesLength)); } } diff --git a/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs b/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs index 1a00144d..f230a784 100644 --- a/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs +++ b/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs @@ -285,6 +285,45 @@ private static unsafe void ComputeGroupRange( /// the Q4_K tiled kernel — activations column-contiguous, output column-major /// (output[c*outputSize + row]). AVX2 + FMA. /// + /// Floats written per weight block by : the eight row scales. + public const int DecodedScalesPerBlock = 8; + + /// + /// Widens every weight block's eight F16 row scales to once, into + /// laid out as [(group·nb + block) · 8]. + /// + /// The Q6_K counterpart of Q4KGemvKernel.DecodeBlockScales, and needed for the same + /// reason: widens these inline once per (group, block), but the kernel itself + /// runs once per column tile, so the same values are decoded as many times as there are tiles. + /// On the Q4_K side hoisting this measured +13% on the projection. Q6_K carries `ffn_down`, 27% of + /// prefill. Bit-identical: the same conversions, fewer times. + /// + /// Q6_K has no dmin, so this writes 8 floats per block against Q4_K's 16. + /// + public static unsafe void DecodeBlockScales( + ReadOnlySpan repacked, + int outputSize, + int inputSize, + Span destination) + { + var nb = inputSize / 256; + var groups = outputSize / 8; + + fixed (byte* w = repacked) + fixed (float* d = destination) + { + for (var x = 0; x < groups; x++) + { + for (var l = 0; l < nb; l++) + { + var index = ((long)x * nb + l) * DecodedScalesPerBlock; + + LoadF16x8Int(w + ((long)x * nb + l) * BlockKx8Bytes).Store(d + index); + } + } + } + } + public static unsafe void GemmTiled( ReadOnlySpan repacked, int outputSize, @@ -292,7 +331,8 @@ public static unsafe void GemmTiled( int cols, ReadOnlySpan actQuants, ReadOnlySpan actScales, - Span output) + Span output, + ReadOnlySpan decodedScales = default) { if (cols is < 1 or > MaxTileCols) { @@ -315,6 +355,7 @@ public static unsafe void GemmTiled( fixed (sbyte* aqAll = actQuants) fixed (float* asc = actScales) fixed (float* outp = output) + fixed (float* dsc = decodedScales) // null when the caller did not pre-decode; see DecodeBlockScales { for (var x = 0; x < outputSize / 8; x++) { @@ -331,7 +372,12 @@ public static unsafe void GemmTiled( var scales = blk + DstScalesOffset; var ql = blk + DstQlOffset; var qh = blk + DstQhOffset; - var dVec = LoadF16x8Int(blk); + + // Pre-decoded when the caller hoisted the F16 widening out of the tile loop; identical + // values either way, so the two paths are bit-identical. + var dVec = dsc is not null + ? Vector256.Load(dsc + (((long)x * nb) + l) * DecodedScalesPerBlock) + : LoadF16x8Int(blk); for (var c = 0; c < cols; c++) { From 34dd755522f3e4d500e1205c236b91a1a8903208 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Wed, 22 Jul 2026 23:07:31 +0200 Subject: [PATCH 19/37] llama --- ROADMAP.md | 35 ++ Sources/Main/Intrinsics/CpuFeatures.cs | 4 + .../Runtime/BatchedQuantProjection.cs | 42 ++- .../LanguageModels/Runtime/Q4KGemvKernel.cs | 321 ++++++++++++++++++ .../Runtime/Avx512PrefillParityTests.cs | 167 +++++++++ 5 files changed, 557 insertions(+), 12 deletions(-) create mode 100644 Tests/LanguageModels/Runtime/Avx512PrefillParityTests.cs diff --git a/ROADMAP.md b/ROADMAP.md index 77c8e599..3e631e4b 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -711,6 +711,41 @@ kernels without re-reading their inputs was a guess. **Both hoists together: prefill 249 → 258–259 tok/s, gap to llama.cpp's AVX-512 build 2.18× → 2.10×.** Suite 1486/0/229. +#### ★★ AVX-512 Q4_K PREFILL KERNEL — prefill 259 → 280 tok/s, gap under 2× for the first time + +`Q4KGemvKernel.GemmTiled512` processes **two activation columns per instruction**: column `2p` in the low 256 +bits of every vector, `2p+1` in the high. Weights are identical for both, so they are broadcast into both +halves; only activations, their scales and their block sums differ. Every shuffle in this kernel is +per-128-bit-lane, so it widens without changing meaning — no new repack layout, `block_q4_Kx8` untouched, +sidecars still valid. + +Two decisions worth keeping: pairing **columns** rather than widening the output-row group avoids a +`block_q4_Kx16` layout and the sidecar invalidation that implies; and the pair loop stays **innermost**, +because hoisting it would re-decode the sixteen weight vectors per pair and throw away the amortisation the +tile-width sweep showed to dominate this kernel. + +| component | before | after | | +|---|---:|---:|---:| +| `ffn_gateup` | 1180 ms | **1017.6** | **−13.8%** | +| `attn_out` | 106.0 | **89.4** | −15.7% | +| `attn_q` | 100.6 | **85.6** | −14.9% | +| `ffn_down` (Q6_K, not ported) | 691 | 658 | −4.8% | +| `attn_scores` (different kernel) | 207.6 | 210.8 | flat | +| **prefill** | **2597 ms / 259 tok/s** | **2398 / 280 tok/s** | **+8.2%** | + +`attn_scores` staying flat while every Q4_K path moves 14–16% is the internal control: this is the change, +not box drift. **Gap to llama.cpp AVX-512 2.10× → 1.93×; to their AVX2 build 1.20×.** + +The kernel itself gained ~1.16×, not the 1.67× its instruction mix promised, because that mix is roughly half +the kernel — loads, the scalar `Unpack` and the stores did not widen. The prior estimate was 1.24×. + +**Bit-identical**, pinned by `Avx512PrefillParityTests` (8 cases: odd and even column counts, bias, and the +pre-decoded scale path) asserting exact equality rather than a tolerance. Gated through +`CpuFeatures.HasAvx512`/`HasAvx512Bw` — the repo's own OVERFIT015 analyzer rejected a direct `IsSupported` +check, which is what that rule is for. Suite 1494/0/229. + +**Next: the same port for `Q6KGemvKernel`** — `ffn_down` is 27% of prefill (658 ms) and still runs 256-bit. + *Invalidated run, kept as a warning:* the first tile sweep ran inside an 11-benchmark class and reported `Tiled` and `Tiled_Cols8` — **the same configuration** — 21% apart, far outside their ±9% bars. Two identical arms in one table is the cheapest canary there is; narrowing the filter so the arms sit adjacent in time made diff --git a/Sources/Main/Intrinsics/CpuFeatures.cs b/Sources/Main/Intrinsics/CpuFeatures.cs index 2ed01935..af943991 100644 --- a/Sources/Main/Intrinsics/CpuFeatures.cs +++ b/Sources/Main/Intrinsics/CpuFeatures.cs @@ -21,6 +21,10 @@ internal static class CpuFeatures public static readonly bool HasAvx512 = Avx512F.IsSupported; + // AVX-512 byte/word ops (vpmaddubsw/vpmaddwd on zmm) — required by the quantized prefill kernels, + // which reach their MACs through those rather than through floating-point FMA. + public static readonly bool HasAvx512Bw = Avx512BW.IsSupported; + public static readonly bool HasAvxVnni = AvxVnni.IsSupported; public static readonly bool HasSse = Sse.IsSupported; diff --git a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs index 02229f8b..7432628d 100644 --- a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs +++ b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs @@ -91,6 +91,16 @@ internal static class BatchedQuantProjection /// internal static bool UsePrecomputedScales = true; + /// + /// Route the tiled Q4_K prefill GEMM through , which processes + /// two activation columns per instruction. Defaults to on wherever the silicon supports it. + /// + /// Measured ceilings on this machine for the kernel's own instruction mix: 4.63 TFLOP/s at 256 + /// bits against 7.71–9.08 at 512. The port is bit-identical, so the existing parity tests apply to it + /// unchanged; Avx512PrefillParityTests pins the two kernels against each other directly. + /// + internal static bool UseAvx512PrefillQ4K = CpuFeatures.HasAvx512 && CpuFeatures.HasAvx512Bw; + /// /// Weight bytes one worker's band may occupy. Half of a 1 MB Zen-5 L2, leaving the rest for the /// activation tile and the output band; the point is residency, not filling the cache exactly. @@ -395,6 +405,7 @@ private static unsafe void DispatchTiledQ4K( Tiles = tiles, DecodedScales = dsc, DecodedScalesLength = scaleCount, + Avx512 = UseAvx512PrefillQ4K && !DisableRepackedKernelsForParity, }; if (!UseOutputBlocking) @@ -555,6 +566,9 @@ private unsafe struct TiledContext /// Length of ; 0 when decoded inline. public int DecodedScalesLength; + + /// Route through the two-columns-per-instruction AVX-512 kernel. + public bool Avx512; } private static unsafe void TiledChunk(int start, int end, void* context) @@ -564,19 +578,23 @@ private static unsafe void TiledChunk(int start, int end, void* context) { var s = t * c.Nr; var cols = Math.Min(c.Nr, c.Rows - s); + var weights = new ReadOnlySpan(c.Repacked, c.RepackedLength); + var quants = new ReadOnlySpan(c.Quants + (long)s * c.InputSize, cols * c.InputSize); + var scales = new ReadOnlySpan(c.Scales + (long)s * c.Spr, cols * c.Spr); + var sums = new ReadOnlySpan(c.Bsums + (long)s * c.BsumsPerRow, cols * c.BsumsPerRow); + var dst = new Span(c.Output + (long)s * c.OutputSize, cols * c.OutputSize); + var bias = new ReadOnlySpan(c.Bias, c.BiasLength); + var decoded = new ReadOnlySpan(c.DecodedScales, c.DecodedScalesLength); + + if (c.Avx512) + { + Q4KGemvKernel.GemmTiled512( + weights, c.OutputSize, c.InputSize, cols, quants, scales, sums, dst, bias, decoded); + continue; + } + Q4KGemvKernel.GemmTiled( - new ReadOnlySpan(c.Repacked, c.RepackedLength), - c.OutputSize, - c.InputSize, - cols, - new ReadOnlySpan(c.Quants + (long)s * c.InputSize, cols * c.InputSize), - new ReadOnlySpan(c.Scales + (long)s * c.Spr, cols * c.Spr), - new ReadOnlySpan(c.Bsums + (long)s * c.BsumsPerRow, cols * c.BsumsPerRow), - new Span(c.Output + (long)s * c.OutputSize, cols * c.OutputSize), - new ReadOnlySpan(c.Bias, c.BiasLength), - 0, - 0, - new ReadOnlySpan(c.DecodedScales, c.DecodedScalesLength)); + weights, c.OutputSize, c.InputSize, cols, quants, scales, sums, dst, bias, 0, 0, decoded); } } diff --git a/Sources/Main/LanguageModels/Runtime/Q4KGemvKernel.cs b/Sources/Main/LanguageModels/Runtime/Q4KGemvKernel.cs index 796b4a2d..9c4ebe40 100644 --- a/Sources/Main/LanguageModels/Runtime/Q4KGemvKernel.cs +++ b/Sources/Main/LanguageModels/Runtime/Q4KGemvKernel.cs @@ -590,6 +590,327 @@ public static void GemmTiled( } } + /// + /// AVX-512 form of : identical arithmetic, but two activation columns per + /// instruction — column 2p in the low 256 bits of every vector, column 2p+1 in the + /// high 256. The weights are the same for both, so they are broadcast into both halves; only the + /// activations, their scales and their block sums differ per half. + /// + /// Why columns and not output rows. Widening the output-row group to 16 would need a new + /// block_q4_Kx16 repack layout and would invalidate every sidecar. Pairing columns reuses + /// block_q4_Kx8 untouched, and every shuffle in this kernel is per-128-bit-lane, so it widens + /// without changing meaning. + /// + /// Why the pair loop stays innermost. Hoisting it would re-decode the sixteen weight + /// vectors per pair. Amortising that fixed per-block work across the whole column tile is precisely + /// what the tile-width sweep showed to dominate this kernel, so the loop order is preserved exactly. + /// + /// Bit-identical to : each column's operations and their order are + /// unchanged, two columns merely execute at once. Measured ceilings on this machine: the kernel's own + /// instruction mix runs at 4.63 TFLOP/s at 256 bits and 7.71–9.08 at 512. + /// + public static void GemmTiled512( + ReadOnlySpan repacked, + int outputSize, + int inputSize, + int cols, + ReadOnlySpan actQuants, + ReadOnlySpan actScales, + ReadOnlySpan actBsums, + Span output, + ReadOnlySpan bias = default, + ReadOnlySpan decodedScales = default) + { + if (cols is < 1 or > MaxTileCols) + { + throw new ArgumentOutOfRangeException(nameof(cols), cols, $"cols must be in [1, {MaxTileCols}]."); + } + + if (!bias.IsEmpty && bias.Length < outputSize) + { + throw new ArgumentException( + $"bias length {bias.Length} < outputSize {outputSize}.", nameof(bias)); + } + + var nb = inputSize / 256; + var pairs = (cols + 1) / 2; + + Span> accRow = stackalloc Vector512[pairs]; + Span> accMin = stackalloc Vector512[pairs]; + Span> iaccB = stackalloc Vector512[pairs]; + Span> iaccMinB = stackalloc Vector512[pairs]; + Span> q8s = stackalloc Vector512[pairs]; + + var m4b = Vector512.Create((byte)0x0F); + var deltamask = Vector128.Create((byte)0, 1, 8, 9, 2, 3, 10, 11, 4, 5, 12, 13, 6, 7, 14, 15); + var scalemask = Vector128.Create((byte)0, 0, 4, 4, 1, 1, 5, 5, 2, 2, 6, 6, 3, 3, 7, 7); + var finalpermute = Vector256.Create(0, 2, 4, 6, 1, 3, 5, 7); + + // Avx2.Blend(..., 170) takes the odd int32 lanes from the right operand; over sixteen lanes that is + // the same alternating pattern, expressed as a mask vector because AVX-512 blends by mask. + var blendMask = Vector512.Create(0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1); + + const uint kmask1 = 0x3f3f3f3f, kmask2 = 0x0f0f0f0f, kmask3 = 0x03030303; + + var u0 = stackalloc uint[4]; + var u1 = stackalloc uint[4]; + + fixed (byte* w = repacked) + fixed (sbyte* aq = actQuants) + fixed (float* asc = actScales) + fixed (short* ab = actBsums) + fixed (float* o = output) + fixed (float* bs = bias) + fixed (float* ds = decodedScales) + { + for (var x = 0; x < outputSize / 8; x++) + { + var bptr = w + (long)x * nb * BlockKx8Bytes; + + for (var p = 0; p < pairs; p++) + { + accRow[p] = Vector512.Zero; + accMin[p] = Vector512.Zero; + } + + for (var b = 0; b < nb; b++) + { + var blk = bptr + (long)b * BlockKx8Bytes; + var decodedAt = ds + (((long)x * nb) + b) * DecodedScalesPerBlock; + + var colScale256 = ds is not null + ? Vector256.Load(decodedAt) + : LoadF16x8Rearrange(blk, deltamask); + var colDmin256 = ds is not null + ? Vector256.Load(decodedAt + 8) + : LoadF16x8(blk + 16); + + var colScale = Vector512.Create(colScale256, colScale256); + var colDmin = Vector512.Create(colDmin256, colDmin256); + + var qsBase = blk + DstQsOffset; + var scBase = blk + DstScalesOffset; + + for (var p = 0; p < pairs; p++) + { + iaccB[p] = Vector512.Zero; + iaccMinB[p] = Vector512.Zero; + q8s[p] = Vector512.Create(BlockSums(ab, PairLow(p), nb, b), BlockSums(ab, PairHigh(p, cols), nb, b)); + } + + for (var sb = 0; sb < 4; sb++) + { + var qs = qsBase + sb * 256; + + var raw0123_0 = Broadcast512(qs); + var raw4567_0 = Broadcast512(qs + 32); + var raw0123_1 = Broadcast512(qs + 64); + var raw4567_1 = Broadcast512(qs + 96); + var raw0123_2 = Broadcast512(qs + 128); + var raw4567_2 = Broadcast512(qs + 160); + var raw0123_3 = Broadcast512(qs + 192); + var raw4567_3 = Broadcast512(qs + 224); + + var v0123_00 = raw0123_0 & m4b; + var v4567_00 = raw4567_0 & m4b; + var v0123_01 = raw0123_1 & m4b; + var v4567_01 = raw4567_1 & m4b; + var v0123_02 = raw0123_2 & m4b; + var v4567_02 = raw4567_2 & m4b; + var v0123_03 = raw0123_3 & m4b; + var v4567_03 = raw4567_3 & m4b; + + var v0123_10 = Hi512(raw0123_0) & m4b; + var v4567_10 = Hi512(raw4567_0) & m4b; + var v0123_11 = Hi512(raw0123_1) & m4b; + var v4567_11 = Hi512(raw4567_1) & m4b; + var v0123_12 = Hi512(raw0123_2) & m4b; + var v4567_12 = Hi512(raw4567_2) & m4b; + var v0123_13 = Hi512(raw0123_3) & m4b; + var v4567_13 = Hi512(raw4567_3) & m4b; + + u0[0] = Unsafe.ReadUnaligned(scBase + 24 * sb); + u0[1] = Unsafe.ReadUnaligned(scBase + 24 * sb + 4); + u0[2] = Unsafe.ReadUnaligned(scBase + 24 * sb + 8); + u1[0] = Unsafe.ReadUnaligned(scBase + 12 + sb * 24); + u1[1] = Unsafe.ReadUnaligned(scBase + 12 + sb * 24 + 4); + u1[2] = Unsafe.ReadUnaligned(scBase + 12 + sb * 24 + 8); + Unpack(u0, kmask1, kmask2, kmask3); + Unpack(u1, kmask1, kmask2, kmask3); + + var ms0 = Vector128.Create(u0[0], u0[1], u0[2], u0[3]).AsByte(); + var ms1 = Vector128.Create(u1[0], u1[1], u1[2], u1[3]).AsByte(); + var s0 = Avx2.ConvertToVector256Int16(Ssse3.Shuffle(ms0, scalemask)); + var s1 = Avx2.ConvertToVector256Int16(Ssse3.Shuffle(ms1, scalemask)); + var mn = Avx2.ConvertToVector256Int16( + Sse2.UnpackLow( + Sse2.Shuffle(ms0.AsInt32(), 78).AsByte(), + Sse2.Shuffle(ms1.AsInt32(), 78).AsByte())); + + var scales0 = Vector512.Create(s0, s0); + var scales1 = Vector512.Create(s1, s1); + var mins01 = Vector512.Create(mn, mn); + + for (var p = 0; p < pairs; p++) + { + var lowCol = PairLow(p); + var highCol = PairHigh(p, cols); + var aLow = aq + (long)lowCol * inputSize + b * 256 + sb * 64; + var aHigh = aq + (long)highCol * inputSize + b * 256 + sb * 64; + + var l00 = BroadcastLo512(aLow, aHigh); + var l01 = BroadcastLo512(aLow + 16, aHigh + 16); + var l10 = BroadcastLo512(aLow + 32, aHigh + 32); + var l11 = BroadcastLo512(aLow + 48, aHigh + 48); + + var iacc0 = Vector512.Zero; + var iacc1 = Vector512.Zero; + + iacc0 = Avx512BW.Add(iacc0, Mul512(Blend512(v0123_00, Sh512(v4567_00, 177), blendMask), Sh32_512(l00, 0))); + iacc0 = Avx512BW.Add(iacc0, Mul512(Blend512(Sh512(v0123_00, 177), v4567_00, blendMask), Sh32_512(l00, 85))); + iacc0 = Avx512BW.Add(iacc0, Mul512(Blend512(v0123_01, Sh512(v4567_01, 177), blendMask), Sh32_512(l00, 170))); + iacc0 = Avx512BW.Add(iacc0, Mul512(Blend512(Sh512(v0123_01, 177), v4567_01, blendMask), Sh32_512(l00, 255))); + iacc0 = Avx512BW.Add(iacc0, Mul512(Blend512(v0123_02, Sh512(v4567_02, 177), blendMask), Sh32_512(l01, 0))); + iacc0 = Avx512BW.Add(iacc0, Mul512(Blend512(Sh512(v0123_02, 177), v4567_02, blendMask), Sh32_512(l01, 85))); + iacc0 = Avx512BW.Add(iacc0, Mul512(Blend512(v0123_03, Sh512(v4567_03, 177), blendMask), Sh32_512(l01, 170))); + iacc0 = Avx512BW.Add(iacc0, Mul512(Blend512(Sh512(v0123_03, 177), v4567_03, blendMask), Sh32_512(l01, 255))); + var iacc0i = Avx512BW.MultiplyAddAdjacent(iacc0, scales0); + + iacc1 = Avx512BW.Add(iacc1, Mul512(Blend512(v0123_10, Sh512(v4567_10, 177), blendMask), Sh32_512(l10, 0))); + iacc1 = Avx512BW.Add(iacc1, Mul512(Blend512(Sh512(v0123_10, 177), v4567_10, blendMask), Sh32_512(l10, 85))); + iacc1 = Avx512BW.Add(iacc1, Mul512(Blend512(v0123_11, Sh512(v4567_11, 177), blendMask), Sh32_512(l10, 170))); + iacc1 = Avx512BW.Add(iacc1, Mul512(Blend512(Sh512(v0123_11, 177), v4567_11, blendMask), Sh32_512(l10, 255))); + iacc1 = Avx512BW.Add(iacc1, Mul512(Blend512(v0123_12, Sh512(v4567_12, 177), blendMask), Sh32_512(l11, 0))); + iacc1 = Avx512BW.Add(iacc1, Mul512(Blend512(Sh512(v0123_12, 177), v4567_12, blendMask), Sh32_512(l11, 85))); + iacc1 = Avx512BW.Add(iacc1, Mul512(Blend512(v0123_13, Sh512(v4567_13, 177), blendMask), Sh32_512(l11, 170))); + iacc1 = Avx512BW.Add(iacc1, Mul512(Blend512(Sh512(v0123_13, 177), v4567_13, blendMask), Sh32_512(l11, 255))); + var iacc1i = Avx512BW.MultiplyAddAdjacent(iacc1, scales1); + + var q8sSb = Avx512F.Shuffle(q8s[p].AsInt32(), 0).AsInt16(); + var iaccMinSb = Avx512BW.MultiplyAddAdjacent(q8sSb, mins01); + q8s[p] = Avx512BW.ShiftRightLogical128BitLane(q8s[p].AsByte(), 4).AsInt16(); + + iaccB[p] = Avx512F.Add(iaccB[p], Avx512F.Add(iacc0i, iacc1i)); + iaccMinB[p] = Avx512F.Add(iaccMinB[p], iaccMinSb); + } + } + + for (var p = 0; p < pairs; p++) + { + var rowScale = Vector512.Create( + Vector256.Create(asc[(long)PairLow(p) * nb + b]), + Vector256.Create(asc[(long)PairHigh(p, cols) * nb + b])); + + accRow[p] = Avx512F.FusedMultiplyAdd( + Avx512F.ConvertToVector512Single(iaccB[p]), + Avx512F.Multiply(colScale, rowScale), + accRow[p]); + accMin[p] = Avx512F.FusedMultiplyAdd( + Avx512F.ConvertToVector512Single(iaccMinB[p]), + Avx512F.Multiply(colDmin, rowScale), + accMin[p]); + } + } + + for (var p = 0; p < pairs; p++) + { + var lowCol = PairLow(p); + var highCol = 2 * p + 1; + + StoreColumn(o, bs, accRow[p].GetLower(), accMin[p].GetLower(), + finalpermute, lowCol, outputSize, x); + + // The odd tail computed a duplicate of the low column in the high half; discard it. + if (highCol < cols) + { + StoreColumn(o, bs, accRow[p].GetUpper(), accMin[p].GetUpper(), + finalpermute, highCol, outputSize, x); + } + } + } + } + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int PairLow(int pair) => 2 * pair; + + /// The odd column of a pair, or the even one again when the tile has an odd column count. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static int PairHigh(int pair, int cols) => Math.Min(2 * pair + 1, cols - 1); + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector256 BlockSums(short* ab, int column, int nb, int b) + { + var q8sums = Vector256.Load(ab + (long)column * nb * 16 + b * 16); + var hadd = Ssse3.HorizontalAdd(q8sums.GetLower(), q8sums.GetUpper()); + + return Vector256.Create(hadd, hadd).AsInt16(); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static void StoreColumn( + float* o, + float* bs, + Vector256 row, + Vector256 min, + Vector256 finalpermute, + int column, + int outputSize, + int x) + { + var permuted = Avx2.PermuteVar8x32(row, finalpermute); + var value = Avx.Subtract(permuted, min); + + // Two stores rather than adding a zero vector: `x + 0f` rewrites -0.0 to +0.0 and would break the + // bit-identity the no-bias path is pinned to. + if (bs is null) + { + value.Store(o + (long)column * outputSize + x * 8); + return; + } + + Avx.Add(value, Vector256.Load(bs + x * 8)).Store(o + (long)column * outputSize + x * 8); + } + + /// The same 32 weight bytes in both halves — weights are shared by the two columns of a pair. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 Broadcast512(byte* p) + { + var v = Vector256.Load(p); + + return Vector512.Create(v, v); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 Hi512(Vector512 v) => + Avx512BW.ShiftRightLogical(v.AsUInt16(), 4).AsByte(); + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 Sh512(Vector512 v, [ConstantExpected] byte imm) => + Avx512F.Shuffle(v.AsInt32(), imm).AsByte(); + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 Sh32_512(Vector512 v, [ConstantExpected] byte imm) => + Avx512F.Shuffle(v.AsInt32(), imm).AsSByte(); + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 Blend512(Vector512 a, Vector512 b, Vector512 mask) => + Avx512F.BlendVariable(a.AsInt32(), b.AsInt32(), mask).AsByte(); + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 Mul512(Vector512 rhs, Vector512 lhs) => + Avx512BW.MultiplyAddAdjacent(rhs, lhs); + + /// One 16-byte activation run per half, duplicated within each half exactly as does. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 BroadcastLo512(sbyte* low, sbyte* high) + { + var l = Vector128.Load(low); + var h = Vector128.Load(high); + + return Vector512.Create(Vector256.Create(l, l), Vector256.Create(h, h)); + } + [MethodImpl(MethodImplOptions.AggressiveInlining)] private static Vector256 Hi(Vector256 v) => Avx2.ShiftRightLogical(v.AsUInt16(), 4).AsByte(); diff --git a/Tests/LanguageModels/Runtime/Avx512PrefillParityTests.cs b/Tests/LanguageModels/Runtime/Avx512PrefillParityTests.cs new file mode 100644 index 00000000..5261ad5d --- /dev/null +++ b/Tests/LanguageModels/Runtime/Avx512PrefillParityTests.cs @@ -0,0 +1,167 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Runtime.Intrinsics.X86; +using DevOnBike.Overfit.LanguageModels.Loading; +using DevOnBike.Overfit.LanguageModels.Runtime; + +namespace DevOnBike.Overfit.Tests.LanguageModels.Runtime +{ + /// + /// Pins against . + /// + /// The AVX-512 kernel packs two activation columns into one instruction — column 2p in the low + /// 256 bits, 2p+1 in the high — but performs each column's operations in the same order as the + /// 256-bit kernel. Nothing is reassociated, so the results must be bit-identical, not merely close; + /// these tests assert exact equality so that any future reordering shows up immediately rather than hiding + /// inside a tolerance. + /// + /// Odd column counts are covered because the tail pair computes a duplicate of its low column in the + /// high half and must discard it — an off-by-one there would silently overwrite the neighbouring column. + /// + public sealed class Avx512PrefillParityTests + { + private const int InputSize = 512; + private const int OutputSize = 64; + + [Theory] + [InlineData(1)] + [InlineData(2)] + [InlineData(3)] + [InlineData(7)] + [InlineData(8)] + [InlineData(16)] + public void GemmTiled512_IsBitIdenticalTo_GemmTiled(int cols) + { + if (!Avx512BW.IsSupported || !Avx512F.IsSupported) + { + // Nothing to compare on a machine without the wider kernel; the 256-bit path is covered + // by Q4KTiledGemmParityTests regardless. + return; + } + + var (weight, quants, scales, bsums) = BuildInputs(cols); + + { + var repacked = weight.EnsureRepacked(); + var reference = new float[cols * OutputSize]; + var wide = new float[cols * OutputSize]; + + Q4KGemvKernel.GemmTiled( + repacked, OutputSize, InputSize, cols, quants, scales, bsums, reference); + Q4KGemvKernel.GemmTiled512( + repacked, OutputSize, InputSize, cols, quants, scales, bsums, wide); + + Assert.Equal(reference, wide); + } + } + + [Fact] + public void GemmTiled512_IsBitIdenticalTo_GemmTiled_WithBias() + { + if (!Avx512BW.IsSupported || !Avx512F.IsSupported) + { + // Nothing to compare on a machine without the wider kernel; the 256-bit path is covered + // by Q4KTiledGemmParityTests regardless. + return; + } + + const int Cols = 8; + var (weight, quants, scales, bsums) = BuildInputs(Cols); + + { + var repacked = weight.EnsureRepacked(); + var bias = new float[OutputSize]; + var rng = new Random(4242); + + for (var i = 0; i < bias.Length; i++) + { + bias[i] = (float)(rng.NextDouble() * 2.0 - 1.0); + } + + var reference = new float[Cols * OutputSize]; + var wide = new float[Cols * OutputSize]; + + Q4KGemvKernel.GemmTiled( + repacked, OutputSize, InputSize, Cols, quants, scales, bsums, reference, bias); + Q4KGemvKernel.GemmTiled512( + repacked, OutputSize, InputSize, Cols, quants, scales, bsums, wide, bias); + + Assert.Equal(reference, wide); + } + } + + /// The pre-decoded scale path must not change the result either — it only widens F16 earlier. + [Fact] + public void GemmTiled512_IsBitIdenticalTo_GemmTiled_WithPrecomputedScales() + { + if (!Avx512BW.IsSupported || !Avx512F.IsSupported) + { + // Nothing to compare on a machine without the wider kernel; the 256-bit path is covered + // by Q4KTiledGemmParityTests regardless. + return; + } + + const int Cols = 8; + var (weight, quants, scales, bsums) = BuildInputs(Cols); + + { + var repacked = weight.EnsureRepacked(); + var decoded = new float[(OutputSize / 8) * (InputSize / 256) * Q4KGemvKernel.DecodedScalesPerBlock]; + + Q4KGemvKernel.DecodeBlockScales(repacked, OutputSize, InputSize, decoded); + + var reference = new float[Cols * OutputSize]; + var wide = new float[Cols * OutputSize]; + + Q4KGemvKernel.GemmTiled( + repacked, OutputSize, InputSize, Cols, quants, scales, bsums, reference); + Q4KGemvKernel.GemmTiled512( + repacked, OutputSize, InputSize, Cols, quants, scales, bsums, wide, [], decoded); + + Assert.Equal(reference, wide); + } + } + + private static (Q4KWeight Weight, sbyte[] Quants, float[] Scales, short[] Bsums) BuildInputs(int cols) + { + var rng = new Random(20260722); + + var f32 = new float[(long)OutputSize * InputSize]; + + for (var i = 0; i < f32.Length; i++) + { + f32[i] = (float)(rng.NextDouble() * 2.0 - 1.0); + } + + var weight = new Q4KWeight(GgmlQuant.QuantizeQ4_K(f32, InputSize, OutputSize), InputSize, OutputSize); + + var spr = weight.SuperBlocksPerRow; + var bsumsPerRow = spr * Q4KDotKernel.GroupsPerSuperBlock; + + var input = new float[cols * InputSize]; + + for (var i = 0; i < input.Length; i++) + { + input[i] = (float)(rng.NextDouble() * 2.0 - 1.0); + } + + var quants = new sbyte[cols * InputSize]; + var scales = new float[cols * spr]; + var bsums = new short[cols * bsumsPerRow]; + + for (var c = 0; c < cols; c++) + { + Q4KDotKernel.QuantizeActivationQ8K( + input.AsSpan(c * InputSize, InputSize), + quants.AsSpan(c * InputSize, InputSize), + scales.AsSpan(c * spr, spr), + bsums.AsSpan(c * bsumsPerRow, bsumsPerRow)); + } + + return (weight, quants, scales, bsums); + } + } +} From d9d80b5f602a1a6d25cf8da0eec1e3af400146a6 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Wed, 22 Jul 2026 23:21:22 +0200 Subject: [PATCH 20/37] llama --- ROADMAP.md | 21 +- .../Runtime/BatchedQuantProjection.cs | 44 +++- .../LanguageModels/Runtime/Q6KGemvKernel.cs | 214 ++++++++++++++++++ .../Runtime/Avx512Q6KPrefillParityTests.cs | 126 +++++++++++ 4 files changed, 396 insertions(+), 9 deletions(-) create mode 100644 Tests/LanguageModels/Runtime/Avx512Q6KPrefillParityTests.cs diff --git a/ROADMAP.md b/ROADMAP.md index 3e631e4b..4bc87510 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -744,7 +744,26 @@ pre-decoded scale path) asserting exact equality rather than a tolerance. Gated `CpuFeatures.HasAvx512`/`HasAvx512Bw` — the repo's own OVERFIT015 analyzer rejected a direct `IsSupported` check, which is what that rule is for. Suite 1494/0/229. -**Next: the same port for `Q6KGemvKernel`** — `ffn_down` is 27% of prefill (658 ms) and still runs 256-bit. +#### ▶ NEGATIVE — the same AVX-512 port for Q6_K is SLOWER, reverted + +`Q6KGemvKernel.GemmTiled512` exists and is bit-identical (`Avx512Q6KPrefillParityTests`, 7 cases), but +`BatchedQuantProjection.UseAvx512PrefillQ6K` is **off**: on the same machine and prompt where the Q4_K port +took `ffn_gateup` down 13.8%, this took `ffn_down` from 658 ms to **794–900 ms** and prefill from 280 back to +256–265 tok/s. Reverting restored 2398.5/2399.6 ms and `ffn_down` 656/659 ms exactly. + +Two things marked it as real rather than drift: the Q4_K components held steady across the same runs +(`ffn_gateup` 1023/1009, `attn_q` 86/84), and the run-to-run spread was concentrated entirely on `ffn_down`. + +**Why the identical technique inverts between the two kernels.** Column pairing pays for the +`vinserti64x4` that builds each broadcast with the arithmetic subsequently done on it. Q4_K broadcasts eight +weight vectors per sub-block and then issues sixteen paired statements against them. Q6_K broadcasts six per +`k`, sixteen times per block, for far less arithmetic each — and its `ReduceRows` cannot widen at all, since +AVX-512 has no `vphaddd` for zmm, adding three more cross-half moves per call across 32 calls per block. The +lane-crossing traffic outruns the arithmetic saved. **A wider vector is not a property of the ISA alone; it +is a ratio between broadcast cost and work done per broadcast, and that ratio is per-kernel.** + +**Where prefill stands: 280 tok/s, gap 1.93×.** Remaining measured items: `attn_scores` 212 ms at 0.27 +TFLOP/s (12% of its ceiling, needs a new kernel), and the scalar `Unpack` at ~3.5%. *Invalidated run, kept as a warning:* the first tile sweep ran inside an 11-benchmark class and reported `Tiled` and `Tiled_Cols8` — **the same configuration** — 21% apart, far outside their ±9% bars. Two identical diff --git a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs index 7432628d..e1f7a0c3 100644 --- a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs +++ b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs @@ -101,6 +101,24 @@ internal static class BatchedQuantProjection /// internal static bool UseAvx512PrefillQ4K = CpuFeatures.HasAvx512 && CpuFeatures.HasAvx512Bw; + /// + /// The same port for Q6_K — measured slower and therefore off. Kept behind the flag with + /// Avx512Q6KPrefillParityTests guarding it, because the negative is the useful part. + /// + /// On the identical machine and prompt where the Q4_K port took ffn_gateup down 13.8%, + /// the Q6_K port took ffn_down from 658 ms to 794–900 ms and prefill from 280 back to + /// 256–265 tok/s. The Q4_K components stayed flat across the same runs, and the run-to-run spread was + /// concentrated entirely on ffn_down, so it is the change and not the box. + /// + /// Why the same technique inverts. Column pairing pays for the vinserti64x4 that + /// builds each broadcast out of arithmetic done on it. Q4_K broadcasts eight weight vectors per + /// sub-block and then issues sixteen paired statements against them. Q6_K broadcasts six per k, + /// sixteen times per block, for far less arithmetic each — and its ReduceRows cannot widen at all + /// (no vphaddd for zmm), adding three more cross-half moves per call, 32 calls per block. The + /// lane-crossing traffic outruns the arithmetic saved. + /// + internal static bool UseAvx512PrefillQ6K; + /// /// Weight bytes one worker's band may occupy. Half of a 1 MB Zen-5 L2, leaving the rest for the /// activation tile and the output band; the point is residency, not filling the cache exactly. @@ -494,6 +512,7 @@ private static unsafe void DispatchTiledQ6K( Rows = rows, DecodedScales = dsc, DecodedScalesLength = scaleCount, + Avx512 = UseAvx512PrefillQ6K && !DisableRepackedKernelsForParity, }; OverfitParallel.For(0, tiles, &TiledQ6KChunk, &ctx); } @@ -517,6 +536,9 @@ private unsafe struct TiledQ6KContext /// Length of ; 0 when decoded inline. public int DecodedScalesLength; + + /// Route through the two-columns-per-instruction AVX-512 kernel. + public bool Avx512; } private static unsafe void TiledQ6KChunk(int start, int end, void* context) @@ -526,15 +548,21 @@ private static unsafe void TiledQ6KChunk(int start, int end, void* context) { var s = t * c.Nr; var cols = Math.Min(c.Nr, c.Rows - s); + var weights = new ReadOnlySpan(c.Repacked, c.RepackedLength); + var quants = new ReadOnlySpan(c.Quants + (long)s * c.InputSize, cols * c.InputSize); + var scales = new ReadOnlySpan(c.Scales + (long)s * c.Spr, cols * c.Spr); + var dst = new Span(c.Output + (long)s * c.OutputSize, cols * c.OutputSize); + var decoded = new ReadOnlySpan(c.DecodedScales, c.DecodedScalesLength); + + if (c.Avx512) + { + Q6KGemvKernel.GemmTiled512( + weights, c.OutputSize, c.InputSize, cols, quants, scales, dst, decoded); + continue; + } + Q6KGemvKernel.GemmTiled( - new ReadOnlySpan(c.Repacked, c.RepackedLength), - c.OutputSize, - c.InputSize, - cols, - new ReadOnlySpan(c.Quants + (long)s * c.InputSize, cols * c.InputSize), - new ReadOnlySpan(c.Scales + (long)s * c.Spr, cols * c.Spr), - new Span(c.Output + (long)s * c.OutputSize, cols * c.OutputSize), - new ReadOnlySpan(c.DecodedScales, c.DecodedScalesLength)); + weights, c.OutputSize, c.InputSize, cols, quants, scales, dst, decoded); } } diff --git a/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs b/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs index f230a784..60e34fc0 100644 --- a/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs +++ b/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs @@ -449,6 +449,220 @@ public static unsafe void GemmTiled( } } + /// + /// AVX-512 form of , structured exactly like + /// Q4KGemvKernel.GemmTiled512: two activation columns per instruction, column 2p + /// in the low 256 bits and 2p+1 in the high, with the shared weights broadcast into both halves + /// and the pair loop left innermost so the per-block weight decode stays amortised across the tile. + /// + /// The one place it cannot widen. ends in vphaddd, which + /// AVX-512 does not provide for zmm at all. The two vpmaddwd steps still run at 512; only the + /// horizontal add and its permute drop to 256-bit halves and are re-joined. That costs one extra + /// instruction per reduction against eighteen saved elsewhere in the same iteration — the arithmetic + /// bulk (eight vpmaddubsw, four subtracts, the scale multiplies) all pairs cleanly. + /// + /// Bit-identical to : each column's operations and their order are + /// unchanged, including the split reduction, which is the same 256-bit sequence applied to the same + /// values. + /// + public static unsafe void GemmTiled512( + ReadOnlySpan repacked, + int outputSize, + int inputSize, + int cols, + ReadOnlySpan actQuants, + ReadOnlySpan actScales, + Span output, + ReadOnlySpan decodedScales = default) + { + if (cols is < 1 or > MaxTileCols) + { + throw new ArgumentOutOfRangeException(nameof(cols), cols, $"cols must be in [1, {MaxTileCols}]."); + } + + var nb = inputSize / 256; + var pairs = (cols + 1) / 2; + + var m4b = Vector512.Create((byte)0x0F); + var m2 = Vector512.Create((byte)0x03); + var m32 = Vector512.Create((byte)32); + var ones = Vector512.Create((short)1); + var reduce = Vector256.Create(0, 1, 4, 5, 2, 3, 6, 7); + + Span> sumf = stackalloc Vector512[pairs]; + Span> iacc = stackalloc Vector512[pairs]; + + fixed (byte* rep = repacked) + fixed (sbyte* aqAll = actQuants) + fixed (float* asc = actScales) + fixed (float* outp = output) + fixed (float* dsc = decodedScales) + { + for (var x = 0; x < outputSize / 8; x++) + { + var bptr = rep + (long)x * nb * BlockKx8Bytes; + + for (var p = 0; p < pairs; p++) + { + sumf[p] = Vector512.Zero; + } + + for (var l = 0; l < nb; l++) + { + var blk = bptr + (long)l * BlockKx8Bytes; + var scales = blk + DstScalesOffset; + var ql = blk + DstQlOffset; + var qh = blk + DstQhOffset; + + var d256 = dsc is not null + ? Vector256.Load(dsc + (((long)x * nb) + l) * DecodedScalesPerBlock) + : LoadF16x8Int(blk); + var dVec = Vector512.Create(d256, d256); + + for (var p = 0; p < pairs; p++) + { + iacc[p] = Vector512.Zero; + } + + for (var k = 0; k < 16; k++) + { + var baseL = (k / 8) * 128 + (k % 8) * 8; + var baseH = baseL + 64; + var qhShiftL = (byte)(((baseL % 128) / 32) * 2); + var qhShiftH = (byte)(((baseH % 128) / 32) * 2); + var qhHalfL = (baseL / 128) * 32; + var qhHalfH = (baseH / 128) * 32; + var qhBlockL = ((qhHalfL + (baseL % 32)) / 8) * 64; + var qhBlockH = ((qhHalfH + (baseH % 32)) / 8) * 64; + + // Weight side: decoded once, shared by both columns of every pair. + var ql03 = Broadcast512(ql + k * 64); + var ql47 = Broadcast512(ql + k * 64 + 32); + var qhL03 = Broadcast512(qh + qhBlockL); + var qhL47 = Broadcast512(qh + qhBlockL + 32); + var qhH03 = Broadcast512(qh + qhBlockH); + var qhH47 = Broadcast512(qh + qhBlockH + 32); + + var qLu03 = LoNib512(ql03, m4b) | QhBits512(qhL03, qhShiftL, m2); + var qLu47 = LoNib512(ql47, m4b) | QhBits512(qhL47, qhShiftL, m2); + var qHu03 = HiNib512(ql03, m4b) | QhBits512(qhH03, qhShiftH, m2); + var qHu47 = HiNib512(ql47, m4b) | QhBits512(qhH47, qhShiftH, m2); + + var sl = ScaleVec(scales + (baseL / 16) * 8); + var sh = ScaleVec(scales + (baseH / 16) * 8); + var scaleL = Vector512.Create(sl, sl); + var scaleH = Vector512.Create(sh, sh); + + for (var p = 0; p < pairs; p++) + { + var lowCol = 2 * p; + var highCol = Math.Min(2 * p + 1, cols - 1); + var aLow = aqAll + (long)lowCol * inputSize + l * 256; + var aHigh = aqAll + (long)highCol * inputSize + l * 256; + + var actL = TileAct512(aLow + baseL, aHigh + baseL); + var actH = TileAct512(aLow + baseH, aHigh + baseH); + + var sumL = ReduceRows512( + Avx512BW.Subtract(Mul512(qLu03, actL), Mul512(m32, actL)), + Avx512BW.Subtract(Mul512(qLu47, actL), Mul512(m32, actL)), + ones, reduce); + var sumH = ReduceRows512( + Avx512BW.Subtract(Mul512(qHu03, actH), Mul512(m32, actH)), + Avx512BW.Subtract(Mul512(qHu47, actH), Mul512(m32, actH)), + ones, reduce); + + iacc[p] = Avx512F.Add(iacc[p], Avx512F.Add( + Avx512F.MultiplyLow(sumL, scaleL), Avx512F.MultiplyLow(sumH, scaleH))); + } + } + + for (var p = 0; p < pairs; p++) + { + var rowScale = Vector512.Create( + Vector256.Create(asc[(long)(2 * p) * nb + l]), + Vector256.Create(asc[(long)Math.Min(2 * p + 1, cols - 1) * nb + l])); + + sumf[p] = Avx512F.FusedMultiplyAdd( + Avx512F.ConvertToVector512Single(iacc[p]), + Avx512F.Multiply(dVec, rowScale), + sumf[p]); + } + } + + for (var p = 0; p < pairs; p++) + { + sumf[p].GetLower().Store(outp + (long)(2 * p) * outputSize + x * 8); + + // The odd tail duplicated its low column into the high half; discard that copy. + if (2 * p + 1 < cols) + { + sumf[p].GetUpper().Store(outp + (long)(2 * p + 1) * outputSize + x * 8); + } + } + } + } + } + + /// The same 32 weight bytes in both halves — weights are shared by the two columns of a pair. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static unsafe Vector512 Broadcast512(byte* p) + { + var v = Vector256.Load(p); + + return Vector512.Create(v, v); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 LoNib512(Vector512 v, Vector512 m4b) => v & m4b; + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 HiNib512(Vector512 v, Vector512 m4b) => + Avx512BW.ShiftRightLogical(v.AsInt16(), 4).AsByte() & m4b; + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 QhBits512(Vector512 qh, byte shift, Vector512 m2) + { + var count = Vector128.CreateScalar((short)shift); + var bits = Avx512BW.ShiftRightLogical(qh.AsInt16(), count).AsByte() & m2; + + return Avx512BW.ShiftLeftLogical(bits.AsInt16(), 4).AsByte(); + } + + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 Mul512(Vector512 rhs, Vector512 lhs) => + Avx512BW.MultiplyAddAdjacent(rhs, lhs); + + /// One column's eight activation bytes per half, duplicated in both 128-bit lanes of that half. + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static unsafe Vector512 TileAct512(sbyte* low, sbyte* high) + { + var lo = Unsafe.ReadUnaligned(low); + var hi = Unsafe.ReadUnaligned(high); + var vl = Vector128.Create(lo, lo).AsSByte(); + var vh = Vector128.Create(hi, hi).AsSByte(); + + return Vector512.Create(Vector256.Create(vl, vl), Vector256.Create(vh, vh)); + } + + /// + /// widened as far as the instruction set allows: both vpmaddwd steps run + /// at 512 bits, then the horizontal add and its permute drop to 256-bit halves because AVX-512 has no + /// vphaddd for zmm. Each half is the identical 256-bit sequence, so the result is bit-identical. + /// + [MethodImpl(MethodImplOptions.AggressiveInlining)] + private static Vector512 ReduceRows512( + Vector512 p03, Vector512 p47, Vector512 ones, Vector256 reduce) + { + var m03 = Avx512BW.MultiplyAddAdjacent(p03, ones); + var m47 = Avx512BW.MultiplyAddAdjacent(p47, ones); + + var lower = Avx2.PermuteVar8x32(Avx2.HorizontalAdd(m03.GetLower(), m47.GetLower()), reduce); + var upper = Avx2.PermuteVar8x32(Avx2.HorizontalAdd(m03.GetUpper(), m47.GetUpper()), reduce); + + return Vector512.Create(lower, upper); + } + [MethodImpl(MethodImplOptions.AggressiveInlining)] private static Vector256 LoNib(Vector256 v, Vector256 m4b) => Avx2.And(v, m4b); diff --git a/Tests/LanguageModels/Runtime/Avx512Q6KPrefillParityTests.cs b/Tests/LanguageModels/Runtime/Avx512Q6KPrefillParityTests.cs new file mode 100644 index 00000000..0aca505e --- /dev/null +++ b/Tests/LanguageModels/Runtime/Avx512Q6KPrefillParityTests.cs @@ -0,0 +1,126 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Buffers.Binary; +using System.Runtime.Intrinsics.X86; +using DevOnBike.Overfit.LanguageModels.Runtime; + +namespace DevOnBike.Overfit.Tests.LanguageModels.Runtime +{ + /// + /// Pins against . + /// + /// Q6_K needs its own parity coverage rather than leaning on the Q4_K tests, because its port is not + /// a pure widening: ReduceRows ends in vphaddd, which AVX-512 does not offer for zmm, so the + /// reduction drops to 256-bit halves while the rest of the loop runs at 512. That split is exactly the kind + /// of seam where a lane ends up in the wrong half, and only an exact comparison catches it. + /// + public sealed class Avx512Q6KPrefillParityTests + { + private const int InputSize = 512; + private const int OutputSize = 64; + + [Theory] + [InlineData(1)] + [InlineData(2)] + [InlineData(3)] + [InlineData(7)] + [InlineData(8)] + [InlineData(16)] + public void GemmTiled512_IsBitIdenticalTo_GemmTiled(int cols) + { + if (!Avx512BW.IsSupported || !Avx512F.IsSupported) + { + return; + } + + var (weight, quants, scales) = BuildInputs(cols); + var repacked = weight.EnsureRepacked(); + + var reference = new float[cols * OutputSize]; + var wide = new float[cols * OutputSize]; + + Q6KGemvKernel.GemmTiled(repacked, OutputSize, InputSize, cols, quants, scales, reference); + Q6KGemvKernel.GemmTiled512(repacked, OutputSize, InputSize, cols, quants, scales, wide); + + Assert.Equal(reference, wide); + } + + [Fact] + public void GemmTiled512_IsBitIdenticalTo_GemmTiled_WithPrecomputedScales() + { + if (!Avx512BW.IsSupported || !Avx512F.IsSupported) + { + return; + } + + const int Cols = 8; + var (weight, quants, scales) = BuildInputs(Cols); + var repacked = weight.EnsureRepacked(); + + var decoded = new float[(OutputSize / 8) * (InputSize / 256) * Q6KGemvKernel.DecodedScalesPerBlock]; + Q6KGemvKernel.DecodeBlockScales(repacked, OutputSize, InputSize, decoded); + + var reference = new float[Cols * OutputSize]; + var wide = new float[Cols * OutputSize]; + + Q6KGemvKernel.GemmTiled(repacked, OutputSize, InputSize, Cols, quants, scales, reference); + Q6KGemvKernel.GemmTiled512(repacked, OutputSize, InputSize, Cols, quants, scales, wide, decoded); + + Assert.Equal(reference, wide); + } + + private static (Q6KWeight Weight, sbyte[] Quants, float[] Scales) BuildInputs(int cols) + { + var rng = new Random(20260722); + + // Synthetic Q6_K blocks, the same construction Q6KDotKernelTests uses: random ql/qh/scales with a + // sane small positive FP16 d. Parity between two kernels does not need realistic weights, only + // identical ones. + var superBlocksPerRow = InputSize / Q6KWeight.SuperBlockElements; + var blocks = new byte[OutputSize * superBlocksPerRow * Q6KWeight.SuperBlockBytes]; + + for (var b = 0; b < OutputSize * superBlocksPerRow; b++) + { + var block = blocks.AsSpan(b * Q6KWeight.SuperBlockBytes, Q6KWeight.SuperBlockBytes); + + for (var i = 0; i < block.Length; i++) + { + block[i] = (byte)rng.Next(256); + } + + var d = (Half)((rng.NextDouble() * 0.05) + 0.001); + BinaryPrimitives.WriteUInt16LittleEndian(block.Slice(208, 2), BitConverter.HalfToUInt16Bits(d)); + } + + var weight = new Q6KWeight(blocks, InputSize, OutputSize); + + var spr = weight.SuperBlocksPerRow; + var bsumsPerRow = spr * Q6KDotKernel.GroupsPerSuperBlock; + + var input = new float[cols * InputSize]; + + for (var i = 0; i < input.Length; i++) + { + input[i] = (float)(rng.NextDouble() * 2.0 - 1.0); + } + + var quants = new sbyte[cols * InputSize]; + var scales = new float[cols * spr]; + var bsums = new short[cols * bsumsPerRow]; + + for (var c = 0; c < cols; c++) + { + Q6KDotKernel.QuantizeActivationQ8K( + input.AsSpan(c * InputSize, InputSize), + quants.AsSpan(c * InputSize, InputSize), + scales.AsSpan(c * spr, spr), + bsums.AsSpan(c * bsumsPerRow, bsumsPerRow)); + } + + return (weight, quants, scales); + } + } +} From cc0a1c8778bf3719099efdfbf8cd479ddae179d7 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Wed, 22 Jul 2026 23:32:59 +0200 Subject: [PATCH 21/37] llama --- ROADMAP.md | 36 ++++++- .../Runtime/CachedAttentionKernel.cs | 96 ++++++++++++++++++- 2 files changed, 128 insertions(+), 4 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 4bc87510..b45058d5 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -762,8 +762,40 @@ AVX-512 has no `vphaddd` for zmm, adding three more cross-half moves per call ac lane-crossing traffic outruns the arithmetic saved. **A wider vector is not a property of the ISA alone; it is a ratio between broadcast cost and work done per broadcast, and that ratio is per-kernel.** -**Where prefill stands: 280 tok/s, gap 1.93×.** Remaining measured items: `attn_scores` 212 ms at 0.27 -TFLOP/s (12% of its ceiling, needs a new kernel), and the scalar `Unpack` at ~3.5%. +#### ★ attn_scores — register-resident value accumulation: −13% on the component + +The softmax-weighted value sum walked every `d` for each `t`, so it loaded **and stored** the whole output +accumulator once per `t`: 512 B of value read against 512 B of accumulator read plus 512 B written — two +thirds of the traffic was the accumulator round-tripping through L1, ~231 MB of ~347 MB per head-layer. +`AccumulateValuesBlocked` blocks `d` into 64 dimensions so eight accumulators stay in registers across the +whole `t` loop; the value stream is unchanged in volume, just read in two passes. Bit-identical — ascending +`t` order per `d` preserved, and the deliberate no-FMA property kept. + +ABAB-interleaved, three rounds, best-of-N: + +| component | baseline | blocked | | +|---|---:|---:|---:| +| **attn_scores** | 213.8 ms | **186.3** | **1.15×** | +| attn_kv / attn_q / attn_out (canaries) | 137.4 / 84.6 / 88.5 | 137.9 / 85.5 / 89.2 | 1.00 / 0.99 / 0.99× | +| ffn_gateup / ffn_down (canaries) | 1006.8 / 659.2 | 1012.8 / 662.6 | 0.99× | +| total | 2385.7 | 2366.6 | 1.01× | + +**End to end this is only +0.8%**, because attn_scores is 8% of prefill. A first single-arm run appeared to +show 280 → 292 tok/s, but `attn_kv` and `ffn_gateup` — neither touched by the change — moved with it, so that +reading was box drift and is withdrawn. Interleaving the arms with those components as canaries is what +separated the two. **Prefill stands at ~283 tok/s.** + +**On the .NET-vs-C++ gaps this work exposed.** Three are real: no `F16C` intrinsic class (nor a `Half` +overload of `Vector128.Widen`), no first-class AVX-512 mask registers, and no `restrict`. All are +dotnet/runtime JIT work, not something a library can supply — F16C in particular is a well-scoped ask with an +existing pattern to follow. `TensorPrimitives` is the right home for the subset expressible as *bulk* +buffer-to-buffer work, and does carry hardware paths not otherwise reachable; it did not fit here because the +values are eight at a time, interleaved every 1152 bytes inside a hot loop. The deeper difference is +optimisation budget — RyuJIT is a fast JIT, and Native AOT uses the same backend, so there is no LLVM-class +scheduling to reach for. **None of this explains the remaining gap**: the Q4_K matmul measured faster than +llama.cpp's at equal ISA and thread count. What is left is AVX-512 coverage and our own kernel structure. + +**Remaining measured item:** the scalar `Unpack` at ~3.5% of the Q4_K kernel. *Invalidated run, kept as a warning:* the first tile sweep ran inside an 11-benchmark class and reported `Tiled` and `Tiled_Cols8` — **the same configuration** — 21% apart, far outside their ±9% bars. Two identical diff --git a/Sources/Main/LanguageModels/Runtime/CachedAttentionKernel.cs b/Sources/Main/LanguageModels/Runtime/CachedAttentionKernel.cs index 10acae4b..210e4a38 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedAttentionKernel.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedAttentionKernel.cs @@ -98,12 +98,26 @@ public static void ComputeSingleHead( var invSum = 1f / sumExp; + // Fold the normalisation into the probabilities once, so the inner loops below read a plain + // coefficient. `scoreScratch[t] * invSum` computed here or there is the same product. for (var t = 0; t < sequenceLength; t++) { - var probability = scoreScratch[t] * invSum; + scoreScratch[t] *= invSum; + } + + var dStart = 0; + + if (CpuFeatures.HasAvx2 && UseRegisterResidentValueSum) + { + dStart = AccumulateValuesBlocked(values, scoreScratch, output, sequenceLength, headDimension); + } + + for (var t = 0; t < sequenceLength; t++) + { + var probability = scoreScratch[t]; var value = values.Slice(t * headDimension, headDimension); - var d = 0; + var d = dStart; if (CpuFeatures.HasAvx2) { // Vectorize over headDim. output[d] accumulates over t in ascending order @@ -127,6 +141,84 @@ public static void ComputeSingleHead( } } + /// A/B switch for ; set OVERFIT_ATTN_REGACC=0 to disable. + internal static bool UseRegisterResidentValueSum = + Environment.GetEnvironmentVariable("OVERFIT_ATTN_REGACC") != "0"; + + /// + /// The softmax-weighted value sum with the accumulators held in registers across the whole + /// t loop, processing 64 output dimensions at a time. Returns the first dimension it did + /// not cover, which the caller finishes with the original loop. + /// + /// What it removes. The straightforward order — for each t, walk every d — + /// loads and stores the whole output accumulator once per t. Per t that is 512 B of value + /// read against 512 B of accumulator read plus 512 B of accumulator write: two thirds of the traffic + /// is the accumulator going out to L1 and back. Across a 672-token prefill head-layer that is + /// ~226k iterations, ~347 MB moved of which ~231 MB is pure round-trip. Blocking d so the + /// accumulators stay in registers reduces that to one load and one store per block per query. + /// + /// The value stream is unchanged in volume — the d blocks partition each value row, so the + /// same bytes are read, just in two passes rather than one. Values fit L2 comfortably at these head + /// dimensions. + /// + /// Bit-identical: for every d the contributions are still summed in ascending + /// t order, and the multiply and add stay separate — the no-FMA property the surrounding method + /// documents is deliberate and preserved here. + /// + private static int AccumulateValuesBlocked( + ReadOnlySpan values, + ReadOnlySpan probabilities, + Span output, + int sequenceLength, + int headDimension) + { + const int Lanes = 8; + const int BlockWidth = 8 * Lanes; + + ref var o = ref MemoryMarshal.GetReference(output); + ref var v = ref MemoryMarshal.GetReference(values); + + var d0 = 0; + + for (; d0 + BlockWidth <= headDimension; d0 += BlockWidth) + { + var a0 = Vector256.Zero; + var a1 = Vector256.Zero; + var a2 = Vector256.Zero; + var a3 = Vector256.Zero; + var a4 = Vector256.Zero; + var a5 = Vector256.Zero; + var a6 = Vector256.Zero; + var a7 = Vector256.Zero; + + for (var t = 0; t < sequenceLength; t++) + { + var probV = Vector256.Create(probabilities[t]); + var b = (nuint)((long)t * headDimension + d0); + + a0 = Avx.Add(a0, Avx.Multiply(probV, Vector256.LoadUnsafe(ref v, b))); + a1 = Avx.Add(a1, Avx.Multiply(probV, Vector256.LoadUnsafe(ref v, b + 8))); + a2 = Avx.Add(a2, Avx.Multiply(probV, Vector256.LoadUnsafe(ref v, b + 16))); + a3 = Avx.Add(a3, Avx.Multiply(probV, Vector256.LoadUnsafe(ref v, b + 24))); + a4 = Avx.Add(a4, Avx.Multiply(probV, Vector256.LoadUnsafe(ref v, b + 32))); + a5 = Avx.Add(a5, Avx.Multiply(probV, Vector256.LoadUnsafe(ref v, b + 40))); + a6 = Avx.Add(a6, Avx.Multiply(probV, Vector256.LoadUnsafe(ref v, b + 48))); + a7 = Avx.Add(a7, Avx.Multiply(probV, Vector256.LoadUnsafe(ref v, b + 56))); + } + + a0.StoreUnsafe(ref o, (nuint)d0); + a1.StoreUnsafe(ref o, (nuint)(d0 + 8)); + a2.StoreUnsafe(ref o, (nuint)(d0 + 16)); + a3.StoreUnsafe(ref o, (nuint)(d0 + 24)); + a4.StoreUnsafe(ref o, (nuint)(d0 + 32)); + a5.StoreUnsafe(ref o, (nuint)(d0 + 40)); + a6.StoreUnsafe(ref o, (nuint)(d0 + 48)); + a7.StoreUnsafe(ref o, (nuint)(d0 + 56)); + } + + return d0; + } + /// /// Q8 KV-cache attend: identical math to but K and V are /// resident as per-position symmetric int8 (one F32 scale per cached vector), so the From 00d050ecb363c232b1414dfa8f8cc72cf487a88e Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Thu, 23 Jul 2026 00:03:53 +0200 Subject: [PATCH 22/37] llama --- ROADMAP.md | 77 +++++++- .../Q4KPrefillProjectionBenchmark.cs | 7 + .../Runtime/BatchedQuantProjection.cs | 5 +- .../Runtime/CachedMultiHeadAttention.cs | 67 ++++++- .../LanguageModels/Runtime/Q6KGemvKernel.cs | 171 +++++++----------- 5 files changed, 217 insertions(+), 110 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index b45058d5..90b22e06 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -744,7 +744,28 @@ pre-decoded scale path) asserting exact equality rather than a tolerance. Gated `CpuFeatures.HasAvx512`/`HasAvx512Bw` — the repo's own OVERFIT015 analyzer rejected a direct `IsSupported` check, which is what that rule is for. Suite 1494/0/229. -#### ▶ NEGATIVE — the same AVX-512 port for Q6_K is SLOWER, reverted +#### ★★ Q6_K AVX-512, SECOND ATTEMPT — pair what is already adjacent: +12% on `ffn_down` + +The failure below was diagnosed as broadcast traffic, not vector width, and that diagnosis held. `ql03`/`ql47` +are stored adjacently (`k*64` and `k*64+32`), as are `qhL03`/`qhL47` — so **one 512-bit load carries real data +in both halves and no weight broadcast is needed at all**. Activations come from a single `vpbroadcastq` +(`Vector512.Create(long)` replicates the 8-byte pattern), which is exactly the tiling the 256-bit path built +by hand from two Create calls. The only cross-half move left is one `GetUpper` per reduction, unavoidable +since AVX-512 has no `vphaddd` for zmm. Accumulators stay 256-bit, so register pressure is unchanged. + +ABAB-interleaved, three rounds: + +| component | 256-bit | 512-bit | | +|---|---:|---:|---:| +| **ffn_down** | 659.7 ms | **589.2** | **1.12×** | +| ffn_gateup / attn_scores / attn_kv / attn_q / attn_out (canaries) | 1017.1 / 187.7 / 138.0 / 84.9 / 89.2 | 1011.3 / 189.0 / 139.0 / 84.7 / 89.0 | 0.99–1.01× | +| total | 2373.6 | 2298.6 | **1.03×** | + +**Prefill 283 → 292 tok/s, gap 1.91× → 1.85×.** Bit-identical, `Avx512Q6KPrefillParityTests` 7/7, suite +1501/0/229. Same kernel, same instruction set, same shape — **only the choice of what shares a register** +turned −20% into +12%. + +#### ▶ NEGATIVE (superseded above) — column-pairing the Q6_K port is SLOWER `Q6KGemvKernel.GemmTiled512` exists and is bit-identical (`Avx512Q6KPrefillParityTests`, 7 cases), but `BatchedQuantProjection.UseAvx512PrefillQ6K` is **off**: on the same machine and prompt where the Q4_K port @@ -795,7 +816,59 @@ optimisation budget — RyuJIT is a fast JIT, and Native AOT uses the same backe scheduling to reach for. **None of this explains the remaining gap**: the Q4_K matmul measured faster than llama.cpp's at equal ISA and thread count. What is left is AVX-512 coverage and our own kernel structure. -**Remaining measured item:** the scalar `Unpack` at ~3.5% of the Q4_K kernel. +#### ▶ NEGATIVE — the unaccounted time holds no surprise; it is spread thin + +A claimed "~192 ms unaccounted" was an arithmetic error: it conflated time outside both blocks with time +inside attention that no sub-slice covers. The profiler's own top-level rows split it properly: + +| | time | share | +|---|---:|---:| +| **attention** (top level) | 585.2 ms | 24.6% | +| — sub-slices (`kv`+`q`+`scores`+`out`) | 504.0 | | +| — **unattributed inside attention** (RoPE, QK-norm, the whole-matrix Q/O gather+scatter) | **81.2** | **3.4%** | +| **ffn** (top level) | 1683.3 ms | 70.8% | +| — sub-slices (`gateup`+`down`) | 1683.1 | | +| — unattributed | **0.2** | **0%** | +| **other** (norms / residual / embed / final norm) | 109.4 | 4.6% | + +**FFN is 100% accounted**, which kills the hypothesis that SwiGLU's 266M `silu` calls were a hidden cost — +the activation lives inside `ffn_gateup` and is not separable at this granularity. The residual is genuinely +thin: halving *both* remaining pieces would buy ~4%. The work is in the large kernels, not hiding beside them. + +**What F16C would be worth now, if .NET exposed it: ~0.1%.** Ablation priced the F16 decode at 12% of the +Q4_K kernel, but hoisting already removed 83/84 of that work by decoding once per projection instead of once +per column tile. The missing instruction would speed up what remains; the restructuring deleted it. Worth +recording as the general shape: **a workaround that removes work beats an instruction that accelerates it**, +and having the instruction available would likely have stopped the search at 12%. Where F16C would still pay +is *model loading* — `GgufReader`, `GgmlDequant`, `SafetensorsReader` and the Whisper loader all widen halves +in scalar loops, hundreds of millions of values per 3B model — but that is startup, not inference. + +#### ★ whole-matrix K/V — one dispatch per projection: prefill 291 → 297 tok/s + +Per-group K/V projects `[dModel → headDim] = [2048 → 128]`, which a micro-bench put at **0.37 TFLOP/s**: +352 MFLOP is too little work to amortise the dispatch's fixed cost, and single-thread was only 1.9× slower +than the pool, so the 84-tile launch — not the matmul — dominates. Ceiling measured before building: two +narrow dispatches 3206 µs vs one wide `[2048 → 256]` 1768 µs = **1.81×** on the projection. Built it: +project all KV heads through `WkWhole`/`WvWhole` once, gather each group's band (adding the per-KV-head bias +in the copy). Gated on both whole handles being Q4_K, so Q6_K `attn_v` layers fall back to per-group. + +ABAB, canaries flat: **attn_kv 139.1 → 96.4 ms (1.44×)**. End-to-end, six paired rounds: + +| | median | min | +|---|---:|---:| +| per-group | 2311 ms / 291 tok/s | 2298 / 292 | +| **whole** | **2262 / 297** | **2251 / 299** | + +**+2.2% e2e**, matching the 1.81×-on-projection ceiling. Bit-identical (each output row's dot product is +unchanged; no reassociation), no parity gate, suite 1501/0/229, `OVERFIT_WHOLE_KV=0` disables. + +*Methodology note kept as a warning:* the first component table read total as 1.00× while attn_kv clearly +dropped — an artifact of taking each component's min from a different run, so total-min and attn_kv-min came +from different rounds. A paired total-only measurement resolved it. **Best-of-N per component does not give a +consistent end-to-end number; measure total paired.** + +**Remaining measured item:** the scalar `Unpack` at ~3.5% of the Q4_K kernel; and `attn_scores` at 187 ms / +0.27 TFLOP/s, which needs a flash-attention-style blocked kernel rather than a loop change. *Invalidated run, kept as a warning:* the first tile sweep ran inside an 11-benchmark class and reported `Tiled` and `Tiled_Cols8` — **the same configuration** — 21% apart, far outside their ±9% bars. Two identical diff --git a/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs b/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs index c9e70e70..a8ec6180 100644 --- a/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs +++ b/Sources/Benchmark/Q4KPrefillProjectionBenchmark.cs @@ -87,6 +87,13 @@ private static (int InputSize, int OutputSize) ShapeOf(string shape) // count (32) - the only like-for-like kernel comparison available without editing their tests. "llama_ref" => (14336, 4096), + // Per-KV-head K or V projection (dModel 2048 -> headDim 128) vs the whole-matrix form for a + // 2-KV-head GQA model (2048 -> 256). The narrow shape is where attn_kv measured 0.37 TFLOP/s: + // 352 MFLOP is too little work to amortise the dispatch's fixed cost (repack check, scale + // decode, 84-tile launch). This pair measures whether widening the output recovers it. + "kv_head" => (2048, 128), + "kv_whole" => (2048, 256), + _ => (2048, 2048), }; } diff --git a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs index e1f7a0c3..231c6873 100644 --- a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs +++ b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs @@ -117,7 +117,10 @@ internal static class BatchedQuantProjection /// (no vphaddd for zmm), adding three more cross-half moves per call, 32 calls per block. The /// lane-crossing traffic outruns the arithmetic saved. /// - internal static bool UseAvx512PrefillQ6K; + internal static bool UseAvx512PrefillQ6K = + CpuFeatures.HasAvx512 + && CpuFeatures.HasAvx512Bw + && Environment.GetEnvironmentVariable("OVERFIT_AVX512_Q6K") != "0"; /// /// Weight bytes one worker's band may occupy. Half of a 1 MB Zen-5 L2, leaving the rest for the diff --git a/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs b/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs index c7750b65..0d508403 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs @@ -570,6 +570,14 @@ internal void DecodeBatched( } } + /// + /// Project all KV heads in one dispatch instead of one per group; A/B via OVERFIT_WHOLE_KV=0. + /// See the measurement at the projection site (1.81× on the projection, 0.37 TFLOP/s per-group is + /// launch-bound). Bit-identical, so it is on by default where the whole handles are Q4_K. + /// + internal static bool UseWholeKv = + Environment.GetEnvironmentVariable("OVERFIT_WHOLE_KV") != "0"; + /// /// Batched (prefill) multi-head attention for the Llama/Qwen quantized path — the /// multi-row counterpart of that supports RoPE + GQA + quantized weights @@ -703,6 +711,33 @@ internal void DecodeBatchedQuant( PrefillProfiler.Stop(PrefillProfiler.Component.AttnQ, profWholeQ); } + // Whole-matrix K/V: the per-group projection is [dModel -> headDim] = [2048 -> 128], and a + // micro-bench put that at 0.37 TFLOP/s — 352 MFLOP is too little work to amortise a dispatch's + // fixed cost (repack check, scale decode, the 84-tile parallel launch), and single-thread was only + // 1.9x slower than the pool, i.e. the launch, not the matmul, dominates. Projecting all KV heads at + // once ([dModel -> kvHeads*headDim]) doubles the work per launch: measured 1.81x on the projection + // (2x narrow 3206 us vs 1x wide 1768 us). Gated on the whole K AND V handles both being Q4_K, so + // Q6_K attn_v layers fall back to per-group. Bias and QK-norm still applied per group below, so the + // result is unchanged from the per-group path (deterministic reduction, no reassociation). + var kvDim = KvHeadCount * headDim; + var useWholeKv = weights.WkWhole.IsQ4K && weights.WvWhole.IsQ4K && UseWholeKv; + using var kAll = new PooledBuffer(useWholeKv ? rows * kvDim : 0, clearMemory: false); + using var vAll = new PooledBuffer(useWholeKv ? rows * kvDim : 0, clearMemory: false); + + if (useWholeKv) + { + var wholeK = weights.WkWhole; // property returns by value - needs a local to pass by `in` + var wholeV = weights.WvWhole; + var profWholeKv = PrefillProfiler.Start(); + BatchedQuantProjection.Dispatch( + hidden, rows, in wholeK, [], kAll.Span.Slice(0, rows * kvDim), + dModel, kvDim, hQuants, hScales, hBsums); + BatchedQuantProjection.Dispatch( + hidden, rows, in wholeV, [], vAll.Span.Slice(0, rows * kvDim), + dModel, kvDim, hQuants, hScales, hBsums); + PrefillProfiler.Stop(PrefillProfiler.Component.AttnKv, profWholeKv); + } + for (var group = 0; group < KvHeadCount; group++) { // K/V weights: GQA shares one KV head per group; MHA uses the head's own. @@ -730,10 +765,34 @@ internal void DecodeBatchedQuant( // K/V projected once per group, RoPE-rotated, stored — every Q head reads the cache. var profKv = PrefillProfiler.Start(); - BatchedQuantProjection.Dispatch( - hidden, rows, in wk, bk, kg.Span, dModel, headDim, hQuants, hScales, hBsums); - BatchedQuantProjection.Dispatch( - hidden, rows, in wv, bv, vg.Span, dModel, headDim, hQuants, hScales, hBsums); + if (useWholeKv) + { + // Gather this group's band out of the whole projection; the bias lives per KV head and is + // added here, exactly as the per-group dispatch would have applied it. + for (var n = 0; n < rows; n++) + { + var dstK = kg.Span.Slice(n * headDim, headDim); + var dstV = vg.Span.Slice(n * headDim, headDim); + kAll.Span.Slice(n * kvDim + group * headDim, headDim).CopyTo(dstK); + vAll.Span.Slice(n * kvDim + group * headDim, headDim).CopyTo(dstV); + if (!bk.IsEmpty) + { + TensorPrimitives.Add(dstK, bk.Slice(0, headDim), dstK); + } + if (!bv.IsEmpty) + { + TensorPrimitives.Add(dstV, bv.Slice(0, headDim), dstV); + } + } + } + + if (!useWholeKv) + { + BatchedQuantProjection.Dispatch( + hidden, rows, in wk, bk, kg.Span, dModel, headDim, hQuants, hScales, hBsums); + BatchedQuantProjection.Dispatch( + hidden, rows, in wv, bv, vg.Span, dModel, headDim, hQuants, hScales, hBsums); + } PrefillProfiler.Stop(PrefillProfiler.Component.AttnKv, profKv); if (weights.HasQkNorm) { diff --git a/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs b/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs index 60e34fc0..34761e50 100644 --- a/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs +++ b/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs @@ -450,20 +450,28 @@ public static unsafe void GemmTiled( } /// - /// AVX-512 form of , structured exactly like - /// Q4KGemvKernel.GemmTiled512: two activation columns per instruction, column 2p - /// in the low 256 bits and 2p+1 in the high, with the shared weights broadcast into both halves - /// and the pair loop left innermost so the per-block weight decode stays amortised across the tile. + /// AVX-512 form of that widens along the weight layout rather than + /// across activation columns. /// - /// The one place it cannot widen. ends in vphaddd, which - /// AVX-512 does not provide for zmm at all. The two vpmaddwd steps still run at 512; only the - /// horizontal add and its permute drop to 256-bit halves and are re-joined. That costs one extra - /// instruction per reduction against eighteen saved elsewhere in the same iteration — the arithmetic - /// bulk (eight vpmaddubsw, four subtracts, the scale multiplies) all pairs cleanly. + /// Why not column pairing. The first attempt put column 2p in the low half and + /// 2p+1 in the high, which meant every weight vector had to be broadcast into both halves with a + /// vinserti64x4. Q6_K issues six such broadcasts per k, sixteen times per block, for far + /// less arithmetic each than Q4_K does — the lane-crossing traffic outran the arithmetic saved and the + /// kernel measured 20% slower. That version is recorded as a negative in ROADMAP. /// - /// Bit-identical to : each column's operations and their order are - /// unchanged, including the split reduction, which is the same 256-bit sequence applied to the same - /// values. + /// What this does instead. ql03 and ql47 are already adjacent in the repacked + /// block — at k*64 and k*64+32 — and so are qhL03/qhL47. One 512-bit load + /// therefore carries real data in both halves, with no broadcast at all, and the halves stay + /// independent all the way through vpmaddubsw and the subtract. Activations arrive as a single + /// vpbroadcastq — Vector512.Create(long) replicates the 8-byte pattern eight times, which + /// is exactly the tiling the 256-bit path built by hand. + /// + /// The only cross-half move left is one GetUpper per reduction, because AVX-512 provides no + /// vphaddd for zmm. Accumulators stay 256-bit, so register pressure is unchanged. + /// + /// Bit-identical to : the 512-bit operations act lanewise on exactly + /// the values the two 256-bit operations did, and the horizontal add receives the same + /// (m03, m47) pair it always did. /// public static unsafe void GemmTiled512( ReadOnlySpan repacked, @@ -481,7 +489,6 @@ public static unsafe void GemmTiled512( } var nb = inputSize / 256; - var pairs = (cols + 1) / 2; var m4b = Vector512.Create((byte)0x0F); var m2 = Vector512.Create((byte)0x03); @@ -489,8 +496,8 @@ public static unsafe void GemmTiled512( var ones = Vector512.Create((short)1); var reduce = Vector256.Create(0, 1, 4, 5, 2, 3, 6, 7); - Span> sumf = stackalloc Vector512[pairs]; - Span> iacc = stackalloc Vector512[pairs]; + Span> sumf = stackalloc Vector256[cols]; + Span> iacc = stackalloc Vector256[cols]; fixed (byte* rep = repacked) fixed (sbyte* aqAll = actQuants) @@ -502,9 +509,9 @@ public static unsafe void GemmTiled512( { var bptr = rep + (long)x * nb * BlockKx8Bytes; - for (var p = 0; p < pairs; p++) + for (var c = 0; c < cols; c++) { - sumf[p] = Vector512.Zero; + sumf[c] = Vector256.Zero; } for (var l = 0; l < nb; l++) @@ -514,14 +521,13 @@ public static unsafe void GemmTiled512( var ql = blk + DstQlOffset; var qh = blk + DstQhOffset; - var d256 = dsc is not null + var dVec = dsc is not null ? Vector256.Load(dsc + (((long)x * nb) + l) * DecodedScalesPerBlock) : LoadF16x8Int(blk); - var dVec = Vector512.Create(d256, d256); - for (var p = 0; p < pairs; p++) + for (var c = 0; c < cols; c++) { - iacc[p] = Vector512.Zero; + iacc[c] = Vector256.Zero; } for (var k = 0; k < 16; k++) @@ -535,84 +541,51 @@ public static unsafe void GemmTiled512( var qhBlockL = ((qhHalfL + (baseL % 32)) / 8) * 64; var qhBlockH = ((qhHalfH + (baseH % 32)) / 8) * 64; - // Weight side: decoded once, shared by both columns of every pair. - var ql03 = Broadcast512(ql + k * 64); - var ql47 = Broadcast512(ql + k * 64 + 32); - var qhL03 = Broadcast512(qh + qhBlockL); - var qhL47 = Broadcast512(qh + qhBlockL + 32); - var qhH03 = Broadcast512(qh + qhBlockH); - var qhH47 = Broadcast512(qh + qhBlockH + 32); - - var qLu03 = LoNib512(ql03, m4b) | QhBits512(qhL03, qhShiftL, m2); - var qLu47 = LoNib512(ql47, m4b) | QhBits512(qhL47, qhShiftL, m2); - var qHu03 = HiNib512(ql03, m4b) | QhBits512(qhH03, qhShiftH, m2); - var qHu47 = HiNib512(ql47, m4b) | QhBits512(qhH47, qhShiftH, m2); - - var sl = ScaleVec(scales + (baseL / 16) * 8); - var sh = ScaleVec(scales + (baseH / 16) * 8); - var scaleL = Vector512.Create(sl, sl); - var scaleH = Vector512.Create(sh, sh); - - for (var p = 0; p < pairs; p++) + // One load each: rows 0-3 in the low half, rows 4-7 in the high half, as stored. + var ql0347 = Vector512.Load(ql + k * 64); + var qhL0347 = Vector512.Load(qh + qhBlockL); + var qhH0347 = Vector512.Load(qh + qhBlockH); + + var qLu = LoNib512(ql0347, m4b) | QhBits512(qhL0347, qhShiftL, m2); + var qHu = HiNib512(ql0347, m4b) | QhBits512(qhH0347, qhShiftH, m2); + + var scaleL = ScaleVec(scales + (baseL / 16) * 8); + var scaleH = ScaleVec(scales + (baseH / 16) * 8); + + for (var c = 0; c < cols; c++) { - var lowCol = 2 * p; - var highCol = Math.Min(2 * p + 1, cols - 1); - var aLow = aqAll + (long)lowCol * inputSize + l * 256; - var aHigh = aqAll + (long)highCol * inputSize + l * 256; + var aqs = aqAll + (long)c * inputSize + l * 256; + var actL = TileAct512(aqs + baseL); + var actH = TileAct512(aqs + baseH); - var actL = TileAct512(aLow + baseL, aHigh + baseL); - var actH = TileAct512(aLow + baseH, aHigh + baseH); + var pL = Avx512BW.Subtract(Mul512(qLu, actL), Mul512(m32, actL)); + var pH = Avx512BW.Subtract(Mul512(qHu, actH), Mul512(m32, actH)); - var sumL = ReduceRows512( - Avx512BW.Subtract(Mul512(qLu03, actL), Mul512(m32, actL)), - Avx512BW.Subtract(Mul512(qLu47, actL), Mul512(m32, actL)), - ones, reduce); - var sumH = ReduceRows512( - Avx512BW.Subtract(Mul512(qHu03, actH), Mul512(m32, actH)), - Avx512BW.Subtract(Mul512(qHu47, actH), Mul512(m32, actH)), - ones, reduce); + var sumL = ReduceRows512(pL, ones, reduce); + var sumH = ReduceRows512(pH, ones, reduce); - iacc[p] = Avx512F.Add(iacc[p], Avx512F.Add( - Avx512F.MultiplyLow(sumL, scaleL), Avx512F.MultiplyLow(sumH, scaleH))); + iacc[c] = Avx2.Add(iacc[c], Avx2.Add( + Avx2.MultiplyLow(sumL, scaleL), Avx2.MultiplyLow(sumH, scaleH))); } } - for (var p = 0; p < pairs; p++) + for (var c = 0; c < cols; c++) { - var rowScale = Vector512.Create( - Vector256.Create(asc[(long)(2 * p) * nb + l]), - Vector256.Create(asc[(long)Math.Min(2 * p + 1, cols - 1) * nb + l])); - - sumf[p] = Avx512F.FusedMultiplyAdd( - Avx512F.ConvertToVector512Single(iacc[p]), - Avx512F.Multiply(dVec, rowScale), - sumf[p]); + sumf[c] = Fma.MultiplyAdd( + Avx.ConvertToVector256Single(iacc[c]), + Avx.Multiply(dVec, Vector256.Create(asc[(long)c * nb + l])), + sumf[c]); } } - for (var p = 0; p < pairs; p++) + for (var c = 0; c < cols; c++) { - sumf[p].GetLower().Store(outp + (long)(2 * p) * outputSize + x * 8); - - // The odd tail duplicated its low column into the high half; discard that copy. - if (2 * p + 1 < cols) - { - sumf[p].GetUpper().Store(outp + (long)(2 * p + 1) * outputSize + x * 8); - } + sumf[c].Store(outp + (long)c * outputSize + x * 8); } } } } - /// The same 32 weight bytes in both halves — weights are shared by the two columns of a pair. - [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static unsafe Vector512 Broadcast512(byte* p) - { - var v = Vector256.Load(p); - - return Vector512.Create(v, v); - } - [MethodImpl(MethodImplOptions.AggressiveInlining)] private static Vector512 LoNib512(Vector512 v, Vector512 m4b) => v & m4b; @@ -633,34 +606,26 @@ private static Vector512 QhBits512(Vector512 qh, byte shift, Vector5 private static Vector512 Mul512(Vector512 rhs, Vector512 lhs) => Avx512BW.MultiplyAddAdjacent(rhs, lhs); - /// One column's eight activation bytes per half, duplicated in both 128-bit lanes of that half. + /// + /// Eight activation bytes replicated across all 64 lanes by a single vpbroadcastq — the same + /// tiling assembles from two Create calls, at no cross-half cost. + /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static unsafe Vector512 TileAct512(sbyte* low, sbyte* high) - { - var lo = Unsafe.ReadUnaligned(low); - var hi = Unsafe.ReadUnaligned(high); - var vl = Vector128.Create(lo, lo).AsSByte(); - var vh = Vector128.Create(hi, hi).AsSByte(); - - return Vector512.Create(Vector256.Create(vl, vl), Vector256.Create(vh, vh)); - } + private static unsafe Vector512 TileAct512(sbyte* a) => + Vector512.Create(Unsafe.ReadUnaligned(a)).AsSByte(); /// - /// widened as far as the instruction set allows: both vpmaddwd steps run - /// at 512 bits, then the horizontal add and its permute drop to 256-bit halves because AVX-512 has no - /// vphaddd for zmm. Each half is the identical 256-bit sequence, so the result is bit-identical. + /// over a vector holding [p03 | p47]. The vpmaddwd runs at 512 + /// bits; the horizontal add then takes the two halves, which is exactly the (m03, m47) pair the + /// 256-bit version passes, so the result is bit-identical. /// [MethodImpl(MethodImplOptions.AggressiveInlining)] - private static Vector512 ReduceRows512( - Vector512 p03, Vector512 p47, Vector512 ones, Vector256 reduce) + private static Vector256 ReduceRows512( + Vector512 p0347, Vector512 ones, Vector256 reduce) { - var m03 = Avx512BW.MultiplyAddAdjacent(p03, ones); - var m47 = Avx512BW.MultiplyAddAdjacent(p47, ones); - - var lower = Avx2.PermuteVar8x32(Avx2.HorizontalAdd(m03.GetLower(), m47.GetLower()), reduce); - var upper = Avx2.PermuteVar8x32(Avx2.HorizontalAdd(m03.GetUpper(), m47.GetUpper()), reduce); + var m = Avx512BW.MultiplyAddAdjacent(p0347, ones); - return Vector512.Create(lower, upper); + return Avx2.PermuteVar8x32(Avx2.HorizontalAdd(m.GetLower(), m.GetUpper()), reduce); } [MethodImpl(MethodImplOptions.AggressiveInlining)] From 1f6cf0b99d5f6399700083175ba8ade60d1a2a5c Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Thu, 23 Jul 2026 00:21:32 +0200 Subject: [PATCH 23/37] llama --- ROADMAP.md | 28 +++++++++++- .../Runtime/CachedAttentionKernel.cs | 43 ++++++++++++++++--- 2 files changed, 64 insertions(+), 7 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 90b22e06..5d67af7a 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -867,8 +867,32 @@ dropped — an artifact of taking each component's min from a different run, so from different rounds. A paired total-only measurement resolved it. **Best-of-N per component does not give a consistent end-to-end number; measure total paired.** -**Remaining measured item:** the scalar `Unpack` at ~3.5% of the Q4_K kernel; and `attn_scores` at 187 ms / -0.27 TFLOP/s, which needs a flash-attention-style blocked kernel rather than a loop change. +#### ▶ attn_scores, split by ablation — and why flash-attention is the WRONG lever + +Before writing a blocked kernel, ablation inside the real kernel split the 189.4 ms three ways: + +| removed | attn_scores | share | +|---|---:|---:| +| Q·Kᵀ dot | 137.7 ms | **27%** | +| softmax exp | 128.0 ms | **32%** | +| both | 60.7 ms | rest **32%** | + +**Neither dominates, and exp is the larger of the two.** A flash-attention rewrite only attacks the dot +(27% of the component = 2.3% of prefill) — the most expensive, highest-risk change aimed at the smaller +piece. Dropped. The exp is the better target and is a *bulk contiguous buffer*, exactly the shape +`TensorPrimitives` serves — the same lever SwiGLU already took (`ApplySiLU` → `TensorPrimitives.Sigmoid`). + +Replacing the scalar `MathF.Exp` loop with `TensorPrimitives.Subtract`/`Exp`/`Sum`: **attn_scores 189.4 → +173.2 ms (1.09×)**, e2e 297 → 299 tok/s (vector won all six paired rounds). Smaller than the 32% ablation +because `TensorPrimitives.Exp` is not free and the fused scalar loop became three passes over a short buffer; +exp itself went ~61 → ~45 ms. Not byte-parity vs the F32 reference (few-ULP, coherence-safe like SwiGLU), but +prefill and decode both reach this method so they stay bit-identical to each other — parity suite green, +1501/0/229. `OVERFIT_ATTN_VEXP=0` disables. + +**attn_scores is now optimised across all three parts** (value sum register-resident, exp vectorised, the dot +is what remains). Further gains need the flash-GEMM for the dot — ~2% e2e at high risk, not worth it now. + +**Remaining measured item:** the scalar `Unpack` at ~3.5% of the Q4_K kernel. *Invalidated run, kept as a warning:* the first tile sweep ran inside an 11-benchmark class and reported `Tiled` and `Tiled_Cols8` — **the same configuration** — 21% apart, far outside their ±9% bars. Two identical diff --git a/Sources/Main/LanguageModels/Runtime/CachedAttentionKernel.cs b/Sources/Main/LanguageModels/Runtime/CachedAttentionKernel.cs index 210e4a38..a7b59d33 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedAttentionKernel.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedAttentionKernel.cs @@ -3,6 +3,7 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com +using System.Numerics.Tensors; using System.Runtime.InteropServices; using System.Runtime.Intrinsics; using System.Runtime.Intrinsics.X86; @@ -65,7 +66,10 @@ public static void ComputeSingleHead( for (var t = 0; t < sequenceLength; t++) { var key = keys.Slice(t * headDimension, headDimension); - var score = Dot(query, key) * scale; + + // AblateScoreDot replaces the query·key GEMV with a cheap constant, to weigh the dot against + // the exp below. Measurement only, never a production path. + var score = (AblateScoreDot ? key[0] : Dot(query, key)) * scale; if (softcap > 0f) // Gemma-2 attn logit soft-cap: tanh(s/cap)·cap { score = MathF.Tanh(score * invCap) * softcap; @@ -81,11 +85,28 @@ public static void ComputeSingleHead( var sumExp = 0.0f; - for (var t = 0; t < sequenceLength; t++) + if (UseVectorizedSoftmaxExp && !AblateSoftmaxExp) { - var exp = MathF.Exp(scoreScratch[t] - maxScore); - scoreScratch[t] = exp; - sumExp += exp; + // Vectorized softmax exp — the same lever SwiGLU already took (ApplySiLU): the scalar + // per-element MathF.Exp was ~32% of attn_scores by ablation. scoreScratch[0..seqLen] is a + // contiguous L1-resident buffer, exactly the bulk shape TensorPrimitives serves. Differs a few + // ULP from scalar, so NOT byte-parity against the F32 reference — but prefill and decode both + // reach this same method, so they stay bit-identical to EACH OTHER (the parity tests compare + // the two paths, not against a stored scalar-exp value). + var scores = scoreScratch.Slice(0, sequenceLength); + TensorPrimitives.Subtract(scores, maxScore, scores); + TensorPrimitives.Exp(scores, scores); + sumExp = TensorPrimitives.Sum(scores); + } + + if (!(UseVectorizedSoftmaxExp && !AblateSoftmaxExp)) + { + for (var t = 0; t < sequenceLength; t++) + { + var exp = AblateSoftmaxExp ? scoreScratch[t] - maxScore : MathF.Exp(scoreScratch[t] - maxScore); + scoreScratch[t] = exp; + sumExp += exp; + } } if (sumExp <= 0f || float.IsNaN(sumExp) || float.IsInfinity(sumExp)) @@ -145,6 +166,18 @@ public static void ComputeSingleHead( internal static bool UseRegisterResidentValueSum = Environment.GetEnvironmentVariable("OVERFIT_ATTN_REGACC") != "0"; + private static readonly string AblateMode = Environment.GetEnvironmentVariable("OVERFIT_ATTN_ABLATE") ?? "none"; + + /// Measurement-only: replace the query·key dot with a constant, to size it against the exp. + internal static bool AblateScoreDot = AblateMode is "dot" or "both"; + + /// Measurement-only: skip the softmax exp, to size it against the query·key dot. + internal static bool AblateSoftmaxExp = AblateMode is "exp" or "both"; + + /// Vectorize the softmax exp via TensorPrimitives; set OVERFIT_ATTN_VEXP=0 to disable. + internal static bool UseVectorizedSoftmaxExp = + Environment.GetEnvironmentVariable("OVERFIT_ATTN_VEXP") != "0"; + /// /// The softmax-weighted value sum with the accumulators held in registers across the whole /// t loop, processing 64 output dimensions at a time. Returns the first dimension it did From 2c958afb417a1e985a1195f06336b4e8fb525fed Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Thu, 23 Jul 2026 00:22:51 +0200 Subject: [PATCH 24/37] llama --- Directory.Packages.props | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Directory.Packages.props b/Directory.Packages.props index 92ed4afd..855a4d48 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -21,8 +21,8 @@ rules silently off). Expect this package to show as permanently "outdated"; that is INTENTIONAL, not debt. Bump it only when the SDK's own Roslyn version moves up. --> - - + + From 9e1579f80475a37813a08d1f236bdba30e8fab50 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Thu, 23 Jul 2026 00:55:35 +0200 Subject: [PATCH 25/37] llama --- ROADMAP.md | 72 ++++++++++++- .../Benchmark/DecodeGemvRooflineBenchmark.cs | 102 ++++++++++++++++++ 2 files changed, 171 insertions(+), 3 deletions(-) create mode 100644 Sources/Benchmark/DecodeGemvRooflineBenchmark.cs diff --git a/ROADMAP.md b/ROADMAP.md index 5d67af7a..948c4780 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -22,7 +22,7 @@ Zero-allocation, pure C# deep-learning framework targeting high-performance CPU | Native C# GGUF loader (F32/F16/BF16/Q8_0/Q4_K/Q6_K) | ✅ Loads `*.gguf` from Ollama/HF directly | | Streaming token generation (`IAsyncEnumerable`) | ✅ Stable, with stop-tokens + cancellation | | LoRA adapter (Enable/Disable, Save/Load) | ✅ Stable, zero-copy weight refs | -| **Quantized weight storage at inference time** | ✅ **Q8_0 + Q4_K_M decode paths done & parity-verified — Qwen2.5-3B Q4_K_M decodes ~19 tok/s @ 3.20 GB RAM, 1 B/token (post-mmap, 2026-05-21). Same-file A/B vs LLamaSharp/llama.cpp: ~1.5× faster on raw tok/s (~29 vs ~19), RAM parity (3.20 GB both), Overfit wins on per-token allocation (1 B vs 21 220 B). Catch-up plan in "Decode throughput catch-up vs llama.cpp" section below.** | +| **Quantized weight storage at inference time** | ✅ **Q8_0 + Q4_K_M decode & prefill paths done & parity-verified.** Decode ~24 tok/s Qwen-3B Q4_K_M, memory-bound (GEMV kernel at 82% of DRAM ceiling — `DecodeGemvRooflineBenchmark`), 1.13× behind llama.cpp. **Prefill ~299 tok/s (pp672), ~1.81× behind their AVX-512 build / 1.14× behind AVX2**, after the AVX-512 Q4_K/Q6_K prefill kernels. Both compute-side perf tracks CLOSED by measurement — see "✅ CLOSED — CPU PREFILL + DECODE PERF TRACK". | | Mixture-of-Experts inference (Qwen-MoE, Mixtral-8x7B) | ✅ Coherent in pure C# (Q8_0 + Q4_K_M); verified "Paris" 2026-05-27 | | Training: gradient checkpointing | ✅ `ComputationGraph.Checkpoint` + `CheckpointedModule` — 24× live-activation cut on 12L GPT-1 | | Training: data parallelism (N replicas) | ✅ `DataParallelTrainer` / `DataParallelSession` + thread-budget fix — ~6× throughput (24 workers) | @@ -77,7 +77,47 @@ So in-place rewrites are free and **the only real risk is extracting a method**. --- -## ▶ NEXT UP — PREFILL. Measured 3.76× behind llama.cpp, and it is compute-bound +## ✅ CLOSED — CPU PREFILL + DECODE PERF TRACK (2026-07-22 → 2026-07-23) + +**Both compute-side performance tracks are closed, by measurement rather than assertion.** The detailed, +chronological record is preserved below — every win, every reverted negative, and the measurement discipline +that produced them. The headline: + +| | start of track | end of track | +|---|---:|---:| +| **prefill** | 143 tok/s (3.76× behind llama.cpp AVX-512) | **~299 tok/s (~1.81×)** — 1.14× to their AVX2 build | +| **decode** | — | **memory-bound, 1.13× behind**, GEMV kernel at 82% of the DRAM ceiling | + +**What shipped this track** (all bit-identical or coherence-safe, pinned by parity tests): Q6_K tiled GEMM, +shared activation quantization, whole-matrix O / Q / K/V projections, register-tiled prefill kernels, the +F16-scale hoist (decode once per projection, not once per column tile), **AVX-512 Q4_K and Q6_K prefill +kernels** (pair what is already adjacent in memory — the choice of what shares a register turned a −20% +port into +12%), register-resident attention value accumulation, and vectorized softmax exp. + +**Why it is closed:** +- **Prefill.** Our Q4_K matmul measured *faster* than llama.cpp's at equal ISA and thread count (1.70 vs + 1.56 TFLOP/s), so the remaining gap is AVX-512 coverage (now largely done) and kernel structure (done). + What is left — a flash-attention GEMM for the `attn_scores` dot (~2% e2e), the scalar `Unpack` (~3.5% of + one kernel) — is high-effort, low-return. +- **Decode.** `DecodeGemvRooflineBenchmark` showed the kernel's compute runs at 132.5 GB/s hot, *above* the + 90 GB/s DRAM ceiling, and 82% of it when streaming from DRAM — so decode is memory-bound and an AVX-512 + decode kernel cannot help (the direct measurement behind the reverted decode-port negative). The whole-model + shortfall is per-token overhead and layer→layer serial latency, where we are already 1.13× of llama.cpp. + +**The most valuable output was the measurement discipline** — roughly eleven mechanism hypotheses refuted, +the rules that survived recorded in the `feedback-measurement-discipline` memory, and permanent infrastructure +left behind: `MachineRooflineBenchmark`, `DecodeGemvRooflineBenchmark`, `Diagnostics/Throughput.cs`, and the +BenchmarkDotNet throughput columns. + +**Next move is a business decision (perf course vs Redaction Gateway), not another kernel.** Any further perf +work should measure the ceiling before writing code — the discipline that made this track pay. + +--- + +
+▼ Full chronological record of the perf track (preserved) + +### PREFILL — starting point: measured 3.76× behind llama.cpp, compute-bound **Measured 2026-07-22, same file (`qwen.q4km.gguf`), same 672-token prompt, best configuration on both sides:** @@ -892,6 +932,29 @@ prefill and decode both reach this method so they stay bit-identical to each oth **attn_scores is now optimised across all three parts** (value sum register-resident, exp vectorised, the dot is what remains). Further gains need the flash-GEMM for the dot — ~2% e2e at high risk, not worth it now. +#### ★ DECODE IS MEMORY-BOUND — measured directly, AVX-512 cannot help + +`DecodeGemvRooflineBenchmark` runs the production decode GEMV (`GemvParallel`, AVX2) on a Q4_K FFN weight at +two sizes — one that fits this box's 128 MB L3, one that does not — to separate the kernel's compute rate +from the memory rate it is fed: + +| weight | source | GB/s | +|---|---|---:| +| 12.7 MB (fits L3) | hot cache | **132.5** | +| 203 MB (exceeds L3) | DRAM | **73.5** | +| DRAM read ceiling | — | ~90 | + +**The kernel's compute (132.5 GB/s hot) is well above the DRAM ceiling (90)**, so the dequant consumes bytes +faster than DRAM delivers them: decode is not compute-bound, and an AVX-512 / VNNI decode kernel cannot help. +This is the direct measurement behind the earlier reverted "AVX-512 decode port" negative. Streaming from +DRAM the GEMV hits **73.5 GB/s = 82% of the ceiling** — the kernel itself is near-optimal. + +The whole-model decode figure (~46 GB/s) is well below the isolated GEMV's 73.5, so that shortfall is **not** +the weight kernel — it is per-token overhead (attention over the growing KV cache, RoPE, norms, sampling) and +the serial layer→layer dependency that leaves memory idle between GEMVs. That is an overlap/latency problem, +not a compute one, and we are already at 1.13× of llama.cpp there. **Decode's compute levers are exhausted, +by measurement.** + **Remaining measured item:** the scalar `Unpack` at ~3.5% of the Q4_K kernel. *Invalidated run, kept as a warning:* the first tile sweep ran inside an 11-benchmark class and reported @@ -990,10 +1053,13 @@ projection with a micro-bench against `sgemm.cpp` before writing any kernel. `ProjectBatched` (re-decode per row), **not** against weight-stationary. Recorded in `CLAUDE.md`. Decode is ~88% quantized GEMV sitting at the DRAM floor (`ffn 69.3% · attention 19.3% · lm_head 10.3%`), so -there is no cheap **decode** kernel win left — see the prefill section above for the path that *is* open. +there is no cheap **decode** kernel win left — this was later confirmed directly by `DecodeGemvRooflineBenchmark` +(kernel compute above the DRAM ceiling; see the closing summary at the top of this track). The product direction (perf course vs. the on-prem commercial track) remains deferred and is a separate, non-technical decision. +
+ --- ## Agentic / interop / vision backlog (2026-06-21) diff --git a/Sources/Benchmark/DecodeGemvRooflineBenchmark.cs b/Sources/Benchmark/DecodeGemvRooflineBenchmark.cs new file mode 100644 index 00000000..493a554f --- /dev/null +++ b/Sources/Benchmark/DecodeGemvRooflineBenchmark.cs @@ -0,0 +1,102 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Running; +using Benchmarks.Helpers; +using DevOnBike.Overfit.LanguageModels.Loading; +using DevOnBike.Overfit.LanguageModels.Runtime; + +namespace Benchmarks +{ + /// + /// Settles one question for the decode path: is the Q4_K GEMV bound by raw memory bandwidth or by + /// dequantization compute? + /// + /// Decode reads the entire weight matrix once per token and does ~one MAC per weight byte, so its + /// arithmetic intensity is tiny and it should be bandwidth-bound. But a whole-model estimate put decode at + /// ≈46 GB/s against a measured 90 GB/s DRAM read ceiling — only ~51%. If the GEMV kernel itself streams its + /// weight near the ceiling, that shortfall is per-token overhead (attention, LM head, sampling) and a wider + /// kernel buys nothing. If the GEMV runs well under the ceiling, the dequant compute cannot consume bytes + /// as fast as memory delivers them, and an AVX-512 / VNNI decode kernel could raise utilisation — the one + /// case where the reverted "AVX-512 decode port" negative would not apply. + /// + /// The GB/s column (from ) reports the repacked weight bytes + /// streamed per call; compare it directly against MachineRooflineBenchmark.ReadBandwidth (~90 GB/s). + /// The shape is one FFN projection at Qwen-3B dimensions; decode processes a single activation row. + /// + /// Run: + /// dotnet run -c Release --project Sources/Benchmark -- --filter "*DecodeGemvRoofline*" + /// + [Config(typeof(BenchmarkConfig))] + public class DecodeGemvRooflineBenchmark + { + private const int InputSize = 2048; + + /// + /// 11008 keeps the repacked weight (~12.7 MB) inside this box's 128 MB L3, so the kernel runs + /// compute-bound with hot data — that number is the kernel's own throughput ceiling. 176128 makes the + /// weight ~203 MB, past L3, so the kernel streams from DRAM — that number is what decode actually sees. + /// The pair separates the kernel's compute rate from the memory rate it is fed. + /// + [Params(11008, 176128)] + public int OutputSize + { + get; set; + } + + private Q4KWeight _q4k = null!; + private byte[] _repacked = null!; + private sbyte[] _quants = null!; + private float[] _scales = null!; + private short[] _bsums = null!; + private float[] _output = null!; + + public static WorkAmount GetWorkAmount(BenchmarkCase benchmarkCase) + { + // Bytes actually streamed: the repacked weight, read once per GEMV. + var outputSize = (int)benchmarkCase.Parameters["OutputSize"]; + var repackedBytes = (long)(outputSize / 8) * (InputSize / 256) * Q4KRepack.BlockKx8Bytes; + + return WorkAmount.Memory(repackedBytes); + } + + [GlobalSetup] + public void Setup() + { + var rng = new Random(20260723); + + var f32 = new float[(long)OutputSize * InputSize]; + for (var i = 0; i < f32.Length; i++) + { + f32[i] = (float)((rng.NextDouble() * 2.0) - 1.0); + } + + _q4k = new Q4KWeight(GgmlQuant.QuantizeQ4_K(f32, InputSize, OutputSize), InputSize, OutputSize); + _repacked = _q4k.EnsureRepacked().ToArray(); + + var spr = _q4k.SuperBlocksPerRow; + var input = new float[InputSize]; + for (var i = 0; i < input.Length; i++) + { + input[i] = (float)((rng.NextDouble() * 2.0) - 1.0); + } + + _quants = new sbyte[InputSize]; + _scales = new float[spr]; + _bsums = new short[spr * Q4KDotKernel.GroupsPerSuperBlock]; + Q4KDotKernel.QuantizeActivationQ8K(input, _quants, _scales, _bsums); + + _output = new float[OutputSize]; + } + + /// The production decode kernel: AVX2 8×8 GEMV, one activation row, parallel over row-groups. + [Benchmark] + public void DecodeGemvParallel() + { + Q4KGemvKernel.GemvParallel(_repacked, OutputSize, InputSize, _quants, _scales, _bsums, _output); + } + } +} From d87f272657acfc75fdb5c6a5e78cf1f2e33d6b8a Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Thu, 23 Jul 2026 11:25:55 +0200 Subject: [PATCH 26/37] llama --- .editorconfig | 1 + Demo/AgentDemo/Program.cs | 3 +- Demo/AgentFrameworkDemo/Program.cs | 3 +- Demo/AnomalyConsoleDemo/Program.cs | 3 +- Demo/EvaluationDemo/Program.cs | 3 +- Demo/Gpt2ConsoleDemo/Program.cs | 3 +- .../Infrastructure/ModelPathResolver.cs | 4 +- Demo/LocalAgentAspNetDemo/Rag/RagService.cs | 3 +- README.md | 15 +-- ROADMAP.md | 22 ++++- .../Analyzers/AnalyzerReleases.Unshipped.md | 1 + .../EnvironmentVariableNameAnalyzer.cs | 96 +++++++++++++++++++ Sources/AndroidBench/DecodeBench.cs | 9 +- .../Benchmark/LargeCnnComparisonBenchmark.cs | 3 +- .../Benchmark/MnistTrainingEpochBenchmark.cs | 3 +- Sources/Cli/HfDownloader.cs | 5 +- Sources/Main/Audio/MelSpectrogram.cs | 17 ++-- Sources/Main/Autograd/ComputationGraph.cs | 2 +- .../LanguageModels/Loading/GgufLlamaLoader.cs | 9 +- .../Runtime/BatchedAttentionKernel.cs | 2 +- .../Runtime/BatchedQuantProjection.cs | 7 +- .../Runtime/CachedAttentionKernel.cs | 7 +- .../Runtime/CachedMultiHeadAttention.cs | 37 ++++--- .../LanguageModels/Runtime/KeyValueCache.cs | 34 +++---- Sources/Main/Onnx/OnnxGraphImporter.cs | 5 +- .../Main/Onnx/Operators/ReduceMeanOperator.cs | 5 +- Sources/Main/Runtime/OverfitEnvironment.cs | 67 ++++++++++++- 27 files changed, 282 insertions(+), 87 deletions(-) create mode 100644 Sources/Analyzers/EnvironmentVariableNameAnalyzer.cs diff --git a/.editorconfig b/.editorconfig index 235ed419..72f2ad22 100644 --- a/.editorconfig +++ b/.editorconfig @@ -268,6 +268,7 @@ dotnet_diagnostic.OVERFIT021.severity = suggestion [Sources/Main/**.cs] dotnet_diagnostic.OVERFIT008.severity = error dotnet_diagnostic.OVERFIT015.severity = error +dotnet_diagnostic.OVERFIT024.severity = warning dotnet_diagnostic.OVERFIT022.severity = error dotnet_diagnostic.OVERFIT023.severity = error dotnet_diagnostic.OVERFIT900.severity = error diff --git a/Demo/AgentDemo/Program.cs b/Demo/AgentDemo/Program.cs index f02e5a8d..f9421589 100644 --- a/Demo/AgentDemo/Program.cs +++ b/Demo/AgentDemo/Program.cs @@ -3,6 +3,7 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com +using DevOnBike.Overfit.Runtime; using System.Text.Json; using DevOnBike.Overfit.LanguageModels.Chat; using DevOnBike.Overfit.LanguageModels.Constraints; @@ -28,7 +29,7 @@ internal static class Program { private static int Main() { - var dir = Environment.GetEnvironmentVariable("OVERFIT_MODEL_DIR") ?? @"C:\qwen3b"; + var dir = Environment.GetEnvironmentVariable(OverfitEnvironment.ModelDir) ?? @"C:\qwen3b"; var ggufPath = ResolveGguf(dir); if (ggufPath is null) { diff --git a/Demo/AgentFrameworkDemo/Program.cs b/Demo/AgentFrameworkDemo/Program.cs index bafbfad6..c9341f6c 100644 --- a/Demo/AgentFrameworkDemo/Program.cs +++ b/Demo/AgentFrameworkDemo/Program.cs @@ -13,6 +13,7 @@ // // Verified against Microsoft.Agents.AI 1.13.0 + DevOnBike.Overfit 10.0.x on a Qwen2.5-0.5B Q4_K_M. +using DevOnBike.Overfit.Runtime; using DevOnBike.Overfit.Extensions.AI; using DevOnBike.Overfit.LanguageModels; using Microsoft.Agents.AI; @@ -26,7 +27,7 @@ public static async Task Main(string[] args) { var modelPath = args.Length > 0 ? args[0] - : Environment.GetEnvironmentVariable("OVERFIT_MODEL_PATH"); + : Environment.GetEnvironmentVariable(OverfitEnvironment.ModelPath); if (string.IsNullOrWhiteSpace(modelPath) || !File.Exists(modelPath)) { diff --git a/Demo/AnomalyConsoleDemo/Program.cs b/Demo/AnomalyConsoleDemo/Program.cs index 04179cec..9246bb31 100644 --- a/Demo/AnomalyConsoleDemo/Program.cs +++ b/Demo/AnomalyConsoleDemo/Program.cs @@ -3,6 +3,7 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com +using DevOnBike.Overfit.Runtime; using DevOnBike.Overfit.Anomalies.Adaptive; using DevOnBike.Overfit.Anomalies.Baseline; using DevOnBike.Overfit.Anomalies.Gpt; @@ -469,7 +470,7 @@ private static bool HasFlag(string[] args, string name) return explicitPath; } - var dir = Environment.GetEnvironmentVariable("OVERFIT_MODEL_DIR"); + var dir = Environment.GetEnvironmentVariable(OverfitEnvironment.ModelDir); var candidates = new[] { dir is null ? null : Path.Combine(dir, "k8s_metrics.csv"), diff --git a/Demo/EvaluationDemo/Program.cs b/Demo/EvaluationDemo/Program.cs index b46b3cba..5cc3cccb 100644 --- a/Demo/EvaluationDemo/Program.cs +++ b/Demo/EvaluationDemo/Program.cs @@ -13,6 +13,7 @@ // judgments; the evaluator prompts are tuned against GPT-4o-class models, so treat small local // judges (<7B) as a demo of the PLUMBING, not a calibrated quality gate. +using DevOnBike.Overfit.Runtime; using DevOnBike.Overfit.Demo.Evaluation; using DevOnBike.Overfit.Extensions.AI; using DevOnBike.Overfit.LanguageModels; @@ -22,7 +23,7 @@ var modelPath = args.Length > 0 ? args[0] - : Environment.GetEnvironmentVariable("OVERFIT_JUDGE") ?? @"C:\qwen3b\qwen.q4km.gguf"; + : Environment.GetEnvironmentVariable(OverfitEnvironment.Judge) ?? @"C:\qwen3b\qwen.q4km.gguf"; if (!File.Exists(modelPath)) { diff --git a/Demo/Gpt2ConsoleDemo/Program.cs b/Demo/Gpt2ConsoleDemo/Program.cs index 686adfb7..41ced354 100644 --- a/Demo/Gpt2ConsoleDemo/Program.cs +++ b/Demo/Gpt2ConsoleDemo/Program.cs @@ -3,6 +3,7 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com +using DevOnBike.Overfit.Runtime; using System.Diagnostics; using DevOnBike.Overfit.DeepLearning; using DevOnBike.Overfit.LanguageModels; @@ -178,7 +179,7 @@ private static string ResolvePath(string? cliValue, string fileName) } // 2. $OVERFIT_MODEL_DIR/ - var envDir = Environment.GetEnvironmentVariable("OVERFIT_MODEL_DIR"); + var envDir = Environment.GetEnvironmentVariable(OverfitEnvironment.ModelDir); if (!string.IsNullOrWhiteSpace(envDir)) { var fromEnv = Path.Combine(envDir, fileName); diff --git a/Demo/LocalAgentAspNetDemo/Infrastructure/ModelPathResolver.cs b/Demo/LocalAgentAspNetDemo/Infrastructure/ModelPathResolver.cs index fc310b10..d7e01dab 100644 --- a/Demo/LocalAgentAspNetDemo/Infrastructure/ModelPathResolver.cs +++ b/Demo/LocalAgentAspNetDemo/Infrastructure/ModelPathResolver.cs @@ -3,6 +3,8 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com +using DevOnBike.Overfit.Runtime; + namespace DevOnBike.Overfit.Demo.LocalAgent.Infrastructure { internal static class ModelPathResolver @@ -25,7 +27,7 @@ public static string Resolve(IConfiguration config) } // 2) Env var `OVERFIT_MODEL_DIR` — a directory; prefer a *.gguf inside, else model.safetensors. - var fromEnv = Environment.GetEnvironmentVariable("OVERFIT_MODEL_DIR"); + var fromEnv = Environment.GetEnvironmentVariable(OverfitEnvironment.ModelDir); if (!string.IsNullOrWhiteSpace(fromEnv) && Directory.Exists(fromEnv)) { var ggufs = Directory.GetFiles(fromEnv, "*.gguf"); diff --git a/Demo/LocalAgentAspNetDemo/Rag/RagService.cs b/Demo/LocalAgentAspNetDemo/Rag/RagService.cs index ec43f2c0..becc981c 100644 --- a/Demo/LocalAgentAspNetDemo/Rag/RagService.cs +++ b/Demo/LocalAgentAspNetDemo/Rag/RagService.cs @@ -3,6 +3,7 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com +using DevOnBike.Overfit.Runtime; using System.Diagnostics; using System.Security.Cryptography; using System.Text; @@ -513,7 +514,7 @@ private string ResolveEmbeddingDir() } // 2) Env var OVERFIT_EMBEDDING_DIR. - var fromEnv = Environment.GetEnvironmentVariable("OVERFIT_EMBEDDING_DIR"); + var fromEnv = Environment.GetEnvironmentVariable(OverfitEnvironment.EmbeddingDir); if (!string.IsNullOrWhiteSpace(fromEnv) && Directory.Exists(fromEnv)) { return fromEnv; diff --git a/README.md b/README.md index 3b570adf..c51724d0 100644 --- a/README.md +++ b/README.md @@ -439,16 +439,18 @@ More details: ## Benchmarks: honest headline -Test machine for current headline numbers: AMD Ryzen 9 9950X3D, Windows 11, -.NET 10, BenchmarkDotNet 0.15.8. +Test machine for current headline numbers: AMD Ryzen 9 9950X3D (Zen 5, 16 cores, AVX-512), Windows 11, +.NET 10, BenchmarkDotNet 0.15.8. LLM figures use Qwen2.5-3B Q4_K_M unless stated. Kernels that need +AVX-512 fall back to AVX2 automatically — the suite is green in both configurations, but the prefill +figure below is the AVX-512 one. | Workload | Result | Allocation | |---|---:|---:| -| Single inference `Linear(784 -> 10)` | ~7.6x faster than ONNX Runtime | 0 B | +| Single inference `Linear(784 -> 10)` | ~8.0x faster than ONNX Runtime (234 ns vs 1883 ns) | 0 B | | GPT-2 Small KV-cache decode | ~6.5x faster than naive O(N²), parity vs PyTorch | 0 B/token | -| Qwen2.5-3B Q4_K_M decode | ~19 tok/s default, **~24 tok/s** with opt-in repacked GEMV (`OVERFIT_REPACK_GEMV=1` + `OVERFIT_DECODE_WORKERS=16`) | ~1 B/token | +| Qwen2.5-3B Q4_K_M decode | ~19 tok/s default, **~25 tok/s** with opt-in repacked GEMV (`OVERFIT_REPACK_GEMV=1` + `OVERFIT_DECODE_WORKERS=16`) | ~1 B/token | | Bielik-4.5B Q4_K_M decode | ~17 tok/s, −36% working set vs same-file llama.cpp | ~1 B/token | -| Bielik-4.5B Q4_K_M prefill / TTFT (410-token prompt) | **1.44× faster** with the weight-stationary Q4_K matmul (10.7 → 7.5 s, bit-identical) | 0 B/request | +| **Qwen2.5-3B Q4_K_M prefill (672-token prompt)** | **~299 tok/s — 2.1× faster than the previous release** (143 → 299), bit-identical. AVX-512 Q4_K/Q6_K kernels where the CPU has them, plus whole-matrix Q/K/V/O projections and hoisted F16 scale decode | **0 B/request** | | MNIST CNN training (60k) | **503 ± 4 ms/epoch** (BenchmarkDotNet) — on par with PyTorch 2.11 CPU at its optimal threads (~524–570 ms, same box/arch); full train ~1.5–2 s with one-cycle LR — [audit](docs/mnist-cnn-training-audit.md) | 1.31 MB/epoch | | Concurrent inference, 8 threads | ~3.6x faster than ONNX Runtime | 0 B | | Batched prefill (272-token prompt, 0.6B) | allocation-free per request (was ~748 MB before pooling), bit-identical output | **0 B/request** | @@ -456,7 +458,8 @@ Test machine for current headline numbers: AMD Ryzen 9 9950X3D, Windows 11, Honest positioning: -- llama.cpp / LLamaSharp are still faster for raw CPU LLM decode (~1.2× same-file vs a current AVX-512 llama.cpp build with our repacked-GEMV flag on, narrowed from ~1.6× — single-stream CPU decode is DRAM-bandwidth-bound). +- llama.cpp / LLamaSharp are still faster for raw CPU LLM decode (~1.15× same-file vs a current AVX-512 llama.cpp build with our repacked-GEMV flag on, narrowed from ~1.6×). Single-stream decode is DRAM-bandwidth-bound and we measured our GEMV at **82% of this box's DRAM read ceiling**, with its compute rate *above* that ceiling — so the remaining gap is memory, not kernel quality, and a wider instruction set cannot close it (`Sources/Benchmark/DecodeGemvRooflineBenchmark.cs`). +- On **prefill** the gap is now ~1.8× against an AVX-512 llama.cpp build and **~1.15× against their AVX2 build** — i.e. on machines without AVX-512 we are close to parity. Measured on one box (Ryzen 9 9950X3D) with one model; treat it as a data point, not a general claim. Our Q4_K matmul measured *faster* than llama.cpp's at equal instruction set and thread count (1.70 vs 1.56 TFLOP/s on their own test shape). - PyTorch CPU is faster for large-scale training. - ONNX Runtime is mature and fast if native dependencies are acceptable. - XGBoost's C++ kernel is still ~1.5× faster for raw batch tree scoring; Overfit wins decisively on in-process online (per-request) latency where the Python/native marshalling tax dominates. diff --git a/ROADMAP.md b/ROADMAP.md index 948c4780..20cefdbb 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -109,8 +109,26 @@ the rules that survived recorded in the `feedback-measurement-discipline` memory left behind: `MachineRooflineBenchmark`, `DecodeGemvRooflineBenchmark`, `Diagnostics/Throughput.cs`, and the BenchmarkDotNet throughput columns. -**Next move is a business decision (perf course vs Redaction Gateway), not another kernel.** Any further perf -work should measure the ceiling before writing code — the discipline that made this track pay. +**Next move is a business decision (perf course vs Redaction Gateway), not another *LLM* kernel.** Any further +perf work should measure the ceiling before writing code — the discipline that made this track pay. + +#### ⚠ BUT: the largest untouched perf reserve in the project is CNN inference, not LLM — 13.2× behind ORT + +Measured 2026-07-23, `LargeCnnComparisonBenchmark`, VGG-16 (~15.5 GFLOPs/inference), same box: + +| | time | GFLOP/s | % of this box's float ceiling (2.19 TFLOP/s) | +|---|---:|---:|---:| +| ONNX Runtime (native MLAS) | **9.96 ms** | 1557 | **71%** | +| Overfit (im2col + GEMM, DAG importer) | **131.7 ms** | 118 | **5.4%** | + +Parity is exact (maxAbsDiff 6.7e-8, cosine 1.000000, same argmax) — this is purely speed. For scale: the +whole prefill sprint above chased a **1.8×** gap on a path already near half of its instruction mix's +ceiling. This is a **13×** gap on a path at 5% of the machine ceiling. + +**A framing correction this exposes.** The README headline "~8× faster than ONNX Runtime" is measured on +`Linear(784 → 10)`, where ORT's *per-call overhead* dominates — it is a real result for small-model, +in-process serving, but it says nothing about kernel quality. VGG-16 is compute-dominated and is the honest +kernel-vs-kernel test. Both statements are true; only the second describes the kernels. --- diff --git a/Sources/Analyzers/AnalyzerReleases.Unshipped.md b/Sources/Analyzers/AnalyzerReleases.Unshipped.md index 6f4b62eb..7a82a650 100644 --- a/Sources/Analyzers/AnalyzerReleases.Unshipped.md +++ b/Sources/Analyzers/AnalyzerReleases.Unshipped.md @@ -28,4 +28,5 @@ OVERFIT020 | Performance | Warning | Primitive-array parameter that never escape OVERFIT021 | Style | Warning | else / else if — use a guard clause + early return, continue, a ternary, or a switch expression OVERFIT022 | Reliability | Warning | Direct recursion — unbounded stack depth; StackOverflowException is uncatchable in .NET OVERFIT023 | Reliability | Warning | Loop with no exit condition in its header (while(true) / for(;;)) — state the bound +OVERFIT024 | Maintainability | Warning | Environment-variable name literal — declare it in OverfitEnvironment so every switch has one audit point OVERFIT900 | Performance | Error | A per-call OVERFIT rule fired inside an [OverfitHotPath] member/type — escalated to a build error diff --git a/Sources/Analyzers/EnvironmentVariableNameAnalyzer.cs b/Sources/Analyzers/EnvironmentVariableNameAnalyzer.cs new file mode 100644 index 00000000..0e02c394 --- /dev/null +++ b/Sources/Analyzers/EnvironmentVariableNameAnalyzer.cs @@ -0,0 +1,96 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Collections.Immutable; +using Microsoft.CodeAnalysis; +using Microsoft.CodeAnalysis.Diagnostics; +using Microsoft.CodeAnalysis.Operations; + +namespace DevOnBike.Overfit.Analyzers +{ + /// + /// OVERFIT024 — a string literal passed as the variable name to + /// Environment.GetEnvironmentVariable / SetEnvironmentVariable. Every environment-variable + /// name the engine reads belongs in DevOnBike.Overfit.Runtime.OverfitEnvironment, referenced as a + /// constant from there. + /// + /// Why. Scattered literals drift: the read site, the docs, the overfit doctor output + /// and the release notes each end up with their own spelling of the same switch, and a typo silently + /// disables the feature instead of failing. One declaration site makes the full set of switches + /// enumerable and reviewable — which is the point, since these are the knobs users are told to set. + /// Tuning flags added during a perf sprint are exactly the ones that leak: six were introduced across four + /// kernels in a single session before this rule existed. + /// + /// Only the literal is flagged; passing OverfitEnvironment.Something — or any other constant + /// reference — is fine. OverfitEnvironment itself is the sanctioned place for the literals. + /// + [DiagnosticAnalyzer(LanguageNames.CSharp)] + public sealed class EnvironmentVariableNameAnalyzer : DiagnosticAnalyzer + { + public const string DiagnosticId = "OVERFIT024"; + + private static readonly DiagnosticDescriptor Rule = new( + DiagnosticId, + title: "Environment-variable name literal — declare it in OverfitEnvironment", + messageFormat: "Environment-variable name \"{0}\" is a literal — declare it as a constant in OverfitEnvironment and reference that: one audit point for every switch, no spelling drift between read site and docs", + category: "Maintainability", + defaultSeverity: DiagnosticSeverity.Warning, + isEnabledByDefault: true, + description: "All OVERFIT_* (and any other) environment-variable names are centralised in DevOnBike.Overfit.Runtime.OverfitEnvironment so the complete set of tuning switches is enumerable in one file and cannot drift between the code that reads it and the documentation that advertises it."); + + public override ImmutableArray SupportedDiagnostics { get; } = [Rule]; + + public override void Initialize(AnalysisContext context) + { + context.ConfigureGeneratedCodeAnalysis(GeneratedCodeAnalysisFlags.None); + context.EnableConcurrentExecution(); + context.RegisterOperationAction(AnalyzeInvocation, OperationKind.Invocation); + } + + private static void AnalyzeInvocation(OperationAnalysisContext context) + { + var invocation = (IInvocationOperation)context.Operation; + var method = invocation.TargetMethod; + + if (method.Name is not ("GetEnvironmentVariable" or "SetEnvironmentVariable")) + { + return; + } + + if (method.ContainingType is not { Name: "Environment" } containing + || containing.ContainingNamespace is not { Name: nameof(System), ContainingNamespace.IsGlobalNamespace: true }) + { + return; + } + + // OverfitEnvironment is where the literals are supposed to live. + if (context.ContainingSymbol.ContainingType is { Name: "OverfitEnvironment" }) + { + return; + } + + if (invocation.Arguments.Length == 0) + { + return; + } + + var name = invocation.Arguments[0].Value; + + // Unwrap an implicit conversion so a literal behind one is still seen. + if (name is IConversionOperation conversion) + { + name = conversion.Operand; + } + + // A constant reference (OverfitEnvironment.X) is an IFieldReferenceOperation with a constant value; + // only a bare literal in source is the violation. + if (name is not ILiteralOperation { ConstantValue: { HasValue: true, Value: string variableName } }) + { + return; + } + + context.ReportDiagnostic(Diagnostic.Create(Rule, name.Syntax.GetLocation(), variableName)); + } + } +} diff --git a/Sources/AndroidBench/DecodeBench.cs b/Sources/AndroidBench/DecodeBench.cs index 2aae7b2e..ccd26919 100644 --- a/Sources/AndroidBench/DecodeBench.cs +++ b/Sources/AndroidBench/DecodeBench.cs @@ -3,6 +3,7 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com +using DevOnBike.Overfit.Runtime; using System.Diagnostics; using System.Runtime.CompilerServices; using System.Runtime.InteropServices; @@ -29,12 +30,12 @@ public static void Run(string modelPath, int genTokens, int repeats, int warmup, // efficiency cores) — fine on a dedicated desktop, wasteful on a phone. Disable it and cap the // worker count to the big cluster. Must be set BEFORE any Overfit type loads (OverfitParallel // reads these at static-init). Read by env name to avoid pulling internals into this project. - var poolEnv = Environment.GetEnvironmentVariable("OVERFIT_BENCH_POOL"); - var workersEnv = Environment.GetEnvironmentVariable("OVERFIT_BENCH_WORKERS"); + var poolEnv = Environment.GetEnvironmentVariable(OverfitEnvironment.BenchPool); + var workersEnv = Environment.GetEnvironmentVariable(OverfitEnvironment.BenchWorkers); Environment.SetEnvironmentVariable("OVERFIT_DECODE_POOL", string.IsNullOrEmpty(poolEnv) ? "0" : poolEnv); Environment.SetEnvironmentVariable("OVERFIT_DECODE_WORKERS", string.IsNullOrEmpty(workersEnv) ? "4" : workersEnv); - log($"config: OVERFIT_DECODE_POOL={Environment.GetEnvironmentVariable("OVERFIT_DECODE_POOL")} " + - $"OVERFIT_DECODE_WORKERS={Environment.GetEnvironmentVariable("OVERFIT_DECODE_WORKERS")}"); + log($"config: OVERFIT_DECODE_POOL={Environment.GetEnvironmentVariable(OverfitEnvironment.DecodePool)} " + + $"OVERFIT_DECODE_WORKERS={Environment.GetEnvironmentVariable(OverfitEnvironment.DecodeWorkers)}"); var fast = Dp.IsSupported ? "NEON(SDOT)" : Avx2.IsSupported ? "AVX2" : "scalar"; diff --git a/Sources/Benchmark/LargeCnnComparisonBenchmark.cs b/Sources/Benchmark/LargeCnnComparisonBenchmark.cs index 6063335d..0118ddd9 100644 --- a/Sources/Benchmark/LargeCnnComparisonBenchmark.cs +++ b/Sources/Benchmark/LargeCnnComparisonBenchmark.cs @@ -3,6 +3,7 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com +using DevOnBike.Overfit.Runtime; using BenchmarkDotNet.Attributes; using DevOnBike.Overfit.Inference; using DevOnBike.Overfit.Onnx; @@ -41,7 +42,7 @@ public class LargeCnnComparisonBenchmark [GlobalSetup] public void Setup() { - var modelPath = Environment.GetEnvironmentVariable("OVERFIT_CNN_ONNX") ?? DefaultModelPath; + var modelPath = Environment.GetEnvironmentVariable(OverfitEnvironment.CnnOnnx) ?? DefaultModelPath; if (!File.Exists(modelPath)) { throw new FileNotFoundException( diff --git a/Sources/Benchmark/MnistTrainingEpochBenchmark.cs b/Sources/Benchmark/MnistTrainingEpochBenchmark.cs index 42e3f284..7de4bfa3 100644 --- a/Sources/Benchmark/MnistTrainingEpochBenchmark.cs +++ b/Sources/Benchmark/MnistTrainingEpochBenchmark.cs @@ -3,6 +3,7 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com +using DevOnBike.Overfit.Runtime; using BenchmarkDotNet.Attributes; using BenchmarkDotNet.Engines; using DevOnBike.Overfit.Autograd; @@ -50,7 +51,7 @@ public class MnistTrainingEpochBenchmark [GlobalSetup] public void Setup() { - var dir = Environment.GetEnvironmentVariable("OVERFIT_MNIST_DIR") ?? @"d:\ml"; + var dir = Environment.GetEnvironmentVariable(OverfitEnvironment.MnistDir) ?? @"d:\ml"; (_trainX, _trainY) = LoadMnist( Path.Combine(dir, "train-images.idx3-ubyte"), Path.Combine(dir, "train-labels.idx1-ubyte")); diff --git a/Sources/Cli/HfDownloader.cs b/Sources/Cli/HfDownloader.cs index 27d56f52..c3337560 100644 --- a/Sources/Cli/HfDownloader.cs +++ b/Sources/Cli/HfDownloader.cs @@ -3,6 +3,7 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com +using DevOnBike.Overfit.Runtime; using System.Net; using System.Net.Http.Headers; using System.Security.Cryptography; @@ -28,7 +29,7 @@ internal static class HfDownloader private static string ResolveEndpoint() { - var endpoint = Environment.GetEnvironmentVariable("HF_ENDPOINT"); + var endpoint = Environment.GetEnvironmentVariable(OverfitEnvironment.HuggingFaceEndpoint); return string.IsNullOrWhiteSpace(endpoint) ? "https://huggingface.co" : endpoint.TrimEnd('/'); } @@ -41,7 +42,7 @@ private static HttpClient CreateClient() client.DefaultRequestHeaders.UserAgent.ParseAdd("overfit-cli/1.0"); - var token = Environment.GetEnvironmentVariable("HF_TOKEN"); + var token = Environment.GetEnvironmentVariable(OverfitEnvironment.HuggingFaceToken); if (!string.IsNullOrWhiteSpace(token)) { diff --git a/Sources/Main/Audio/MelSpectrogram.cs b/Sources/Main/Audio/MelSpectrogram.cs index 8544fe94..8411dd51 100644 --- a/Sources/Main/Audio/MelSpectrogram.cs +++ b/Sources/Main/Audio/MelSpectrogram.cs @@ -62,19 +62,16 @@ public MelSpectrogram(int nMels = DefaultMelCount, ReadOnlySpan melFilter _hann[n] = 0.5f * (1f - MathF.Cos(2f * MathF.PI * n / NFft)); } - if (melFilters.IsEmpty) + if (!melFilters.IsEmpty && melFilters.Length != nMels * _nFreqs) { - _melFilters = BuildSlaneyMelFilters(nMels, _nFreqs, SampleRate, NFft); + throw new ArgumentException($"melFilters must be [{nMels} × {_nFreqs}] = {nMels * _nFreqs}, got {melFilters.Length}.", nameof(melFilters)); } - if (!(melFilters.IsEmpty)) - { - if (melFilters.Length != nMels * _nFreqs) - { - throw new ArgumentException($"melFilters must be [{nMels} × {_nFreqs}] = {nMels * _nFreqs}, got {melFilters.Length}.", nameof(melFilters)); - } - _melFilters = melFilters.ToArray(); - } + // One conditional expression rather than two mirrored `if` blocks: the compiler cannot see such a + // pair as exhaustive, so the field read as CS8618 "uninitialised" and broke the AOT guard. + _melFilters = melFilters.IsEmpty + ? BuildSlaneyMelFilters(nMels, _nFreqs, SampleRate, NFft) + : melFilters.ToArray(); // ── Bluestein tables ── var m = 1; diff --git a/Sources/Main/Autograd/ComputationGraph.cs b/Sources/Main/Autograd/ComputationGraph.cs index d57375fc..a6ed6e2a 100644 --- a/Sources/Main/Autograd/ComputationGraph.cs +++ b/Sources/Main/Autograd/ComputationGraph.cs @@ -559,7 +559,7 @@ private static int ResolveDefaultTapeBufferElements() } var isGitHubActions = string.Equals( - Environment.GetEnvironmentVariable("GITHUB_ACTIONS"), + Environment.GetEnvironmentVariable(OverfitEnvironment.GitHubActions), "true", StringComparison.OrdinalIgnoreCase); diff --git a/Sources/Main/LanguageModels/Loading/GgufLlamaLoader.cs b/Sources/Main/LanguageModels/Loading/GgufLlamaLoader.cs index e714323c..862a8bad 100644 --- a/Sources/Main/LanguageModels/Loading/GgufLlamaLoader.cs +++ b/Sources/Main/LanguageModels/Loading/GgufLlamaLoader.cs @@ -1293,14 +1293,15 @@ private static void LoadTensor(GgufReader reader, string name, Span dst) private static void LoadTensorOrZeros(GgufReader reader, string name, Span dst) { - var found = reader.Tensors.TryGetValue(name, out var info); - - if (found) + // Test the TryGetValue directly rather than through a `found` bool: routed through a separate + // variable the compiler loses the link to `info`'s null state and reports CS8604 on the call + // below, which the AOT guard promotes to an error. + if (reader.Tensors.TryGetValue(name, out var info)) { reader.LoadTensorAsF32(info, dst); } - if (!found) + if (info is null) { dst.Clear(); } diff --git a/Sources/Main/LanguageModels/Runtime/BatchedAttentionKernel.cs b/Sources/Main/LanguageModels/Runtime/BatchedAttentionKernel.cs index 2c9fbc9c..393c2f9b 100644 --- a/Sources/Main/LanguageModels/Runtime/BatchedAttentionKernel.cs +++ b/Sources/Main/LanguageModels/Runtime/BatchedAttentionKernel.cs @@ -39,7 +39,7 @@ public static unsafe class BatchedAttentionKernel /// query order. A/B switch for the measurement below; set OVERFIT_BALANCED_ATTN=0 to disable. /// internal static bool UseBalancedQueryOrder = - Environment.GetEnvironmentVariable("OVERFIT_BALANCED_ATTN") != "0"; + Environment.GetEnvironmentVariable(OverfitEnvironment.BalancedAttention) != "0"; /// /// Sequential batched attention. is row-major diff --git a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs index 231c6873..23f9ffbf 100644 --- a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs +++ b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs @@ -99,7 +99,10 @@ internal static class BatchedQuantProjection /// bits against 7.71–9.08 at 512. The port is bit-identical, so the existing parity tests apply to it /// unchanged; Avx512PrefillParityTests pins the two kernels against each other directly. /// - internal static bool UseAvx512PrefillQ4K = CpuFeatures.HasAvx512 && CpuFeatures.HasAvx512Bw; + internal static bool UseAvx512PrefillQ4K = + CpuFeatures.HasAvx512 + && CpuFeatures.HasAvx512Bw + && Environment.GetEnvironmentVariable(OverfitEnvironment.Avx512PrefillQ4K) != "0"; /// /// The same port for Q6_K — measured slower and therefore off. Kept behind the flag with @@ -120,7 +123,7 @@ internal static class BatchedQuantProjection internal static bool UseAvx512PrefillQ6K = CpuFeatures.HasAvx512 && CpuFeatures.HasAvx512Bw - && Environment.GetEnvironmentVariable("OVERFIT_AVX512_Q6K") != "0"; + && Environment.GetEnvironmentVariable(OverfitEnvironment.Avx512PrefillQ6K) != "0"; /// /// Weight bytes one worker's band may occupy. Half of a 1 MB Zen-5 L2, leaving the rest for the diff --git a/Sources/Main/LanguageModels/Runtime/CachedAttentionKernel.cs b/Sources/Main/LanguageModels/Runtime/CachedAttentionKernel.cs index a7b59d33..bfd35340 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedAttentionKernel.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedAttentionKernel.cs @@ -8,6 +8,7 @@ using System.Runtime.Intrinsics; using System.Runtime.Intrinsics.X86; using DevOnBike.Overfit.Intrinsics; +using DevOnBike.Overfit.Runtime; namespace DevOnBike.Overfit.LanguageModels.Runtime { @@ -164,9 +165,9 @@ public static void ComputeSingleHead( /// A/B switch for ; set OVERFIT_ATTN_REGACC=0 to disable. internal static bool UseRegisterResidentValueSum = - Environment.GetEnvironmentVariable("OVERFIT_ATTN_REGACC") != "0"; + Environment.GetEnvironmentVariable(OverfitEnvironment.AttentionRegisterAccumulate) != "0"; - private static readonly string AblateMode = Environment.GetEnvironmentVariable("OVERFIT_ATTN_ABLATE") ?? "none"; + private static readonly string AblateMode = Environment.GetEnvironmentVariable(OverfitEnvironment.AttentionAblate) ?? "none"; /// Measurement-only: replace the query·key dot with a constant, to size it against the exp. internal static bool AblateScoreDot = AblateMode is "dot" or "both"; @@ -176,7 +177,7 @@ public static void ComputeSingleHead( /// Vectorize the softmax exp via TensorPrimitives; set OVERFIT_ATTN_VEXP=0 to disable. internal static bool UseVectorizedSoftmaxExp = - Environment.GetEnvironmentVariable("OVERFIT_ATTN_VEXP") != "0"; + Environment.GetEnvironmentVariable(OverfitEnvironment.AttentionVectorizedExp) != "0"; /// /// The softmax-weighted value sum with the accumulators held in registers across the whole diff --git a/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs b/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs index 0d508403..afa9bbe2 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedMultiHeadAttention.cs @@ -111,25 +111,22 @@ public CachedMultiHeadAttention( dModel, HeadDimension, maxSequenceLength, attnLogitSoftcap); } - // Whole-matrix attention scratch only when OVERFIT_REPACK_ATTN is on (else empty, no RAM cost). - if (Q4KGemvKernel.AttnEnabled) - { - var wholeSize = headCount * HeadDimension; - _qWhole = new float[wholeSize]; - _attnBands = new float[wholeSize]; - _attnQuants = new sbyte[wholeSize]; - _attnScales = new float[(wholeSize + Q4KDotKernel.SuperBlockElements - 1) / Q4KDotKernel.SuperBlockElements]; - _attnBsums = new short[(wholeSize + Q4KDotKernel.GroupSize - 1) / Q4KDotKernel.GroupSize]; - } - - if (!(Q4KGemvKernel.AttnEnabled)) - { - _qWhole = []; - _attnBands = []; - _attnQuants = []; - _attnScales = []; - _attnBsums = []; - } + // Whole-matrix attention scratch only when OVERFIT_REPACK_ATTN is on; `[]` otherwise, no RAM cost. + // Conditional expressions rather than two mirrored `if` blocks — a pair of `if (x)` / `if (!x)` + // statements is not visibly exhaustive to the compiler, so these read as CS8618 "uninitialised" + // and broke the AOT guard, which promotes warnings to errors. + var wholeAttn = Q4KGemvKernel.AttnEnabled; + var wholeSize = headCount * HeadDimension; + + _qWhole = wholeAttn ? new float[wholeSize] : []; + _attnBands = wholeAttn ? new float[wholeSize] : []; + _attnQuants = wholeAttn ? new sbyte[wholeSize] : []; + _attnScales = wholeAttn + ? new float[(wholeSize + Q4KDotKernel.SuperBlockElements - 1) / Q4KDotKernel.SuperBlockElements] + : []; + _attnBsums = wholeAttn + ? new short[(wholeSize + Q4KDotKernel.GroupSize - 1) / Q4KDotKernel.GroupSize] + : []; } /// Gemma-2 attention logit soft-cap applied to pre-softmax scores (0 = off). @@ -576,7 +573,7 @@ internal void DecodeBatched( /// launch-bound). Bit-identical, so it is on by default where the whole handles are Q4_K. /// internal static bool UseWholeKv = - Environment.GetEnvironmentVariable("OVERFIT_WHOLE_KV") != "0"; + Environment.GetEnvironmentVariable(OverfitEnvironment.WholeMatrixKv) != "0"; /// /// Batched (prefill) multi-head attention for the Llama/Qwen quantized path — the diff --git a/Sources/Main/LanguageModels/Runtime/KeyValueCache.cs b/Sources/Main/LanguageModels/Runtime/KeyValueCache.cs index b5f354ed..4af3928b 100644 --- a/Sources/Main/LanguageModels/Runtime/KeyValueCache.cs +++ b/Sources/Main/LanguageModels/Runtime/KeyValueCache.cs @@ -69,26 +69,20 @@ public KeyValueCache(KeyValueCacheShape shape, KvCacheDType dtype) Dtype = dtype; var elems = (int)shape.ElementsPerCache; - if (dtype == KvCacheDType.Q8) - { - var vectors = shape.LayerCount * shape.KvHeadCount * shape.MaxSequenceLength; - _keysQ = new sbyte[elems]; - _valuesQ = new sbyte[elems]; - _keyScales = new float[vectors]; - _valueScales = new float[vectors]; - _keys = []; - _values = []; - } - - if (!(dtype == KvCacheDType.Q8)) - { - _keys = new float[elems]; - _values = new float[elems]; - _keysQ = []; - _valuesQ = []; - _keyScales = []; - _valueScales = []; - } + var vectors = shape.LayerCount * shape.KvHeadCount * shape.MaxSequenceLength; + var quantized = dtype == KvCacheDType.Q8; + + // Conditional expressions rather than two mirrored `if` blocks: the compiler cannot see that a pair + // of `if (x)` / `if (!x)` statements is exhaustive, so every field read as CS8618 "uninitialised" + // and the AOT guard (which promotes warnings to errors) failed on it. A ternary is one definite + // assignment per field, needs no `else` (OVERFIT021), and the unused side is `[]` = Array.Empty, + // so the discarded branch allocates nothing. + _keys = quantized ? [] : new float[elems]; + _values = quantized ? [] : new float[elems]; + _keysQ = quantized ? new sbyte[elems] : []; + _valuesQ = quantized ? new sbyte[elems] : []; + _keyScales = quantized ? new float[vectors] : []; + _valueScales = quantized ? new float[vectors] : []; } public KeyValueCacheShape Shape diff --git a/Sources/Main/Onnx/OnnxGraphImporter.cs b/Sources/Main/Onnx/OnnxGraphImporter.cs index 45948e48..6c86c642 100644 --- a/Sources/Main/Onnx/OnnxGraphImporter.cs +++ b/Sources/Main/Onnx/OnnxGraphImporter.cs @@ -92,7 +92,10 @@ public static OnnxGraphModel LoadFromBytes( slotMap[outName] = slot; } - if (!hasSlot && hasInitializer) + // Test the out-value rather than the `hasInitializer` bool: routed through a separate + // variable the compiler loses the link to initTensor's null state (CS8601), which the + // AOT guard promotes to an error. Same condition, analysable. + if (!hasSlot && initTensor is not null) { // Relabels a CONSTANT: a folded/deduplicated weight or bias routed to its // consumer under a new name (e.g. torch's constant-folding aliases equal biases diff --git a/Sources/Main/Onnx/Operators/ReduceMeanOperator.cs b/Sources/Main/Onnx/Operators/ReduceMeanOperator.cs index 4a44fbc7..f6e674e7 100644 --- a/Sources/Main/Onnx/Operators/ReduceMeanOperator.cs +++ b/Sources/Main/Onnx/Operators/ReduceMeanOperator.cs @@ -34,7 +34,10 @@ public static IModule Build( var hasAxesAttribute = node.Attributes.TryGetValue("axes", out var axesAttr); - if (hasAxesAttribute) + // Test the out-value, not the bool: through a separate variable the compiler loses axesAttr's + // null state (CS8602), which the AOT guard promotes to an error. hasAxesAttribute still drives + // the fallback below, so the control flow is unchanged. + if (axesAttr is not null) { axes = axesAttr.IntArray; } diff --git a/Sources/Main/Runtime/OverfitEnvironment.cs b/Sources/Main/Runtime/OverfitEnvironment.cs index 95e3a5ac..fe127a74 100644 --- a/Sources/Main/Runtime/OverfitEnvironment.cs +++ b/Sources/Main/Runtime/OverfitEnvironment.cs @@ -41,6 +41,32 @@ public static class OverfitEnvironment /// KV-cache element type — e.g. q8 for the int8 KV cache (default F32). public const string KvDType = "OVERFIT_KV_DTYPE"; + // ── Prefill kernel switches (all default ON where the hardware allows; set to 0 to opt out) ── + // These exist so a measured win can be A/B'd against its predecessor without a rebuild, and so a + // regression on unfamiliar hardware can be bisected in the field rather than only on the dev box. + + /// Set to 0 to force the 256-bit Q4_K prefill GEMM instead of the AVX-512 two-column kernel. + public const string Avx512PrefillQ4K = "OVERFIT_AVX512_Q4K"; + + /// Set to 0 to force the 256-bit Q6_K prefill GEMM instead of the AVX-512 kernel. + public const string Avx512PrefillQ6K = "OVERFIT_AVX512_Q6K"; + + /// Set to 0 to project K/V once per KV head instead of one whole-matrix dispatch. + public const string WholeMatrixKv = "OVERFIT_WHOLE_KV"; + + /// Set to 0 to hand attention queries to workers in raw order instead of load-balanced pairs. + public const string BalancedAttention = "OVERFIT_BALANCED_ATTN"; + + /// Set to 0 to accumulate the attention value sum through memory instead of in registers. + public const string AttentionRegisterAccumulate = "OVERFIT_ATTN_REGACC"; + + /// Set to 0 to use scalar MathF.Exp for the softmax instead of the vectorized path. + public const string AttentionVectorizedExp = "OVERFIT_ATTN_VEXP"; + + /// Diagnostics only: dot / exp / both removes that part of the attention + /// kernel to size its share. Produces WRONG results by construction — never set in production. + public const string AttentionAblate = "OVERFIT_ATTN_ABLATE"; + /// Diagnostics A/B switch: set to 1/true to force the scalar Q4_K main-dot (skip AVX2/NEON). /// For measuring SIMD-vs-scalar on one device — not a production tuning knob. public const string ForceScalar = "OVERFIT_FORCE_SCALAR"; @@ -57,5 +83,44 @@ public static class OverfitEnvironment /// Default SNAC decoder-weights directory for the TTS commands. public const string SnacDir = "OVERFIT_SNAC_DIR"; + + // ── Model/asset path hints read by the CLI, demos and benchmarks ────────── + + /// Directory holding the default model for the demos and the local-agent host. + public const string ModelDir = "OVERFIT_MODEL_DIR"; + + /// Explicit path to a single model file, where a demo takes a file rather than a directory. + public const string ModelPath = "OVERFIT_MODEL_PATH"; + + /// Directory holding the sentence-embedding model used by the RAG demo. + public const string EmbeddingDir = "OVERFIT_EMBEDDING_DIR"; + + /// Model used as the judge in the evaluation demo. + public const string Judge = "OVERFIT_JUDGE"; + + /// ONNX model path for the large-CNN comparison benchmark. + public const string CnnOnnx = "OVERFIT_CNN_ONNX"; + + /// MNIST data directory for the training benchmarks. + public const string MnistDir = "OVERFIT_MNIST_DIR"; + + // ── Android decode bench (Sources/AndroidBench) ─────────────────────────── + + /// Overrides the decode-pool setting for the on-device bench. + public const string BenchPool = "OVERFIT_BENCH_POOL"; + + /// Overrides the worker count for the on-device bench. + public const string BenchWorkers = "OVERFIT_BENCH_WORKERS"; + + // ── Third-party / host environment (not ours, but read by us) ───────────── + + /// Hugging Face API endpoint override for the model downloader. + public const string HuggingFaceEndpoint = "HF_ENDPOINT"; + + /// Hugging Face access token for gated repositories. + public const string HuggingFaceToken = "HF_TOKEN"; + + /// Set by GitHub Actions; used to detect a CI run. + public const string GitHubActions = "GITHUB_ACTIONS"; } -} +} \ No newline at end of file From da75c93d4fd0d984b767abb7737ac0bb1e8800cb Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Thu, 23 Jul 2026 11:52:42 +0200 Subject: [PATCH 27/37] onnx --- Directory.Packages.props | 2 +- ROADMAP.md | 49 +++++++ Sources/Main/Kernels/Conv2DGemmKernels.cs | 126 ++++++++++++++---- .../LanguageModels/Runtime/PrefillProfiler.cs | 24 ++++ Sources/Main/Runtime/OverfitEnvironment.cs | 3 + .../Diagnostics/PrefillCallCountTests.cs | 117 ++++++++++++++++ 6 files changed, 294 insertions(+), 27 deletions(-) create mode 100644 Tests/LanguageModels/Diagnostics/PrefillCallCountTests.cs diff --git a/Directory.Packages.props b/Directory.Packages.props index 855a4d48..4ba073d9 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -21,7 +21,7 @@ rules silently off). Expect this package to show as permanently "outdated"; that is INTENTIONAL, not debt. Bump it only when the SDK's own Roslyn version moves up. --> - + diff --git a/ROADMAP.md b/ROADMAP.md index 20cefdbb..a49958aa 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -130,6 +130,55 @@ ceiling. This is a **13×** gap on a path at 5% of the machine ceiling. in-process serving, but it says nothing about kernel quality. VGG-16 is compute-dominated and is the honest kernel-vs-kernel test. Both statements are true; only the second describes the kernels. +**Where the 13× splits — worker sweep, ORT stable at 11.8–12.0 ms throughout as the canary:** + +| workers | Overfit | GFLOP/s | speedup | +|---|---:|---:|---:| +| 1 | 663.2 ms | 23 | 1.00× | +| 4 | 256.4 ms | 60 | 2.59× | +| 16 | 138.4 ms | 112 | 4.79× | +| default (32) | 141.3 ms | 110 | 4.69× | + +Two independent problems, both large: +1. **Per-core kernel: 23 GFLOP/s against a ~137 GFLOP/s single-core ceiling — 17% efficiency.** +2. **Parallel scaling: 16 workers buy 4.79×, not ~14× — 30% efficiency.** + +**The most useful comparison is internal.** Our Q4_K prefill GEMM runs at 2.15 TFLOP/s over 32 threads +≈ **134 GFLOP/s per core** — the same project, the same machine, ~6× the per-core efficiency of the conv +GEMM. We demonstrably know how to write a competitive GEMM (it measured faster than llama.cpp's at equal +ISA); the convolution path simply does not use that class of kernel. The techniques that paid there — +register tiling, weight-stationary reuse, hoisting fixed per-block work, counting how often work repeats — +have not been applied here at all. + +*Caveat before targeting a number:* VGG-16 is entirely 3×3 convs, where ORT's MLAS may use Winograd (a +2.25× FLOP reduction), so its 1557 GFLOP/s is not necessarily 71% of the hardware ceiling in executed FLOPs. +This repo measured Winograd as a **negative** (+79% on deepcnn) with its current infrastructure. Wall-clock +is what matters, and wall-clock says 13×. + +#### ★★ PARALLEL im2col — VGG-16 141 → 73 ms, gap to ORT 13.2× → 6.3× + +The GEMM was already parallel; **the im2col patch gather never was**. On VGG-16 that gather is enormous — +`conv1_2` alone materialises a `[576 × 50176]` matrix (115 MB) one scalar element at a time — and an Amdahl +fit over the worker sweep put the serial fraction at ~15.5%, i.e. **~103 of the 138 ms at 16 workers**. +`Im2Col` now fans out over `krow`: each row owns a disjoint `n`-element slice of `cols` and only reads the +input, so no synchronisation is needed and the result is bit-identical. Gated below `K·N < 65536` so small +convs keep the serial path; `OVERFIT_PARALLEL_IM2COL=0` restores it. + +| workers | before | after | | +|---|---:|---:|---| +| 1 | 663 ms | 668 ms | unchanged — the control: no work was added, only spread | +| 4 | 256 | 226 | | +| 16 | 138 | **87.7** | | +| default (32) | 141 | **73.0** | 110 → **212 GFLOP/s** | + +Speedup over one worker **4.69× → 9.16×**; serial fraction **15.5% → 7.3%**. Parity unchanged +(maxAbsDiff 6.7e-8, cosine 1.000000, same argmax), conv tests 56/0, suite 1499/0/232. The paired A/B run had +ORT flat at 11.66 vs 11.75 ms as the canary. + +**Next: the remaining ~49 ms of serial work**, which now *dominates* at high worker counts (49 of 73 ms at +32 workers). Candidates not yet measured: the elementwise activations, MaxPool, the FC layers, and the +graph executor's inter-node buffer handling. Measure which before touching any of them. + ---
diff --git a/Sources/Main/Kernels/Conv2DGemmKernels.cs b/Sources/Main/Kernels/Conv2DGemmKernels.cs index d4249805..fdd36b08 100644 --- a/Sources/Main/Kernels/Conv2DGemmKernels.cs +++ b/Sources/Main/Kernels/Conv2DGemmKernels.cs @@ -75,45 +75,119 @@ public static void Forward( } } + /// + /// Minimum K·N before im2col fans out. Below this the patch gather is a fraction of a + /// millisecond and the dispatch would cost more than the work. + /// + private const int ParallelIm2ColMinElements = 1 << 16; + + /// Set OVERFIT_PARALLEL_IM2COL=0 to force the original serial gather (A/B switch). + internal static bool UseParallelIm2Col = + Environment.GetEnvironmentVariable(OverfitEnvironment.ParallelIm2Col) != "0"; + // cols[krow, pos] = input[ic, oy·stride-pad+ky, ox·stride-pad+kx] (0 outside the image), where // krow = (ic·k + ky)·k + kx (matches the [outC, inC, k, k] kernel's flattened K), pos = oy·outW + ox. - private static void Im2Col( + // + // Parallelised over krow. Each krow owns a disjoint n-element row of `cols` and only ever READS the + // input, so the fan-out needs no synchronisation and is bit-identical to the serial gather. + // + // This matters more than it looks. The GEMM below was already parallel but the gather was not, and on + // VGG-16 the gather is enormous — conv1_2 alone materialises a [576 × 50176] matrix (115 MB) one + // scalar element at a time. An Amdahl fit over the measured worker sweep (663 ms at 1 worker, 138 ms + // at 16) put the serial fraction at ~15.5%, i.e. **~103 of those 138 ms were serial**, which is why 16 + // workers bought only 4.79× instead of ~14×. + private static unsafe void Im2Col( ReadOnlySpan input, Span cols, int inChannels, int inputH, int inputW, int kernelSize, int padding, int stride, int outH, int outW) { var n = outH * outW; - for (var ic = 0; ic < inChannels; ic++) + var kRows = inChannels * kernelSize * kernelSize; + + if (!UseParallelIm2Col || (long)kRows * n < ParallelIm2ColMinElements) { - var inChanBase = ic * inputH * inputW; - for (var ky = 0; ky < kernelSize; ky++) + for (var krow = 0; krow < kRows; krow++) { - for (var kx = 0; kx < kernelSize; kx++) - { - var krow = ((ic * kernelSize) + ky) * kernelSize + kx; - var dst = cols.Slice(krow * n, n); + Im2ColRow(input, cols, krow, inputH, inputW, kernelSize, padding, stride, outH, outW, n); + } - for (var oy = 0; oy < outH; oy++) - { - var iy = oy * stride - padding + ky; - var rowDst = dst.Slice(oy * outW, outW); - if ((uint)iy >= (uint)inputH) - { - rowDst.Clear(); - continue; - } - - var inRowBase = inChanBase + iy * inputW; - for (var ox = 0; ox < outW; ox++) - { - var ix = ox * stride - padding + kx; - rowDst[ox] = (uint)ix < (uint)inputW ? input[inRowBase + ix] : 0f; - } - } - } + return; + } + + fixed (float* pIn = input, pCols = cols) + { + var ctx = new Im2ColCtx( + pIn, pCols, inChannels, inputH, inputW, kernelSize, padding, stride, outH, outW, n); + + OverfitParallel.For(0, kRows, 1, &Im2ColRowRange, &ctx); + } + } + + private static unsafe void Im2ColRowRange(int start, int end, void* context) + { + ref var ctx = ref Unsafe.AsRef(context); + + var input = new ReadOnlySpan(ctx.Input, ctx.InChannels * ctx.InputH * ctx.InputW); + var cols = new Span(ctx.Cols, ctx.KRows * ctx.N); + + for (var krow = start; krow < end; krow++) + { + Im2ColRow( + input, cols, krow, + ctx.InputH, ctx.InputW, ctx.KernelSize, ctx.Padding, ctx.Stride, ctx.OutH, ctx.OutW, ctx.N); + } + } + + /// One row of the im2col matrix — the unit of both the serial and the parallel path, so the + /// two cannot drift apart. + private static void Im2ColRow( + ReadOnlySpan input, Span cols, int krow, + int inputH, int inputW, int kernelSize, int padding, int stride, int outH, int outW, int n) + { + var kx = krow % kernelSize; + var ky = (krow / kernelSize) % kernelSize; + var ic = krow / (kernelSize * kernelSize); + + var inChanBase = ic * inputH * inputW; + var dst = cols.Slice(krow * n, n); + + for (var oy = 0; oy < outH; oy++) + { + var iy = oy * stride - padding + ky; + var rowDst = dst.Slice(oy * outW, outW); + if ((uint)iy >= (uint)inputH) + { + rowDst.Clear(); + continue; + } + + var inRowBase = inChanBase + iy * inputW; + for (var ox = 0; ox < outW; ox++) + { + var ix = ox * stride - padding + kx; + rowDst[ox] = (uint)ix < (uint)inputW ? input[inRowBase + ix] : 0f; } } } + private readonly unsafe struct Im2ColCtx( + float* input, float* cols, + int inChannels, int inputH, int inputW, int kernelSize, + int padding, int stride, int outH, int outW, int n) + { + public readonly float* Input = input; + public readonly float* Cols = cols; + public readonly int InChannels = inChannels; + public readonly int InputH = inputH; + public readonly int InputW = inputW; + public readonly int KernelSize = kernelSize; + public readonly int Padding = padding; + public readonly int Stride = stride; + public readonly int OutH = outH; + public readonly int OutW = outW; + public readonly int N = n; + public readonly int KRows = inChannels * kernelSize * kernelSize; + } + // C[M,N] = A[M,K] @ B[K,N], parallelised over N-panels (each worker packs its 8-col B panel and sweeps M // with the full-K register-blocked micro-kernel). NOTE: a BLIS-style K-blocked + A-packed variant was // tried and MEASURED to regress on these CNN dims (deepcnn 101→125, vgg 140→189, resnet 45→118 ms) — diff --git a/Sources/Main/LanguageModels/Runtime/PrefillProfiler.cs b/Sources/Main/LanguageModels/Runtime/PrefillProfiler.cs index 03436f63..44909706 100644 --- a/Sources/Main/LanguageModels/Runtime/PrefillProfiler.cs +++ b/Sources/Main/LanguageModels/Runtime/PrefillProfiler.cs @@ -126,6 +126,30 @@ public static void Reset() /// Prompt tokens prefilled since the last . public static long Rows => _rows; + /// Requests measured since the last . + public static long Requests => _requests; + + /// + /// How many times ran per request — the number that exposes *redundant* + /// work, as opposed to slow work. + /// + /// A component's cost can be perfectly optimised internally and still be paid many more times + /// than the algorithm requires; that is invisible in a timing column and obvious in a call count. Two + /// real cases in this project: attn_out reported 306 calls instead of 36, which is how a + /// too-strict whole-matrix gate was found (it engaged in 18 of 36 layers); and the per-block F16 scale + /// decode looked amortised at "once per weight block" while the kernel holding it ran once per column + /// tile — 84 times per projection — which ablation priced at 12% of the kernel. + /// + /// Pinned by PrefillCallCountTests so a future gate or dispatch change that multiplies the + /// work fails a test instead of quietly costing throughput. + /// + public static double CallsPerRequest(Component component) + { + var requests = _requests == 0 ? 1 : _requests; + + return (double)_calls[(int)component] / requests; + } + /// /// Per-request breakdown: ms, % of prefill wall time, and calls. Also prints the prefill rate in /// tok/s, which is directly comparable to llama-bench -p N -n 0. diff --git a/Sources/Main/Runtime/OverfitEnvironment.cs b/Sources/Main/Runtime/OverfitEnvironment.cs index fe127a74..09e854b2 100644 --- a/Sources/Main/Runtime/OverfitEnvironment.cs +++ b/Sources/Main/Runtime/OverfitEnvironment.cs @@ -41,6 +41,9 @@ public static class OverfitEnvironment /// KV-cache element type — e.g. q8 for the int8 KV cache (default F32). public const string KvDType = "OVERFIT_KV_DTYPE"; + /// Set to 0 to force the serial im2col patch gather in the conv GEMM path (A/B switch). + public const string ParallelIm2Col = "OVERFIT_PARALLEL_IM2COL"; + // ── Prefill kernel switches (all default ON where the hardware allows; set to 0 to opt out) ── // These exist so a measured win can be A/B'd against its predecessor without a rebuild, and so a // regression on unfamiliar hardware can be bisected in the field rather than only on the dev box. diff --git a/Tests/LanguageModels/Diagnostics/PrefillCallCountTests.cs b/Tests/LanguageModels/Diagnostics/PrefillCallCountTests.cs new file mode 100644 index 00000000..999ab7e8 --- /dev/null +++ b/Tests/LanguageModels/Diagnostics/PrefillCallCountTests.cs @@ -0,0 +1,117 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.LanguageModels.Contracts; +using DevOnBike.Overfit.LanguageModels.Runtime; +using DevOnBike.Overfit.LanguageModels.Tokenizers; +using DevOnBike.Overfit.Tests.TestSupport; +using Xunit.Abstractions; + +namespace DevOnBike.Overfit.Tests.LanguageModels.Diagnostics +{ + /// + /// Pins how many times each prefill component runs per request — the measurement that catches + /// redundant work, which a timing column cannot. + /// + /// Why a separate test from the timing profile. A component can be optimal internally and + /// still be executed many more times than the algorithm needs. That shows up as "everything is a bit slow" + /// and is invisible unless something counts the calls. Two cases from this project: + /// + /// attn_out once reported 306 calls instead of 36 — the whole-matrix O gate demanded + /// Q/K/V/O all be Q4_K, but attn_v is Q6_K in half the layers under Q4_K_M, so 18 layers silently + /// fell back to 16 per-head dispatches. Output was correct; only the count showed it. + /// The per-block F16 scale decode read as amortised ("once per weight block") while the kernel + /// holding it ran once per column tile — 84 times per projection. Ablation priced it at 12%. + /// + /// + /// The invariant. Every projection below is a per-layer operation, so with a whole-matrix path + /// engaged each must run exactly layerCount times per request — never layerCount × headCount. + /// The assertions are upper bounds keyed to the layer count rather than exact equalities, so an + /// architecture with a different KV-group structure does not fail spuriously; what they forbid is the + /// per-head explosion, which is off by more than 10×. + /// + /// Model-gated like the other diagnostics: without the Qwen-3B fixture it logs and returns, so CI + /// (which has no fixtures) stays green. + /// + public sealed class PrefillCallCountTests + { + private readonly ITestOutputHelper _out; + + public PrefillCallCountTests(ITestOutputHelper output) => _out = output; + + [LongFact] + public void Prefill_ComponentCallCounts_AreOncePerLayer_NotOncePerHead() + { + var path = TestModelPaths.Qwen3B.Q4KmGgufPath; + if (!File.Exists(path)) + { + _out.WriteLine($"missing {path}"); + return; + } + + using var engine = CachedLlamaInferenceEngine.LoadGguf(path); + var tok = GgufTokenizer.Load(path); + var sampling = SamplingOptions.Greedy; + + var paragraph = string.Join(" ", + Enumerable.Repeat( + "The history of computing began with mechanical calculators and evolved through vacuum tubes, " + + "transistors, integrated circuits and finally the microprocessor era.", 24)); + var ids = tok.Encode(paragraph); + + // Warm up outside the counted region: the one-off repack must not be attributed here. + using (var warm = engine.CreateSession(1024)) + { + warm.Reset(ids); + warm.GenerateNextToken(in sampling); + } + + PrefillProfiler.Reset(); + PrefillProfiler.Enabled = true; + try + { + using var session = engine.CreateSession(1024); + session.Reset(ids); + } + finally + { + PrefillProfiler.Enabled = false; + } + + // The FFN runs once per layer by construction, so its own count IS the layer count — no model + // metadata needed, and the test stays correct for any depth. + var layers = PrefillProfiler.CallsPerRequest(PrefillProfiler.Component.Ffn); + + _out.WriteLine($"layers (from ffn count): {layers:F0}"); + foreach (var component in Enum.GetValues()) + { + _out.WriteLine($" {component,-12} {PrefillProfiler.CallsPerRequest(component),8:F1} calls/request"); + } + + Assert.True(layers > 0, "profiler recorded no FFN calls — hooks not reached"); + + // One dispatch per layer each. The per-head failure mode would be ~16× these numbers, so a 2× + // allowance leaves room for legitimate structure (K and V are two dispatches, GQA groups) while + // still failing hard on the explosion this test exists to catch. + AssertPerLayer(PrefillProfiler.Component.FfnGateUp, layers, 1); + AssertPerLayer(PrefillProfiler.Component.FfnDown, layers, 1); + AssertPerLayer(PrefillProfiler.Component.AttnQ, layers, 2); + AssertPerLayer(PrefillProfiler.Component.AttnOut, layers, 2); + AssertPerLayer(PrefillProfiler.Component.AttnKv, layers, 4); + } + + private void AssertPerLayer(PrefillProfiler.Component component, double layers, double allowance) + { + var calls = PrefillProfiler.CallsPerRequest(component); + var limit = layers * allowance; + + Assert.True( + calls <= limit, + $"{component} ran {calls:F0}× per request against a {limit:F0}× budget ({layers:F0} layers). " + + "That is the per-head dispatch pattern the whole-matrix projections exist to remove — check " + + "the gate that selects them (a too-strict condition silently falls back per head)."); + } + } +} From 74258f3b9ff8304ee1b3d6bef4309b7706d690a6 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Thu, 23 Jul 2026 12:00:41 +0200 Subject: [PATCH 28/37] onnx --- ROADMAP.md | 28 +++++- Sources/Main/Onnx/OnnxGraphModel.cs | 129 +++++++++++++++++++++++++++- 2 files changed, 150 insertions(+), 7 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index a49958aa..71200074 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -175,9 +175,31 @@ Speedup over one worker **4.69× → 9.16×**; serial fraction **15.5% → 7.3%* (maxAbsDiff 6.7e-8, cosine 1.000000, same argmax), conv tests 56/0, suite 1499/0/232. The paired A/B run had ORT flat at 11.66 vs 11.75 ms as the canary. -**Next: the remaining ~49 ms of serial work**, which now *dominates* at high worker counts (49 of 73 ms at -32 workers). Candidates not yet measured: the elementwise activations, MaxPool, the FC layers, and the -graph executor's inter-node buffer handling. Measure which before touching any of them. +#### ▶ RETRACTED — there is no "~49 ms of serial work". Conv is simply 91% of the time + +The Amdahl fit above predicted ~49 ms of serial residue dominating at 32 workers. `OnnxGraphModel.ProfileNodes` +(new, opt-in, the CNN counterpart of `PrefillProfiler`) measured it per operator instead: + +| operator | 32 workers | share | 1 worker | +|---|---:|---:|---:| +| **ConvLayer** (13 nodes) | **72.10 ms** | **90.7%** | 1557 ms | +| MaxPool2DLayer (5) | 5.22 | 6.6% | 5.46 | +| ReluActivation (13) | 1.92 | 2.4% | 1.70 | +| LinearLayer (1) | 0.21 | 0.3% | 0.23 | +| GlobalAveragePool2DLayer | 0.01 | 0.0% | 0.01 | + +MaxPool and ReLU *are* serial — they do not move between 1 and 32 workers — but together they are **7.1 ms, +9% of the total, not 49 ms**. The Amdahl model overestimated the serial share by more than 6×, because a +clean serial/parallel split does not describe this system: Conv's scaling is imperfect (21.6×), not absent, +and imperfect scaling reads as "serial fraction" to that fit. **Treat Amdahl fits as a pointer, not a +measurement — it was right that something was wrong, and wrong about what and how much.** + +**So the next lever is kernel quality, not parallelism.** Conv does 15.5 GFLOP in 72.1 ms = **215 GFLOP/s +against a 2190 GFLOP/s ceiling — 10%**. Parallelising MaxPool + ReLU is real but capped at ~1.1× overall. + +*Unexplained and therefore not built on:* the standalone driver measures 1564 ms at one worker where the +BenchmarkDotNet sweep measured 668 ms — same variable, same box. The per-operator conclusion rests on the +default-worker numbers, where the two agree (73 vs 79.5 ms); the single-worker column is indicative only. --- diff --git a/Sources/Main/Onnx/OnnxGraphModel.cs b/Sources/Main/Onnx/OnnxGraphModel.cs index 73c9440a..5ec5aed7 100644 --- a/Sources/Main/Onnx/OnnxGraphModel.cs +++ b/Sources/Main/Onnx/OnnxGraphModel.cs @@ -3,6 +3,8 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com +using System.Diagnostics; +using System.Text; using DevOnBike.Overfit.DeepLearning; using DevOnBike.Overfit.Tensors.Core; @@ -70,8 +72,10 @@ public void RunInference(ReadOnlySpan input, Span output) // Slot 0 = model input. input.CopyTo(_buffers[0].AsSpan()); - foreach (var node in _nodes) + for (var i = 0; i < _nodes.Length; i++) { + var node = _nodes[i]; + var started = ProfileNodes ? Stopwatch.GetTimestamp() : 0L; var outBuf = _buffers[node.OutputSlot].AsSpan().Slice(0, node.OutputSize); if (node.InputSlots.Length == 2 && node.Module is OnnxAddLayer addLayer) @@ -80,11 +84,18 @@ public void RunInference(ReadOnlySpan input, Span output) var left = _buffers[node.InputSlots[0]].AsSpan(); var right = _buffers[node.InputSlots[1]].AsSpan(); addLayer.ForwardInference(left, right, outBuf); - continue; } - var inBuf = _buffers[node.InputSlots[0]].AsSpan(); - node.Module.ForwardInference(inBuf, outBuf); + if (!(node.InputSlots.Length == 2 && node.Module is OnnxAddLayer)) + { + var inBuf = _buffers[node.InputSlots[0]].AsSpan(); + node.Module.ForwardInference(inBuf, outBuf); + } + + if (ProfileNodes) + { + RecordNode(i, Stopwatch.GetTimestamp() - started); + } } // Last node's output slot → caller's output span. @@ -92,6 +103,116 @@ public void RunInference(ReadOnlySpan input, Span output) _buffers[lastNode.OutputSlot].AsSpan().Slice(0, _outputSize).CopyTo(output); } + /// + /// Opt-in per-node timing. Off by default and checked before any timestamp is taken, so the inference + /// path is unchanged when it is off. + /// + /// This is the CNN counterpart of PrefillProfiler, and exists for the same reason: after + /// parallelising the im2col gather, VGG-16's Amdahl serial fraction fell from ~15.5% to ~7.3% — but at + /// 32 workers that residue still dominates (~49 of 73 ms). Guessing which operator holds it + /// would be guessing about mechanism, which this project has been wrong about repeatedly; this + /// measures it per operator instead. + /// + public static bool ProfileNodes; + + private long[]? _nodeTicks; + private long[]? _nodeCalls; + + private void RecordNode(int index, long ticks) + { + _nodeTicks ??= new long[_nodes.Length]; + _nodeCalls ??= new long[_nodes.Length]; + + _nodeTicks[index] += ticks; + _nodeCalls[index]++; + } + + /// Clears the per-node accumulators (call before the measured segment). + public void ResetNodeProfile() + { + _nodeTicks = null; + _nodeCalls = null; + } + + /// + /// Per-operator totals, heaviest first: total ms, share of the measured wall time, and node count. + /// Grouped by module type, because "which operator" is the actionable unit — not which of 40 nodes. + /// + public string NodeProfileReport() + { + if (_nodeTicks is null) + { + return "(no node profile recorded — set OnnxGraphModel.ProfileNodes before running)"; + } + + // Aggregate into parallel arrays and selection-sort them. A Dictionary + OrderByDescending would + // read better, but LINQ is banned in Sources/Main (RS0030) and the node count is tiny, so an + // O(n^2) sort over distinct operator names costs nothing. + var names = new string[_nodes.Length]; + var ticks = new long[_nodes.Length]; + var counts = new int[_nodes.Length]; + var distinct = 0; + var total = 0L; + + for (var i = 0; i < _nodes.Length; i++) + { + var name = _nodes[i].Module.GetType().Name; + total += _nodeTicks[i]; + + var slot = -1; + for (var j = 0; j < distinct; j++) + { + if (string.Equals(names[j], name, StringComparison.Ordinal)) + { + slot = j; + break; + } + } + + if (slot < 0) + { + slot = distinct; + names[slot] = name; + distinct++; + } + + ticks[slot] += _nodeTicks[i]; + counts[slot]++; + } + + for (var a = 0; a < distinct - 1; a++) + { + var best = a; + for (var bIdx = a + 1; bIdx < distinct; bIdx++) + { + if (ticks[bIdx] > ticks[best]) + { + best = bIdx; + } + } + + (names[a], names[best]) = (names[best], names[a]); + (ticks[a], ticks[best]) = (ticks[best], ticks[a]); + (counts[a], counts[best]) = (counts[best], counts[a]); + } + + var toMs = 1000.0 / Stopwatch.Frequency; + var runs = _nodeCalls is null || _nodeCalls.Length == 0 ? 1L : Math.Max(1L, _nodeCalls[0]); + var sb = new StringBuilder(); + + sb.AppendLine($"=== OnnxGraphModel node profile ({runs} run(s), {_nodes.Length} nodes) ==="); + sb.AppendLine($" total: {total * toMs / runs:F2} ms/run"); + + for (var i = 0; i < distinct; i++) + { + var ms = ticks[i] * toMs / runs; + sb.AppendLine( + $" {names[i],-28} {ms,8:F2} ms {(total == 0 ? 0 : 100.0 * ticks[i] / total),6:F1}% ({counts[i]} nodes)"); + } + + return sb.ToString(); + } + /// /// Sets all modules to evaluation mode (uses running stats for BatchNorm, etc.). /// Should be called before inference. From 20550e388db2c26b5a38cd55cc448f0a9586c84f Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Thu, 23 Jul 2026 13:14:23 +0200 Subject: [PATCH 29/37] onnx --- Overfit.sln | 2 - ROADMAP.md | 234 ++++++- Sources/Benchmark/GemmKBlockingBenchmark.cs | 204 ++++++ .../GemmMicroKernelShapeBenchmark.cs | 451 +++++++++++++ Sources/Main/Kernels/Conv2DGemmKernels.cs | 315 ++++++++- Sources/Main/Onnx/OnnxGraphModel.cs | 31 + Sources/Main/Runtime/OverfitEnvironment.cs | 6 + Tests/Diagnostics/ConvGemmPartProfileTests.cs | 227 +++++++ Tests/Diagnostics/MachineProbeTests.cs | 598 ++++++++++++++++++ 9 files changed, 2049 insertions(+), 19 deletions(-) create mode 100644 Sources/Benchmark/GemmKBlockingBenchmark.cs create mode 100644 Sources/Benchmark/GemmMicroKernelShapeBenchmark.cs create mode 100644 Tests/Diagnostics/ConvGemmPartProfileTests.cs create mode 100644 Tests/Diagnostics/MachineProbeTests.cs diff --git a/Overfit.sln b/Overfit.sln index 5999aeba..7a3bee7f 100644 --- a/Overfit.sln +++ b/Overfit.sln @@ -53,8 +53,6 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "MnistWpfDemo", "Demo\MnistW EndProject Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "AgentFrameworkDemo", "Demo\AgentFrameworkDemo\AgentFrameworkDemo.csproj", "{B790BE61-A6E4-4556-8019-08778CF2172A}" EndProject -Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Sources", "Sources", "{C16B2082-E6A6-C480-36D0-FC08AA18D453}" -EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|Any CPU = Debug|Any CPU diff --git a/ROADMAP.md b/ROADMAP.md index 71200074..b3e8a75f 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -194,8 +194,238 @@ clean serial/parallel split does not describe this system: Conv's scaling is imp and imperfect scaling reads as "serial fraction" to that fit. **Treat Amdahl fits as a pointer, not a measurement — it was right that something was wrong, and wrong about what and how much.** -**So the next lever is kernel quality, not parallelism.** Conv does 15.5 GFLOP in 72.1 ms = **215 GFLOP/s -against a 2190 GFLOP/s ceiling — 10%**. Parallelising MaxPool + ReLU is real but capped at ~1.1× overall. +**So the next lever is kernel quality, not parallelism.** Parallelising MaxPool + ReLU is real but capped at +~1.1× overall. + +#### ★ im2col vs GEMM, and a FLOP-counting correction that changes the target + +`Conv2DGemmKernels.ProfileParts` (opt-in) + `ConvGemmPartProfileTests` split conv time on VGG-16: + +| part | per run | share of conv | +|---|---:|---:| +| im2col gather | 9.33 ms | 14.7% | +| **GEMM** | **54.22 ms** | **85.3%** | + +So after parallelising the gather, **the micro-kernel is the target** — confirmed rather than assumed. + +**Correction to every VGG GFLOP/s figure above.** They used "15.5 GFLOPs/inference", which is the commonly +quoted VGG-16 **MAC** count. Under this project's convention (MAC = 2 ops, matching `Throughput` and +llama.cpp's `test-backend-ops`) VGG-16's conv layers are **30.7 GFLOP**. The earlier rates were understated 2×: + +| | time | GFLOP/s | % of the 2190 GFLOP/s ceiling | +|---|---:|---:|---:| +| our GEMM alone | 54.2 ms | **566** | **26%** | +| whole model, Overfit | 73 ms | 420 | — | +| whole model, ORT | 11.6 ms | **2647** | **121%** ⚠ | + +**ORT "achieves" 121% of this machine's measured float ceiling, which is impossible** — so it is executing +fewer operations than the formula counts. That is direct evidence for the Winograd hypothesis flagged +earlier: MLAS uses a FLOP-reducing transform on 3×3 convs (F(2,3) cuts FLOPs 2.25×). Corrected, ORT runs at +roughly **1176 GFLOP/s ≈ 54% of ceiling**. + +**This reframes the gap: part of ORT's lead is algorithmic, not kernel craft.** Our GEMM at 26% against +their ~54% is about **2×** of kernel-quality difference, with the rest coming from doing less work. + +#### ▶ REFUTED — the micro-kernel tile shape is NOT the problem; it is already at hardware peak + +The hypothesis was that `Mr=8 × Nr=8` is load-port bound (9 loads per 8 FMAs) and that a `6×16` tile would +pay. `GemmMicroKernelShapeBenchmark` measured the candidate shapes single-threaded, accumulators in named +locals, panels L1-resident: + +| shape (1 thread) | GFLOP/s | vs today | +|---|---:|---:| +| **AVX2 8×8 (today)** | **148** | 1.00× | +| AVX2 6×16 | 182 | 1.23× | +| AVX2 4×24 | 182 | 1.23× | +| AVX-512 8×16 | 276 | 1.86× | +| **AVX-512 8×32** | **337** | **2.27×** | +| AVX-512 6×48 | 337 | 2.27× | + +**AVX2's single-core FMA peak is ≈138 GFLOP/s** (8 lanes × 2 ops × 2 FMA units × ~4.3 GHz), and the current +shape measures **148** — it is already at the hardware ceiling, boost clock and all. The load-port argument +was wrong: Zen 5 sustains those loads. Reshaping the AVX2 tile is worth ~1.2×, not the 4× the production +deficit implies. + +**What this reveals instead.** The micro-kernel can do 148 GFLOP/s per core → ~2370 GFLOP/s across 16 cores. +Production conv GEMM does **566 — 24% of what its own micro-kernel achieves when fed properly.** The kernel +is fine; **everything around it is not**: B-panel packing, the memory traffic of a `[K, N]` im2col matrix +(conv1_2's is 115 MB, far past any cache), and panel scheduling. That is the target, not the tile. + +AVX-512 is separately worth **2.27×** on the micro-kernel — but only to the extent production is +compute-bound, and at 24% efficiency it plainly is not. Expect far less than 2.27× end to end. + +#### ▶ REFUTED AGAIN — packing is not it either. The micro-kernel is starved by the cache hierarchy + +Ablation inside `GemmNPanelWorker` (`AblatePackB` / `AblateMicroKernel`, measurement-only), VGG-16: + +| arm | ms/run | share | +|---|---:|---:| +| baseline (pack + micro) | 76.04 | 100% | +| pack only | 27.61 | 36.3% | +| micro only | 67.66 | 89.0% | + +Netting out the rest of the model (im2col 9.3, MaxPool+ReLU 7.1): **packing ≈ 11 ms, micro-kernel ≈ 51 ms.** +The negative "unattributed" (−19 ms) is expected overlap — removing either side frees cache for the other — +so both figures are upper bounds. Either way the micro-kernel dominates the GEMM, and the strided scalar +pack, plausible as it looked, is the minority cost. + +**The one number that matters.** The same micro-kernel measures **148 GFLOP/s per core in isolation** and +**~38 GFLOP/s per core in production** (51 ms for 30.7 GFLOP over 16 cores) — **4× slower running the same +instructions.** The difference is the memory feed: `packB` is `K × Nr` floats, which at K=2304 is **73 KB +against a 32–48 KB L1**, so every row-block re-streams the panel from L2, and the A rows (8 × K floats, +another 73 KB) do the same. The isolation benchmark had both in L1, which is exactly why it hit peak. + +**So the target is K-blocking** — split the contraction so `packB` and the A slice fit L1. One caveat that is +mine to state: the kernel's own comment records that a BLIS-style **K-blocked + A-packed** variant was tried +and regressed (vgg 140 → 189 ms). That measurement predates the parallel im2col and bundled A-packing, whose +one-time `O(M·K)` cost may have dominated it. It is not proof that K-blocking alone fails — and equally, not +licence to repeat it blind. Measure the L1-residency effect on a single VGG layer shape first. + +#### ▶ REFUTED — K-blocking does not help either, and the prototype exposes where the loss really is + +`GemmKBlockingBenchmark` runs the full conv5_1 GEMM (M=512, K=4608, N=196) single-threaded at several +contraction blocks. `Kc=4608` is today's unblocked kernel (144 KB packed panel); the rest bring it inside L1: + +| Kc | packed panel | ms | TFLOP/s | +|---|---:|---:|---:| +| **4608 (today)** | 144 KB | **6.99** | **0.13** | +| 1152 | 36 KB | 7.24 | 0.13 | +| 512 | 16 KB | 7.25 | 0.13 | +| 256 | 8 KB | 8.00 | 0.12 | +| 128 | 4 KB | 7.96 | 0.12 | + +**Unblocked wins.** Making the panel L1-resident is flat to slightly worse, so the L1-residency hypothesis is +refuted and the earlier K-blocking negative is independently confirmed — this time without A-packing to +confound it. + +**But the prototype answers a better question than the one asked.** It does everything production does — +pack, micro-kernel, real memory — single-threaded at **132 GFLOP/s**, while production conv at one worker +runs at **19.7 GFLOP/s** (30.7 GFLOP in 1557 ms). Same structure, **6.7× apart**. So the deficit is neither +the tile, nor the pack, nor cache blocking: it is **shape-dependent**, and this prototype picked a shape +where everything is fine (small N, large K). + +The suspicion now points at the early layers, where the arithmetic-to-overhead ratio inverts. `conv1_2` is +M=64, K=576, **N=50176**: A (147 KB) is re-read for each of **6272 panels**, and the pack does one scalar +branchy copy per FMA issued. **Next measurement: per-layer conv timing, not per-operator** — then each +layer's achieved GFLOP/s against its own shape. + +**Four hypotheses refuted in a row on this path** — tile shape, packing, L1 blocking, and the "49 ms serial" +model. Each cost minutes to measure; the rewrites they prevented would have cost days. + +#### ★★★ FOUND IT — A is re-read once per N-panel: 7.5 GB of traffic for 30.7 GFLOP of work + +The per-layer profile (`PerNodeProfileReport`) shows the conv layers are **uniform**, 347–581 GFLOP/s, with +no outlier — so "the early layers are the problem" is refuted too. The shape table is where it shows: +`GemmNPanelWorker` sweeps M *inside* the panel loop, so the whole A matrix is re-read **for every N-panel**. + +| layer | panels | A | A traffic | +|---|---:|---:|---:| +| conv2 | 6272 | 144 KB | 903 MB | +| conv4 | 1568 | 576 KB | 903 MB | +| conv6 / conv7 | 392 | 2304 KB | 903 MB each | +| conv9 / conv10 | 98 | 9216 KB | 903 MB each | +| others | | | ~2.1 GB | +| **total** | | | **≈7.5 GB per inference** | + +**7.5 GB moved for 30.7 GFLOP computed = 0.24 bytes/FLOP**, where a well-blocked GEMM runs at ~0.01 — **24× +more traffic than the arithmetic requires**. And it matches the clock: 7.5 GB in 73 ms is **103 GB/s**, +against a measured 90 GB/s DRAM read ceiling (L3 is faster, but finite and shared by 16 cores). + +**The conv GEMM is bandwidth-bound on re-reading A** — which is why a single-threaded prototype hit +132 GFLOP/s while production gets ~30 per core: one thread has the cache to itself. + +**Fix: block over N-panels.** Process a group of panels (e.g. 8 = 64 columns) and sweep M once per group, +cutting A traffic by the group size — 7.5 GB → ~0.94 GB at 8 panels. This is the outer half of the standard +Goto/BLIS structure, and it is the piece this kernel has never had. + +**Why every earlier hypothesis missed it:** tile shape, packing and K-blocking are all *within* one panel. +The waste is *between* panels, which no measurement scoped to a single panel could see. + +#### ▶ REFUTED — N-panel grouping does nothing, and the traffic argument was wrong about *where* + +`Conv2DGemmKernels.NPanelGroup` (default **1**, `OVERFIT_CONV_PANEL_GROUP`) packs and sweeps several panels +together so A is read once per group. Interleaved against an ORT canary (4% spread over the whole sweep): + +| group | 1 | 2 | 4 | +|---|---:|---:|---:| +| VGG-16 | 72.7 ms | 73.0 | 73.3 | + +0.8% apart — inside the noise. **The 7.5 GB figure was right; the conclusion drawn from it was not.** That +traffic never reaches DRAM: this CPU has **128 MB of L3 (V-cache)**, so every layer's A (≤9 MB) is re-read +from L3. Counting bytes without asking *which cache level serves them* is worthless. + +#### ★★ `Sources/MachineProbe` — a standalone hardware probe, and it explains both blocking failures + +A console app with **no reference to Overfit, no BenchmarkDotNet, no packages** — one file, `Stopwatch` only, +so it can be run on a customer box, a CI runner or a cloud VM before anyone reads meaning into an Overfit +number. `dotnet run -c Release --project Sources/MachineProbe`. + +On the 9950X3D: + +| peak FMA | 1 core | all cores | scaling | +|---|---:|---:|---:| +| 128-bit | 84 GF/s | 1336 | 15.8× | +| 256-bit | **179** | **2310** | 12.9× | +| 512-bit | 351 | 4200 | 12.0× | + +Memory: read **89.0 GB/s**, copy 74.0, triad 49.5. + +**Correction it forces:** the AVX2 single-core peak was *estimated* at 138 GF/s, which made the 148 GF/s +micro-kernel look like it exceeded the hardware. Measured, the peak is **179** — the micro-kernel is at +**83% of it**, still high, but the earlier claim was arithmetic, not measurement. + +**The working-set sweep is the real payload** (one core, sequential read): + +| 8 KB | 48 KB | 512 KB | 8 MB | 32 MB | 128 MB | +|---:|---:|---:|---:|---:|---:| +| 72.9 | 74.6 | 75.3 | 76.0 GB/s | 68.9 | 58.5 | + +**Flat from L1 to 8 MB.** One core reads ~75 GB/s *wherever the data lives* — for streaming access this +machine has **no L1/L2/L3 cliff at all**, because the prefetcher keeps up. + +*The first version of that sweep was wrong and the conclusion drawn from it is withdrawn.* It used a single +`Vector` accumulator, so it measured the chain's **latency** (~75 GB/s, below every cache level's +bandwidth) and produced a perfectly flat curve that appeared to prove "this machine has no cache cliff". With +eight independent streams the structure appears — see below. The probe now lives in +`Tests/Diagnostics/MachineProbeTests.cs` (xUnit, `ValueStopwatch`, asserts its own loops allocate 0 B). + +#### ★★★ WHY PARALLEL SCALING IS POOR — bandwidth stops scaling past ~2 MB per core + +| working set | 1 core | all cores | scaling | +|---|---:|---:|---:| +| 16 KB – 2 MB | ~76 GB/s | 700–900 GB/s | **9–12×** | +| 8 MB | 67.3 | 112.8 | **1.7×** | +| 32 MB | 64.4 | 67.6 | 1.1× | +| 128 MB (DRAM) | 59.6 | 63.6 | 1.1× | + +**Compute scales 12–15×; bandwidth scales 10× only while the per-core working set fits private cache, then +collapses to ~1×.** One core already draws 60% of total DRAM bandwidth; the other fifteen add 60%. + +The cliff lands exactly where **16 cores × 8 MB = 128 MB = this chip's L3 including V-cache** — the +measurement validates itself against a number it was never given. + +**So any kernel that outruns private cache cannot be fixed by more cores or by blocking — only by needing +fewer bytes per FLOP.** That reframes conv: the fix is arithmetic intensity, not scheduling. + +#### ★★ AVX-512 8×32 conv micro-kernel — VGG-16 72.7 → 63.7 ms (1.14×) + +An `Mr×Nr` tile loads `Mr+Nr` floats per k-step and performs `2·Mr·Nr` FLOPs, so intensity is +`Mr·Nr / (2(Mr+Nr))`: **2.0 FLOP/byte at 8×8, 3.2 at 8×32**. AVX-512's 32 registers make 16 accumulators +plus 2 B vectors and a broadcast fit. Interleaved A/B, three rounds, ORT canary within 2%: + +| | median | GFLOP/s | +|---|---:|---:| +| AVX2 8×8 | 72.7 ms | 422 | +| **AVX-512 8×32** | **63.7 ms** | **482** | + +Parity exact in every round, conv tests 56/0, `OVERFIT_CONV_AVX512=0` falls back. **Gap to ORT 6.3× → 5.35×.** + +*Honest note on the model:* intensity predicted up to 1.6× and delivered 1.14×, so intensity is a real but +not dominant term — do not extrapolate a further tile widening from it without measuring. + +**Machine identity, measured rather than reported** (`MachineProbeTests`): AMD Ryzen 9 9950X3D, **5.59 GHz** +from a dependent-add chain — cross-checked against 5.53 GHz derived independently from the AVX2 FMA peak, +agreeing to 1%. *Unexplained and therefore not built on:* the standalone driver measures 1564 ms at one worker where the BenchmarkDotNet sweep measured 668 ms — same variable, same box. The per-operator conclusion rests on the diff --git a/Sources/Benchmark/GemmKBlockingBenchmark.cs b/Sources/Benchmark/GemmKBlockingBenchmark.cs new file mode 100644 index 00000000..0b1337c4 --- /dev/null +++ b/Sources/Benchmark/GemmKBlockingBenchmark.cs @@ -0,0 +1,204 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Runtime.Intrinsics; +using System.Runtime.Intrinsics.X86; +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Running; +using Benchmarks.Helpers; + +namespace Benchmarks +{ + /// + /// Measures whether making the micro-kernel's working set L1-resident recovers the 4× it loses + /// between isolation and production — on one real VGG layer shape, single-threaded, before any of this is + /// written into Conv2DGemmKernels. + /// + /// The finding this tests. The 8×8 micro-kernel measures 148 GFLOP/s per core in isolation + /// (its AVX2 hardware peak) but ~38 GFLOP/s per core inside the conv GEMM — the same instructions, 4× + /// slower. The suspected cause is the working set: the packed B panel is K × 8 floats, which at + /// K=4608 is 147 KB against a 32–48 KB L1, and the A row-block is another 147 KB. Splitting the + /// contraction into Kc-sized blocks shrinks both to Kc × 8 × 4 bytes. + /// + /// Why this is a prototype, not a micro-benchmark. It performs the entire layer GEMM at each + /// blocking factor, so the arms differ only in Kc: total FLOPs, total A traffic and total B traffic + /// are identical by construction (each element is still touched once per panel sweep). What changes is + /// solely what fits in L1 — and the extra C read-modify-write that blocking forces, which is part of the + /// cost being measured and is deliberately not excluded. + /// + /// Shape: VGG-16 conv5_1 — M=512 output channels, K=4608 (512·3·3), N=196 (14×14). + /// Chosen because K is at its largest here, the extreme case for L1 overflow, while the small N keeps a + /// single-threaded run short. Kc = K reproduces today's unblocked kernel exactly. + /// + /// Single-threaded on purpose: parallel scaling was measured separately at 21.6×, so mixing it in + /// would only add noise to a cache question. + /// + /// Run: + /// dotnet run -c Release --project Sources/Benchmark -- --filter "*GemmKBlocking*" + /// + [Config(typeof(BenchmarkConfig))] + public class GemmKBlockingBenchmark + { + /// VGG-16 conv5_1: output channels. + public const int M = 512; + + /// VGG-16 conv5_1: inChannels·3·3 — the contraction length. + public const int K = 4608; + + /// VGG-16 conv5_1: outH·outW = 14·14. + public const int N = 196; + + private const int Mr = 8; + private const int Nr = 8; + + /// + /// Contraction block. 4608 is today's behaviour (no blocking, 147 KB panel); the smaller values + /// bring the packed panel to 16 KB / 8 KB / 4 KB, i.e. inside L1 with room for the A block beside it. + /// + [Params(4608, 1152, 512, 256, 128)] + public int Kc + { + get; set; + } + + private float[] _a = null!; + private float[] _b = null!; + private float[] _c = null!; + private float[] _packB = null!; + + public float Sink; + + public static WorkAmount GetWorkAmount(BenchmarkCase benchmarkCase) + { + return WorkAmount.Matmul(M, K, N); + } + + [GlobalSetup] + public void Setup() + { + var rng = new Random(20260723); + + _a = new float[(long)M * K]; + _b = new float[(long)K * N]; + _c = new float[(long)M * N]; + _packB = new float[(long)K * Nr]; + + for (var i = 0; i < _a.Length; i++) + { + _a[i] = (float)((rng.NextDouble() * 2.0) - 1.0); + } + + for (var i = 0; i < _b.Length; i++) + { + _b[i] = (float)((rng.NextDouble() * 2.0) - 1.0); + } + } + + /// + /// The layer GEMM with the contraction split into -sized blocks. At Kc == K this + /// is exactly the production loop: pack the whole panel, then sweep M once. + /// + [Benchmark] + public unsafe void BlockedGemm() + { + var panels = (N + Nr - 1) / Nr; + + fixed (float* a = _a, b = _b, c = _c, packB = _packB) + { + new Span(c, M * N).Clear(); + + for (var k0 = 0; k0 < K; k0 += Kc) + { + var kcEff = Math.Min(Kc, K - k0); + + for (var np = 0; np < panels; np++) + { + var n0 = np * Nr; + var nrEff = Math.Min(Nr, N - n0); + + for (var kk = 0; kk < kcEff; kk++) + { + var src = ((k0 + kk) * N) + n0; + var dst = kk * Nr; + for (var j = 0; j < Nr; j++) + { + packB[dst + j] = j < nrEff ? b[src + j] : 0f; + } + } + + for (var m0 = 0; m0 + Mr <= M; m0 += Mr) + { + Accumulate8x8(a, m0, k0, kcEff, packB, c, n0, nrEff); + } + } + } + } + + Sink = _c[0]; + } + + /// + /// One 8×8 tile, accumulated over a -long slice of the contraction. + /// + /// Unlike the production kernel this loads C in and stores it back, because a K-block only holds + /// a partial sum. That extra read-modify-write per (K-block, panel, row-block) is the price of blocking + /// and is measured here rather than assumed away — at Kc = K it happens once and costs nothing. + /// + private static unsafe void Accumulate8x8( + float* a, int m0, int k0, int kcEff, float* packB, float* c, int n0, int nrEff) + { + var a0 = a + ((long)(m0 + 0) * K) + k0; + var a1 = a + ((long)(m0 + 1) * K) + k0; + var a2 = a + ((long)(m0 + 2) * K) + k0; + var a3 = a + ((long)(m0 + 3) * K) + k0; + var a4 = a + ((long)(m0 + 4) * K) + k0; + var a5 = a + ((long)(m0 + 5) * K) + k0; + var a6 = a + ((long)(m0 + 6) * K) + k0; + var a7 = a + ((long)(m0 + 7) * K) + k0; + + var acc0 = Vector256.Zero; + var acc1 = Vector256.Zero; + var acc2 = Vector256.Zero; + var acc3 = Vector256.Zero; + var acc4 = Vector256.Zero; + var acc5 = Vector256.Zero; + var acc6 = Vector256.Zero; + var acc7 = Vector256.Zero; + + for (var kk = 0; kk < kcEff; kk++) + { + var bv = Vector256.Load(packB + (kk * Nr)); + + acc0 = Fma.MultiplyAdd(Vector256.Create(a0[kk]), bv, acc0); + acc1 = Fma.MultiplyAdd(Vector256.Create(a1[kk]), bv, acc1); + acc2 = Fma.MultiplyAdd(Vector256.Create(a2[kk]), bv, acc2); + acc3 = Fma.MultiplyAdd(Vector256.Create(a3[kk]), bv, acc3); + acc4 = Fma.MultiplyAdd(Vector256.Create(a4[kk]), bv, acc4); + acc5 = Fma.MultiplyAdd(Vector256.Create(a5[kk]), bv, acc5); + acc6 = Fma.MultiplyAdd(Vector256.Create(a6[kk]), bv, acc6); + acc7 = Fma.MultiplyAdd(Vector256.Create(a7[kk]), bv, acc7); + } + + StoreRow(c, (m0 + 0) * N + n0, acc0, nrEff); + StoreRow(c, (m0 + 1) * N + n0, acc1, nrEff); + StoreRow(c, (m0 + 2) * N + n0, acc2, nrEff); + StoreRow(c, (m0 + 3) * N + n0, acc3, nrEff); + StoreRow(c, (m0 + 4) * N + n0, acc4, nrEff); + StoreRow(c, (m0 + 5) * N + n0, acc5, nrEff); + StoreRow(c, (m0 + 6) * N + n0, acc6, nrEff); + StoreRow(c, (m0 + 7) * N + n0, acc7, nrEff); + } + + private static unsafe void StoreRow(float* c, int offset, Vector256 acc, int nrEff) + { + var dst = c + offset; + + for (var j = 0; j < nrEff; j++) + { + dst[j] += acc.GetElement(j); + } + } + } +} diff --git a/Sources/Benchmark/GemmMicroKernelShapeBenchmark.cs b/Sources/Benchmark/GemmMicroKernelShapeBenchmark.cs new file mode 100644 index 00000000..dbc14eb3 --- /dev/null +++ b/Sources/Benchmark/GemmMicroKernelShapeBenchmark.cs @@ -0,0 +1,451 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Runtime.Intrinsics; +using System.Runtime.Intrinsics.X86; +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Running; +using Benchmarks.Helpers; + +namespace Benchmarks +{ + /// + /// Prices the candidate SGEMM micro-kernel tile shapes before any of them is written into + /// Conv2DGemmKernels — the ceiling-first step that made the LLM prefill track pay. + /// + /// The question. The conv GEMM runs at 566 GFLOP/s against a 2190 GFLOP/s machine ceiling + /// (26%). Its micro-kernel is Mr=8 × Nr=8, which per k-step issues 8 FMAs against 9 loads + /// (one B vector + eight A broadcasts) — arithmetic that says it is limited by the load ports, not the FMA + /// units. Widening the tile amortises the B loads over more FMAs. Whether that actually pays, and which + /// shape pays most on this silicon, is measured here rather than argued. + /// + /// What each shape costs per k-step (V = floats per vector: 8 for AVX2, 16 for AVX-512): + /// + /// 8×1V8 accumulators · 1 B load + 8 broadcasts = 9 loads · 8 FMAs → 0.89 FMA/load (today's shape) + /// 6×2V12 accumulators · 2 + 6 = 8 loads · 12 FMAs → 1.50 + /// 4×3V12 accumulators · 3 + 4 = 7 loads · 12 FMAs → 1.71 + /// 8×2V16 accumulators · 2 + 8 = 10 loads · 16 FMAs → 1.60 (needs AVX-512's 32 registers to hold) + /// + /// + /// Method notes that this project has paid to learn. Accumulators are named locals, never + /// a stackalloc span — a span forces an L1 round-trip per accumulator per iteration and measures + /// cache latency instead of issue rate (it cost a 2.8× error in an earlier roofline). A and B panels are + /// sized to sit in L1 so the result reflects instruction issue, not memory bandwidth; the real kernel's + /// memory behaviour is a separate question from its tile shape. + /// + /// Run: + /// dotnet run -c Release --project Sources/Benchmark -- --filter "*GemmMicroKernelShape*" + /// + [Config(typeof(BenchmarkConfig))] + public class GemmMicroKernelShapeBenchmark + { + /// Contraction length per call — a typical VGG im2col K (inC 256 · 3 · 3 = 2304 rounds here). + public const int K = 2304; + + /// Times the tile is swept, to give BenchmarkDotNet a multi-millisecond subject. + public const int Sweeps = 2000; + + private const int MaxRows = 8; + private const int MaxCols = 48; + + private float[] _a = null!; + private float[] _b = null!; + private float[] _c = null!; + + public float Sink; + + /// + /// FLOPs for the shape a benchmark method encodes, read from its name suffix (rows × cols). + /// A MAC counts as 2, matching and the rest of this project. + /// + public static WorkAmount GetWorkAmount(BenchmarkCase benchmarkCase) + { + var (rows, cols) = benchmarkCase.Descriptor.WorkloadMethod.Name switch + { + nameof(Avx2_8x8) => (8, 8), + nameof(Avx2_6x16) => (6, 16), + nameof(Avx2_4x24) => (4, 24), + nameof(Avx512_8x16) => (8, 16), + nameof(Avx512_8x32) => (8, 32), + nameof(Avx512_6x48) => (6, 48), + _ => (0, 0), + }; + + return WorkAmount.Matmul(rows, K * (long)Sweeps, cols); + } + + [GlobalSetup] + public void Setup() + { + var rng = new Random(20260723); + + _a = new float[MaxRows * K]; + _b = new float[K * MaxCols]; + _c = new float[MaxRows * MaxCols]; + + for (var i = 0; i < _a.Length; i++) + { + _a[i] = (float)((rng.NextDouble() * 2.0) - 1.0); + } + + for (var i = 0; i < _b.Length; i++) + { + _b[i] = (float)((rng.NextDouble() * 2.0) - 1.0); + } + } + + /// Today's shape: 8 rows × one 256-bit vector. 9 loads per 8 FMAs. + [Benchmark(Baseline = true)] + public unsafe void Avx2_8x8() + { + fixed (float* a = _a, b = _b, c = _c) + { + for (var s = 0; s < Sweeps; s++) + { + var a0 = Vector256.Zero; + var a1 = Vector256.Zero; + var a2 = Vector256.Zero; + var a3 = Vector256.Zero; + var a4 = Vector256.Zero; + var a5 = Vector256.Zero; + var a6 = Vector256.Zero; + var a7 = Vector256.Zero; + + for (var k = 0; k < K; k++) + { + var bv = Vector256.Load(b + (k * 8)); + var ak = a + k; + + a0 = Fma.MultiplyAdd(Vector256.Create(ak[0 * K]), bv, a0); + a1 = Fma.MultiplyAdd(Vector256.Create(ak[1 * K]), bv, a1); + a2 = Fma.MultiplyAdd(Vector256.Create(ak[2 * K]), bv, a2); + a3 = Fma.MultiplyAdd(Vector256.Create(ak[3 * K]), bv, a3); + a4 = Fma.MultiplyAdd(Vector256.Create(ak[4 * K]), bv, a4); + a5 = Fma.MultiplyAdd(Vector256.Create(ak[5 * K]), bv, a5); + a6 = Fma.MultiplyAdd(Vector256.Create(ak[6 * K]), bv, a6); + a7 = Fma.MultiplyAdd(Vector256.Create(ak[7 * K]), bv, a7); + } + + a0.Store(c); + a1.Store(c + 8); + a2.Store(c + 16); + a3.Store(c + 24); + a4.Store(c + 32); + a5.Store(c + 40); + a6.Store(c + 48); + a7.Store(c + 56); + } + } + + Sink = _c[0]; + } + + /// 6 rows × two 256-bit vectors: 12 accumulators, 8 loads per 12 FMAs. + [Benchmark] + public unsafe void Avx2_6x16() + { + fixed (float* a = _a, b = _b, c = _c) + { + for (var s = 0; s < Sweeps; s++) + { + Vector256 a00 = default, a01 = default, a10 = default, a11 = default; + Vector256 a20 = default, a21 = default, a30 = default, a31 = default; + Vector256 a40 = default, a41 = default, a50 = default, a51 = default; + + for (var k = 0; k < K; k++) + { + var b0 = Vector256.Load(b + (k * 16)); + var b1 = Vector256.Load(b + (k * 16) + 8); + var ak = a + k; + + var r = Vector256.Create(ak[0 * K]); + a00 = Fma.MultiplyAdd(r, b0, a00); + a01 = Fma.MultiplyAdd(r, b1, a01); + r = Vector256.Create(ak[1 * K]); + a10 = Fma.MultiplyAdd(r, b0, a10); + a11 = Fma.MultiplyAdd(r, b1, a11); + r = Vector256.Create(ak[2 * K]); + a20 = Fma.MultiplyAdd(r, b0, a20); + a21 = Fma.MultiplyAdd(r, b1, a21); + r = Vector256.Create(ak[3 * K]); + a30 = Fma.MultiplyAdd(r, b0, a30); + a31 = Fma.MultiplyAdd(r, b1, a31); + r = Vector256.Create(ak[4 * K]); + a40 = Fma.MultiplyAdd(r, b0, a40); + a41 = Fma.MultiplyAdd(r, b1, a41); + r = Vector256.Create(ak[5 * K]); + a50 = Fma.MultiplyAdd(r, b0, a50); + a51 = Fma.MultiplyAdd(r, b1, a51); + } + + a00.Store(c); + a01.Store(c + 8); + a10.Store(c + 16); + a11.Store(c + 24); + a20.Store(c + 32); + a21.Store(c + 40); + a30.Store(c + 48); + a31.Store(c + 56); + a40.Store(c + 64); + a41.Store(c + 72); + a50.Store(c + 80); + a51.Store(c + 88); + } + } + + Sink = _c[0]; + } + + /// 4 rows × three 256-bit vectors: 12 accumulators, 7 loads per 12 FMAs — the best AVX2 ratio. + [Benchmark] + public unsafe void Avx2_4x24() + { + fixed (float* a = _a, b = _b, c = _c) + { + for (var s = 0; s < Sweeps; s++) + { + Vector256 a00 = default, a01 = default, a02 = default; + Vector256 a10 = default, a11 = default, a12 = default; + Vector256 a20 = default, a21 = default, a22 = default; + Vector256 a30 = default, a31 = default, a32 = default; + + for (var k = 0; k < K; k++) + { + var b0 = Vector256.Load(b + (k * 24)); + var b1 = Vector256.Load(b + (k * 24) + 8); + var b2 = Vector256.Load(b + (k * 24) + 16); + var ak = a + k; + + var r = Vector256.Create(ak[0 * K]); + a00 = Fma.MultiplyAdd(r, b0, a00); + a01 = Fma.MultiplyAdd(r, b1, a01); + a02 = Fma.MultiplyAdd(r, b2, a02); + r = Vector256.Create(ak[1 * K]); + a10 = Fma.MultiplyAdd(r, b0, a10); + a11 = Fma.MultiplyAdd(r, b1, a11); + a12 = Fma.MultiplyAdd(r, b2, a12); + r = Vector256.Create(ak[2 * K]); + a20 = Fma.MultiplyAdd(r, b0, a20); + a21 = Fma.MultiplyAdd(r, b1, a21); + a22 = Fma.MultiplyAdd(r, b2, a22); + r = Vector256.Create(ak[3 * K]); + a30 = Fma.MultiplyAdd(r, b0, a30); + a31 = Fma.MultiplyAdd(r, b1, a31); + a32 = Fma.MultiplyAdd(r, b2, a32); + } + + a00.Store(c); + a01.Store(c + 8); + a02.Store(c + 16); + a10.Store(c + 24); + a11.Store(c + 32); + a12.Store(c + 40); + a20.Store(c + 48); + a21.Store(c + 56); + a22.Store(c + 64); + a30.Store(c + 72); + a31.Store(c + 80); + a32.Store(c + 88); + } + } + + Sink = _c[0]; + } + + /// AVX-512, 8 rows × one 512-bit vector: the direct widening of today's shape. + [Benchmark] + public unsafe void Avx512_8x16() + { + if (!Avx512F.IsSupported) + { + return; + } + + fixed (float* a = _a, b = _b, c = _c) + { + for (var s = 0; s < Sweeps; s++) + { + Vector512 a0 = default, a1 = default, a2 = default, a3 = default; + Vector512 a4 = default, a5 = default, a6 = default, a7 = default; + + for (var k = 0; k < K; k++) + { + var bv = Vector512.Load(b + (k * 16)); + var ak = a + k; + + a0 = Avx512F.FusedMultiplyAdd(Vector512.Create(ak[0 * K]), bv, a0); + a1 = Avx512F.FusedMultiplyAdd(Vector512.Create(ak[1 * K]), bv, a1); + a2 = Avx512F.FusedMultiplyAdd(Vector512.Create(ak[2 * K]), bv, a2); + a3 = Avx512F.FusedMultiplyAdd(Vector512.Create(ak[3 * K]), bv, a3); + a4 = Avx512F.FusedMultiplyAdd(Vector512.Create(ak[4 * K]), bv, a4); + a5 = Avx512F.FusedMultiplyAdd(Vector512.Create(ak[5 * K]), bv, a5); + a6 = Avx512F.FusedMultiplyAdd(Vector512.Create(ak[6 * K]), bv, a6); + a7 = Avx512F.FusedMultiplyAdd(Vector512.Create(ak[7 * K]), bv, a7); + } + + a0.Store(c); + a1.Store(c + 16); + a2.Store(c + 32); + a3.Store(c + 48); + a4.Store(c + 64); + a5.Store(c + 80); + a6.Store(c + 96); + a7.Store(c + 112); + } + } + + Sink = _c[0]; + } + + /// AVX-512, 8 rows × two 512-bit vectors: 16 accumulators, 10 loads per 16 FMAs. + [Benchmark] + public unsafe void Avx512_8x32() + { + if (!Avx512F.IsSupported) + { + return; + } + + fixed (float* a = _a, b = _b, c = _c) + { + for (var s = 0; s < Sweeps; s++) + { + Vector512 a00 = default, a01 = default, a10 = default, a11 = default; + Vector512 a20 = default, a21 = default, a30 = default, a31 = default; + Vector512 a40 = default, a41 = default, a50 = default, a51 = default; + Vector512 a60 = default, a61 = default, a70 = default, a71 = default; + + for (var k = 0; k < K; k++) + { + var b0 = Vector512.Load(b + (k * 32)); + var b1 = Vector512.Load(b + (k * 32) + 16); + var ak = a + k; + + var r = Vector512.Create(ak[0 * K]); + a00 = Avx512F.FusedMultiplyAdd(r, b0, a00); + a01 = Avx512F.FusedMultiplyAdd(r, b1, a01); + r = Vector512.Create(ak[1 * K]); + a10 = Avx512F.FusedMultiplyAdd(r, b0, a10); + a11 = Avx512F.FusedMultiplyAdd(r, b1, a11); + r = Vector512.Create(ak[2 * K]); + a20 = Avx512F.FusedMultiplyAdd(r, b0, a20); + a21 = Avx512F.FusedMultiplyAdd(r, b1, a21); + r = Vector512.Create(ak[3 * K]); + a30 = Avx512F.FusedMultiplyAdd(r, b0, a30); + a31 = Avx512F.FusedMultiplyAdd(r, b1, a31); + r = Vector512.Create(ak[4 * K]); + a40 = Avx512F.FusedMultiplyAdd(r, b0, a40); + a41 = Avx512F.FusedMultiplyAdd(r, b1, a41); + r = Vector512.Create(ak[5 * K]); + a50 = Avx512F.FusedMultiplyAdd(r, b0, a50); + a51 = Avx512F.FusedMultiplyAdd(r, b1, a51); + r = Vector512.Create(ak[6 * K]); + a60 = Avx512F.FusedMultiplyAdd(r, b0, a60); + a61 = Avx512F.FusedMultiplyAdd(r, b1, a61); + r = Vector512.Create(ak[7 * K]); + a70 = Avx512F.FusedMultiplyAdd(r, b0, a70); + a71 = Avx512F.FusedMultiplyAdd(r, b1, a71); + } + + a00.Store(c); + a01.Store(c + 16); + a10.Store(c + 32); + a11.Store(c + 48); + a20.Store(c + 64); + a21.Store(c + 80); + a30.Store(c + 96); + a31.Store(c + 112); + a40.Store(c + 128); + a41.Store(c + 144); + a50.Store(c + 160); + a51.Store(c + 176); + a60.Store(c + 192); + a61.Store(c + 208); + a70.Store(c + 224); + a71.Store(c + 240); + } + } + + Sink = _c[0]; + } + + /// AVX-512, 6 rows × three 512-bit vectors: 18 accumulators, 9 loads per 18 FMAs — the best ratio tried. + [Benchmark] + public unsafe void Avx512_6x48() + { + if (!Avx512F.IsSupported) + { + return; + } + + fixed (float* a = _a, b = _b, c = _c) + { + for (var s = 0; s < Sweeps; s++) + { + Vector512 a00 = default, a01 = default, a02 = default; + Vector512 a10 = default, a11 = default, a12 = default; + Vector512 a20 = default, a21 = default, a22 = default; + Vector512 a30 = default, a31 = default, a32 = default; + Vector512 a40 = default, a41 = default, a42 = default; + Vector512 a50 = default, a51 = default, a52 = default; + + for (var k = 0; k < K; k++) + { + var b0 = Vector512.Load(b + (k * 48)); + var b1 = Vector512.Load(b + (k * 48) + 16); + var b2 = Vector512.Load(b + (k * 48) + 32); + var ak = a + k; + + var r = Vector512.Create(ak[0 * K]); + a00 = Avx512F.FusedMultiplyAdd(r, b0, a00); + a01 = Avx512F.FusedMultiplyAdd(r, b1, a01); + a02 = Avx512F.FusedMultiplyAdd(r, b2, a02); + r = Vector512.Create(ak[1 * K]); + a10 = Avx512F.FusedMultiplyAdd(r, b0, a10); + a11 = Avx512F.FusedMultiplyAdd(r, b1, a11); + a12 = Avx512F.FusedMultiplyAdd(r, b2, a12); + r = Vector512.Create(ak[2 * K]); + a20 = Avx512F.FusedMultiplyAdd(r, b0, a20); + a21 = Avx512F.FusedMultiplyAdd(r, b1, a21); + a22 = Avx512F.FusedMultiplyAdd(r, b2, a22); + r = Vector512.Create(ak[3 * K]); + a30 = Avx512F.FusedMultiplyAdd(r, b0, a30); + a31 = Avx512F.FusedMultiplyAdd(r, b1, a31); + a32 = Avx512F.FusedMultiplyAdd(r, b2, a32); + r = Vector512.Create(ak[4 * K]); + a40 = Avx512F.FusedMultiplyAdd(r, b0, a40); + a41 = Avx512F.FusedMultiplyAdd(r, b1, a41); + a42 = Avx512F.FusedMultiplyAdd(r, b2, a42); + r = Vector512.Create(ak[5 * K]); + a50 = Avx512F.FusedMultiplyAdd(r, b0, a50); + a51 = Avx512F.FusedMultiplyAdd(r, b1, a51); + a52 = Avx512F.FusedMultiplyAdd(r, b2, a52); + } + + a00.Store(c); + a01.Store(c + 16); + a02.Store(c + 32); + a10.Store(c + 48); + a11.Store(c + 64); + a12.Store(c + 80); + a20.Store(c + 96); + a21.Store(c + 112); + a22.Store(c + 128); + a30.Store(c + 144); + a31.Store(c + 160); + a32.Store(c + 176); + a40.Store(c + 192); + a41.Store(c + 208); + a42.Store(c + 224); + a50.Store(c + 240); + a51.Store(c + 256); + a52.Store(c + 272); + } + } + + Sink = _c[0]; + } + } +} diff --git a/Sources/Main/Kernels/Conv2DGemmKernels.cs b/Sources/Main/Kernels/Conv2DGemmKernels.cs index fdd36b08..fc42b31a 100644 --- a/Sources/Main/Kernels/Conv2DGemmKernels.cs +++ b/Sources/Main/Kernels/Conv2DGemmKernels.cs @@ -3,6 +3,7 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com +using System.Diagnostics; using System.Runtime.CompilerServices; using System.Runtime.Intrinsics; using System.Runtime.Intrinsics.X86; @@ -28,6 +29,53 @@ internal static class Conv2DGemmKernels public static bool IsSupported => CpuFeatures.HasFma; + /// + /// Opt-in split of conv time into the im2col patch gather versus the GEMM. Off by default and checked + /// before any timestamp, so the inference path is unchanged when it is off. + /// + /// Needed because "Conv is 90.7% of VGG-16" does not say which half. The gather moves + /// O(K·N) floats and the GEMM does O(M·N·K) FLOPs, so their ratio varies enormously across layers — + /// and the answer decides whether the next lever is the micro-kernel or the gather. Guessing which + /// would be guessing about mechanism. + /// + public static bool ProfileParts; + + private static long _im2colTicks; + private static long _gemmTicks; + + /// + /// Measurement-only: skip the B-panel pack, leaving the micro-kernel to run over whatever the previous + /// panel left in the buffer. Produces WRONG results by construction. + /// + /// Splits the GEMM's 54 ms into packing versus arithmetic without putting timestamps inside the + /// parallel region (where per-thread accumulation and Interlocked would distort what is being measured). + /// The pack is the prime suspect: it reads B[kk·n + n0] with stride n — 200 KB apart on + /// VGG's early layers — one scalar element at a time with a bounds branch each, and across a whole GEMM + /// it moves the entire im2col matrix once more. + /// + internal static bool AblatePackB; + + /// Measurement-only: skip the micro-kernel, leaving only the pack. Wrong results by construction. + internal static bool AblateMicroKernel; + + /// Clears the part accumulators (call before the measured segment). + public static void ResetPartProfile() + { + _im2colTicks = 0; + _gemmTicks = 0; + } + + /// im2col versus GEMM, in milliseconds and as a share of the two combined. + public static string PartProfileReport() + { + var toMs = 1000.0 / Stopwatch.Frequency; + var total = _im2colTicks + _gemmTicks; + var share = total == 0 ? 1.0 : total; + + return $"im2col {_im2colTicks * toMs,8:F2} ms {100.0 * _im2colTicks / share,5:F1}% " + + $"gemm {_gemmTicks * toMs,8:F2} ms {100.0 * _gemmTicks / share,5:F1}%"; + } + public static void Forward( ReadOnlySpan input, // [batch, inChannels, H, W] ReadOnlySpan kernels, // [outChannels, inChannels, k, k] == [outChannels, K] @@ -67,11 +115,21 @@ public static void Forward( for (var b = 0; b < batchSize; b++) { + var t0 = ProfileParts ? Stopwatch.GetTimestamp() : 0L; + Im2Col( input.Slice(b * inputPlane, inputPlane), cols, inChannels, inputH, inputW, kernelSize, padding, stride, outH, outW); + var t1 = ProfileParts ? Stopwatch.GetTimestamp() : 0L; + Gemm(kernels, cols, output.Slice(b * outputPlane, outputPlane), m, n, k); + + if (ProfileParts) + { + _im2colTicks += t1 - t0; + _gemmTicks += Stopwatch.GetTimestamp() - t1; + } } } @@ -194,16 +252,190 @@ private readonly unsafe struct Im2ColCtx( // most im2col K values are ≤ a few hundred (single K-block → no blocking benefit) while the one-time A // pack adds single-threaded O(M·K) overhead. Cache-blocking pays on large dense GEMM, not CNN-shaped im2col. // Internal so the Winograd path can reuse the same tuned micro-kernel for its 16 element-wise GEMMs. + /// Micro-kernel columns for the AVX-512 path: 8 rows × 32 columns = 16 zmm accumulators. + private const int Nr512 = 32; + + /// + /// Route the conv GEMM through the AVX-512 8×32 micro-kernel; OVERFIT_CONV_AVX512=0 forces the + /// AVX2 8×8 path. + /// + /// Why a wider tile and not a faster one. Shape benchmarks showed the 8×8 kernel already + /// runs at 83% of this machine's single-core FMA peak, so there is no instruction-level headroom. What + /// a wider tile changes is arithmetic intensity: per k-step an Mr×Nr tile loads + /// Mr + Nr floats and performs 2·Mr·Nr FLOPs, giving Mr·Nr / (2(Mr+Nr)) FLOP per + /// byte — 2.0 at 8×8, 3.2 at 8×32. That is 38% less memory traffic for the same arithmetic. + /// + /// Why that is the lever here. The machine probe found bandwidth scales 10–13× across + /// cores only while the per-core working set stays under ~2 MB, and collapses to 1–2× beyond it — so a + /// parallel kernel that outruns its cache cannot be fixed by adding cores or by blocking, only by + /// needing fewer bytes per FLOP. + /// + internal static bool UseAvx512Conv = + CpuFeatures.HasAvx512 + && Environment.GetEnvironmentVariable(OverfitEnvironment.ConvAvx512) != "0"; + + // C[M,N] = A[M,K] @ B[K,N], parallelised over N-panels (each worker packs its B panel and sweeps M + // with the full-K register-blocked micro-kernel). NOTE: a BLIS-style K-blocked + A-packed variant was + // tried and MEASURED to regress on these CNN dims (deepcnn 101→125, vgg 140→189, resnet 45→118 ms) — + // most im2col K values are ≤ a few hundred (single K-block → no blocking benefit) while the one-time A + // pack adds single-threaded O(M·K) overhead. Cache-blocking pays on large dense GEMM, not CNN-shaped im2col. + // Internal so the Winograd path can reuse the same tuned micro-kernel for its 16 element-wise GEMMs. internal static unsafe void Gemm(ReadOnlySpan a, ReadOnlySpan b, Span c, int m, int n, int k) { - var nPanels = (n + Nr - 1) / Nr; + var nr = UseAvx512Conv ? Nr512 : Nr; + var nPanels = (n + nr - 1) / nr; + fixed (float* pa = a, pb = b, pc = c) { var ctx = new GemmCtx(pa, pb, pc, m, n, k); + + if (UseAvx512Conv) + { + OverfitParallel.For(0, nPanels, 1, &GemmNPanelWorker512, &ctx); + return; + } + OverfitParallel.For(0, nPanels, 1, &GemmNPanelWorker, &ctx); } } + /// + /// One 32-column panel per work item: pack it, then sweep M with the 8×32 AVX-512 micro-kernel. + /// Structurally identical to , only wider. + /// + private static unsafe void GemmNPanelWorker512(int npStart, int npEnd, void* ctxPtr) + { + ref readonly var c = ref Unsafe.AsRef(ctxPtr); + var k = c.K; + var n = c.N; + var m = c.M; + + using var packBuf = new PooledBuffer(checked(k * Nr512), clearMemory: false); + var packB = packBuf.Span; + + for (var np = npStart; np < npEnd; np++) + { + var n0 = np * Nr512; + var nrEff = Math.Min(Nr512, n - n0); + + for (var kk = 0; kk < k; kk++) + { + var srcBase = (kk * n) + n0; + var dstBase = kk * Nr512; + + for (var j = 0; j < Nr512; j++) + { + packB[dstBase + j] = j < nrEff ? c.B[srcBase + j] : 0f; + } + } + + fixed (float* pPackB = packB) + { + for (var m0 = 0; m0 < m; m0 += Mr) + { + var mrEff = Math.Min(Mr, m - m0); + + MicroKernel8x32Avx512(c.A, m0, mrEff, k, pPackB, c.C, n, n0, nrEff); + } + } + } + } + + /// + /// 8 rows × 32 columns in sixteen accumulators, held in registers across the + /// whole K contraction. Per k-step: two 512-bit B loads and up to eight A broadcasts feed sixteen FMAs + /// — 3.2 FLOP per byte loaded, against 2.0 for the 8×8 AVX2 kernel. + /// + /// Handles a partial row block ( < 8) and a partial column tail + /// ( < 32) with scalar stores rather than a separate kernel; both are edge + /// cases of the last block, not the steady state. + /// + private static unsafe void MicroKernel8x32Avx512( + float* a, int m0, int mrEff, int k, float* packB, float* c, int n, int n0, int nrEff) + { + // Row pointers, clamped to the last valid row so a short block reads in-bounds; the extra rows' + // results are simply not stored below. + var rows = stackalloc float*[Mr]; + for (var r = 0; r < Mr; r++) + { + rows[r] = a + ((long)(m0 + Math.Min(r, mrEff - 1)) * k); + } + + Vector512 c00 = default, c01 = default, c10 = default, c11 = default; + Vector512 c20 = default, c21 = default, c30 = default, c31 = default; + Vector512 c40 = default, c41 = default, c50 = default, c51 = default; + Vector512 c60 = default, c61 = default, c70 = default, c71 = default; + + for (var kk = 0; kk < k; kk++) + { + var b0 = Vector512.Load(packB + (kk * Nr512)); + var b1 = Vector512.Load(packB + (kk * Nr512) + 16); + + var r = Vector512.Create(rows[0][kk]); + c00 = Avx512F.FusedMultiplyAdd(r, b0, c00); + c01 = Avx512F.FusedMultiplyAdd(r, b1, c01); + r = Vector512.Create(rows[1][kk]); + c10 = Avx512F.FusedMultiplyAdd(r, b0, c10); + c11 = Avx512F.FusedMultiplyAdd(r, b1, c11); + r = Vector512.Create(rows[2][kk]); + c20 = Avx512F.FusedMultiplyAdd(r, b0, c20); + c21 = Avx512F.FusedMultiplyAdd(r, b1, c21); + r = Vector512.Create(rows[3][kk]); + c30 = Avx512F.FusedMultiplyAdd(r, b0, c30); + c31 = Avx512F.FusedMultiplyAdd(r, b1, c31); + r = Vector512.Create(rows[4][kk]); + c40 = Avx512F.FusedMultiplyAdd(r, b0, c40); + c41 = Avx512F.FusedMultiplyAdd(r, b1, c41); + r = Vector512.Create(rows[5][kk]); + c50 = Avx512F.FusedMultiplyAdd(r, b0, c50); + c51 = Avx512F.FusedMultiplyAdd(r, b1, c51); + r = Vector512.Create(rows[6][kk]); + c60 = Avx512F.FusedMultiplyAdd(r, b0, c60); + c61 = Avx512F.FusedMultiplyAdd(r, b1, c61); + r = Vector512.Create(rows[7][kk]); + c70 = Avx512F.FusedMultiplyAdd(r, b0, c70); + c71 = Avx512F.FusedMultiplyAdd(r, b1, c71); + } + + var tile = stackalloc float[Nr512]; + + StoreTile(c, n, n0, m0, 0, mrEff, nrEff, c00, c01, tile); + StoreTile(c, n, n0, m0, 1, mrEff, nrEff, c10, c11, tile); + StoreTile(c, n, n0, m0, 2, mrEff, nrEff, c20, c21, tile); + StoreTile(c, n, n0, m0, 3, mrEff, nrEff, c30, c31, tile); + StoreTile(c, n, n0, m0, 4, mrEff, nrEff, c40, c41, tile); + StoreTile(c, n, n0, m0, 5, mrEff, nrEff, c50, c51, tile); + StoreTile(c, n, n0, m0, 6, mrEff, nrEff, c60, c61, tile); + StoreTile(c, n, n0, m0, 7, mrEff, nrEff, c70, c71, tile); + } + + private static unsafe void StoreTile( + float* c, int n, int n0, int m0, int row, int mrEff, int nrEff, + Vector512 lo, Vector512 hi, float* scratch) + { + if (row >= mrEff) + { + return; + } + + var dst = c + ((long)(m0 + row) * n) + n0; + + if (nrEff == Nr512) + { + lo.Store(dst); + hi.Store(dst + 16); + return; + } + + lo.Store(scratch); + hi.Store(scratch + 16); + + for (var j = 0; j < nrEff; j++) + { + dst[j] = scratch[j]; + } + } + private readonly unsafe struct GemmCtx { public readonly float* A; @@ -224,43 +456,96 @@ public GemmCtx(float* a, float* b, float* c, int m, int n, int k) } } + /// + /// N-panels packed and swept together, so the A row-block is read once per group rather than + /// once per panel. 1 reproduces the original loop exactly. + /// + /// Measured motivation. The original loop sweeps M inside the panel loop, so the whole A + /// matrix is re-read for every N-panel: on VGG-16 that is ~7.5 GB of A traffic for 30.7 GFLOP of + /// arithmetic — 0.24 bytes/FLOP where a blocked GEMM runs at ~0.01, and 7.5 GB in 73 ms is + /// ≈103 GB/s against a 90 GB/s DRAM read ceiling. It is why one thread reaches 132 GFLOP/s on this + /// kernel while sixteen reach only ~30 each: alone, a core has the cache to itself. + /// + /// Grouping trades A traffic for a larger packed-B working set (K · Nr · group floats), so + /// the useful group size is bounded by cache, not by the arithmetic — hence a flag rather than a + /// constant, and a measurement rather than a guess. + /// + internal static int NPanelGroup = ResolveNPanelGroup(); + + private static int ResolveNPanelGroup() + { + var raw = Environment.GetEnvironmentVariable(OverfitEnvironment.ConvPanelGroup); + + // Default 1 — the original per-panel loop. Grouping was built to cut A re-reads and MEASURED to do + // nothing: interleaved with an ORT canary (4% spread), groups 1/2/4 came out 72.7/73.0/73.3 ms. + // The traffic argument that motivated it (7.5 GB of A re-reads against a 90 GB/s DRAM ceiling) was + // wrong about WHERE the traffic goes: this CPU has 128 MB of L3, so every layer's A (≤9 MB) is + // re-read from L3, not DRAM. Kept behind the flag because the negative is worth preserving. + return int.TryParse(raw, out var parsed) && parsed >= 1 ? parsed : 1; + } + private static unsafe void GemmNPanelWorker(int npStart, int npEnd, void* ctxPtr) { ref readonly var c = ref Unsafe.AsRef(ctxPtr); var k = c.K; var n = c.N; var m = c.M; + var group = NPanelGroup; - using var packBuf = new PooledBuffer(checked(k * Nr), clearMemory: false); + using var packBuf = new PooledBuffer(checked(k * Nr * group), clearMemory: false); var packB = packBuf.Span; - for (var np = npStart; np < npEnd; np++) + for (var gStart = npStart; gStart < npEnd; gStart += group) { - var n0 = np * Nr; - var nrEff = Math.Min(Nr, n - n0); + var gCount = Math.Min(group, npEnd - gStart); - for (var kk = 0; kk < k; kk++) + if (!AblatePackB) { - var srcBase = kk * n + n0; - var dstBase = kk * Nr; - for (var j = 0; j < Nr; j++) + for (var g = 0; g < gCount; g++) { - packB[dstBase + j] = j < nrEff ? c.B[srcBase + j] : 0f; + var n0 = (gStart + g) * Nr; + var nrEff = Math.Min(Nr, n - n0); + var panelBase = g * k * Nr; + + for (var kk = 0; kk < k; kk++) + { + var srcBase = (kk * n) + n0; + var dstBase = panelBase + (kk * Nr); + for (var j = 0; j < Nr; j++) + { + packB[dstBase + j] = j < nrEff ? c.B[srcBase + j] : 0f; + } + } } } + if (AblateMicroKernel) + { + continue; + } + fixed (float* pPackB = packB) { + // M outermost: each 8-row block of A is loaded once and reused across every panel in the + // group, which is the whole point of grouping. for (var m0 = 0; m0 < m; m0 += Mr) { var mrEff = Math.Min(Mr, m - m0); - if (mrEff == Mr) + + for (var g = 0; g < gCount; g++) { - MicroKernel8x8(c.A, m0, k, pPackB, c.C, n, n0, nrEff); - continue; - } + var n0 = (gStart + g) * Nr; + var nrEff = Math.Min(Nr, n - n0); + var panel = pPackB + ((long)g * k * Nr); + + if (mrEff == Mr) + { + MicroKernel8x8(c.A, m0, k, panel, c.C, n, n0, nrEff); + continue; + } - MicroKernelTail(c.A, m0, mrEff, k, pPackB, c.C, n, n0, nrEff); + MicroKernelTail(c.A, m0, mrEff, k, panel, c.C, n, n0, nrEff); + } } } } diff --git a/Sources/Main/Onnx/OnnxGraphModel.cs b/Sources/Main/Onnx/OnnxGraphModel.cs index 5ec5aed7..17c53410 100644 --- a/Sources/Main/Onnx/OnnxGraphModel.cs +++ b/Sources/Main/Onnx/OnnxGraphModel.cs @@ -127,6 +127,37 @@ private void RecordNode(int index, long ticks) _nodeCalls[index]++; } + /// + /// Every node individually — index, operator, output size and ms — rather than grouped by operator. + /// + /// Grouping answers "which operator"; this answers "which layer", which is the question + /// once an operator's cost is known to be shape-dependent. On VGG-16 a standalone prototype of the conv + /// GEMM reached 132 GFLOP/s single-threaded on a late-layer shape while production conv averaged + /// 19.7 GFLOP/s across all layers — a 6.7× spread that only a per-layer view can locate. + /// + public string PerNodeProfileReport() + { + if (_nodeTicks is null) + { + return "(no node profile recorded — set OnnxGraphModel.ProfileNodes before running)"; + } + + var toMs = 1000.0 / Stopwatch.Frequency; + var runs = _nodeCalls is null || _nodeCalls.Length == 0 ? 1L : Math.Max(1L, _nodeCalls[0]); + var sb = new StringBuilder(); + + sb.AppendLine($"=== per-node ({runs} run(s)) ==="); + + for (var i = 0; i < _nodes.Length; i++) + { + var node = _nodes[i]; + sb.AppendLine( + $" [{i,2}] {node.Module.GetType().Name,-26} out={node.OutputSize,9} {_nodeTicks[i] * toMs / runs,8:F2} ms"); + } + + return sb.ToString(); + } + /// Clears the per-node accumulators (call before the measured segment). public void ResetNodeProfile() { diff --git a/Sources/Main/Runtime/OverfitEnvironment.cs b/Sources/Main/Runtime/OverfitEnvironment.cs index 09e854b2..63e6a121 100644 --- a/Sources/Main/Runtime/OverfitEnvironment.cs +++ b/Sources/Main/Runtime/OverfitEnvironment.cs @@ -44,6 +44,12 @@ public static class OverfitEnvironment /// Set to 0 to force the serial im2col patch gather in the conv GEMM path (A/B switch). public const string ParallelIm2Col = "OVERFIT_PARALLEL_IM2COL"; + /// N-panels packed and swept together in the conv GEMM; 1 = the original per-panel loop. + public const string ConvPanelGroup = "OVERFIT_CONV_PANEL_GROUP"; + + /// Set to 0 to force the AVX2 8×8 conv micro-kernel instead of the AVX-512 8×32 one. + public const string ConvAvx512 = "OVERFIT_CONV_AVX512"; + // ── Prefill kernel switches (all default ON where the hardware allows; set to 0 to opt out) ── // These exist so a measured win can be A/B'd against its predecessor without a rebuild, and so a // regression on unfamiliar hardware can be bisected in the field rather than only on the dev box. diff --git a/Tests/Diagnostics/ConvGemmPartProfileTests.cs b/Tests/Diagnostics/ConvGemmPartProfileTests.cs new file mode 100644 index 00000000..89b718f0 --- /dev/null +++ b/Tests/Diagnostics/ConvGemmPartProfileTests.cs @@ -0,0 +1,227 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.Diagnostics; +using DevOnBike.Overfit.Inference; +using DevOnBike.Overfit.Kernels; +using DevOnBike.Overfit.Onnx; +using DevOnBike.Overfit.Runtime; +using Xunit.Abstractions; + +namespace DevOnBike.Overfit.Tests.Diagnostics +{ + /// + /// Splits convolution time into the im2col patch gather versus the GEMM, on a real + /// ImageNet-sized CNN. + /// + /// The question. A per-operator profile put ConvLayer at 90.7% of VGG-16's 79 ms — but that + /// does not say which half of the conv. The gather moves O(K·N) floats; the GEMM does + /// O(M·N·K) FLOPs, and their ratio swings by an order of magnitude across VGG's layers (early + /// layers: huge N, small K; late layers: small N, large K). Whether the next optimisation target is the + /// micro-kernel or the gather depends entirely on this split, and it is cheap to measure and expensive to + /// guess. + /// + /// Reference points for reading the output on the dev box: the conv GEMMs total ≈30.7 GFLOP per + /// VGG-16 inference, the machine sustains ≈2190 GFLOP/s of float FMA across all cores + /// (MachineRooflineBenchmark), and ONNX Runtime runs the whole model in ≈10–12 ms. + /// + /// Needs an exported VGG-16 (python Scripts/export_cnn_onnx.py --arch vgg16); logs and + /// returns without it, so CI stays green. + /// + public sealed class ConvGemmPartProfileTests + { + private const int InputSize = 3 * 224 * 224; + private const int OutputSize = 1000; + private const int Runs = 10; + + private readonly ITestOutputHelper _out; + + public ConvGemmPartProfileTests(ITestOutputHelper output) => _out = output; + + [LongFact] + public void Conv_Im2ColVersusGemm_Split() + { + var path = Environment.GetEnvironmentVariable(OverfitEnvironment.CnnOnnx) + ?? @"C:\onnxmodels\cnn.onnx"; + + if (!File.Exists(path)) + { + _out.WriteLine($"missing {path} — export with Scripts/export_cnn_onnx.py --arch vgg16"); + return; + } + + using var model = OnnxGraphImporter.Load(path, InputSize, OutputSize); + model.Eval(); + using var engine = InferenceEngine.FromBackend(new OnnxGraphInferenceBackend(model)); + + var input = new float[InputSize]; + var output = new float[OutputSize]; + var rng = new Random(1234); + for (var i = 0; i < input.Length; i++) + { + input[i] = (float)rng.NextDouble(); + } + + // Warm up outside the measured region: JIT, the pooled im2col buffer, page-in. + for (var i = 0; i < 3; i++) + { + engine.Run(input, output); + } + + Conv2DGemmKernels.ResetPartProfile(); + Conv2DGemmKernels.ProfileParts = true; + var started = ValueStopwatch.StartNew(); + try + { + for (var i = 0; i < Runs; i++) + { + engine.Run(input, output); + } + } + finally + { + Conv2DGemmKernels.ProfileParts = false; + } + + var wallMs = started.GetElapsedTime().TotalMilliseconds / Runs; + + _out.WriteLine($"wall : {wallMs,8:F2} ms/run"); + _out.WriteLine($"conv parts (×{Runs}): {Conv2DGemmKernels.PartProfileReport()}"); + _out.WriteLine("conv GEMM work : 30.7 GFLOP/run | machine 2190 GFLOP/s | ORT whole model ~10-12 ms"); + + // Per layer, with each conv's own GFLOP/s — the view that locates a shape-dependent deficit. + model.ResetNodeProfile(); + OnnxGraphModel.ProfileNodes = true; + try + { + for (var i = 0; i < Runs; i++) + { + engine.Run(input, output); + } + } + finally + { + OnnxGraphModel.ProfileNodes = false; + } + + _out.WriteLine(model.PerNodeProfileReport()); + _out.WriteLine(VggConvShapeTable()); + + Assert.True(wallMs > 0, "no inference time recorded"); + } + + /// + /// Splits the GEMM itself into B-panel packing versus the micro-kernel, by ablation. + /// + /// The micro-kernel was measured in isolation at 148 GFLOP/s per core — essentially the AVX2 + /// hardware peak — yet the production GEMM reaches only 566 GFLOP/s across 16 cores, about 24% of what + /// that kernel can do. So the loss is not the arithmetic. Packing is the suspect: it reads + /// B[kk·n + n0] with stride n (200 KB apart on VGG's early layers) one scalar at a time, + /// and over a whole GEMM it moves the entire im2col matrix a second time. + /// + /// Ablation rather than timers, because timestamps inside the parallel region would need + /// per-thread accumulation and would perturb the thing being measured. Each arm produces wrong output + /// by construction — this measures cost, never correctness. Read the two as upper bounds: removing one + /// side also frees the other's cache pressure. + /// + [LongFact] + public void ConvGemm_PackVersusMicroKernel_Split() + { + var path = Environment.GetEnvironmentVariable(OverfitEnvironment.CnnOnnx) + ?? @"C:\onnxmodels\cnn.onnx"; + + if (!File.Exists(path)) + { + _out.WriteLine($"missing {path}"); + return; + } + + using var model = OnnxGraphImporter.Load(path, InputSize, OutputSize); + model.Eval(); + using var engine = InferenceEngine.FromBackend(new OnnxGraphInferenceBackend(model)); + + var input = new float[InputSize]; + var output = new float[OutputSize]; + var rng = new Random(1234); + for (var i = 0; i < input.Length; i++) + { + input[i] = (float)rng.NextDouble(); + } + + for (var i = 0; i < 3; i++) + { + engine.Run(input, output); + } + + var baseline = TimeArm(engine, input, output, pack: true, micro: true); + var packOnly = TimeArm(engine, input, output, pack: true, micro: false); + var microOnly = TimeArm(engine, input, output, pack: false, micro: true); + + _out.WriteLine($"baseline (pack + micro) : {baseline,8:F2} ms/run"); + _out.WriteLine($"pack only : {packOnly,8:F2} ms/run ({100 * packOnly / baseline,5:F1}% of baseline)"); + _out.WriteLine($"micro only : {microOnly,8:F2} ms/run ({100 * microOnly / baseline,5:F1}% of baseline)"); + _out.WriteLine($"unattributed : {baseline - packOnly - microOnly,8:F2} ms/run"); + + Assert.True(baseline > 0, "no inference time recorded"); + } + + /// + /// VGG-16's thirteen conv layers as GEMM shapes, in graph order, so the per-node timings above can be + /// read as GFLOP/s per layer. M = output channels, K = inChannels·3·3, N = outH·outW. + /// + private static string VggConvShapeTable() + { + (int M, int K, int N)[] layers = + [ + (64, 3 * 9, 224 * 224), (64, 64 * 9, 224 * 224), + (128, 64 * 9, 112 * 112), (128, 128 * 9, 112 * 112), + (256, 128 * 9, 56 * 56), (256, 256 * 9, 56 * 56), (256, 256 * 9, 56 * 56), + (512, 256 * 9, 28 * 28), (512, 512 * 9, 28 * 28), (512, 512 * 9, 28 * 28), + (512, 512 * 9, 14 * 14), (512, 512 * 9, 14 * 14), (512, 512 * 9, 14 * 14), + ]; + + var sb = new System.Text.StringBuilder(); + sb.AppendLine("=== VGG-16 conv shapes, in graph order (match against the conv nodes above) ==="); + + for (var i = 0; i < layers.Length; i++) + { + var (m, k, n) = layers[i]; + var gflop = 2.0 * m * k * n / 1e9; + var panels = (n + 7) / 8; + sb.AppendLine( + $" conv{i + 1,-2} M={m,4} K={k,5} N={n,6} {gflop,6:F2} GFLOP " + + $"panels={panels,5} A={m * k * 4 / 1024,6} KB re-read per panel"); + } + + return sb.ToString(); + } + + private static double TimeArm( + InferenceEngine engine, float[] input, float[] output, bool pack, bool micro) + { + Conv2DGemmKernels.AblatePackB = !pack; + Conv2DGemmKernels.AblateMicroKernel = !micro; + try + { + // One warm pass under this arm's configuration, so a branch-predictor or cache state change + // between arms is not charged to the measured loop. + engine.Run(input, output); + + var started = ValueStopwatch.StartNew(); + for (var i = 0; i < Runs; i++) + { + engine.Run(input, output); + } + + return started.GetElapsedTime().TotalMilliseconds / Runs; + } + finally + { + Conv2DGemmKernels.AblatePackB = false; + Conv2DGemmKernels.AblateMicroKernel = false; + } + } + } +} diff --git a/Tests/Diagnostics/MachineProbeTests.cs b/Tests/Diagnostics/MachineProbeTests.cs new file mode 100644 index 00000000..7a421b50 --- /dev/null +++ b/Tests/Diagnostics/MachineProbeTests.cs @@ -0,0 +1,598 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Numerics; +using System.Runtime.Intrinsics; +using System.Runtime.Intrinsics.Arm; +using System.Runtime.Intrinsics.X86; +using DevOnBike.Overfit.Diagnostics; +using Xunit.Abstractions; + +namespace DevOnBike.Overfit.Tests.Diagnostics +{ + /// + /// Measures this machine's hardware ceilings — peak FMA throughput, sustained memory bandwidth, and + /// bandwidth versus working-set size — so every performance number elsewhere can be read as a fraction of + /// what the box can actually do. A bare "1.7 TFLOP/s" means nothing until you know whether the machine + /// tops out at 2 or at 20. + /// + /// Two method traps this file exists to avoid, both of which this project has fallen into. + /// + /// Accumulators must be named locals, never a stackalloc span: a span forces an L1 + /// round-trip per accumulator per iteration and measures cache latency rather than issue rate. That + /// mistake once reported a float peak of 0.79 TFLOP/s — below what a real matmul achieved, which is + /// impossible for a loop that touches no memory. + /// Several independent chains, never one. A single accumulator measures the dependency + /// chain's latency, not throughput. An earlier version of the working-set sweep used one + /// Vector<float> accumulator and produced a perfectly flat ~75 GB/s from L1 to DRAM — not + /// because the machine has no cache cliff, but because 75 GB/s was the latency ceiling of that chain, + /// below every cache level's bandwidth. The flat curve was an artefact, and the conclusion drawn from + /// it ("this CPU has no cache cliff, so blocking cannot pay") was withdrawn. + /// + /// + /// A multiply-accumulate counts as 2 operations, matching and llama.cpp's + /// test-backend-ops, so figures are directly comparable across the two projects. Every number is a + /// best-of-N: noise only ever makes a machine look slower. + /// + /// rather than throughout — + /// the allocation-free timer this repo standardises on. + /// + public sealed class MachineProbeTests + { + /// Independent accumulator chains — enough to hide FMA latency without exhausting registers. + private const int Chains = 12; + + /// Independent load streams in the bandwidth loops, for the same reason. + private const int Streams = 8; + + private const int ComputeIterations = 2_000_000; + private const int Repeats = 5; + + /// 256 MB — must dwarf any last-level cache, including a 128 MB V-cache. + private const int MemoryFloats = 64 * 1024 * 1024; + + private static float _sink; + + private readonly ITestOutputHelper _out; + + public MachineProbeTests(ITestOutputHelper output) => _out = output; + + [LongFact] + public void Probe_HardwareCeilings() + { + var cores = Environment.ProcessorCount; + + _out.WriteLine("=== machine ceilings ==="); + _out.WriteLine($" CPU : {ReadCpuName()}"); + _out.WriteLine($" clock : {MeasureClockGhz(),0:F2} GHz (measured, not reported)"); + _out.WriteLine($" logical CPUs : {cores}"); + _out.WriteLine($" vector width : {Vector.Count * 32} bit ({Vector.Count} floats)"); + _out.WriteLine($" ISA : {DescribeIsa()}"); + _out.WriteLine($" cache : {ReadCacheTopology()}"); + _out.WriteLine(string.Empty); + + ReportCompute(cores); + _out.WriteLine(string.Empty); + ReportMemory(cores); + _out.WriteLine(string.Empty); + ReportWorkingSetSweep(cores); + _out.WriteLine(string.Empty); + ReportAllocations(); + + Assert.True(cores > 0); + } + + /// + /// Confirms the measured loops themselves allocate nothing, so the figures above are not partly a GC + /// measurement. + /// + /// The buffers are allocated during setup and that is unavoidable; what must be zero is the + /// timed region. The parallel arms are reported separately and are not expected to be + /// zero — allocates its own state per call — which is + /// exactly why the single-core figures are the ones to trust for a clean rate. + /// + private void ReportAllocations() + { + _out.WriteLine("--- allocation check on the measured loops ---"); + + var data = new float[1024 * 1024]; + for (var i = 0; i < data.Length; i++) + { + data[i] = i; + } + + _out.WriteLine($" FMA chain (256-bit) {AllocatedBy(() => FmaChains256()),8} B"); + _out.WriteLine($" read loop {AllocatedBy(() => ReadRepeated(data, 0, data.Length, 4)),8} B"); + + var cores = Environment.ProcessorCount; + _out.WriteLine($" parallel read {AllocatedBy(() => ParallelRead(data, cores)),8} B" + + " (Parallel.For state — expected, not part of any reported rate)"); + + var single = AllocatedBy(() => ReadRepeated(data, 0, data.Length, 4)); + var fma = AllocatedBy(() => FmaChains256()); + + Assert.True(single == 0, $"read loop allocated {single} B — the sweep would be measuring GC, not bandwidth"); + Assert.True(fma == 0, $"FMA chain allocated {fma} B — the peak figures would include GC work"); + } + + private static long AllocatedBy(Func body) + { + body(); + GC.Collect(); + GC.WaitForPendingFinalizers(); + + var before = GC.GetAllocatedBytesForCurrentThread(); + _sink = body(); + + return GC.GetAllocatedBytesForCurrentThread() - before; + } + + /// + /// Core clock, measured rather than read from a spec sheet: a chain of dependent integer adds + /// retires exactly one per cycle on every mainstream core, so iterations per second is the clock. + /// Reporting the measured value matters because boost, thermal and power state make the nameplate + /// figure wrong most of the time — and every GF/s number above has to be divided by the clock that was + /// actually in effect. + /// + private static double MeasureClockGhz() + { + const int Iterations = 200_000_000; + + var best = double.MaxValue; + + for (var r = 0; r < 3; r++) + { + var x = 1; + var started = ValueStopwatch.StartNew(); + + for (var i = 0; i < Iterations; i++) + { + x += x & 1; // dependent on the previous value: one add per cycle + } + + var seconds = started.GetElapsedTime().TotalSeconds; + _sink = x; + best = Math.Min(best, seconds); + } + + // Two dependent ops per iteration (the AND and the ADD) plus loop overhead the JIT folds away; + // treat this as a lower bound on the true clock rather than a precise figure. + return Iterations * 2.0 / best / 1e9; + } + + /// CPU model from the OS, best-effort and non-fatal — the measurements stand without it. + private static string ReadCpuName() + { + try + { + if (OperatingSystem.IsWindows()) + { + var value = Microsoft.Win32.Registry.GetValue( + @"HKEY_LOCAL_MACHINE\HARDWARE\DESCRIPTION\System\CentralProcessor\0", + "ProcessorNameString", null); + + return value?.ToString()?.Trim() ?? "(unknown)"; + } + + if (OperatingSystem.IsLinux() && File.Exists("/proc/cpuinfo")) + { + foreach (var line in File.ReadLines("/proc/cpuinfo")) + { + if (line.StartsWith("model name", StringComparison.Ordinal)) + { + return line[(line.IndexOf(':') + 1)..].Trim(); + } + } + } + } + catch (Exception ex) + { + return $"(unavailable: {ex.GetType().Name})"; + } + + return "(unknown)"; + } + + /// + /// Cache sizes per level, best-effort from the OS. Where it is unavailable the working-set sweep below + /// still shows the effective boundaries, which is the number that actually governs a kernel — reported + /// capacity and usable capacity are not the same thing on a CPU with stacked cache. + /// + private static string ReadCacheTopology() + { + try + { + if (OperatingSystem.IsLinux()) + { + var parts = new List(); + + for (var index = 0; index < 6; index++) + { + var levelPath = $"/sys/devices/system/cpu/cpu0/cache/index{index}/level"; + var sizePath = $"/sys/devices/system/cpu/cpu0/cache/index{index}/size"; + + if (!File.Exists(levelPath) || !File.Exists(sizePath)) + { + continue; + } + + parts.Add($"L{File.ReadAllText(levelPath).Trim()}={File.ReadAllText(sizePath).Trim()}"); + } + + if (parts.Count > 0) + { + return string.Join(" ", parts); + } + } + } + catch (Exception ex) + { + return $"(unavailable: {ex.GetType().Name})"; + } + + return "(see the working-set sweep below — effective sizes are measured there)"; + } + + private static string DescribeIsa() + { + var parts = new List(); + + if (Avx2.IsSupported) + { + parts.Add("AVX2"); + } + if (Fma.IsSupported) + { + parts.Add("FMA"); + } + if (Avx512F.IsSupported) + { + parts.Add("AVX512F"); + } + if (Avx512BW.IsSupported) + { + parts.Add("AVX512BW"); + } + if (AdvSimd.IsSupported) + { + parts.Add("NEON"); + } + + return parts.Count == 0 ? "(baseline)" : string.Join(" ", parts); + } + + private void ReportCompute(int cores) + { + _out.WriteLine("--- peak FMA (register-resident, no memory traffic) ---"); + _out.WriteLine($" {"width",-10}{"1 core",13}{"all cores",14}{"scaling",10}"); + + MeasureCompute("128-bit", FmaChains128, Vector128.Count, cores); + + if (Avx.IsSupported && Fma.IsSupported) + { + MeasureCompute("256-bit", FmaChains256, Vector256.Count, cores); + } + + if (Avx512F.IsSupported) + { + MeasureCompute("512-bit", FmaChains512, Vector512.Count, cores); + } + } + + private void MeasureCompute(string label, Func body, int lanes, int cores) + { + var flopsPerCall = 2.0 * Chains * ComputeIterations * lanes; + + var single = BestSeconds(body); + var all = BestSeconds(() => + { + var partial = new float[cores]; + Parallel.For(0, cores, new ParallelOptions { MaxDegreeOfParallelism = cores }, + i => partial[i] = body()); + return partial[0]; + }); + + var oneCore = flopsPerCall / single / 1e9; + var allCores = flopsPerCall * cores / all / 1e9; + + _out.WriteLine($" {label,-10}{oneCore,8:F0} GF/s{allCores,9:F0} GF/s{allCores / oneCore,9:F1}x"); + } + + private void ReportMemory(int cores) + { + _out.WriteLine("--- sustained bandwidth, 256 MB buffers (past any cache) ---"); + + var x = new float[MemoryFloats]; + var y = new float[MemoryFloats]; + var z = new float[MemoryFloats]; + var rng = new Random(20260723); + + for (var i = 0; i < MemoryFloats; i++) + { + x[i] = (float)rng.NextDouble(); + y[i] = (float)rng.NextDouble(); + z[i] = (float)rng.NextDouble(); + } + + const long Bytes = (long)MemoryFloats * sizeof(float); + + var oneCoreRead = Bytes / BestSeconds(() => ReadRange(x, 0, x.Length)); + var allCoreRead = Bytes / BestSeconds(() => ParallelRead(x, cores)); + + _out.WriteLine($" read 1 core {oneCoreRead / 1e9,8:F1} GB/s"); + _out.WriteLine($" read all {allCoreRead / 1e9,8:F1} GB/s {allCoreRead / oneCoreRead,5:F2}x scaling"); + _out.WriteLine($" copy all {2 * Bytes / BestSeconds(() => { ParallelCopy(x, y, cores); return y[0]; }) / 1e9,8:F1} GB/s"); + _out.WriteLine($" triad all {3 * Bytes / BestSeconds(() => { ParallelTriad(x, y, z, cores); return x[0]; }) / 1e9,8:F1} GB/s"); + } + + /// + /// Read bandwidth against working-set size, at one core and at all cores, using + /// independent accumulators so the loop is throughput-bound rather than latency-bound. + /// + /// The all-core column is the one that explains parallel scaling. Private L1/L2 scale with + /// cores; a shared L3 and DRAM do not. Comparing the two columns at each size shows exactly where + /// adding cores stops buying bandwidth — which is the difference between "this kernel is slow" and + /// "this kernel is fed slowly", and no compute measurement can distinguish them. + /// + /// Each core reads its own private buffer in the all-core arm, not a shared one: sharing + /// would measure cache-line replication rather than aggregate bandwidth. + /// + private void ReportWorkingSetSweep(int cores) + { + _out.WriteLine("--- read bandwidth vs working set (independent streams) ---"); + _out.WriteLine($" {"size",9}{"1 core",12}{"all cores",13}{"scaling",10}"); + + int[] kilobytes = [16, 32, 48, 64, 128, 256, 512, 1024, 2048, 8192, 32768, 131072]; + + foreach (var kb in kilobytes) + { + var floats = kb * 1024 / sizeof(float); + var passes = Math.Max(8, (int)(64L * 1024 * 1024 / ((long)floats * sizeof(float)))); + + var single = new float[floats]; + for (var i = 0; i < floats; i++) + { + single[i] = i; + } + + var oneCore = (double)floats * sizeof(float) * passes + / BestSeconds(() => ReadRepeated(single, 0, floats, passes)) / 1e9; + + // One private buffer per worker: a shared buffer would measure replication, not bandwidth. + var buffers = new float[cores][]; + for (var w = 0; w < cores; w++) + { + buffers[w] = new float[floats]; + Array.Copy(single, buffers[w], floats); + } + + var allSeconds = BestSeconds(() => + { + var partial = new float[cores]; + Parallel.For(0, cores, new ParallelOptions { MaxDegreeOfParallelism = cores }, + w => partial[w] = ReadRepeated(buffers[w], 0, floats, passes)); + return partial[0]; + }); + + var allCores = (double)floats * sizeof(float) * passes * cores / allSeconds / 1e9; + + _out.WriteLine( + $" {kb,6} KB {oneCore,8:F1} GB/s{allCores,9:F1} GB/s{allCores / oneCore,8:F1}x " + + new string('#', Math.Min(40, (int)(allCores / 25)))); + } + } + + /// + /// Sums a range times through independent accumulators. + /// + /// The pass loop lives inside the accumulator setup deliberately. With the repetition + /// outside, a small window's cost is dominated by re-initialising eight accumulators and doing a + /// horizontal reduction per call — which is why an earlier version reported 7.5 GB/s at 32 KB, below + /// its own DRAM figure. That was measurement overhead, not bandwidth. + /// + private static float ReadRepeated(float[] data, int from, int to, int passes) + { + var width = Vector.Count; + var step = width * Streams; + + Vector a0 = default, a1 = default, a2 = default, a3 = default; + Vector a4 = default, a5 = default, a6 = default, a7 = default; + + for (var p = 0; p < passes; p++) + { + var i = from; + for (; i <= to - step; i += step) + { + a0 += new Vector(data.AsSpan(i, width)); + a1 += new Vector(data.AsSpan(i + width, width)); + a2 += new Vector(data.AsSpan(i + (2 * width), width)); + a3 += new Vector(data.AsSpan(i + (3 * width), width)); + a4 += new Vector(data.AsSpan(i + (4 * width), width)); + a5 += new Vector(data.AsSpan(i + (5 * width), width)); + a6 += new Vector(data.AsSpan(i + (6 * width), width)); + a7 += new Vector(data.AsSpan(i + (7 * width), width)); + } + } + + return Vector.Sum(a0 + a1 + a2 + a3 + a4 + a5 + a6 + a7); + } + + /// Single pass over a range — used by the whole-buffer bandwidth figures. + private static float ReadRange(float[] data, int from, int to) + { + return ReadRepeated(data, from, to, 1); + } + + private static float ParallelRead(float[] source, int cores) + { + var partial = new float[cores]; + + Parallel.For(0, cores, new ParallelOptions { MaxDegreeOfParallelism = cores }, w => + { + var (from, to) = Slice(source.Length, cores, w); + partial[w] = ReadRange(source, from, to); + }); + + var total = 0f; + for (var i = 0; i < partial.Length; i++) + { + total += partial[i]; + } + + return total; + } + + private static void ParallelCopy(float[] source, float[] destination, int cores) + { + Parallel.For(0, cores, new ParallelOptions { MaxDegreeOfParallelism = cores }, w => + { + var (from, to) = Slice(source.Length, cores, w); + source.AsSpan(from, to - from).CopyTo(destination.AsSpan(from, to - from)); + }); + } + + private static void ParallelTriad(float[] a, float[] b, float[] c, int cores) + { + Parallel.For(0, cores, new ParallelOptions { MaxDegreeOfParallelism = cores }, w => + { + var (from, to) = Slice(a.Length, cores, w); + var scalar = new Vector(3f); + var width = Vector.Count; + var i = from; + + for (; i <= to - width; i += width) + { + var vb = new Vector(b.AsSpan(i, width)); + var vc = new Vector(c.AsSpan(i, width)); + (vb + (scalar * vc)).CopyTo(a.AsSpan(i, width)); + } + + for (; i < to; i++) + { + a[i] = b[i] + (3f * c[i]); + } + }); + } + + private static (int From, int To) Slice(int length, int workers, int worker) + { + var width = Vector.Count; + var per = length / workers / width * width; + var from = worker * per; + + return (from, worker == workers - 1 ? length : from + per); + } + + private static double BestSeconds(Func body) + { + body(); + + var best = double.MaxValue; + + for (var i = 0; i < Repeats; i++) + { + var started = ValueStopwatch.StartNew(); + _sink = body(); + var elapsed = started.GetElapsedTime().TotalSeconds; + best = Math.Min(best, elapsed); + } + + return best; + } + + private static float FmaChains128() + { + var m = Vector128.Create(1.000001f); + var a = Vector128.Create(0.000001f); + + Vector128 c0 = Vector128.Create(1f), c1 = Vector128.Create(2f); + Vector128 c2 = Vector128.Create(3f), c3 = Vector128.Create(4f); + Vector128 c4 = Vector128.Create(5f), c5 = Vector128.Create(6f); + Vector128 c6 = Vector128.Create(7f), c7 = Vector128.Create(8f); + Vector128 c8 = Vector128.Create(9f), c9 = Vector128.Create(10f); + Vector128 c10 = Vector128.Create(11f), c11 = Vector128.Create(12f); + + for (var i = 0; i < ComputeIterations; i++) + { + c0 = (c0 * m) + a; + c1 = (c1 * m) + a; + c2 = (c2 * m) + a; + c3 = (c3 * m) + a; + c4 = (c4 * m) + a; + c5 = (c5 * m) + a; + c6 = (c6 * m) + a; + c7 = (c7 * m) + a; + c8 = (c8 * m) + a; + c9 = (c9 * m) + a; + c10 = (c10 * m) + a; + c11 = (c11 * m) + a; + } + + return Vector128.Sum(c0 + c1 + c2 + c3 + c4 + c5 + c6 + c7 + c8 + c9 + c10 + c11); + } + + private static float FmaChains256() + { + var m = Vector256.Create(1.000001f); + var a = Vector256.Create(0.000001f); + + Vector256 c0 = Vector256.Create(1f), c1 = Vector256.Create(2f); + Vector256 c2 = Vector256.Create(3f), c3 = Vector256.Create(4f); + Vector256 c4 = Vector256.Create(5f), c5 = Vector256.Create(6f); + Vector256 c6 = Vector256.Create(7f), c7 = Vector256.Create(8f); + Vector256 c8 = Vector256.Create(9f), c9 = Vector256.Create(10f); + Vector256 c10 = Vector256.Create(11f), c11 = Vector256.Create(12f); + + for (var i = 0; i < ComputeIterations; i++) + { + c0 = Fma.MultiplyAdd(c0, m, a); + c1 = Fma.MultiplyAdd(c1, m, a); + c2 = Fma.MultiplyAdd(c2, m, a); + c3 = Fma.MultiplyAdd(c3, m, a); + c4 = Fma.MultiplyAdd(c4, m, a); + c5 = Fma.MultiplyAdd(c5, m, a); + c6 = Fma.MultiplyAdd(c6, m, a); + c7 = Fma.MultiplyAdd(c7, m, a); + c8 = Fma.MultiplyAdd(c8, m, a); + c9 = Fma.MultiplyAdd(c9, m, a); + c10 = Fma.MultiplyAdd(c10, m, a); + c11 = Fma.MultiplyAdd(c11, m, a); + } + + return Vector256.Sum(c0 + c1 + c2 + c3 + c4 + c5 + c6 + c7 + c8 + c9 + c10 + c11); + } + + private static float FmaChains512() + { + var m = Vector512.Create(1.000001f); + var a = Vector512.Create(0.000001f); + + Vector512 c0 = Vector512.Create(1f), c1 = Vector512.Create(2f); + Vector512 c2 = Vector512.Create(3f), c3 = Vector512.Create(4f); + Vector512 c4 = Vector512.Create(5f), c5 = Vector512.Create(6f); + Vector512 c6 = Vector512.Create(7f), c7 = Vector512.Create(8f); + Vector512 c8 = Vector512.Create(9f), c9 = Vector512.Create(10f); + Vector512 c10 = Vector512.Create(11f), c11 = Vector512.Create(12f); + + for (var i = 0; i < ComputeIterations; i++) + { + c0 = Avx512F.FusedMultiplyAdd(c0, m, a); + c1 = Avx512F.FusedMultiplyAdd(c1, m, a); + c2 = Avx512F.FusedMultiplyAdd(c2, m, a); + c3 = Avx512F.FusedMultiplyAdd(c3, m, a); + c4 = Avx512F.FusedMultiplyAdd(c4, m, a); + c5 = Avx512F.FusedMultiplyAdd(c5, m, a); + c6 = Avx512F.FusedMultiplyAdd(c6, m, a); + c7 = Avx512F.FusedMultiplyAdd(c7, m, a); + c8 = Avx512F.FusedMultiplyAdd(c8, m, a); + c9 = Avx512F.FusedMultiplyAdd(c9, m, a); + c10 = Avx512F.FusedMultiplyAdd(c10, m, a); + c11 = Avx512F.FusedMultiplyAdd(c11, m, a); + } + + return Vector512.Sum(c0 + c1 + c2 + c3 + c4 + c5 + c6 + c7 + c8 + c9 + c10 + c11); + } + } +} From 18e3e3b9e305eb10c0404753928026c33efd6ca7 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Thu, 23 Jul 2026 13:40:18 +0200 Subject: [PATCH 30/37] onnx --- ROADMAP.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ROADMAP.md b/ROADMAP.md index b3e8a75f..03ce0cec 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -109,7 +109,7 @@ the rules that survived recorded in the `feedback-measurement-discipline` memory left behind: `MachineRooflineBenchmark`, `DecodeGemvRooflineBenchmark`, `Diagnostics/Throughput.cs`, and the BenchmarkDotNet throughput columns. -**Next move is a business decision (perf course vs Redaction Gateway), not another *LLM* kernel.** Any further +**Next move is a business decision about product direction, not another *LLM* kernel.** Any further perf work should measure the ceiling before writing code — the discipline that made this track pay. #### ⚠ BUT: the largest untouched perf reserve in the project is CNN inference, not LLM — 13.2× behind ORT From 22d190a570822337ac47cf5ffde28259466c0bb4 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Thu, 23 Jul 2026 14:35:11 +0200 Subject: [PATCH 31/37] onnx --- Sources/Analyzers/CpuFeaturesGateAnalyzer.cs | 9 +- Sources/Main/Intrinsics/CpuFeatures.cs | 20 ++ .../Runtime/BatchedQuantProjection.cs | 93 +++++++- .../LanguageModels/Runtime/Q6KGemvKernel.cs | 22 +- .../Main/Randomization/VectorizedRandom.cs | 3 +- Sources/Main/Runtime/OverfitEnvironment.cs | 3 + Sources/Main/Runtime/OverfitParallel.cs | 27 +++ .../Diagnostics/PrefillFixedCostTests.cs | 191 ++++++++++++++++ .../PrefillLatencyByPromptLengthTests.cs | 104 +++++++++ .../PrefillScaleDecodeCostTests.cs | 193 ++++++++++++++++ .../Diagnostics/ShortPrefillBreakdownTests.cs | 87 ++++++++ docs/draft-eleven-refuted-hypotheses.md | 208 ++++++++++++++++++ 12 files changed, 949 insertions(+), 11 deletions(-) create mode 100644 Tests/LanguageModels/Diagnostics/PrefillFixedCostTests.cs create mode 100644 Tests/LanguageModels/Diagnostics/PrefillLatencyByPromptLengthTests.cs create mode 100644 Tests/LanguageModels/Diagnostics/PrefillScaleDecodeCostTests.cs create mode 100644 Tests/LanguageModels/Diagnostics/ShortPrefillBreakdownTests.cs create mode 100644 docs/draft-eleven-refuted-hypotheses.md diff --git a/Sources/Analyzers/CpuFeaturesGateAnalyzer.cs b/Sources/Analyzers/CpuFeaturesGateAnalyzer.cs index de858903..af7ab184 100644 --- a/Sources/Analyzers/CpuFeaturesGateAnalyzer.cs +++ b/Sources/Analyzers/CpuFeaturesGateAnalyzer.cs @@ -27,7 +27,7 @@ public sealed class CpuFeaturesGateAnalyzer : DiagnosticAnalyzer private static readonly DiagnosticDescriptor Rule = new( DiagnosticId, title: "Direct intrinsics IsSupported — use CpuFeatures", - messageFormat: "Direct '{0}.IsSupported' — gate ISA paths through CpuFeatures (CpuFeatures.Has{0}): one audit point, composed flags, same JIT constant-folding", + messageFormat: "Direct '{0}.{1}' — gate ISA and vector-width checks through CpuFeatures (CpuFeatures.Has{0}): one audit point, composed flags, same JIT constant-folding", category: "Performance", defaultSeverity: DiagnosticSeverity.Warning, isEnabledByDefault: true, @@ -47,7 +47,10 @@ private static void AnalyzePropertyReference(OperationAnalysisContext context) var operation = (IPropertyReferenceOperation)context.Operation; var property = operation.Property; - if (property.Name != "IsSupported" || !property.IsStatic) + // IsHardwareAccelerated is the portable width check (Vector128/256/512) that TensorPrimitives and + // ImageSharp both use; it is a second door to the same room and belongs behind the same facade, or + // the centralisation only holds for the x86 half. + if (property.Name is not ("IsSupported" or "IsHardwareAccelerated") || !property.IsStatic) { return; } @@ -64,7 +67,7 @@ private static void AnalyzePropertyReference(OperationAnalysisContext context) } context.ReportDiagnostic(Diagnostic.Create( - Rule, operation.Syntax.GetLocation(), property.ContainingType.Name)); + Rule, operation.Syntax.GetLocation(), property.ContainingType.Name, property.Name)); } private static bool IsHardwareIntrinsicsType(INamedTypeSymbol? type) diff --git a/Sources/Main/Intrinsics/CpuFeatures.cs b/Sources/Main/Intrinsics/CpuFeatures.cs index af943991..ba06f3fb 100644 --- a/Sources/Main/Intrinsics/CpuFeatures.cs +++ b/Sources/Main/Intrinsics/CpuFeatures.cs @@ -3,6 +3,7 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com +using System.Runtime.Intrinsics; using System.Runtime.Intrinsics.Arm; using System.Runtime.Intrinsics.X86; @@ -31,6 +32,25 @@ internal static class CpuFeatures public static readonly bool HasSse3 = Sse3.IsSupported; + // ── Portable vector widths ──────────────────────────────────────────────── + // + // `Vector512.IsHardwareAccelerated` asks "is this width fast here?" rather than "is this x86 ISA + // present?", so the same branch covers AVX-512 on x86 and SVE on arm64 — which matters because this + // library also ships to Android. The flags above stay for kernels that genuinely need an x86-specific + // instruction (`vpmaddubsw`, `vpdpbusd`); these are for kernels whose only question is vector width. + // + // The three-step cascade (512 → 256 → 128 → scalar) is what both the BCL's own TensorPrimitives and + // ImageSharp converged on independently — two production codebases, same shape, so it is adopted here + // rather than reinvented. Note the property is IsHardwareAccelerated, not IsSupported: a CPU can + // *support* a width while executing it at half rate (Zen 4 double-pumps 512-bit through a 256-bit + // datapath), and the runtime reports the useful answer rather than the nominal one. + + public static readonly bool HasVector128 = Vector128.IsHardwareAccelerated; + + public static readonly bool HasVector256 = Vector256.IsHardwareAccelerated; + + public static readonly bool HasVector512 = Vector512.IsHardwareAccelerated; + // to zawsze na koncu - bo zalezy od HasFma i HasAvx2 (pola sa inicjowane od gory do dolu) public static readonly bool HasAvx2Fma = HasAvx2 && HasFma; } diff --git a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs index 23f9ffbf..b7f455bd 100644 --- a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs +++ b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs @@ -79,6 +79,33 @@ internal static class BatchedQuantProjection /// internal static bool UseOutputBlocking; + /// + /// Chooses the parallelisation axis from the tile count: band over output rows when fanning out over + /// column tiles would leave most of the pool idle, tile otherwise. + /// + /// Why this is not a preference but a measurement. Prefill fans out over column tiles and + /// tiles = rows / NR, so a 16-token prompt yields two work items for sixteen cores while a + /// 672-token prompt yields eighty-four. Measured on Qwen-3B, banding versus tiling: + /// + /// + /// 16 tokens (2 tiles)224.8 → 177.3 ms — 1.27× + /// 32 tokens (4 tiles)267.6 → 235.0 ms — 1.14× + /// 64 tokens (8 tiles)352.0 → 356.5 ms — 0.99× (crossover) + /// 672 tokens (84 tiles)2231 → 2680 ms — 0.83× + /// + /// + /// Banding was built earlier, measured at −20% on a 672-token prompt, and left off — a correct + /// decision from an incomplete experiment, because only the long prompt was ever tried. The property + /// that makes banding pointless when tiles are plentiful is exactly what is missing when they are not. + /// This matters for latency users actually feel: chat prompts are tens of tokens, not hundreds. + /// + private static bool ShouldBandOutputRows(int tiles, int cores) + { + // Half the cores is where the measured curve crosses: at 8 tiles on 16 physical cores the two + // axes tie, below it banding wins, above it the extra packing cost dominates. + return tiles < cores / 2; + } + /// /// Decode every weight block's F16 scale/min pair to once per projection instead of /// once per column tile. @@ -89,7 +116,8 @@ internal static class BatchedQuantProjection /// it is fixed work per block, so it is exactly the term the tile-width sweep showed being amortised /// across columns. Hoisting it divides the work by the tile count. /// - internal static bool UsePrecomputedScales = true; + internal static bool UsePrecomputedScales = + Environment.GetEnvironmentVariable(OverfitEnvironment.PrecomputedScales) != "0"; /// /// Route the tiled Q4_K prefill GEMM through , which processes @@ -432,7 +460,7 @@ private static unsafe void DispatchTiledQ4K( Avx512 = UseAvx512PrefillQ4K && !DisableRepackedKernelsForParity, }; - if (!UseOutputBlocking) + if (!UseOutputBlocking && !ShouldBandOutputRows(tiles, Environment.ProcessorCount / 2)) { OverfitParallel.For(0, tiles, &TiledChunk, &ctx); return; @@ -519,8 +547,24 @@ private static unsafe void DispatchTiledQ6K( DecodedScales = dsc, DecodedScalesLength = scaleCount, Avx512 = UseAvx512PrefillQ6K && !DisableRepackedKernelsForParity, + Tiles = tiles, }; - OverfitParallel.For(0, tiles, &TiledQ6KChunk, &ctx); + + // Same axis choice as the Q4_K path. Without it `ffn_down` — 36% of a 16-token prefill — + // keeps fanning out over two column tiles while fourteen cores idle. + if (!UseOutputBlocking && !ShouldBandOutputRows(tiles, Environment.ProcessorCount / 2)) + { + OverfitParallel.For(0, tiles, &TiledQ6KChunk, &ctx); + return; + } + + var totalGroupsQ6 = outputSize / 8; + ctx.GroupsPerBand = ResolveGroupsPerBand( + totalGroupsQ6, spr, Q6KRepack.BlockKx8Bytes, cores); + + var bandsQ6 = (totalGroupsQ6 + ctx.GroupsPerBand - 1) / ctx.GroupsPerBand; + + OverfitParallel.For(0, bandsQ6, &TiledQ6KBandChunk, &ctx); } } @@ -545,6 +589,49 @@ private unsafe struct TiledQ6KContext /// Route through the two-columns-per-instruction AVX-512 kernel. public bool Avx512; + + /// Column tiles per projection — the inner loop when banding over output rows. + public int Tiles; + + /// Output groups per band; see . + public int GroupsPerBand; + } + + // One band of Q6_K output groups, swept by every column tile. Mirrors TiledBandChunk: bands are + // disjoint in weights (read-only) and in output rows, so no worker writes where another reads. + private static unsafe void TiledQ6KBandChunk(int start, int end, void* context) + { + ref var c = ref Unsafe.AsRef(context); + var totalGroups = c.OutputSize / 8; + + for (var band = start; band < end; band++) + { + var groupStart = band * c.GroupsPerBand; + var groupCount = Math.Min(c.GroupsPerBand, totalGroups - groupStart); + + for (var t = 0; t < c.Tiles; t++) + { + var s = t * c.Nr; + var cols = Math.Min(c.Nr, c.Rows - s); + var weights = new ReadOnlySpan(c.Repacked, c.RepackedLength); + var quants = new ReadOnlySpan(c.Quants + (long)s * c.InputSize, cols * c.InputSize); + var scales = new ReadOnlySpan(c.Scales + (long)s * c.Spr, cols * c.Spr); + var dst = new Span(c.Output + (long)s * c.OutputSize, cols * c.OutputSize); + var decoded = new ReadOnlySpan(c.DecodedScales, c.DecodedScalesLength); + + if (c.Avx512) + { + Q6KGemvKernel.GemmTiled512( + weights, c.OutputSize, c.InputSize, cols, quants, scales, dst, decoded, + groupStart, groupCount); + continue; + } + + Q6KGemvKernel.GemmTiled( + weights, c.OutputSize, c.InputSize, cols, quants, scales, dst, decoded, + groupStart, groupCount); + } + } } private static unsafe void TiledQ6KChunk(int start, int end, void* context) diff --git a/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs b/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs index 34761e50..9bf78b0f 100644 --- a/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs +++ b/Sources/Main/LanguageModels/Runtime/Q6KGemvKernel.cs @@ -332,7 +332,9 @@ public static unsafe void GemmTiled( ReadOnlySpan actQuants, ReadOnlySpan actScales, Span output, - ReadOnlySpan decodedScales = default) + ReadOnlySpan decodedScales = default, + int groupStart = 0, + int groupCount = 0) { if (cols is < 1 or > MaxTileCols) { @@ -357,7 +359,12 @@ public static unsafe void GemmTiled( fixed (float* outp = output) fixed (float* dsc = decodedScales) // null when the caller did not pre-decode; see DecodeBlockScales { - for (var x = 0; x < outputSize / 8; x++) + // Absolute group index, so a caller can hand this kernel one band of output rows and every + // weight/output offset below still lands in the right place. + var totalGroups = outputSize / 8; + var groupEnd = groupCount <= 0 ? totalGroups : Math.Min(groupStart + groupCount, totalGroups); + + for (var x = groupStart; x < groupEnd; x++) { var bptr = rep + (long)x * nb * BlockKx8Bytes; @@ -481,7 +488,9 @@ public static unsafe void GemmTiled512( ReadOnlySpan actQuants, ReadOnlySpan actScales, Span output, - ReadOnlySpan decodedScales = default) + ReadOnlySpan decodedScales = default, + int groupStart = 0, + int groupCount = 0) { if (cols is < 1 or > MaxTileCols) { @@ -505,7 +514,12 @@ public static unsafe void GemmTiled512( fixed (float* outp = output) fixed (float* dsc = decodedScales) { - for (var x = 0; x < outputSize / 8; x++) + // Absolute group index, so a caller can hand this kernel one band of output rows and every + // weight/output offset below still lands in the right place. + var totalGroups = outputSize / 8; + var groupEnd = groupCount <= 0 ? totalGroups : Math.Min(groupStart + groupCount, totalGroups); + + for (var x = groupStart; x < groupEnd; x++) { var bptr = rep + (long)x * nb * BlockKx8Bytes; diff --git a/Sources/Main/Randomization/VectorizedRandom.cs b/Sources/Main/Randomization/VectorizedRandom.cs index 6b883181..60864b3a 100644 --- a/Sources/Main/Randomization/VectorizedRandom.cs +++ b/Sources/Main/Randomization/VectorizedRandom.cs @@ -6,6 +6,7 @@ using System.Buffers.Binary; using System.Runtime.CompilerServices; using System.Runtime.Intrinsics; +using DevOnBike.Overfit.Intrinsics; using System.Security.Cryptography; namespace DevOnBike.Overfit.Randomization @@ -44,7 +45,7 @@ public sealed class VectorizedRandom : IRandom static VectorizedRandom() { - if (!Vector256.IsHardwareAccelerated) + if (!CpuFeatures.HasVector256) { throw new PlatformNotSupportedException("VectorizedRandom requires 256-bit SIMD hardware acceleration."); } diff --git a/Sources/Main/Runtime/OverfitEnvironment.cs b/Sources/Main/Runtime/OverfitEnvironment.cs index 63e6a121..91196a47 100644 --- a/Sources/Main/Runtime/OverfitEnvironment.cs +++ b/Sources/Main/Runtime/OverfitEnvironment.cs @@ -50,6 +50,9 @@ public static class OverfitEnvironment /// Set to 0 to force the AVX2 8×8 conv micro-kernel instead of the AVX-512 8×32 one. public const string ConvAvx512 = "OVERFIT_CONV_AVX512"; + /// Set to 0 to decode Q4_K/Q6_K F16 scales inside the tile loop instead of once per projection. + public const string PrecomputedScales = "OVERFIT_PRECOMPUTED_SCALES"; + // ── Prefill kernel switches (all default ON where the hardware allows; set to 0 to opt out) ── // These exist so a measured win can be A/B'd against its predecessor without a rebuild, and so a // regression on unfamiliar hardware can be bisected in the field rather than only on the dev box. diff --git a/Sources/Main/Runtime/OverfitParallel.cs b/Sources/Main/Runtime/OverfitParallel.cs index ffd38577..e6e13fa5 100644 --- a/Sources/Main/Runtime/OverfitParallel.cs +++ b/Sources/Main/Runtime/OverfitParallel.cs @@ -383,6 +383,28 @@ public static bool SuppressParallelismOnCurrentThread } /// + /// + /// Diagnostics only: count every real fan-out (the inline fast path is not counted, since it costs + /// nothing to launch). Off by default and checked before the interlocked increment. + /// + /// Exists because a prefill's fixed cost had to be attributed. Prompt-length sweeps showed + /// ~175 ms that does not scale with the prompt — 8% of a 672-token prefill but ~70% of a chat-sized + /// one — and the two candidates were per-dispatch launch overhead and the unavoidable walk over the + /// weight matrix. Counting the dispatches turns that from an argument into arithmetic. + /// + public static bool CountDispatches; + + private static long _dispatchCount; + + /// Fan-outs since the last . + public static long DispatchCount => Interlocked.Read(ref _dispatchCount); + + /// Clears the dispatch counter. + public static void ResetDispatchCount() + { + Interlocked.Exchange(ref _dispatchCount, 0); + } + /// Executes over chunks of /// [rangeStart, rangeEnd) across the worker pool. Equivalent to /// the grained overload with minItemsPerWorker = 1. @@ -472,6 +494,11 @@ public static void For( return; } + if (CountDispatches) + { + Interlocked.Increment(ref _dispatchCount); + } + var cap = maxWorkers < 1 ? 1 : Math.Min(maxWorkers, _workerCount); var chunkCount = Math.Min(cap, totalWork); var perChunk = (totalWork + chunkCount - 1) / chunkCount; diff --git a/Tests/LanguageModels/Diagnostics/PrefillFixedCostTests.cs b/Tests/LanguageModels/Diagnostics/PrefillFixedCostTests.cs new file mode 100644 index 00000000..bccce278 --- /dev/null +++ b/Tests/LanguageModels/Diagnostics/PrefillFixedCostTests.cs @@ -0,0 +1,191 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.Diagnostics; +using DevOnBike.Overfit.LanguageModels.Contracts; +using DevOnBike.Overfit.LanguageModels.Runtime; +using DevOnBike.Overfit.LanguageModels.Tokenizers; +using DevOnBike.Overfit.Runtime; +using DevOnBike.Overfit.Tests.TestSupport; +using Xunit.Abstractions; + +namespace DevOnBike.Overfit.Tests.LanguageModels.Diagnostics +{ + /// + /// Attributes the fixed cost of a prefill call — the part that does not scale with prompt length. + /// + /// The number being explained. A prompt-length sweep fitted prefill as + /// ~175 ms + 3.06 ms per token: the constant is 8% of a 672-token prefill and about 70% of a + /// chat-sized one. Every optimisation in the prefill campaign moved the per-token term. Nothing has ever + /// touched the constant, and for interactive use the constant is most of the latency. + /// + /// Two candidates, measured rather than argued: + /// + /// Parallel launch overhead. A prefill issues one fan-out per projection per layer. If the + /// count is in the hundreds and each launch costs hundreds of microseconds, that alone is the + /// constant. Counted with OverfitParallel.CountDispatches, priced with an empty-body loop. + /// The weight walk. Prefill must read every weight once no matter how short the prompt, so + /// there is a floor of model bytes ÷ bandwidth that no compute optimisation can remove. If that + /// floor is most of the constant, short prefill is memory-bound exactly like decode — and the same + /// conclusion applies: a wider kernel cannot help. + /// + /// + public sealed class PrefillFixedCostTests + { + private const int ShortPrompt = 16; + private const int LongPrompt = 672; + + private readonly ITestOutputHelper _out; + + public PrefillFixedCostTests(ITestOutputHelper output) => _out = output; + + [LongFact] + public unsafe void Prefill_FixedCost_DispatchesVersusWeightWalk() + { + var path = TestModelPaths.Qwen3B.Q4KmGgufPath; + if (!File.Exists(path)) + { + _out.WriteLine($"missing {path}"); + return; + } + + var modelBytes = new FileInfo(path).Length; + + using var engine = CachedLlamaInferenceEngine.LoadGguf(path); + var tok = GgufTokenizer.Load(path); + var sampling = SamplingOptions.Greedy; + + var paragraph = string.Join(" ", + Enumerable.Repeat( + "The history of computing began with mechanical calculators and evolved through vacuum tubes, " + + "transistors, integrated circuits and finally the microprocessor era.", 40)); + var allIds = tok.Encode(paragraph); + + using (var warm = engine.CreateSession(1024)) + { + warm.Reset(allIds.AsSpan(0, 64).ToArray()); + warm.GenerateNextToken(in sampling); + } + + // ── 1. How many fan-outs does one prefill issue, and does the count depend on prompt length? ── + var shortDispatches = CountDispatches(engine, allIds, ShortPrompt); + var longDispatches = CountDispatches(engine, allIds, LongPrompt); + + _out.WriteLine($"dispatches per prefill: {ShortPrompt,4} tokens -> {shortDispatches,6}"); + _out.WriteLine($" {LongPrompt,4} tokens -> {longDispatches,6}"); + + // ── 2. What does one fan-out cost when the body does nothing? ── + var perDispatchUs = MeasureEmptyDispatchMicroseconds(); + var launchMs = shortDispatches * perDispatchUs / 1000.0; + + _out.WriteLine(string.Empty); + _out.WriteLine($"empty fan-out cost : {perDispatchUs,8:F1} us"); + _out.WriteLine($"launch cost at {ShortPrompt} tokens: {launchMs,8:F1} ms" + + $" ({100 * launchMs / 175.0,5:F0}% of the ~175 ms constant)"); + + // ── 3. The floor: every weight must be read once regardless of prompt length. ── + var readGbps = MeasureReadGigabytesPerSecond(); + var walkMs = modelBytes / (readGbps * 1e9) * 1000.0; + + _out.WriteLine(string.Empty); + _out.WriteLine($"model on disk : {modelBytes / 1e9,8:F2} GB"); + _out.WriteLine($"measured read bandwidth : {readGbps,8:F1} GB/s"); + _out.WriteLine($"weight-walk floor : {walkMs,8:F1} ms" + + $" ({100 * walkMs / 175.0,5:F0}% of the ~175 ms constant)"); + + _out.WriteLine(string.Empty); + _out.WriteLine($"accounted : {100 * (launchMs + walkMs) / 175.0,5:F0}% of the constant"); + + Assert.True(shortDispatches > 0, "no dispatches counted — the counter hook was not reached"); + } + + private static long CountDispatches(CachedLlamaInferenceEngine engine, int[] allIds, int length) + { + var ids = allIds.AsSpan(0, length).ToArray(); + + using var session = engine.CreateSession(1024); + + OverfitParallel.ResetDispatchCount(); + OverfitParallel.CountDispatches = true; + try + { + session.Reset(ids); + } + finally + { + OverfitParallel.CountDispatches = false; + } + + return OverfitParallel.DispatchCount; + } + + /// + /// Cost of one fan-out with a body that does nothing — pure launch, wake and join. The range is wide + /// enough that the pool actually fans out rather than taking the inline fast path, which is the whole + /// point: the inline path is free and is not what a prefill pays. + /// + private static unsafe double MeasureEmptyDispatchMicroseconds() + { + const int Iterations = 2000; + + // Warm the pool so thread wake-up is not charged to the first samples. + for (var i = 0; i < 100; i++) + { + OverfitParallel.For(0, 1024, 1, &NoOp, null); + } + + var best = double.MaxValue; + + for (var r = 0; r < 5; r++) + { + var started = ValueStopwatch.StartNew(); + + for (var i = 0; i < Iterations; i++) + { + OverfitParallel.For(0, 1024, 1, &NoOp, null); + } + + best = Math.Min(best, started.GetElapsedTime().TotalMilliseconds); + } + + return best * 1000.0 / Iterations; + } + + private static unsafe void NoOp(int start, int end, void* context) + { + } + + /// Single-core sequential read rate, the rate a weight walk can realistically achieve. + private static double MeasureReadGigabytesPerSecond() + { + const int Floats = 64 * 1024 * 1024; // 256 MB — past any cache + + var data = new float[Floats]; + for (var i = 0; i < Floats; i++) + { + data[i] = i; + } + + var best = double.MaxValue; + + for (var r = 0; r < 3; r++) + { + var started = ValueStopwatch.StartNew(); + var total = 0f; + + for (var i = 0; i < Floats; i += 8) + { + total += data[i]; + } + + var elapsed = started.GetElapsedTime().TotalSeconds; + GC.KeepAlive(total); + best = Math.Min(best, elapsed); + } + + return (double)Floats * sizeof(float) / best / 1e9; + } + } +} diff --git a/Tests/LanguageModels/Diagnostics/PrefillLatencyByPromptLengthTests.cs b/Tests/LanguageModels/Diagnostics/PrefillLatencyByPromptLengthTests.cs new file mode 100644 index 00000000..c9566e5d --- /dev/null +++ b/Tests/LanguageModels/Diagnostics/PrefillLatencyByPromptLengthTests.cs @@ -0,0 +1,104 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.Diagnostics; +using DevOnBike.Overfit.LanguageModels.Contracts; +using DevOnBike.Overfit.LanguageModels.Runtime; +using DevOnBike.Overfit.LanguageModels.Tokenizers; +using DevOnBike.Overfit.Tests.TestSupport; +using Xunit.Abstractions; + +namespace DevOnBike.Overfit.Tests.LanguageModels.Diagnostics +{ + /// + /// Prefill cost as a function of prompt length — the axis every prefill measurement in this project + /// has ignored. + /// + /// Why this exists. A whole optimisation campaign tuned prefill on a 672-token prompt and took + /// it from 143 to ~299 tok/s. A later comparison against another pure-.NET engine measured our + /// time-to-first-token on a ~25-token prompt at 410 ms against their 66 ms, while per-token decode + /// was slightly in our favour. 410 ms over 25 tokens is 16 ms per prompt token, against 3.3 ms per token at + /// 672 — the same engine, five times worse per token, purely because the prompt is short. + /// + /// Long prompts are what benchmarks use; short prompts are what interactive chat actually sends. If + /// per-token cost climbs as the prompt shrinks, the campaign optimised the benchmark rather than the user's + /// experience, and this test is what would have caught it. + /// + /// What it separates. Engine-level prefill only — no HTTP, no serialisation, no sampling. Put + /// next to the 410 ms measured through the server, the difference is per-request overhead rather than + /// kernel cost, and the two need very different fixes. + /// + /// The OVERFIT_PRECOMPUTED_SCALES arm tests the leading suspect: the F16 scale decode is + /// hoisted to once per projection, and that cost is independent of row count — so it amortises over 672 + /// rows and may not over 16. + /// + public sealed class PrefillLatencyByPromptLengthTests + { + private static readonly int[] PromptLengths = [8, 16, 32, 64, 128, 256, 512, 672]; + + private const int Repeats = 3; + + private readonly ITestOutputHelper _out; + + public PrefillLatencyByPromptLengthTests(ITestOutputHelper output) => _out = output; + + [LongFact] + public void Prefill_CostPerPromptToken_AcrossLengths() + { + var path = TestModelPaths.Qwen3B.Q4KmGgufPath; + if (!File.Exists(path)) + { + _out.WriteLine($"missing {path}"); + return; + } + + using var engine = CachedLlamaInferenceEngine.LoadGguf(path); + var tok = GgufTokenizer.Load(path); + var sampling = SamplingOptions.Greedy; + + var paragraph = string.Join(" ", + Enumerable.Repeat( + "The history of computing began with mechanical calculators and evolved through vacuum tubes, " + + "transistors, integrated circuits and finally the microprocessor era.", 40)); + var allIds = tok.Encode(paragraph); + + // Warm up outside every measurement: JIT, page-in the weights, the one-off repack. + using (var warm = engine.CreateSession(1024)) + { + warm.Reset(allIds.AsSpan(0, Math.Min(64, allIds.Length)).ToArray()); + warm.GenerateNextToken(in sampling); + } + + _out.WriteLine($"batched-prefill threshold is 16 tokens; hoisted scales = " + + $"{(Environment.GetEnvironmentVariable("OVERFIT_PRECOMPUTED_SCALES") != "0" ? "ON" : "OFF")}"); + _out.WriteLine(string.Empty); + _out.WriteLine($" {"tokens",7}{"prefill ms",12}{"ms/token",11}{"tok/s",10}"); + + foreach (var length in PromptLengths) + { + if (length > allIds.Length) + { + continue; + } + + var ids = allIds.AsSpan(0, length).ToArray(); + var best = double.MaxValue; + + for (var r = 0; r < Repeats; r++) + { + using var session = engine.CreateSession(1024); + var started = ValueStopwatch.StartNew(); + session.Reset(ids); + best = Math.Min(best, started.GetElapsedTime().TotalMilliseconds); + } + + _out.WriteLine( + $" {length,7}{best,10:F1} ms{best / length,10:F2}{length / (best / 1000.0),10:F0}"); + } + + Assert.True(allIds.Length >= 16, "prompt corpus too short to cross the batched threshold"); + } + } +} diff --git a/Tests/LanguageModels/Diagnostics/PrefillScaleDecodeCostTests.cs b/Tests/LanguageModels/Diagnostics/PrefillScaleDecodeCostTests.cs new file mode 100644 index 00000000..41980b18 --- /dev/null +++ b/Tests/LanguageModels/Diagnostics/PrefillScaleDecodeCostTests.cs @@ -0,0 +1,193 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.Diagnostics; +using DevOnBike.Overfit.LanguageModels.Contracts; +using DevOnBike.Overfit.LanguageModels.Runtime; +using DevOnBike.Overfit.LanguageModels.Tokenizers; +using DevOnBike.Overfit.Tests.TestSupport; +using Xunit.Abstractions; + +namespace DevOnBike.Overfit.Tests.LanguageModels.Diagnostics +{ + /// + /// Prices the F16 scale decode inside a real prefill, at a chat-sized prompt. + /// + /// Why an earlier A/B could not see this. Toggling UsePrecomputedScales compares two + /// ways of doing the same fixed work — hoisted to once per projection, or inline once per column tile. + /// Neither arm removes it, so the comparison came out neutral and the cost stayed invisible. Only + /// AblateF16Scales actually deletes the work, and it is bypassed when the hoist is on (the kernel + /// then reads a precomputed buffer). Removing the cost therefore needs both: hoist off, ablate on. + /// + /// The arithmetic that makes it a suspect. A 2.1 GB Q4_K model holds roughly 1.8 million + /// weight blocks, each carrying 16 F16 scale/min values, and every one is widened to float on every + /// prefill — about 29 million scalar conversions, independent of prompt length. A prompt-length + /// sweep fitted prefill at ~175 ms + 3.06 ms/token, and dispatch overhead plus the weight walk + /// account for under 16% of that constant. + /// + /// Why it matters. At 672 tokens the constant is 8% of prefill; at a chat-sized ~25 tokens it + /// is about 70%, which is what a user waiting for the first token actually experiences. Results are wrong + /// while ablated — this measures cost, never correctness. + /// + public sealed class PrefillScaleDecodeCostTests + { + private static readonly int[] Lengths = [16, 64, 672]; + + private const int Repeats = 3; + + private readonly ITestOutputHelper _out; + + public PrefillScaleDecodeCostTests(ITestOutputHelper output) => _out = output; + + [LongFact] + public void Prefill_ScaleDecodeShare_ByPromptLength() + { + var path = TestModelPaths.Qwen3B.Q4KmGgufPath; + if (!File.Exists(path)) + { + _out.WriteLine($"missing {path}"); + return; + } + + using var engine = CachedLlamaInferenceEngine.LoadGguf(path); + var tok = GgufTokenizer.Load(path); + var sampling = SamplingOptions.Greedy; + + var paragraph = string.Join(" ", + Enumerable.Repeat( + "The history of computing began with mechanical calculators and evolved through vacuum tubes, " + + "transistors, integrated circuits and finally the microprocessor era.", 40)); + var allIds = tok.Encode(paragraph); + + using (var warm = engine.CreateSession(1024)) + { + warm.Reset(allIds.AsSpan(0, 64).ToArray()); + warm.GenerateNextToken(in sampling); + } + + _out.WriteLine($" {"tokens",7}{"baseline",12}{"no scales",12}{"scale cost",12}{"share",8}"); + + foreach (var length in Lengths) + { + var ids = allIds.AsSpan(0, length).ToArray(); + + var baseline = Best(engine, ids, ablate: false); + var ablated = Best(engine, ids, ablate: true); + var cost = baseline - ablated; + + _out.WriteLine( + $" {length,7}{baseline,9:F1} ms{ablated,9:F1} ms{cost,9:F1} ms{100 * cost / baseline,7:F0}%"); + } + + Assert.True(allIds.Length >= 672, "prompt corpus too short"); + } + + /// + /// Short prompts starve the parallel pool: prefill fans out over column tiles, and + /// tiles = rows / 8, so a 16-token prompt produces two work items for sixteen cores while + /// a 672-token prompt produces eighty-four. The component breakdown shows every component about 4× + /// less efficient per token at 16 tokens, which is what two cores instead of ~thirteen looks like. + /// + /// UseOutputBlocking fans out over output-row bands instead, so the work-item count + /// stops depending on prompt length. It was built earlier and measured at −20% on a 672-token prompt, + /// where tiles are plentiful and banding only adds overhead — and was therefore left off. This tests + /// the case it was never tried on, where the same property that made it useless is exactly what is + /// missing. + /// + [LongFact] + public void Prefill_OutputBlocking_OnShortPrompts() + { + var path = TestModelPaths.Qwen3B.Q4KmGgufPath; + if (!File.Exists(path)) + { + _out.WriteLine($"missing {path}"); + return; + } + + using var engine = CachedLlamaInferenceEngine.LoadGguf(path); + var tok = GgufTokenizer.Load(path); + var sampling = SamplingOptions.Greedy; + + var paragraph = string.Join(" ", + Enumerable.Repeat( + "The history of computing began with mechanical calculators and evolved through vacuum tubes, " + + "transistors, integrated circuits and finally the microprocessor era.", 40)); + var allIds = tok.Encode(paragraph); + + using (var warm = engine.CreateSession(1024)) + { + warm.Reset(allIds.AsSpan(0, 64).ToArray()); + warm.GenerateNextToken(in sampling); + } + + _out.WriteLine($" {"tokens",7}{"tiles",7}{"per-tile",12}{"banded",12}{"speedup",10}"); + + foreach (var length in new[] { 16, 32, 64, 128, 256, 672 }) + { + var ids = allIds.AsSpan(0, length).ToArray(); + + var perTile = BestWithBanding(engine, ids, banded: false); + var banded = BestWithBanding(engine, ids, banded: true); + + _out.WriteLine( + $" {length,7}{(length + 7) / 8,7}{perTile,9:F1} ms{banded,9:F1} ms{perTile / banded,9:F2}x"); + } + + Assert.True(allIds.Length >= 672, "prompt corpus too short"); + } + + private static double BestWithBanding(CachedLlamaInferenceEngine engine, int[] ids, bool banded) + { + BatchedQuantProjection.UseOutputBlocking = banded; + + try + { + var best = double.MaxValue; + + for (var r = 0; r < Repeats; r++) + { + using var session = engine.CreateSession(1024); + var started = ValueStopwatch.StartNew(); + session.Reset(ids); + best = Math.Min(best, started.GetElapsedTime().TotalMilliseconds); + } + + return best; + } + finally + { + BatchedQuantProjection.UseOutputBlocking = false; + } + } + + private static double Best(CachedLlamaInferenceEngine engine, int[] ids, bool ablate) + { + // Both switches are needed: the ablation flag only reaches the inline decode, which the hoist + // bypasses by reading a precomputed buffer. + BatchedQuantProjection.UsePrecomputedScales = !ablate; + Q4KGemvKernel.AblateF16Scales = ablate; + + try + { + var best = double.MaxValue; + + for (var r = 0; r < Repeats; r++) + { + using var session = engine.CreateSession(1024); + var started = ValueStopwatch.StartNew(); + session.Reset(ids); + best = Math.Min(best, started.GetElapsedTime().TotalMilliseconds); + } + + return best; + } + finally + { + BatchedQuantProjection.UsePrecomputedScales = true; + Q4KGemvKernel.AblateF16Scales = false; + } + } + } +} diff --git a/Tests/LanguageModels/Diagnostics/ShortPrefillBreakdownTests.cs b/Tests/LanguageModels/Diagnostics/ShortPrefillBreakdownTests.cs new file mode 100644 index 00000000..6a033929 --- /dev/null +++ b/Tests/LanguageModels/Diagnostics/ShortPrefillBreakdownTests.cs @@ -0,0 +1,87 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.LanguageModels.Contracts; +using DevOnBike.Overfit.LanguageModels.Runtime; +using DevOnBike.Overfit.LanguageModels.Tokenizers; +using DevOnBike.Overfit.Tests.TestSupport; +using Xunit.Abstractions; + +namespace DevOnBike.Overfit.Tests.LanguageModels.Diagnostics +{ + /// + /// The component breakdown of a chat-sized prefill, next to the 672-token one every optimisation in + /// this project was tuned against. + /// + /// Why. A prompt-length sweep fitted prefill at ~175 ms + 3.06 ms/token. Three + /// candidates for that constant have been measured and refuted: parallel dispatch launches (2%), the + /// unavoidable weight walk (7–13%), and the F16 scale decode (2%). Roughly 85% remains unattributed, and + /// guessing a fourth candidate would repeat a mistake this project has made repeatedly today. The profiler + /// already splits prefill by component and by call count; running it at both lengths and comparing the + /// shares says where the constant lives without another hypothesis. + /// + /// Read the two tables against each other: a component whose absolute milliseconds barely + /// change between 16 and 672 tokens is the constant. One whose milliseconds scale with the prompt is the + /// per-token term, and irrelevant to time-to-first-token on a short prompt. + /// + public sealed class ShortPrefillBreakdownTests + { + private readonly ITestOutputHelper _out; + + public ShortPrefillBreakdownTests(ITestOutputHelper output) => _out = output; + + [LongFact] + public void Prefill_Breakdown_ShortVersusLongPrompt() + { + var path = TestModelPaths.Qwen3B.Q4KmGgufPath; + if (!File.Exists(path)) + { + _out.WriteLine($"missing {path}"); + return; + } + + using var engine = CachedLlamaInferenceEngine.LoadGguf(path); + var tok = GgufTokenizer.Load(path); + var sampling = SamplingOptions.Greedy; + + var paragraph = string.Join(" ", + Enumerable.Repeat( + "The history of computing began with mechanical calculators and evolved through vacuum tubes, " + + "transistors, integrated circuits and finally the microprocessor era.", 40)); + var allIds = tok.Encode(paragraph); + + using (var warm = engine.CreateSession(1024)) + { + warm.Reset(allIds.AsSpan(0, 64).ToArray()); + warm.GenerateNextToken(in sampling); + } + + foreach (var length in new[] { 16, 672 }) + { + var ids = allIds.AsSpan(0, length).ToArray(); + + PrefillProfiler.Reset(); + PrefillProfiler.Enabled = true; + try + { + for (var r = 0; r < 3; r++) + { + using var session = engine.CreateSession(1024); + session.Reset(ids); + } + } + finally + { + PrefillProfiler.Enabled = false; + } + + _out.WriteLine($"───────── prompt = {length} tokens ─────────"); + _out.WriteLine(PrefillProfiler.Report()); + } + + Assert.True(allIds.Length >= 672, "prompt corpus too short"); + } + } +} diff --git a/docs/draft-eleven-refuted-hypotheses.md b/docs/draft-eleven-refuted-hypotheses.md new file mode 100644 index 00000000..deae0f5f --- /dev/null +++ b/docs/draft-eleven-refuted-hypotheses.md @@ -0,0 +1,208 @@ +# DRAFT — Our C# matmul turned out to be faster than llama.cpp's. It took eleven refuted hypotheses to find out. + +> Status: draft. Every number here is measured on one machine (AMD Ryzen 9 9950X3D, .NET 10, Windows 11) +> and reproducible from the repository. Nothing is estimated; where something is estimated it says so. + +--- + +## The claim, and why you should not believe it yet + +Overfit is a pure-C# CPU inference engine — no native binaries, no Python, no ONNX Runtime. Its Q4_K +matrix-multiply kernel, run on llama.cpp's own benchmark shape, at the same instruction set and the same +thread count: + +| | time | TFLOP/s | +|---|---:|---:| +| llama.cpp (AVX2 build, `test-backend-ops`) | 38 559 µs | 1.56 | +| **Overfit `GemmTiled`** | **35 308 µs** | **1.70** | + +That is a 1.09× win in managed C# over hand-written C++ SIMD. + +Now the honest part: **the whole model is still slower than llama.cpp**, and I spent most of a day being +wrong about why. This article is about the being-wrong, because that is the part nobody publishes and the +only part that generalises. + +--- + +## Act I: the comfortable, wrong story + +Prefill — processing the prompt before the first token — was **3.76× behind** llama.cpp. Decode was only +1.13× behind. Two very different numbers for the same engine, which should have been the first clue. + +I read llama.cpp's `ggml_gemm_q4_K_8x8_q8_K`. Beautiful code: constant-index unrolled accumulators, a +`block_q8_Kx4` activation interleaving, four hand-tuned ISA variants. I formed a comfortable narrative: +*their kernel craft is better than ours; the gap decomposes as 2.34× kernel quality × 1.60× AVX-512.* + +I then spent several hours building things that followed from that narrative. Every single one failed: + +- **Register pressure.** Their kernel keeps a tile in registers; ours spills to `stackalloc` scratch. + Obvious cause. I split the pass in half to reduce live state — **exact tie**. +- **Activation interleaving.** Copy their `block_q8_Kx4` layout — never got built, because of Act II. +- **Fixed-tile specialisation.** Unrolled the tile loop to constant indices — **inconclusive, reverted**. + +Three failures, one after another, each of which sounded correct while I was writing it. + +**The mistake was not the hypotheses. It was that I never measured their kernel.** I read it and inferred. + +--- + +## Act II: measure the thing you are comparing against + +llama.cpp ships `test-backend-ops`, which times individual operators. Thirty seconds of work: + +``` +q4_K m=4096 n=512 k=14336 → 38558.65 us/run, 60.13 GFLOP/run → 1.56 TFLOPS +``` + +I put that exact shape into our benchmark. We came out at **1.70 TFLOP/s**. + +**The premise of everything I had built that day was false.** Their kernel was not better. The gap lived +somewhere else entirely, and three days of planned work evaporated in one measurement. + +> **Lesson 1.** Reading someone else's code produces *plausible explanations*, which are worse than no +> explanation because they feel like knowledge. If you are comparing against a project, measure it. + +--- + +## Act III: the ceiling is not where you think + +With the kernel exonerated, I needed a real ceiling. So I wrote one — a standalone probe that measures what +the machine can actually do, since a bare "1.7 TFLOP/s" is meaningless until you know whether the box tops +out at 2 or at 20. + +The first version reported a peak of **0.79 TFLOP/s** — *below* what our real matmul achieved. Impossible +for a loop that touches no memory. + +The bug: I had put the accumulator chains in a `stackalloc` span. + +```csharp +Span> acc = stackalloc Vector256[Chains]; // an L1 round-trip per accumulator +``` + +A span indexed by a loop variable does not live in registers. It forced a load and a store per accumulator +per iteration, so the benchmark measured L1 latency instead of FMA issue rate. Named locals with constant +indices fixed it: **2.19 TFLOP/s**. + +> **Lesson 2.** `stackalloc` is not "registers". Constant-index named locals are. And if a synthetic peak +> comes out below your real code, the benchmark is broken — real code cannot exceed a true ceiling. + +And then I made a *different* error with the same shape. A working-set sweep, meant to expose the L1→L2→L3 +steps, came out perfectly flat at ~75 GB/s from 8 KB to 128 MB. I concluded the machine had no cache cliff +and that cache blocking therefore could not pay — a conclusion I acted on. + +It had one accumulator. It was measuring the dependency chain's *latency*, which was below every cache +level's bandwidth, so no level could show. With eight independent streams: + +| working set | 1 core | all cores | scaling | +|---|---:|---:|---:| +| 16 KB – 2 MB | ~76 GB/s | 700–900 GB/s | **9–12×** | +| 8 MB | 67 | 113 | **1.7×** | +| 128 MB | 60 | 64 | 1.1× | + +There is a cliff, and it lands exactly where **16 cores × 8 MB = 128 MB = this chip's L3 including +V-cache** — a number the benchmark was never given. That self-consistency is the only reason I now trust it. + +> **Lesson 3.** One accumulator measures latency. Several measure throughput. This is the same mistake as +> Lesson 2 wearing a different hat, and I made both in the same afternoon. + +--- + +## Act IV: what actually paid + +Once the ceilings were honest, the wins were unglamorous. + +**The biggest one was not making anything faster — it was doing it less often.** The Q4_K kernel widens F16 +scales to float once per weight block. That reads as amortised. But the kernel is invoked once per *column +tile* — 84 times for a 672-token prompt — so every scale was widened **84 times over**. Ablation priced it +at 12% of the kernel. + +x86 has `vcvtph2ps`, which would widen eight halves in one instruction. .NET exposes neither an `F16C` +intrinsic class nor a `Half` overload of `Vector128.Widen`, so that instruction was unavailable. + +I hoisted the decode to once per projection instead. **12% → 0.14%.** + +The missing instruction would have made the work ~4× faster. Restructuring deleted 83/84 of it. And had the +instruction been available, I would very likely have used it, banked the 4×, and never asked the better +question. + +> **Lesson 4.** Before looking for a faster way to do the work, count how many times you do it. + +**Second: the same technique inverted between two kernels.** Porting the Q4_K kernel to AVX-512 by pairing +two activation columns per instruction gave **+13.8%**. The identical port applied to the Q6_K kernel gave +**−20%**, and was reverted. + +Why: pairing pays for the `vinserti64x4` that broadcasts shared weights with the arithmetic subsequently done +on them. Q4_K broadcasts eight vectors per sub-block and then issues sixteen paired statements against them. +Q6_K broadcasts six per `k`, sixteen times per block, for far less arithmetic each. The lane-crossing traffic +outran the savings. + +A second attempt on Q6_K — pairing what was *already adjacent in memory* rather than broadcasting — gave +**+12%**. Same kernel, same instruction set, same shape. Only the choice of what shares a register changed. + +> **Lesson 5.** A wider vector is not a property of the ISA. It is a ratio between broadcast cost and work +> done per broadcast, and that ratio is per-kernel. Do not extrapolate a port from one kernel to another. + +--- + +## Act V: the measurement discipline, in one page + +Everything above reduces to a handful of rules that survived the day: + +1. **Two identical arms are the cheapest canary there is.** One sweep reported the same configuration 21% + apart under two different names. Without that accidental duplicate I would have believed the whole table. +2. **Interleave arms; never run all-A-then-all-B.** A 20% thermal drift between two sequential runs once + inverted a result completely. +3. **A single-arm measurement after a hot-path change is worthless.** One change "gave" +4% while an + untouched component moved with it — that was the machine, not the change. +4. **An impossible ordering means a broken benchmark, not a discovery.** When 512-bit measured *slower* than + 256-bit, the cause was a helper method the JIT declined to inline: I was timing the calling convention. +5. **Ablate inside the real kernel; do not micro-benchmark the part.** Toggling a piece off in production + measures its real share. A synthetic harness measures the harness. +6. **Match the ceiling to the instruction mix your code actually issues.** I claimed our kernel ran at 78% + of the float ceiling. It performs one `vpmaddubsw` per 32 MACs; against the ceiling that applies it was + at 15%. The number was arithmetic, not measurement. + +--- + +## Where it ended + +Prefill went from **143 to ~299 tok/s** — the gap to llama.cpp's AVX-512 build from 3.76× to **1.81×**, and +to their AVX2 build to about **1.14×**. On machines without AVX-512, a managed C# engine is roughly at +parity with llama.cpp on prompt processing. + +Decode is memory-bound and stays 1.13× behind; the probe shows our GEMV already runs at 82% of the DRAM read +ceiling with compute headroom to spare, so a wider instruction set cannot help there. That is a closed +question rather than an open one, which is worth as much as a speedup. + +Final tally for the campaign: **six wins, about fifteen refuted hypotheses, three corrected arithmetic errors +of my own.** The corrections included counting FLOPs at half their true value for an entire afternoon, +because the commonly quoted "15.5 GFLOPs" for VGG-16 is a *MAC* count. + +I do not think the wins are the interesting part. + +--- + +## Try it on your machine + +The probe is a single xUnit test with no external dependencies. It reports peak FMA at every vector width +(one core and all cores), sustained memory bandwidth, and the working-set sweep — and asserts that its own +measurement loops allocate zero bytes, so it cannot silently degrade into a GC benchmark. + +``` +dotnet test -c Release --filter FullyQualifiedName~MachineProbe +``` + +Read your numbers before you believe anyone's — including mine. + +--- + +### Notes for revision (remove before publishing) + +- Decide the audience: kernel authors (narrow) vs anyone who optimises anything (wide). Lessons 1–6 are + discipline and generalise to SQL, pipelines, allocation work. The SIMD is the *illustration*. Leaning wide + is probably the difference between a few hundred readers and a few thousand. +- Charts worth making from data already in the repo: the worker sweep, the working-set cliff, the + before/after prefill bar, the tile-shape ceilings. +- Consider splitting Act V into its own follow-up piece — it is the most reusable and the most quotable. +- Verify every figure against ROADMAP before publishing; several numbers in this draft were themselves + corrected mid-session. From 3dad18b2b444da5fa39c63c8cf50048521b7bd59 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Thu, 23 Jul 2026 16:24:47 +0200 Subject: [PATCH 32/37] dotllm --- .../Main/LanguageModels/Chat/ChatSession.cs | 17 +- .../LanguageModels/Contracts/ISlmSession.cs | 16 ++ .../Runtime/BatchedQuantProjection.cs | 84 +++++- .../Runtime/CachedLlamaSession.cs | 92 +++++++ Sources/Server/OverfitOpenAiServer.cs | 41 ++- .../Diagnostics/PrefillAxisSweepTests.cs | 143 ++++++++++ .../Runtime/PromptCacheReuseTests.cs | 251 ++++++++++++++++++ 7 files changed, 631 insertions(+), 13 deletions(-) create mode 100644 Tests/LanguageModels/Diagnostics/PrefillAxisSweepTests.cs create mode 100644 Tests/LanguageModels/Runtime/PromptCacheReuseTests.cs diff --git a/Sources/Main/LanguageModels/Chat/ChatSession.cs b/Sources/Main/LanguageModels/Chat/ChatSession.cs index c784bbc5..f075eab3 100644 --- a/Sources/Main/LanguageModels/Chat/ChatSession.cs +++ b/Sources/Main/LanguageModels/Chat/ChatSession.cs @@ -90,6 +90,16 @@ public GenerationStats LastStats get; private set; } + /// + /// How many prompt tokens the most recent turn took from the KV cache instead of re-encoding — + /// 0 on the first turn of a conversation, and typically the whole preceding conversation + /// afterwards. LastStats.PromptTokens minus this is what was actually forwarded. + /// + public int CachedPromptTokens + { + get; private set; + } + public void AddSystem(string content) => _history.Add(ChatMessage.System(content)); /// @@ -183,7 +193,12 @@ private string GenerateFor( var tokenCount = _tokenizer.CountTokens(promptText); var promptTokens = new int[tokenCount]; var written = _tokenizer.Encode(promptText, promptTokens); - _session.Reset(promptTokens.AsSpan(0, written)); + + // Reuse the KV already built for the shared prefix of the previous turn. Every turn re-sends the + // whole conversation, so the tokens up to the end of the last assistant reply are byte-identical + // to what this session just encoded — re-prefilling them is pure duplicate work. Falls back to a + // full prefill on its own when the session is fresh or the conversation diverged. + CachedPromptTokens = _session.PrefillReusingCache(promptTokens.AsSpan(0, written)); var stopwatch = ValueStopwatch.StartNew(); var reply = Generate(promptTokens.AsSpan(0, written), in options, onText, constraint, out var generatedTokens); diff --git a/Sources/Main/LanguageModels/Contracts/ISlmSession.cs b/Sources/Main/LanguageModels/Contracts/ISlmSession.cs index a7da4b8e..0226847c 100644 --- a/Sources/Main/LanguageModels/Contracts/ISlmSession.cs +++ b/Sources/Main/LanguageModels/Contracts/ISlmSession.cs @@ -31,6 +31,22 @@ bool HasKeyValueCache void Reset(ReadOnlySpan promptTokens); + /// + /// Prefills , reusing whatever leading portion is already in this + /// session's KV cache, and returns how many tokens that saved. The end state matches + /// exactly — reuse is an optimisation, never a + /// behaviour change. The default implementation reuses nothing, so sessions that do not track + /// their cached tokens keep working unchanged. + /// + /// This is the multi-turn chat lever: every turn re-sends the whole conversation, so without + /// reuse turn N re-encodes everything turns 1..N-1 already encoded. + /// + int PrefillReusingCache(ReadOnlySpan promptTokens) + { + Reset(promptTokens); + return 0; + } + int GenerateNextToken(in SamplingOptions sampling); /// diff --git a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs index b7f455bd..c2f7e1b4 100644 --- a/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs +++ b/Sources/Main/LanguageModels/Runtime/BatchedQuantProjection.cs @@ -79,9 +79,12 @@ internal static class BatchedQuantProjection /// internal static bool UseOutputBlocking; + /// Test hook: pin the column-tile axis so the automatic rule can be A/B'd against it. + internal static bool DisableAutoAxisSelection; + /// - /// Chooses the parallelisation axis from the tile count: band over output rows when fanning out over - /// column tiles would leave most of the pool idle, tile otherwise. + /// Chooses the parallelisation axis: band over output rows when column tiling at the widest tile + /// the kernel supports could not fill the machine, tile otherwise. /// /// Why this is not a preference but a measurement. Prefill fans out over column tiles and /// tiles = rows / NR, so a 16-token prompt yields two work items for sixteen cores while a @@ -98,12 +101,53 @@ internal static class BatchedQuantProjection /// decision from an incomplete experiment, because only the long prompt was ever tried. The property /// that makes banding pointless when tiles are plentiful is exactly what is missing when they are not. /// This matters for latency users actually feel: chat prompts are tens of tokens, not hundreds. + /// + /// Why the count is taken at the maximum width, not at the resolved one. The first version + /// of this rule fed it 's answer, which collapses to NR=4 exactly when rows + /// are scarce — so a 48-token prompt reported twelve tiles, cleared the threshold and never banded, and + /// the rule fired only below 32 tokens. The question the axis decision is actually asking is "can column + /// tiling fill the machine at a width worth using", so it must be evaluated at that width. Measured on + /// Qwen-3B with the width free to follow the axis (best of three, ms): + /// + /// + /// 96 tokens (6 tiles)442.1 tiled → 428.8 banded + /// 128 tokens (8 tiles)610.2 → 545.4 + /// 192 tokens (12 tiles)807.7 → 799.6 + /// 256 tokens (16 tiles)1114.6 → 1034.9 + /// 384 tokens (24 tiles)1303.1 tiled → 1476.9 banded (banding loses) + /// /// - private static bool ShouldBandOutputRows(int tiles, int cores) + private static bool ShouldBandOutputRows(int tilesAtMaxWidth, int cores) { - // Half the cores is where the measured curve crosses: at 8 tiles on 16 physical cores the two - // axes tie, below it banding wins, above it the extra packing cost dominates. - return tiles < cores / 2; + // One tile per physical core is where the measured curve crosses: at 16 tiles on 16 physical cores + // banding still wins by 7%, at 24 it loses by 13%. + return tilesAtMaxWidth <= cores; + } + + /// + /// Column-tile width once the axis is known: the widest the kernel supports whenever the work is + /// banded over output rows, and 's parallelism-constrained choice + /// otherwise. + /// + /// Why the two decisions are linked. ResolveTileCols caps the width so that + /// rows / NR still leaves a tile per core — a necessary rule while tiles are the work + /// items. Banding makes output-row bands the work items, so tile count stops driving parallelism and + /// the cap has nothing left to protect. Widening then costs nothing and halves the number of passes + /// over the weights. + /// + /// Measured on Qwen-3B at chat prompt lengths, NR=8 against NR=16 with banding in effect: + /// 24 tokens 165.2 → 154.2 ms, 48 tokens 281.6 → 238.8 ms, 96 tokens 441.0 → 417.3 ms. + /// The same NR=16 with tiling forced instead is 457–640 ms, which is why the width can only be + /// widened together with the axis change and not on its own. + /// + private static int ResolveTileColsForAxis(int rows, int cores, int maxTileCols, bool banding) + { + if (TileColsOverride > 0) + { + return Math.Min(TileColsOverride, maxTileCols); + } + + return banding ? maxTileCols : ResolveTileCols(rows, cores, maxTileCols); } /// @@ -220,7 +264,13 @@ private static int ResolveTileCols(int rows, int cores, int maxTileCols) } } - return 4; + // Nothing reaches two tiles per core, so narrowing cannot buy the granularity it is meant to buy — + // it only multiplies passes over the weights. Measured at 384 rows, where the old fallback of 4 + // (96 tiles) ran 1617.9 ms against 1303.1 ms at NR=16 (24 tiles): a 1.24x loss for a tile count the + // machine could not use anyway. Falling back to the widest tile is the opposite direction from the + // rule above and deliberately so — the rule protects granularity while granularity is still + // purchasable, and this handles the case where it is not. + return maxTileCols; } /// @@ -413,7 +463,14 @@ private static unsafe void DispatchTiledQ4K( var repacked = w.EnsureRepacked(); var cores = Environment.ProcessorCount; - var nr = ResolveTileCols(rows, cores, Q4KGemvKernel.MaxTileCols); + + // Decide the axis first from the width the old rule would pick, then let the axis choose the + // final width: banding frees the tile width from the parallelism constraint that capped it. + var banding = !DisableAutoAxisSelection + && ShouldBandOutputRows( + (rows + Q4KGemvKernel.MaxTileCols - 1) / Q4KGemvKernel.MaxTileCols, cores / 2); + + var nr = ResolveTileColsForAxis(rows, cores, Q4KGemvKernel.MaxTileCols, banding); var tiles = (rows + nr - 1) / nr; // One decode of the F16 scales for the whole projection, reused by every column tile. Skipped when @@ -460,7 +517,7 @@ private static unsafe void DispatchTiledQ4K( Avx512 = UseAvx512PrefillQ4K && !DisableRepackedKernelsForParity, }; - if (!UseOutputBlocking && !ShouldBandOutputRows(tiles, Environment.ProcessorCount / 2)) + if (!UseOutputBlocking && !banding) { OverfitParallel.For(0, tiles, &TiledChunk, &ctx); return; @@ -509,7 +566,12 @@ private static unsafe void DispatchTiledQ6K( var repacked = w.EnsureRepacked(); var cores = Environment.ProcessorCount; - var nr = ResolveTileCols(rows, cores, Q6KGemvKernel.MaxTileCols); + + var banding = !DisableAutoAxisSelection + && ShouldBandOutputRows( + (rows + Q6KGemvKernel.MaxTileCols - 1) / Q6KGemvKernel.MaxTileCols, cores / 2); + + var nr = ResolveTileColsForAxis(rows, cores, Q6KGemvKernel.MaxTileCols, banding); var tiles = (rows + nr - 1) / nr; // Same hoist as the Q4_K path: widen the F16 row scales once per projection rather than once per @@ -552,7 +614,7 @@ private static unsafe void DispatchTiledQ6K( // Same axis choice as the Q4_K path. Without it `ffn_down` — 36% of a 16-token prefill — // keeps fanning out over two column tiles while fourteen cores idle. - if (!UseOutputBlocking && !ShouldBandOutputRows(tiles, Environment.ProcessorCount / 2)) + if (!UseOutputBlocking && !banding) { OverfitParallel.For(0, tiles, &TiledQ6KChunk, &ctx); return; diff --git a/Sources/Main/LanguageModels/Runtime/CachedLlamaSession.cs b/Sources/Main/LanguageModels/Runtime/CachedLlamaSession.cs index b1f5b5aa..9bc83d39 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedLlamaSession.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedLlamaSession.cs @@ -53,6 +53,18 @@ public sealed class CachedLlamaSession : ISlmSession private readonly int[] _dryRev = new int[DryHistoryCap]; private readonly int[] _dryZ = new int[DryHistoryCap]; + // Prompt cache: the token ids currently represented in the KV cache, indexed BY CACHE POSITION. + // The live region is always [0, _cache.CurrentLength) — which is what makes truncation free + // bookkeeping-wise: dropping KV state past N automatically drops these too, and the stale tail is + // overwritten when the cache refills. Sized to the context length once, so recording a token is a + // single array store and the zero-allocation decode invariant is preserved. + private readonly int[] _cacheTokens; + + // Cleared whenever cache positions stop corresponding to recorded ids — sliding-window eviction + // (every id shifts down) and prefix restore (ids belong to whoever took the snapshot). Reuse then + // falls back to a full prefill rather than attending over K/V that does not match the prompt. + private bool _cacheTokensValid = true; + private bool _disposed; private bool _slidingWindow; private int _evictBlock; @@ -79,6 +91,7 @@ internal CachedLlamaSession( _logits = new float[config.VocabSize]; _indexScratch = new int[config.VocabSize]; _scoreScratch = new float[config.VocabSize]; + _cacheTokens = new int[cache.MaxLength]; _random = new Random(); } @@ -137,6 +150,10 @@ private void MakeRoomIfSliding() var count = Math.Min(_evictBlock, _cache.CurrentLength - 1); if (count > 0) { + // Eviction shifts every surviving token down by `count`, so the recorded ids no longer sit + // at their own positions. Rebuilding the map would be cheap, but a slid session's prompt no + // longer starts at position 0 either — prefix reuse is meaningless once the head is gone. + _cacheTokensValid = false; _cache.Evict(count); } } @@ -155,6 +172,7 @@ public void Reset() ThrowIfDisposed(); _cache.Reset(); _generatedTokens.Clear(); + _cacheTokensValid = true; } /// @@ -255,9 +273,75 @@ public void RestorePrefix(KvCacheSnapshot prefix) { ThrowIfDisposed(); ArgumentNullException.ThrowIfNull(prefix); + + // The restored K/V belongs to whoever took the snapshot; this session never saw those token ids, + // so it cannot claim any prefix matches them. + _cacheTokensValid = false; _cache.RestoreFrom(prefix); } + /// + /// Prefills reusing the longest prefix already present in the KV + /// cache, and returns how many tokens that reuse saved. The remainder is prefilled normally, so + /// the resulting state is the same one would leave — + /// this trades no accuracy for the saving, because K/V for a given position depends only on the + /// tokens at and before it, which are by construction identical across the matched prefix. + /// + /// What it is for. In a chat server every turn re-sends the whole conversation, so turn + /// N re-encodes everything turns 1..N-1 already encoded. Measured against a competing pure-.NET + /// engine through the same load driver, its prompt cache answered a repeated prompt in 47 ms where + /// its own cold prefill of the same prompt took 865 ms — an 18x difference that has nothing to do + /// with kernel quality and everything to do with not doing the work twice. + /// + /// One token is always re-forwarded. Even on an exact match the last token is dropped + /// and re-run, because _logits must predict the token that follows the prompt, and those + /// logits are a by-product of the forward pass rather than cache state. So a fully cached prompt + /// still costs one decode step, not zero. + /// + /// Falls back to a full reset+prefill when the recorded ids cannot be trusted (after + /// sliding-window eviction or ) or when nothing matches. The DRY history + /// is deliberately not rewound — it is a bounded anti-repetition heuristic over what this + /// session emitted, not part of the cache contract. + /// + public int PrefillReusingCache(ReadOnlySpan promptTokens) + { + ThrowIfDisposed(); + + var reusable = ReusablePrefixLength(promptTokens); + if (reusable <= 0) + { + Reset(promptTokens); + return 0; + } + + _cache.TruncateTo(reusable); + Prefill(promptTokens[reusable..]); + return reusable; + } + + /// + /// How many leading tokens of are already in the cache at the very + /// positions they would occupy. Always leaves at least one token for + /// to forward. + /// + private int ReusablePrefixLength(ReadOnlySpan promptTokens) + { + if (!_cacheTokensValid || _slidingWindow || _cache.BasePosition != 0) + { + return 0; + } + + var limit = Math.Min(_cache.CurrentLength, promptTokens.Length); + var match = 0; + while (match < limit && _cacheTokens[match] == promptTokens[match]) + { + match++; + } + + // Never reuse the whole prompt: the final token must go through the stack to produce logits. + return Math.Min(match, promptTokens.Length - 1); + } + /// /// Generates the next token using the current cache state. /// The generated token is automatically fed back as context. @@ -533,10 +617,16 @@ private int GenerateSpeculativeCore( var hidden = hiddenArr.Span; _embedWeights.DequantizeRow(t0, hidden.Slice(0, dModel)); ApplyEmbeddingScale(hidden.Slice(0, dModel)); + _cacheTokens[basePosition] = t0; for (var j = 0; j < dn; j++) { _embedWeights.DequantizeRow(draft[j], hidden.Slice((1 + j) * dModel, dModel)); ApplyEmbeddingScale(hidden.Slice((1 + j) * dModel, dModel)); + + // Record the drafts too: this batch bypasses EmbedAndAdvance, and the truncation below keeps + // exactly the accepted prefix — so recording all of them and letting TruncateTo cut the + // rejected tail leaves the map correct without a second pass. + _cacheTokens[basePosition + 1 + j] = draft[j]; } _cache.Advance(batch); @@ -892,6 +982,7 @@ private void PrefillBatchedQuant(ReadOnlySpan promptTokens) { _embedWeights.DequantizeRow(promptTokens[i], hidden.Span.Slice(i * dModel, dModel)); ApplyEmbeddingScale(hidden.Span.Slice(i * dModel, dModel)); + _cacheTokens[basePosition + i] = promptTokens[i]; _cache.Advance(); } @@ -928,6 +1019,7 @@ private int EmbedAndAdvance(int tokenId) // No additive positional embedding — RoPE handles positions inside attention. var position = _cache.CurrentLength; + _cacheTokens[position] = tokenId; _cache.Advance(); return position; } diff --git a/Sources/Server/OverfitOpenAiServer.cs b/Sources/Server/OverfitOpenAiServer.cs index e8fe549f..ed1c33f8 100644 --- a/Sources/Server/OverfitOpenAiServer.cs +++ b/Sources/Server/OverfitOpenAiServer.cs @@ -14,6 +14,7 @@ using DevOnBike.Overfit.LanguageModels.Embeddings; using DevOnBike.Overfit.Server.OpenAi; using DevOnBike.Overfit.Serving; +using DevOnBike.Overfit.Diagnostics; namespace DevOnBike.Overfit.Server { @@ -380,6 +381,10 @@ private static byte[] ToPcm16Bytes(float[] samples) return bytes; } + /// Opt-in per-request phase trace (OVERFIT_SERVER_TRACE=1) for TTFT attribution. + private static readonly bool ServerTrace = + Environment.GetEnvironmentVariable("OVERFIT_SERVER_TRACE") == "1"; + private static void HandleChatCompletions(HttpListenerContext ctx, OverfitClient client, string modelName, string systemMessage) { ChatCompletionRequest? req; @@ -428,7 +433,21 @@ private static void HandleChatCompletions(HttpListenerContext ctx, OverfitClient // generation is zero-allocation (0 GC, 0 B over 150 prefill+decode cycles), so there are no gen-2 // pauses to suppress, while the mode would only trade RAM for nothing. Left off by design; // GcLatencyScope stays an opt-in primitive for genuinely allocation-heavy host workloads.) + // Opt-in phase timing (OVERFIT_SERVER_TRACE=1). TTFT measured through this server ran ~405 ms + // while the engine's own prefill for the same prompt measured ~150 ms — so most of the latency + // a client feels is NOT the prefill kernel. Rather than guess which of replay, templating or + // the first decode holds it, each phase is timed. + var trace = ServerTrace; + var phase = trace ? ValueStopwatch.StartNew() : default; + OpenAiChatMapping.ReplayHistory(client.Chat, req.Messages); + + if (trace) + { + Console.WriteLine($"[trace] replay {phase.GetElapsedTime().TotalMilliseconds:F1} ms " + + $"({req.Messages.Count} message(s))"); + } + var userContent = last.Content ?? string.Empty; if (!req.Stream) @@ -469,11 +488,31 @@ private static void HandleChatCompletions(HttpListenerContext ctx, OverfitClient resp.SendChunked = true; WriteChunk(resp, id, ts, modelName, new OpenAiMessage { Role = "assistant" }, finishReason: null); + + var sendStarted = trace ? ValueStopwatch.StartNew() : default; + var firstDelta = true; + client.Chat.Send(userContent, in options, - onText: delta => WriteChunk(resp, id, ts, modelName, new OpenAiMessage { Content = delta }, finishReason: null), + onText: delta => + { + if (trace && firstDelta) + { + firstDelta = false; + Console.WriteLine($"[trace] first token {sendStarted.GetElapsedTime().TotalMilliseconds:F1} ms"); + } + + WriteChunk(resp, id, ts, modelName, new OpenAiMessage { Content = delta }, finishReason: null); + }, constraint: constraint); var streamStats = client.Chat.LastStats; + + if (trace) + { + Console.WriteLine($"[trace] prompt {streamStats.PromptTokens} tok, " + + $"{client.Chat.CachedPromptTokens} reused from the KV cache"); + } + var streamFinish = streamStats.GeneratedTokens >= maxTokens ? "length" : "stop"; WriteChunk(resp, id, ts, modelName, new OpenAiMessage(), finishReason: streamFinish); WriteSseRaw(resp, "[DONE]"); diff --git a/Tests/LanguageModels/Diagnostics/PrefillAxisSweepTests.cs b/Tests/LanguageModels/Diagnostics/PrefillAxisSweepTests.cs new file mode 100644 index 00000000..f8820a4a --- /dev/null +++ b/Tests/LanguageModels/Diagnostics/PrefillAxisSweepTests.cs @@ -0,0 +1,143 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.Diagnostics; +using DevOnBike.Overfit.LanguageModels.Contracts; +using DevOnBike.Overfit.LanguageModels.Runtime; +using DevOnBike.Overfit.LanguageModels.Tokenizers; +using DevOnBike.Overfit.Tests.TestSupport; +using Xunit.Abstractions; + +namespace DevOnBike.Overfit.Tests.LanguageModels.Diagnostics +{ + /// + /// Both parallelisation axes and both tile widths, at the prompt lengths a chat server actually sees, + /// against the floor set by a single decode step. + /// + /// The gap being chased. Measured through an OpenAI-protocol load test, a competing pure-.NET + /// engine answers a chat-sized prompt with ~44 ms to first token; we take ~300 ms. A server phase trace + /// showed only ~26 ms of that is HTTP and JSON — the rest is prefill of the chat-templated prompt (roughly + /// 50 tokens once role markers and the system turn are added). + /// + /// The floor. One decode step reads every weight once and costs ~56 ms. A 50-token prefill + /// should cost about one such pass plus arithmetic; theirs does, and ours costs about five. Whatever the + /// cause, it is structural rather than kernel quality — the same kernels decode at a competitive rate. + /// + /// This measures the three candidate configurations side by side so the choice is made on numbers: + /// the automatic axis rule, forced banding, and forced column tiling, each at the tile width in use and at + /// the widest the kernel supports. + /// + public sealed class PrefillAxisSweepTests + { + private static readonly int[] Lengths = [24, 48, 96]; + + private const int Repeats = 3; + + private readonly ITestOutputHelper _out; + + public PrefillAxisSweepTests(ITestOutputHelper output) => _out = output; + + [LongFact] + public void Prefill_AxisAndTileWidth_AtChatPromptLengths() + { + var path = TestModelPaths.Qwen3B.Q4KmGgufPath; + if (!File.Exists(path)) + { + _out.WriteLine($"missing {path}"); + return; + } + + using var engine = CachedLlamaInferenceEngine.LoadGguf(path); + var tok = GgufTokenizer.Load(path); + var sampling = SamplingOptions.Greedy; + + var paragraph = string.Join(" ", + Enumerable.Repeat( + "The history of computing began with mechanical calculators and evolved through vacuum tubes, " + + "transistors, integrated circuits and finally the microprocessor era.", 40)); + var allIds = tok.Encode(paragraph); + + using (var warm = engine.CreateSession(1024)) + { + warm.Reset(allIds.AsSpan(0, 64).ToArray()); + warm.GenerateNextToken(in sampling); + } + + // The floor: one decode step is one full pass over the weights. + double decodeMs; + { + using var session = engine.CreateSession(1024); + session.Reset(allIds.AsSpan(0, 16).ToArray()); + session.GenerateNextToken(in sampling); + + var started = ValueStopwatch.StartNew(); + for (var i = 0; i < 5; i++) + { + session.GenerateNextToken(in sampling); + } + + decodeMs = started.GetElapsedTime().TotalMilliseconds / 5; + } + + _out.WriteLine($"one decode step (= one weight pass): {decodeMs:F1} ms"); + _out.WriteLine(string.Empty); + _out.WriteLine($" {"tokens",7}{"NR",4}{"auto",11}{"banded",11}{"tiled",11}{"best/decode",13}"); + + foreach (var length in Lengths) + { + var ids = allIds.AsSpan(0, length).ToArray(); + + foreach (var nr in new[] { 0, 16 }) + { + BatchedQuantProjection.TileColsOverride = nr; + + var auto = Best(engine, ids, forceBanding: null); + var banded = Best(engine, ids, forceBanding: true); + var tiled = Best(engine, ids, forceBanding: false); + var best = Math.Min(auto, Math.Min(banded, tiled)); + + _out.WriteLine( + $" {length,7}{(nr == 0 ? 8 : nr),4}{auto,8:F1} ms{banded,8:F1} ms{tiled,8:F1} ms" + + $"{best / decodeMs,12:F1}x"); + } + + BatchedQuantProjection.TileColsOverride = 0; + } + + Assert.True(decodeMs > 0); + } + + /// + /// null leaves the automatic rule in charge; true or false pins the + /// axis. Forcing "tiled" needs the rule disabled, which is what the negative TileColsOverride + /// path cannot express — hence the explicit switch. + /// + private static double Best(CachedLlamaInferenceEngine engine, int[] ids, bool? forceBanding) + { + BatchedQuantProjection.UseOutputBlocking = forceBanding == true; + BatchedQuantProjection.DisableAutoAxisSelection = forceBanding == false; + + try + { + var best = double.MaxValue; + + for (var r = 0; r < Repeats; r++) + { + using var session = engine.CreateSession(1024); + var started = ValueStopwatch.StartNew(); + session.Reset(ids); + best = Math.Min(best, started.GetElapsedTime().TotalMilliseconds); + } + + return best; + } + finally + { + BatchedQuantProjection.UseOutputBlocking = false; + BatchedQuantProjection.DisableAutoAxisSelection = false; + } + } + } +} diff --git a/Tests/LanguageModels/Runtime/PromptCacheReuseTests.cs b/Tests/LanguageModels/Runtime/PromptCacheReuseTests.cs new file mode 100644 index 00000000..186647f7 --- /dev/null +++ b/Tests/LanguageModels/Runtime/PromptCacheReuseTests.cs @@ -0,0 +1,251 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.LanguageModels.Contracts; +using DevOnBike.Overfit.LanguageModels.Runtime; +using DevOnBike.Overfit.LanguageModels.Tokenizers; +using DevOnBike.Overfit.Tests.TestSupport; +using Xunit.Abstractions; + +namespace DevOnBike.Overfit.Tests.LanguageModels.Runtime +{ + /// + /// Pins the contract of : reuse is an optimisation, + /// never a behaviour change. + /// + /// The property being tested is the one that makes the whole feature safe — K/V at a position + /// depends only on the tokens at and before that position, so a prompt sharing a prefix with what the + /// cache already holds must produce bit-identical logits whether that prefix was re-encoded or + /// reused. If this ever fails, the cache is silently answering from the wrong context, which is far + /// worse than being slow. + /// + /// The one thing these tests must hold constant. Bit-identity holds per kernel path. + /// Prefill has two of them — the batched multi-row GEMM (prompts ≥ 16 tokens) and the single-token loop — + /// and they are not bit-identical to each other; that is a pre-existing, documented and accepted property + /// (see BatchedQuantProjection.DisableRepackedKernelsForParity, which exists precisely because a + /// test comparing them has to pin the layout or it stops testing what it claims to). Reuse necessarily + /// splits one prefill into two, so a test that lets the split cross that boundary measures the kernel + /// difference, not the cache. Measured on Qwen-0.5B, same reuse mechanism throughout: + /// + /// + /// both sides single-tokenmaxAbsLogitDiff 0 + /// both sides batched (26 + 23 vs 49 rows)maxAbsLogitDiff 0 + /// split crosses the boundary (14 batched vs 14+7 mixed)0.756, argmax flips + /// + /// + /// So row count does not affect the batched kernel's result, and reuse itself is exact. What this + /// means in production is stated honestly: a cached turn can sample a different token than the same turn + /// would uncached, because the cached prefix contains reply tokens produced by the decode loop where a + /// cold prefill would have run them through the batched kernel. That is the same accepted trade the + /// repacked kernels already carry — validated by end-to-end coherence, not byte-parity. + /// + public sealed class PromptCacheReuseTests + { + private readonly ITestOutputHelper _out; + + public PromptCacheReuseTests(ITestOutputHelper output) => _out = output; + + /// + /// The realistic server shape — a long cached turn extended by a long new turn, so every prefill on + /// both sides goes through the batched kernel and the comparison isolates reuse itself. + /// + [SmallModelFact] + public void ReusedPrefix_ProducesIdenticalLogits_WhenBothSidesUseTheBatchedKernel() + { + var path = TestModelPaths.Qwen05B.RequireQ4KmGgufPath(); + + using var engine = CachedLlamaInferenceEngine.LoadGguf(path); + var tok = GgufTokenizer.Load(path); + + // Both the cached head and the appended tail sit comfortably above the 16-token batched + // threshold, so only the row count differs between the arms. + var headText = "The history of computing began with mechanical calculators and evolved " + + "through vacuum tubes, transistors and integrated circuits into the modern era."; + var turn1 = tok.Encode(headText); + var turn2 = tok.Encode(headText + + " Today, running a language model on a plain desktop processor without any " + + "dedicated accelerator hardware is entirely practical and quite common."); + + // Reference: a session that has never seen turn 1 — the full-prefill answer. + float[] reference; + using (var fresh = engine.CreateSession(512)) + { + fresh.Reset(turn2); + reference = new float[fresh.VocabularySize]; + fresh.GetLastLogits(reference); + } + + // Under test: prefill turn 1, then turn 2 reusing the shared prefix. + using var session = engine.CreateSession(512); + session.Reset(turn1); + var reused = session.PrefillReusingCache(turn2); + + var actual = new float[session.VocabularySize]; + session.GetLastLogits(actual); + + _out.WriteLine($"turn1 {turn1.Length} tok, turn2 {turn2.Length} tok, reused {reused} tok, " + + $"tail {turn2.Length - reused} tok"); + + Assert.True(reused > 0, "the shared prefix should have been reused"); + Assert.True(reused < turn2.Length, "the last token must always be re-forwarded for logits"); + Assert.True(turn2.Length - reused >= 16, "tail must stay on the batched path for this comparison"); + + var maxDiff = 0f; + for (var i = 0; i < reference.Length; i++) + { + maxDiff = Math.Max(maxDiff, Math.Abs(reference[i] - actual[i])); + } + + _out.WriteLine($"maxAbsLogitDiff = {maxDiff:G6}"); + Assert.Equal(0f, maxDiff); + } + + /// + /// The same property with both arms pinned to the single-token loop, which removes the kernel + /// variable entirely: whatever the split, reuse must reproduce a full prefill exactly. + /// + [SmallModelFact] + public void ReusedPrefix_ProducesIdenticalLogits_WhenTheKernelPathIsPinned() + { + var path = TestModelPaths.Qwen05B.RequireQ4KmGgufPath(); + + using var engine = CachedLlamaInferenceEngine.LoadGguf(path); + var tok = GgufTokenizer.Load(path); + + var turn1 = tok.Encode("The capital of France is Paris, a city known for its museums."); + var turn2 = tok.Encode( + "The capital of France is Paris, a city known for its museums. What is the capital of Italy?"); + + float[] reference; + using (var fresh = (CachedLlamaSession)engine.CreateSession(512)) + { + fresh.DisableBatchedPrefillForParity = true; + fresh.Reset(turn2); + reference = new float[fresh.VocabularySize]; + fresh.GetLastLogits(reference); + } + + using var session = (CachedLlamaSession)engine.CreateSession(512); + session.DisableBatchedPrefillForParity = true; + session.Reset(turn1); + var reused = session.PrefillReusingCache(turn2); + + var actual = new float[session.VocabularySize]; + session.GetLastLogits(actual); + + _out.WriteLine($"turn1 {turn1.Length} tok, turn2 {turn2.Length} tok, reused {reused} tok"); + + Assert.True(reused > 0, "the shared prefix should have been reused"); + + var maxDiff = 0f; + for (var i = 0; i < reference.Length; i++) + { + maxDiff = Math.Max(maxDiff, Math.Abs(reference[i] - actual[i])); + } + + _out.WriteLine($"maxAbsLogitDiff = {maxDiff:G6}"); + Assert.Equal(0f, maxDiff); + } + + /// + /// A prompt that diverges from the cached one must not silently attend over the stale tail. The + /// divergence here is deliberately placed mid-prompt, so a naive "reuse everything up to the shorter + /// length" implementation would pass a length check and still be wrong. + /// + [SmallModelFact] + public void DivergentPrompt_FallsBackToTheMatchingPrefixOnly() + { + var path = TestModelPaths.Qwen05B.RequireQ4KmGgufPath(); + + using var engine = CachedLlamaInferenceEngine.LoadGguf(path); + var tok = GgufTokenizer.Load(path); + + var cached = tok.Encode("The quick brown fox jumps over the lazy dog near the river bank."); + var diverged = tok.Encode("The quick brown fox sleeps under the old oak tree in the meadow."); + + float[] reference; + using (var fresh = engine.CreateSession(512)) + { + fresh.Reset(diverged); + reference = new float[fresh.VocabularySize]; + fresh.GetLastLogits(reference); + } + + using var session = engine.CreateSession(512); + session.Reset(cached); + var reused = session.PrefillReusingCache(diverged); + + var actual = new float[session.VocabularySize]; + session.GetLastLogits(actual); + + _out.WriteLine($"reused {reused} of {diverged.Length} tokens (divergence point)"); + + var maxDiff = 0f; + for (var i = 0; i < reference.Length; i++) + { + maxDiff = Math.Max(maxDiff, Math.Abs(reference[i] - actual[i])); + } + + _out.WriteLine($"maxAbsLogitDiff = {maxDiff:G6}"); + Assert.Equal(0f, maxDiff); + } + + /// + /// Re-sending the identical prompt is the load-test shape and the degenerate case of the matcher: + /// everything matches, so the implementation must still hold one token back to refresh the logits. + /// + [SmallModelFact] + public void IdenticalPrompt_ReusesAllButTheLastToken() + { + var path = TestModelPaths.Qwen05B.RequireQ4KmGgufPath(); + + using var engine = CachedLlamaInferenceEngine.LoadGguf(path); + var tok = GgufTokenizer.Load(path); + var prompt = tok.Encode("Explain in one sentence why local inference matters."); + + using var session = engine.CreateSession(512); + session.Reset(prompt); + + var expected = new float[session.VocabularySize]; + session.GetLastLogits(expected); + + var reused = session.PrefillReusingCache(prompt); + + var actual = new float[session.VocabularySize]; + session.GetLastLogits(actual); + + Assert.Equal(prompt.Length - 1, reused); + + var maxDiff = 0f; + for (var i = 0; i < expected.Length; i++) + { + maxDiff = Math.Max(maxDiff, Math.Abs(expected[i] - actual[i])); + } + + _out.WriteLine($"reused {reused}/{prompt.Length}, maxAbsLogitDiff = {maxDiff:G6}"); + Assert.Equal(0f, maxDiff); + } + + /// + /// Sliding-window sessions evict from the head, so recorded ids stop matching cache positions. The + /// matcher must refuse to reuse rather than attend over shifted K/V. + /// + [SmallModelFact] + public void SlidingWindowSession_DoesNotReuse() + { + var path = TestModelPaths.Qwen05B.RequireQ4KmGgufPath(); + + using var engine = CachedLlamaInferenceEngine.LoadGguf(path); + var tok = GgufTokenizer.Load(path); + var prompt = tok.Encode("Sliding windows drop the oldest tokens as the context fills up."); + + using var session = engine.CreateSession(512); + session.EnableSlidingWindow(); + session.Reset(prompt); + + Assert.Equal(0, session.PrefillReusingCache(prompt)); + } + } +} From f3c4925c985fb9c43acb040ded536bfabd6b67e3 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Thu, 23 Jul 2026 20:32:38 +0200 Subject: [PATCH 33/37] dotllm --- Sources/Cli/Commands.cs | 15 +- Sources/Cli/ServingBenchmark.cs | 1 - .../Adaptive/AdaptiveAnomalyMonitor.cs | 2 - .../Anomalies/Live/LiveMonitoringPipeline.cs | 1 - .../Abstractions/IRawMetricSource.cs | 1 - .../Anomalies/Training/OfflineTrainingJob.cs | 1 - Sources/Main/Audio/WavReader.cs | 1 - .../ComputationGraph.FrozenQuantizedLinear.cs | 1 - .../ScaledDotProductAttentionLayer.cs | 1 - .../Main/LanguageModels/Agents/CriticLoop.cs | 2 - .../LanguageModels/Agents/CriticResult.cs | 2 - .../Main/LanguageModels/Agents/ReActAgent.cs | 1 - .../Main/LanguageModels/Agents/ReActResult.cs | 2 - .../Main/LanguageModels/Chat/ChatSession.cs | 64 +++++++- .../LanguageModels/Contracts/ISlmSession.cs | 18 +++ .../LanguageModels/Embeddings/BertEncoder.cs | 2 - .../Embeddings/BertSafetensorsLoader.cs | 1 - .../LanguageModels/LoRA/QLoRAFineTuner.cs | 1 - .../LanguageModels/Loading/GgufLlamaLoader.cs | 1 - .../Loading/RepackedWeightsFile.cs | 1 - .../Memory/ChatHistoryCompactor.cs | 1 - .../LanguageModels/Memory/CompactionPlan.cs | 1 - .../Memory/SummarizingChatSession.cs | 1 - .../Runtime/CachedLlamaSession.cs | 145 ++++++++++++++++-- .../Runtime/MoeFeedForwardBlock.cs | 1 - .../Runtime/Qwen2MoeFeedForwardBlock.cs | 1 - .../Tokenizers/GgufTokenizer.cs | 1 - .../Tokenizers/HuggingFaceBpeTokenizer.cs | 2 - .../Tokenizers/WordPieceTokenizer.cs | 2 - .../Whisper/WhisperGgmlLoader.cs | 1 - Sources/Main/Optimizers/Adam.cs | 1 - Sources/Main/Runtime/OverfitEnvironment.cs | 27 ++++ Sources/Main/Runtime/OverfitParallel.cs | 2 +- Sources/Main/Training/DataParallelTrainer.cs | 1 - Sources/Server/OpenAi/OpenAiChatMapping.cs | 1 - Sources/Server/OverfitOpenAiServer.cs | 27 ++-- Sources/Server/RedactionGateway.cs | 9 +- Tests/Adapters/MeaiAdapterEndToEndTests.cs | 1 - Tests/Adapters/OverfitChatClientTests.cs | 1 - .../GptVsEwmaBaselineComparisonTests.cs | 1 - Tests/Audio/WavReaderTests.cs | 1 - .../Autograd/FrozenQuantizedLinearTests.cs | 1 - Tests/Core/Autograd/GqaAttentionTests.cs | 1 - Tests/Core/Autograd/QLoRATrainingTests.cs | 1 - Tests/Core/Autograd/RmsNormTests.cs | 1 - Tests/Core/Autograd/RopeTests.cs | 1 - Tests/Core/Autograd/SiLUTests.cs | 1 - Tests/Data/Mnist/MnistAllocBreakdownTests.cs | 1 - Tests/Data/Mnist/MnistOneCycleBenchTests.cs | 1 - Tests/DeepLearning/CheckpointParityTests.cs | 1 - .../CifarCnnBackwardOpProfilerTests.cs | 1 - .../CifarCnnForwardLayerProfilerTests.cs | 1 - .../MnistCnnBackwardOpProfilerTests.cs | 1 - .../MnistCnnForwardLayerProfilerTests.cs | 1 - .../DeepLearning/TrainableLlamaBlockTests.cs | 1 - .../DeepLearning/TrainableLlamaModelTests.cs | 2 - .../UninitializedStrategyGuardTests.cs | 1 - Tests/Examples/CtcOcrDemoTests.cs | 2 - .../Agents/ReActAgentEndToEndTests.cs | 1 - .../LanguageModels/Agents/ReActAgentTests.cs | 1 - .../Chat/HuggingFaceChatModelTests.cs | 1 - .../Chat/HuggingFaceLlamaModelTests.cs | 2 - .../LanguageModels/Chat/QwenChatModelTests.cs | 1 - .../Diagnostics/DecodeCostAblationTests.cs | 111 ++++++++++++++ .../Diagnostics/OrpheusPromptTokenTests.cs | 1 - .../TinyBlasProjectionHeadroomPhase05Tests.cs | 1 - .../Embeddings/MiniLmFullLengthTests.cs | 1 - .../Loading/BielikSafetensorsParityTests.cs | 1 - .../Loading/GgufNestedArrayDepthTests.cs | 1 - .../Loading/QLoraGgufBridgeTests.cs | 1 - .../QwenGgufQLoraAdapterRoundTripTests.cs | 1 - .../Loading/QwenGgufQLoraE2ETests.cs | 1 - .../Loading/QwenGgufTrainingRamTests.cs | 1 - .../Loading/RepackedSidecarEngineE2ETests.cs | 1 - .../Loading/RepackedWeightsFileTests.cs | 1 - .../LanguageModels/Runtime/EmbeddingsTests.cs | 1 - .../Runtime/MoeFeedForwardBlockTests.cs | 1 - .../Parity/BielikDraftSpeculativeBench.cs | 1 - .../Parity/DraftModelSpeculativeBench.cs | 1 - .../Q4KBatchedProjectionScalingBench.cs | 1 - .../Parity/Q4KWeightStationaryParityTests.cs | 1 - .../Parity/SmallModelAgenticProbeTests.cs | 1 - .../Parity/SpeculativeDecodeParityTests.cs | 1 - .../Runtime/PromptCacheReuseTests.cs | 65 +++++++- .../Runtime/Q4KDotKernelNeonParityTests.cs | 1 - .../Runtime/Q6KTiledGemmParityTests.cs | 1 - .../Runtime/Qwen2MoeFeedForwardBlockTests.cs | 1 - .../Evaluation/OverfitSkillRunnerLongTests.cs | 2 - .../Skills/Evaluation/SkillEvaluatorTests.cs | 2 - .../Optimization/SkillOptimizerTests.cs | 3 - .../Tokenization/WordPieceTokenizerTests.cs | 1 - .../Whisper/WhisperGgmlLoaderTests.cs | 1 - Tests/Optimizers/AdamCheckpointTests.cs | 2 - Tests/Redaction/AllowlistAndEntropyTests.cs | 1 - Tests/Redaction/GatewayConfigTests.cs | 1 - Tests/Redaction/RedactionGatewayAuthTests.cs | 1 - Tests/Redaction/RedactionGatewayE2ETests.cs | 1 - .../RedactionGatewayEndpointsTests.cs | 1 - .../RedactionGatewayHeaderForwardingTests.cs | 1 - .../RedactionGatewayResponseScanTests.cs | 1 - .../RedactionGatewayStreamingScanTests.cs | 1 - .../RedactionGatewayStreamingTests.cs | 1 - Tests/Redaction/ResponseScanTests.cs | 1 - .../StreamingResponseScannerTests.cs | 1 - Tests/Serving/ServingLoadReportTests.cs | 1 - .../Helpers/SafetensorsTestWriter.cs | 1 - Tests/Training/DataParallelSessionTests.cs | 2 - Tests/Trees/XgboostParityTests.cs | 1 - docs/overfit-vs-dotllm.md | 113 ++++++++++++++ 109 files changed, 561 insertions(+), 149 deletions(-) create mode 100644 Tests/LanguageModels/Diagnostics/DecodeCostAblationTests.cs create mode 100644 docs/overfit-vs-dotllm.md diff --git a/Sources/Cli/Commands.cs b/Sources/Cli/Commands.cs index cdf44321..46000db8 100644 --- a/Sources/Cli/Commands.cs +++ b/Sources/Cli/Commands.cs @@ -1333,11 +1333,12 @@ public static int Chat( try { - while (true) + // Bound stated in the header (OVERFIT023): the REPL ends when stdin closes — ReadLine + // returns null on EOF, a closed pipe or Ctrl+Z/Ctrl+D. `/exit` is the interactive shortcut + // for the same thing and stays an explicit break. + for (var line = ReadCommand(); line is not null; line = ReadCommand()) { - Console.Write("> "); - var line = Console.ReadLine(); - if (line is null || line.Equals("/exit", StringComparison.OrdinalIgnoreCase)) + if (line.Equals("/exit", StringComparison.OrdinalIgnoreCase)) { break; } @@ -1366,6 +1367,12 @@ public static int Chat( client.Dispose(); } return 0; + + static string? ReadCommand() + { + Console.Write("> "); + return Console.ReadLine(); + } } } } diff --git a/Sources/Cli/ServingBenchmark.cs b/Sources/Cli/ServingBenchmark.cs index 3ca04da2..d342f8e3 100644 --- a/Sources/Cli/ServingBenchmark.cs +++ b/Sources/Cli/ServingBenchmark.cs @@ -4,7 +4,6 @@ // For commercial licensing options, contact: devonbike@gmail.com using System.Buffers; -using System.Net.Http; using System.Net.Http.Headers; using System.Text.Json; using DevOnBike.Overfit.Diagnostics; diff --git a/Sources/Main/Anomalies/Adaptive/AdaptiveAnomalyMonitor.cs b/Sources/Main/Anomalies/Adaptive/AdaptiveAnomalyMonitor.cs index 339a0c5e..77d1456a 100644 --- a/Sources/Main/Anomalies/Adaptive/AdaptiveAnomalyMonitor.cs +++ b/Sources/Main/Anomalies/Adaptive/AdaptiveAnomalyMonitor.cs @@ -3,8 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; -using System.IO; using DevOnBike.Overfit.Anomalies.Gpt; using DevOnBike.Overfit.Anomalies.Monitoring.Contracts; using DevOnBike.Overfit.DeepLearning; diff --git a/Sources/Main/Anomalies/Live/LiveMonitoringPipeline.cs b/Sources/Main/Anomalies/Live/LiveMonitoringPipeline.cs index 5b5cf9be..98ba7f6c 100644 --- a/Sources/Main/Anomalies/Live/LiveMonitoringPipeline.cs +++ b/Sources/Main/Anomalies/Live/LiveMonitoringPipeline.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; using DevOnBike.Overfit.Anomalies.Adaptive; using DevOnBike.Overfit.Anomalies.Alerting.Abstractions; using DevOnBike.Overfit.Anomalies.Alerting.Contracts; diff --git a/Sources/Main/Anomalies/Monitoring/Abstractions/IRawMetricSource.cs b/Sources/Main/Anomalies/Monitoring/Abstractions/IRawMetricSource.cs index e0c7fd94..e3ebbdd8 100644 --- a/Sources/Main/Anomalies/Monitoring/Abstractions/IRawMetricSource.cs +++ b/Sources/Main/Anomalies/Monitoring/Abstractions/IRawMetricSource.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; using DevOnBike.Overfit.Anomalies.Monitoring.Contracts; namespace DevOnBike.Overfit.Anomalies.Monitoring.Abstractions diff --git a/Sources/Main/Anomalies/Training/OfflineTrainingJob.cs b/Sources/Main/Anomalies/Training/OfflineTrainingJob.cs index 54eb8ef9..acd08802 100644 --- a/Sources/Main/Anomalies/Training/OfflineTrainingJob.cs +++ b/Sources/Main/Anomalies/Training/OfflineTrainingJob.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Diagnostics; using DevOnBike.Overfit.Anomalies.Gpt; using DevOnBike.Overfit.Anomalies.Monitoring; using DevOnBike.Overfit.Autograd; diff --git a/Sources/Main/Audio/WavReader.cs b/Sources/Main/Audio/WavReader.cs index 3a4ce466..593f876a 100644 --- a/Sources/Main/Audio/WavReader.cs +++ b/Sources/Main/Audio/WavReader.cs @@ -5,7 +5,6 @@ // For commercial licensing options, contact: devonbike@gmail.com using System.Buffers.Binary; -using System.IO; namespace DevOnBike.Overfit.Audio { diff --git a/Sources/Main/Autograd/ComputationGraph.FrozenQuantizedLinear.cs b/Sources/Main/Autograd/ComputationGraph.FrozenQuantizedLinear.cs index 93145f3e..197ad4d3 100644 --- a/Sources/Main/Autograd/ComputationGraph.FrozenQuantizedLinear.cs +++ b/Sources/Main/Autograd/ComputationGraph.FrozenQuantizedLinear.cs @@ -8,7 +8,6 @@ using DevOnBike.Overfit.Intrinsics; using DevOnBike.Overfit.Runtime; using DevOnBike.Overfit.Tensors; -using DevOnBike.Overfit.Tensors.Core; namespace DevOnBike.Overfit.Autograd { diff --git a/Sources/Main/DeepLearning/ScaledDotProductAttentionLayer.cs b/Sources/Main/DeepLearning/ScaledDotProductAttentionLayer.cs index 34d0e3cd..49ed6d68 100644 --- a/Sources/Main/DeepLearning/ScaledDotProductAttentionLayer.cs +++ b/Sources/Main/DeepLearning/ScaledDotProductAttentionLayer.cs @@ -10,7 +10,6 @@ using DevOnBike.Overfit.Ops; using DevOnBike.Overfit.Parameters; using DevOnBike.Overfit.Tensors; -using DevOnBike.Overfit.Tensors.Core; namespace DevOnBike.Overfit.DeepLearning { diff --git a/Sources/Main/LanguageModels/Agents/CriticLoop.cs b/Sources/Main/LanguageModels/Agents/CriticLoop.cs index 5c1ef26a..7b2c1a3a 100644 --- a/Sources/Main/LanguageModels/Agents/CriticLoop.cs +++ b/Sources/Main/LanguageModels/Agents/CriticLoop.cs @@ -3,8 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; - namespace DevOnBike.Overfit.LanguageModels.Agents { /// diff --git a/Sources/Main/LanguageModels/Agents/CriticResult.cs b/Sources/Main/LanguageModels/Agents/CriticResult.cs index d8d3518c..c2e0aec5 100644 --- a/Sources/Main/LanguageModels/Agents/CriticResult.cs +++ b/Sources/Main/LanguageModels/Agents/CriticResult.cs @@ -3,8 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; - namespace DevOnBike.Overfit.LanguageModels.Agents { /// Result of CriticLoop.Run: final candidate + per-iteration trace + exit reason. diff --git a/Sources/Main/LanguageModels/Agents/ReActAgent.cs b/Sources/Main/LanguageModels/Agents/ReActAgent.cs index 599b92d3..518c79bf 100644 --- a/Sources/Main/LanguageModels/Agents/ReActAgent.cs +++ b/Sources/Main/LanguageModels/Agents/ReActAgent.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; using System.Text; using System.Text.Json; using DevOnBike.Overfit.LanguageModels.Chat; diff --git a/Sources/Main/LanguageModels/Agents/ReActResult.cs b/Sources/Main/LanguageModels/Agents/ReActResult.cs index 1e3fa4c6..bf25eaf0 100644 --- a/Sources/Main/LanguageModels/Agents/ReActResult.cs +++ b/Sources/Main/LanguageModels/Agents/ReActResult.cs @@ -3,8 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; - namespace DevOnBike.Overfit.LanguageModels.Agents { /// Result of running a ReActAgent loop: final answer + per-step trace + exit reason. diff --git a/Sources/Main/LanguageModels/Chat/ChatSession.cs b/Sources/Main/LanguageModels/Chat/ChatSession.cs index f075eab3..2ca667ed 100644 --- a/Sources/Main/LanguageModels/Chat/ChatSession.cs +++ b/Sources/Main/LanguageModels/Chat/ChatSession.cs @@ -8,6 +8,7 @@ using DevOnBike.Overfit.Diagnostics; using DevOnBike.Overfit.LanguageModels.Contracts; using DevOnBike.Overfit.LanguageModels.Runtime; +using DevOnBike.Overfit.Runtime; namespace DevOnBike.Overfit.LanguageModels.Chat { @@ -35,6 +36,18 @@ public sealed class ChatSession private readonly bool _slidingWindow; private readonly List _history = []; + /// Test hook: restore the pre-early-emit ordering (emit after the forward pass, not before). + /// Defaults from so both orderings can be served by + /// two processes and compared inside a single interleaved run. + internal static bool DisableEarlyEmit = + Environment.GetEnvironmentVariable(OverfitEnvironment.DisableEarlyEmit) == "1"; + + /// Test hook: force the exact single-token decode loop instead of the speculative path. + /// Defaults from so both can be served side by + /// side and measured in one interleaved run. + internal static bool DisableSpeculative = + Environment.GetEnvironmentVariable(OverfitEnvironment.DisableSpeculative) == "1"; + /// Underlying SLM session that runs prefill/decode and owns the KV cache. /// Tokenizer used to encode prompts and decode generated tokens. /// Chat template that formats messages into the model's prompt format. @@ -264,6 +277,23 @@ bool EmitToken(int token) return stops.Stopped || constraint is { IsComplete: true }; } + // Emit the token the instant it is sampled, before the forward pass that prepares the NEXT + // logits — otherwise every token, the first one included, arrives one whole weight-pass late. + // Returning the stop decision straight back lets the session skip that pass when the answer is + // over. Allocated once per generation, not per token. + var stopped = false; + var onSampled = new Func(token => + { + stopped = EmitToken(token); + return stopped; + }); + + // Test hook: emit after the step instead of during it, i.e. the pre-early-emit ordering. + if (DisableEarlyEmit) + { + onSampled = null!; + } + // Speculative fast path (prompt-lookup, adaptively gated): commits ≥1 token per batched // verify, sampling-correct, and ~free when drafts don't fire — but it can't mask the draft // against a per-token constraint, so it only runs unconstrained on a speculation-capable @@ -271,7 +301,7 @@ bool EmitToken(int token) // Hoisted out of the condition: the speculative session is needed inside the branch, and a // second (negated) test could not re-introduce a pattern variable in the same scope. var spec = _session as CachedLlamaSession; - var useSpeculative = constraint is null && spec is not null && spec.CanSpeculate; + var useSpeculative = constraint is null && spec is not null && spec.CanSpeculate && !DisableSpeculative; if (useSpeculative) { @@ -286,12 +316,32 @@ bool EmitToken(int token) while (generated.Count < maxNew && (_slidingWindow || _session.CurrentPosition < _session.MaxContextLength)) { - var n = spec!.GenerateSpeculative(CollectionsMarshal.AsSpan(history), committed, in sampling, maxDraft); + var n = spec!.GenerateSpeculative( + CollectionsMarshal.AsSpan(history), committed, in sampling, maxDraft, onSampled); var stop = false; for (var c = 0; c < n; c++) { var token = committed[c]; history.Add(token); + + // committed[0] is the token the hook already emitted before the verify forward ran; + // re-emitting it would duplicate it in the stream. + if (c == 0) + { + if (DisableEarlyEmit) + { + stopped = EmitToken(token); + } + + if (stopped || generated.Count >= maxNew) + { + stop = true; + break; + } + + continue; + } + if (EmitToken(token) || generated.Count >= maxNew) { stop = true; @@ -312,7 +362,15 @@ bool EmitToken(int token) for (var i = 0; i < maxNew && (_slidingWindow || _session.CurrentPosition < _session.MaxContextLength); i++) { - if (EmitToken(_session.GenerateNextToken(in sampling, constraint))) + // The hook emits; `stopped` carries its verdict back out. Sessions without early-emit + // support still invoke it exactly once per token, just after their forward. + var produced = _session.GenerateNextToken(in sampling, constraint, onSampled); + if (DisableEarlyEmit) + { + stopped = EmitToken(produced); + } + + if (stopped) { break; } diff --git a/Sources/Main/LanguageModels/Contracts/ISlmSession.cs b/Sources/Main/LanguageModels/Contracts/ISlmSession.cs index 0226847c..15ef7afa 100644 --- a/Sources/Main/LanguageModels/Contracts/ISlmSession.cs +++ b/Sources/Main/LanguageModels/Contracts/ISlmSession.cs @@ -81,6 +81,24 @@ int GenerateNextToken(in SamplingOptions sampling, ITokenConstraint? constraint) $"{GetType().Name} does not support constrained generation."); #pragma warning restore RS0030 + /// + /// Generates the next token and hands it to as early as the + /// implementation can — ideally before the forward pass that prepares the following logits, which is + /// what lets a streaming caller put the token on the wire a whole weight-pass sooner. Returning + /// true from the hook says the caller is finished with generation, letting the implementation + /// skip that pass entirely. + /// + /// The default implementation invokes the hook after the step instead, so a caller can + /// rely on it firing exactly once per token whatever the session: it is a latency optimisation where + /// supported, never a difference in what gets emitted. + /// + int GenerateNextToken(in SamplingOptions sampling, ITokenConstraint? constraint, Func? onSampled) + { + var token = GenerateNextToken(in sampling, constraint); + onSampled?.Invoke(token); + return token; + } + int Generate( ReadOnlySpan promptTokens, Span outputTokens, diff --git a/Sources/Main/LanguageModels/Embeddings/BertEncoder.cs b/Sources/Main/LanguageModels/Embeddings/BertEncoder.cs index 152f127b..46b8ab99 100644 --- a/Sources/Main/LanguageModels/Embeddings/BertEncoder.cs +++ b/Sources/Main/LanguageModels/Embeddings/BertEncoder.cs @@ -7,8 +7,6 @@ using DevOnBike.Overfit.DeepLearning; using DevOnBike.Overfit.LanguageModels.Contracts; using DevOnBike.Overfit.Ops; -using DevOnBike.Overfit.Tensors; -using DevOnBike.Overfit.Tensors.Core; namespace DevOnBike.Overfit.LanguageModels.Embeddings { diff --git a/Sources/Main/LanguageModels/Embeddings/BertSafetensorsLoader.cs b/Sources/Main/LanguageModels/Embeddings/BertSafetensorsLoader.cs index ccede0c5..694b4f51 100644 --- a/Sources/Main/LanguageModels/Embeddings/BertSafetensorsLoader.cs +++ b/Sources/Main/LanguageModels/Embeddings/BertSafetensorsLoader.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; using DevOnBike.Overfit.DeepLearning; using DevOnBike.Overfit.LanguageModels.Loading; using DevOnBike.Overfit.Parameters; diff --git a/Sources/Main/LanguageModels/LoRA/QLoRAFineTuner.cs b/Sources/Main/LanguageModels/LoRA/QLoRAFineTuner.cs index c5f23c47..ff06b826 100644 --- a/Sources/Main/LanguageModels/LoRA/QLoRAFineTuner.cs +++ b/Sources/Main/LanguageModels/LoRA/QLoRAFineTuner.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.IO; using DevOnBike.Overfit.Autograd; using DevOnBike.Overfit.DeepLearning; using DevOnBike.Overfit.LanguageModels.Runtime; diff --git a/Sources/Main/LanguageModels/Loading/GgufLlamaLoader.cs b/Sources/Main/LanguageModels/Loading/GgufLlamaLoader.cs index 862a8bad..304c281d 100644 --- a/Sources/Main/LanguageModels/Loading/GgufLlamaLoader.cs +++ b/Sources/Main/LanguageModels/Loading/GgufLlamaLoader.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.IO; using DevOnBike.Overfit.DeepLearning; using DevOnBike.Overfit.LanguageModels.Runtime; using DevOnBike.Overfit.Runtime; diff --git a/Sources/Main/LanguageModels/Loading/RepackedWeightsFile.cs b/Sources/Main/LanguageModels/Loading/RepackedWeightsFile.cs index 5fe10519..387113b6 100644 --- a/Sources/Main/LanguageModels/Loading/RepackedWeightsFile.cs +++ b/Sources/Main/LanguageModels/Loading/RepackedWeightsFile.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.IO; using System.Text; using DevOnBike.Overfit.LanguageModels.Runtime; diff --git a/Sources/Main/LanguageModels/Memory/ChatHistoryCompactor.cs b/Sources/Main/LanguageModels/Memory/ChatHistoryCompactor.cs index 4cf6dd44..26e70207 100644 --- a/Sources/Main/LanguageModels/Memory/ChatHistoryCompactor.cs +++ b/Sources/Main/LanguageModels/Memory/ChatHistoryCompactor.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; using DevOnBike.Overfit.LanguageModels.Chat; namespace DevOnBike.Overfit.LanguageModels.Memory diff --git a/Sources/Main/LanguageModels/Memory/CompactionPlan.cs b/Sources/Main/LanguageModels/Memory/CompactionPlan.cs index 01bbb4c1..39702b70 100644 --- a/Sources/Main/LanguageModels/Memory/CompactionPlan.cs +++ b/Sources/Main/LanguageModels/Memory/CompactionPlan.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; using DevOnBike.Overfit.LanguageModels.Chat; namespace DevOnBike.Overfit.LanguageModels.Memory diff --git a/Sources/Main/LanguageModels/Memory/SummarizingChatSession.cs b/Sources/Main/LanguageModels/Memory/SummarizingChatSession.cs index 8da548c4..121a0497 100644 --- a/Sources/Main/LanguageModels/Memory/SummarizingChatSession.cs +++ b/Sources/Main/LanguageModels/Memory/SummarizingChatSession.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; using DevOnBike.Overfit.LanguageModels.Chat; using DevOnBike.Overfit.LanguageModels.Contracts; diff --git a/Sources/Main/LanguageModels/Runtime/CachedLlamaSession.cs b/Sources/Main/LanguageModels/Runtime/CachedLlamaSession.cs index 9bc83d39..94947db8 100644 --- a/Sources/Main/LanguageModels/Runtime/CachedLlamaSession.cs +++ b/Sources/Main/LanguageModels/Runtime/CachedLlamaSession.cs @@ -7,8 +7,8 @@ using DevOnBike.Overfit.DeepLearning; using DevOnBike.Overfit.LanguageModels.Contracts; using DevOnBike.Overfit.LanguageModels.Rope; +using DevOnBike.Overfit.Runtime; using DevOnBike.Overfit.Tensors; -using DevOnBike.Overfit.Tensors.Core; namespace DevOnBike.Overfit.LanguageModels.Runtime { @@ -65,6 +65,19 @@ public sealed class CachedLlamaSession : ISlmSession // falls back to a full prefill rather than attending over K/V that does not match the prompt. private bool _cacheTokensValid = true; + // The logits left by the most recent prefill, plus the cache length they belong to (-1 = none). + // + // K/V reuse alone still costs one forward pass, because logits are a by-product of the stack rather + // than cache state: even a prompt the cache holds in full has to re-run its last token to learn what + // comes next. Keeping the end-of-prompt logits removes that last pass — an exact match restores them + // with a copy and forwards nothing at all. Valid for as long as tokens [0, position) are untouched, + // which a whole turn of generation is, since decoding only ever appends. + // + // Costs one float[vocab] per session (~608 KB for Qwen-3B's 151936-wide vocabulary) against a KV + // cache measured in tens of megabytes, and one memcpy per prefill. + private readonly float[] _promptLogits; + private int _promptLogitsPosition = -1; + private bool _disposed; private bool _slidingWindow; private int _evictBlock; @@ -92,6 +105,7 @@ internal CachedLlamaSession( _indexScratch = new int[config.VocabSize]; _scoreScratch = new float[config.VocabSize]; _cacheTokens = new int[cache.MaxLength]; + _promptLogits = new float[config.VocabSize]; _random = new Random(); } @@ -154,6 +168,7 @@ private void MakeRoomIfSliding() // at their own positions. Rebuilding the map would be cheap, but a slid session's prompt no // longer starts at position 0 either — prefix reuse is meaningless once the head is gone. _cacheTokensValid = false; + _promptLogitsPosition = -1; _cache.Evict(count); } } @@ -173,6 +188,7 @@ public void Reset() _cache.Reset(); _generatedTokens.Clear(); _cacheTokensValid = true; + _promptLogitsPosition = -1; } /// @@ -213,6 +229,7 @@ public void Prefill(ReadOnlySpan promptTokens) PrefillProfiler.BeginRequest(promptTokens.Length); PrefillBatchedQuant(promptTokens); PrefillProfiler.EndRequest(); + SnapshotPromptLogits(); return; } @@ -240,6 +257,18 @@ public void Prefill(ReadOnlySpan promptTokens) DecodeToken(promptTokens[i]); } } + + SnapshotPromptLogits(); + } + + /// + /// Records the logits this prefill just produced against the cache length they describe, so a later + /// prompt that the cache already holds in full can skip the forward pass entirely. + /// + private void SnapshotPromptLogits() + { + _logits.AsSpan(0, VocabularySize).CopyTo(_promptLogits.AsSpan(0, VocabularySize)); + _promptLogitsPosition = _cache.CurrentLength; } /// @@ -277,6 +306,7 @@ public void RestorePrefix(KvCacheSnapshot prefix) // The restored K/V belongs to whoever took the snapshot; this session never saw those token ids, // so it cannot claim any prefix matches them. _cacheTokensValid = false; + _promptLogitsPosition = -1; _cache.RestoreFrom(prefix); } @@ -307,13 +337,48 @@ public int PrefillReusingCache(ReadOnlySpan promptTokens) { ThrowIfDisposed(); - var reusable = ReusablePrefixLength(promptTokens); + var match = MatchingPrefixLength(promptTokens); + + if (DisableLogitsCache) + { + return PrefillReusingKeyValuesOnly(promptTokens, match); + } + + // Everything matches AND we kept the logits this exact prompt produced: restore them and forward + // nothing. This is the re-sent-prompt case (a retry, a regenerate, a load test), where even the + // one-token fallback below would be re-deriving something already computed. + if (match == promptTokens.Length && _promptLogitsPosition == promptTokens.Length) + { + _cache.TruncateTo(promptTokens.Length); + _promptLogits.AsSpan(0, VocabularySize).CopyTo(_logits.AsSpan(0, VocabularySize)); + return promptTokens.Length; + } + + return PrefillReusingKeyValuesOnly(promptTokens, match); + } + + /// Test hook: skip the kept-logits fast path so the K/V-only behaviour can be A/B'd against + /// it. Defaults from so a server process can be + /// started in either configuration and both measured in one interleaved run. + internal static bool DisableLogitsCache = + Environment.GetEnvironmentVariable(OverfitEnvironment.DisableLogitsCache) == "1"; + + /// + /// Reuse K/V only: hold one token back — logits are a by-product of the stack, so the last token has + /// to go through it for the session to learn what follows the prompt. + /// + private int PrefillReusingKeyValuesOnly(ReadOnlySpan promptTokens, int match) + { + var reusable = Math.Min(match, promptTokens.Length - 1); if (reusable <= 0) { Reset(promptTokens); return 0; } + // Any truncation below the snapshot leaves it describing K/V the cache no longer holds. The + // Prefill below re-establishes it; dropping it first means no window where it could be believed. + _promptLogitsPosition = -1; _cache.TruncateTo(reusable); Prefill(promptTokens[reusable..]); return reusable; @@ -321,10 +386,9 @@ public int PrefillReusingCache(ReadOnlySpan promptTokens) /// /// How many leading tokens of are already in the cache at the very - /// positions they would occupy. Always leaves at least one token for - /// to forward. + /// positions they would occupy — the raw match, before any decision about holding a token back. /// - private int ReusablePrefixLength(ReadOnlySpan promptTokens) + private int MatchingPrefixLength(ReadOnlySpan promptTokens) { if (!_cacheTokensValid || _slidingWindow || _cache.BasePosition != 0) { @@ -338,8 +402,7 @@ private int ReusablePrefixLength(ReadOnlySpan promptTokens) match++; } - // Never reuse the whole prompt: the final token must go through the stack to produce logits. - return Math.Min(match, promptTokens.Length - 1); + return match; } /// @@ -358,6 +421,29 @@ public int GenerateNextToken(in SamplingOptions sampling) /// logits, overwritten by the next decode, so masking in place is safe. /// public int GenerateNextToken(in SamplingOptions sampling, ITokenConstraint? constraint) + => GenerateNextToken(in sampling, constraint, onSampled: null); + + /// + /// As , but hands the sampled + /// token to before the forward pass that follows it. + /// + /// Why the ordering is worth an API. A decode step samples token N from the logits it + /// already holds, then runs a full pass over the weights so that logits predict token N+1. Emitting + /// after that pass makes every token — including the first — arrive one whole pass late. Measured + /// through the server on Qwen-3B with a fully cached prompt: time to first token was 74.8 ms against + /// an inter-token latency of 37.2 ms, i.e. exactly two passes, where one is all the answer needs. + /// + /// Returning true from the hook means the caller is finished with this token (a stop + /// sequence, end-of-text, a closed constraint), so the trailing pass is skipped entirely — it would + /// only have prepared logits nobody reads. The token is then not fed back into the cache, which + /// is the honest state: the cache holds what was forwarded. Do not continue generating on the same + /// session after returning true without resetting or prefilling — the logits still predict the + /// token just sampled, so the next step would draw it again. + /// + public int GenerateNextToken( + in SamplingOptions sampling, + ITokenConstraint? constraint, + Func? onSampled) { ThrowIfDisposed(); @@ -384,6 +470,16 @@ public int GenerateNextToken(in SamplingOptions sampling, ITokenConstraint? cons constraint?.Accept(token); TrackGenerated(token); + + // Hand the token over before the pass that prepares the NEXT logits, so a streaming caller can + // put it on the wire a full weight-pass earlier — and can tell us the answer is finished, in + // which case that pass is pure waste and is skipped. + if (onSampled is not null && onSampled(token)) + { + DecodeProfiler.EndToken(); + return token; + } + DecodeToken(token); DecodeProfiler.EndToken(); return token; @@ -503,7 +599,25 @@ public int GenerateSpeculative( int maxDraft = 4, int ngramMin = 1, int ngramMax = 3) - => GenerateSpeculativeCore(history, committed, in sampling, maxDraft, ngramMin, ngramMax, drafter: null); + => GenerateSpeculativeCore( + history, committed, in sampling, maxDraft, ngramMin, ngramMax, drafter: null, onSampled: null); + + /// + /// Speculative step with the same early-emit hook as + /// : the first + /// token of the step is drawn from the logits already held, so it can go out before the verify + /// forward runs. Returning true ends the step immediately — the whole verify is skipped, not + /// just a single pass. + /// + public int GenerateSpeculative( + ReadOnlySpan history, + Span committed, + in SamplingOptions sampling, + int maxDraft, + Func? onSampled) + => GenerateSpeculativeCore( + history, committed, in sampling, maxDraft, ngramMin: 1, ngramMax: 3, drafter: null, + onSampled: onSampled); /// /// Draft-MODEL speculative overload: proposals come from (a small draft @@ -516,7 +630,9 @@ internal int GenerateSpeculative( in SamplingOptions sampling, int maxDraft, ISpeculativeDrafter drafter) - => GenerateSpeculativeCore(history, committed, in sampling, maxDraft, ngramMin: 1, ngramMax: 3, drafter: drafter); + => GenerateSpeculativeCore( + history, committed, in sampling, maxDraft, ngramMin: 1, ngramMax: 3, drafter: drafter, + onSampled: null); private int GenerateSpeculativeCore( ReadOnlySpan history, @@ -525,7 +641,8 @@ private int GenerateSpeculativeCore( int maxDraft, int ngramMin, int ngramMax, - ISpeculativeDrafter? drafter) + ISpeculativeDrafter? drafter, + Func? onSampled) { ThrowIfDisposed(); ArgumentOutOfRangeException.ThrowIfNegativeOrZero(maxDraft); @@ -544,6 +661,14 @@ private int GenerateSpeculativeCore( // Next token from the current (target) distribution — the same draw a normal step would make. var t0 = TokenSampler.Sample(_logits, in sampling, _random, _indexScratch, _scoreScratch); + // Early emit: t0 comes from logits we already hold, so it can reach the client before the verify + // forward. If the caller says the answer ends here, the entire verify is wasted work — skip it. + if (onSampled is not null && onSampled(t0)) + { + committed[0] = t0; + return 1; + } + var canSpeculate = !_slidingWindow && _config.FfnActivation is FeedForwardActivation.SwiGLU or FeedForwardActivation.GeGLU && maxDraft > 0; diff --git a/Sources/Main/LanguageModels/Runtime/MoeFeedForwardBlock.cs b/Sources/Main/LanguageModels/Runtime/MoeFeedForwardBlock.cs index f11f4c9c..3e91955f 100644 --- a/Sources/Main/LanguageModels/Runtime/MoeFeedForwardBlock.cs +++ b/Sources/Main/LanguageModels/Runtime/MoeFeedForwardBlock.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using DevOnBike.Overfit.DeepLearning; using DevOnBike.Overfit.Tensors; namespace DevOnBike.Overfit.LanguageModels.Runtime diff --git a/Sources/Main/LanguageModels/Runtime/Qwen2MoeFeedForwardBlock.cs b/Sources/Main/LanguageModels/Runtime/Qwen2MoeFeedForwardBlock.cs index 1757d4e5..a72b50cf 100644 --- a/Sources/Main/LanguageModels/Runtime/Qwen2MoeFeedForwardBlock.cs +++ b/Sources/Main/LanguageModels/Runtime/Qwen2MoeFeedForwardBlock.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using DevOnBike.Overfit.DeepLearning; using DevOnBike.Overfit.Tensors; namespace DevOnBike.Overfit.LanguageModels.Runtime diff --git a/Sources/Main/LanguageModels/Tokenizers/GgufTokenizer.cs b/Sources/Main/LanguageModels/Tokenizers/GgufTokenizer.cs index ea044f42..49632939 100644 --- a/Sources/Main/LanguageModels/Tokenizers/GgufTokenizer.cs +++ b/Sources/Main/LanguageModels/Tokenizers/GgufTokenizer.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; using System.Text; using System.Text.RegularExpressions; using DevOnBike.Overfit.LanguageModels.Loading; diff --git a/Sources/Main/LanguageModels/Tokenizers/HuggingFaceBpeTokenizer.cs b/Sources/Main/LanguageModels/Tokenizers/HuggingFaceBpeTokenizer.cs index 13145459..0a8b70b5 100644 --- a/Sources/Main/LanguageModels/Tokenizers/HuggingFaceBpeTokenizer.cs +++ b/Sources/Main/LanguageModels/Tokenizers/HuggingFaceBpeTokenizer.cs @@ -3,8 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; -using System.IO; using System.Text; using System.Text.Json; using System.Text.RegularExpressions; diff --git a/Sources/Main/LanguageModels/Tokenizers/WordPieceTokenizer.cs b/Sources/Main/LanguageModels/Tokenizers/WordPieceTokenizer.cs index ecfef6bb..458e3f4f 100644 --- a/Sources/Main/LanguageModels/Tokenizers/WordPieceTokenizer.cs +++ b/Sources/Main/LanguageModels/Tokenizers/WordPieceTokenizer.cs @@ -3,9 +3,7 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; using System.Globalization; -using System.IO; using System.Text; using DevOnBike.Overfit.LanguageModels.Contracts; diff --git a/Sources/Main/LanguageModels/Whisper/WhisperGgmlLoader.cs b/Sources/Main/LanguageModels/Whisper/WhisperGgmlLoader.cs index dcfc966f..5bc5957e 100644 --- a/Sources/Main/LanguageModels/Whisper/WhisperGgmlLoader.cs +++ b/Sources/Main/LanguageModels/Whisper/WhisperGgmlLoader.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.IO; using System.Text; namespace DevOnBike.Overfit.LanguageModels.Whisper diff --git a/Sources/Main/Optimizers/Adam.cs b/Sources/Main/Optimizers/Adam.cs index 3d108d2a..9e8aee58 100644 --- a/Sources/Main/Optimizers/Adam.cs +++ b/Sources/Main/Optimizers/Adam.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.IO; using System.Numerics; using System.Runtime.CompilerServices; using System.Runtime.InteropServices; diff --git a/Sources/Main/Runtime/OverfitEnvironment.cs b/Sources/Main/Runtime/OverfitEnvironment.cs index 91196a47..310562b2 100644 --- a/Sources/Main/Runtime/OverfitEnvironment.cs +++ b/Sources/Main/Runtime/OverfitEnvironment.cs @@ -124,6 +124,33 @@ public static class OverfitEnvironment /// Overrides the worker count for the on-device bench. public const string BenchWorkers = "OVERFIT_BENCH_WORKERS"; + // ── HTTP server (Sources/Server) ────────────────────────────────────────── + + /// + /// Set to 1 to print a per-request phase trace (history replay, prompt-cache reuse, time to + /// first token) — the attribution used to tell server overhead apart from engine work. + /// + public const string ServerTrace = "OVERFIT_SERVER_TRACE"; + + /// + /// Set to 1 to skip the kept-end-of-prompt-logits fast path, so a re-sent prompt costs one + /// forward pass instead of none. Exists so both configurations can be measured side by side in the + /// same interleaved run rather than across processes. + /// + public const string DisableLogitsCache = "OVERFIT_DISABLE_LOGITS_CACHE"; + + /// + /// Set to 1 to emit each token after the forward pass that follows it rather than before — + /// the ordering that predates the early-emit change. Same purpose: an in-run A/B. + /// + public const string DisableEarlyEmit = "OVERFIT_DISABLE_EARLY_EMIT"; + + /// + /// Set to 1 to force the exact single-token decode loop instead of the speculative path. + /// Same purpose: an in-run A/B of speculative decode against plain decode. + /// + public const string DisableSpeculative = "OVERFIT_DISABLE_SPECULATIVE"; + // ── Third-party / host environment (not ours, but read by us) ───────────── /// Hugging Face API endpoint override for the model downloader. diff --git a/Sources/Main/Runtime/OverfitParallel.cs b/Sources/Main/Runtime/OverfitParallel.cs index e6e13fa5..1d5ff133 100644 --- a/Sources/Main/Runtime/OverfitParallel.cs +++ b/Sources/Main/Runtime/OverfitParallel.cs @@ -382,7 +382,6 @@ public static bool SuppressParallelismOnCurrentThread set => _suppressOnThisThread = value; } - /// /// /// Diagnostics only: count every real fan-out (the inline fast path is not counted, since it costs /// nothing to launch). Off by default and checked before the interlocked increment. @@ -405,6 +404,7 @@ public static void ResetDispatchCount() Interlocked.Exchange(ref _dispatchCount, 0); } + /// /// Executes over chunks of /// [rangeStart, rangeEnd) across the worker pool. Equivalent to /// the grained overload with minItemsPerWorker = 1. diff --git a/Sources/Main/Training/DataParallelTrainer.cs b/Sources/Main/Training/DataParallelTrainer.cs index da239492..607af0ff 100644 --- a/Sources/Main/Training/DataParallelTrainer.cs +++ b/Sources/Main/Training/DataParallelTrainer.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Threading.Tasks; using DevOnBike.Overfit.Optimizers.Abstractions; using DevOnBike.Overfit.Parameters; using DevOnBike.Overfit.Runtime; diff --git a/Sources/Server/OpenAi/OpenAiChatMapping.cs b/Sources/Server/OpenAi/OpenAiChatMapping.cs index 58bd8333..a55adbbf 100644 --- a/Sources/Server/OpenAi/OpenAiChatMapping.cs +++ b/Sources/Server/OpenAi/OpenAiChatMapping.cs @@ -4,7 +4,6 @@ // For commercial licensing options, contact: devonbike@gmail.com using System.Text.Json; -using DevOnBike.Overfit.LanguageModels; using DevOnBike.Overfit.LanguageModels.Chat; using DevOnBike.Overfit.LanguageModels.Constraints; using DevOnBike.Overfit.LanguageModels.Contracts; diff --git a/Sources/Server/OverfitOpenAiServer.cs b/Sources/Server/OverfitOpenAiServer.cs index ed1c33f8..414d981c 100644 --- a/Sources/Server/OverfitOpenAiServer.cs +++ b/Sources/Server/OverfitOpenAiServer.cs @@ -15,6 +15,7 @@ using DevOnBike.Overfit.Server.OpenAi; using DevOnBike.Overfit.Serving; using DevOnBike.Overfit.Diagnostics; +using DevOnBike.Overfit.Runtime; namespace DevOnBike.Overfit.Server { @@ -383,7 +384,7 @@ private static byte[] ToPcm16Bytes(float[] samples) /// Opt-in per-request phase trace (OVERFIT_SERVER_TRACE=1) for TTFT attribution. private static readonly bool ServerTrace = - Environment.GetEnvironmentVariable("OVERFIT_SERVER_TRACE") == "1"; + Environment.GetEnvironmentVariable(OverfitEnvironment.ServerTrace) == "1"; private static void HandleChatCompletions(HttpListenerContext ctx, OverfitClient client, string modelName, string systemMessage) { @@ -620,21 +621,23 @@ private static void TryWriteError(HttpListenerResponse resp, HttpStatusCode stat /// request at a time (single-threaded accept loop), so a lock-free lazy init is safe here. private static string OpenApiYaml() { - if (_openApiYaml is null) + // Returning from each branch rather than falling through to a shared `return`: the field is + // nullable, and the two assignments above a common exit are not enough for the compiler to prove + // it was set (CS8603). With `else` banned, an early return per branch is the honest shape. + if (_openApiYaml is not null) { - using var stream = typeof(OverfitOpenAiServer).Assembly.GetManifestResourceStream("openapi.yaml"); - if (stream is null) - { - _openApiYaml = "openapi: 3.0.3\ninfo:\n title: Overfit\n version: '1.0.0'\npaths: {}\n"; - } + return _openApiYaml; + } - if (!(stream is null)) - { - using var reader = new StreamReader(stream, Encoding.UTF8); - _openApiYaml = reader.ReadToEnd(); - } + using var stream = typeof(OverfitOpenAiServer).Assembly.GetManifestResourceStream("openapi.yaml"); + if (stream is null) + { + _openApiYaml = "openapi: 3.0.3\ninfo:\n title: Overfit\n version: '1.0.0'\npaths: {}\n"; + return _openApiYaml; } + using var reader = new StreamReader(stream, Encoding.UTF8); + _openApiYaml = reader.ReadToEnd(); return _openApiYaml; } diff --git a/Sources/Server/RedactionGateway.cs b/Sources/Server/RedactionGateway.cs index af16ffe3..661bc2db 100644 --- a/Sources/Server/RedactionGateway.cs +++ b/Sources/Server/RedactionGateway.cs @@ -67,7 +67,10 @@ public static void Serve( Console.WriteLine(" response scanning: ON (model-generated secrets/PII masked on non-streaming responses)."); } - while (true) + // Bound stated in the header (OVERFIT023): the loop runs exactly as long as the listener is up. + // Stopping or disposing it both clear IsListening and make a pending GetContext throw, so no path + // keeps accepting after shutdown. + while (listener.IsListening) { HttpListenerContext ctx; try @@ -78,6 +81,10 @@ public static void Serve( { break; } + catch (ObjectDisposedException) + { + break; + } // Dispatch each request to the thread pool so a slow (or streaming) call never blocks the next caller. var captured = ctx; diff --git a/Tests/Adapters/MeaiAdapterEndToEndTests.cs b/Tests/Adapters/MeaiAdapterEndToEndTests.cs index d585c287..3ce61bdf 100644 --- a/Tests/Adapters/MeaiAdapterEndToEndTests.cs +++ b/Tests/Adapters/MeaiAdapterEndToEndTests.cs @@ -4,7 +4,6 @@ // For commercial licensing options, contact: devonbike@gmail.com using System.Text; -using System.Threading.Tasks; using DevOnBike.Overfit.Extensions.AI; using DevOnBike.Overfit.LanguageModels; using DevOnBike.Overfit.LanguageModels.Embeddings; diff --git a/Tests/Adapters/OverfitChatClientTests.cs b/Tests/Adapters/OverfitChatClientTests.cs index 0b96ed56..5b2d0205 100644 --- a/Tests/Adapters/OverfitChatClientTests.cs +++ b/Tests/Adapters/OverfitChatClientTests.cs @@ -4,7 +4,6 @@ // For commercial licensing options, contact: devonbike@gmail.com using System.Text; -using System.Threading.Tasks; using DevOnBike.Overfit.Extensions.AI; using DevOnBike.Overfit.LanguageModels.Chat; using DevOnBike.Overfit.LanguageModels.Contracts; diff --git a/Tests/Anomalies/GptVsEwmaBaselineComparisonTests.cs b/Tests/Anomalies/GptVsEwmaBaselineComparisonTests.cs index d32dae37..3772104d 100644 --- a/Tests/Anomalies/GptVsEwmaBaselineComparisonTests.cs +++ b/Tests/Anomalies/GptVsEwmaBaselineComparisonTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Linq; using DevOnBike.Overfit.Anomalies.Baseline; using DevOnBike.Overfit.Anomalies.Gpt; using DevOnBike.Overfit.Anomalies.Monitoring; diff --git a/Tests/Audio/WavReaderTests.cs b/Tests/Audio/WavReaderTests.cs index 688a5c40..16daf8c3 100644 --- a/Tests/Audio/WavReaderTests.cs +++ b/Tests/Audio/WavReaderTests.cs @@ -4,7 +4,6 @@ // For commercial licensing options, contact: devonbike@gmail.com using System.Buffers.Binary; -using System.IO; using System.Text; using DevOnBike.Overfit.Audio; diff --git a/Tests/Core/Autograd/FrozenQuantizedLinearTests.cs b/Tests/Core/Autograd/FrozenQuantizedLinearTests.cs index dd286d52..3d510d39 100644 --- a/Tests/Core/Autograd/FrozenQuantizedLinearTests.cs +++ b/Tests/Core/Autograd/FrozenQuantizedLinearTests.cs @@ -8,7 +8,6 @@ using DevOnBike.Overfit.Tensors; using DevOnBike.Overfit.Tensors.Core; using Xunit.Abstractions; -using Ops = DevOnBike.Overfit.Ops; namespace DevOnBike.Overfit.Tests.Core.Autograd { diff --git a/Tests/Core/Autograd/GqaAttentionTests.cs b/Tests/Core/Autograd/GqaAttentionTests.cs index 0f23a9f6..b1ffe297 100644 --- a/Tests/Core/Autograd/GqaAttentionTests.cs +++ b/Tests/Core/Autograd/GqaAttentionTests.cs @@ -7,7 +7,6 @@ using DevOnBike.Overfit.Tensors; using DevOnBike.Overfit.Tensors.Core; using Xunit.Abstractions; -using Ops = DevOnBike.Overfit.Ops; namespace DevOnBike.Overfit.Tests.Core.Autograd { diff --git a/Tests/Core/Autograd/QLoRATrainingTests.cs b/Tests/Core/Autograd/QLoRATrainingTests.cs index 40f6df7a..094c647e 100644 --- a/Tests/Core/Autograd/QLoRATrainingTests.cs +++ b/Tests/Core/Autograd/QLoRATrainingTests.cs @@ -10,7 +10,6 @@ using DevOnBike.Overfit.Tensors; using DevOnBike.Overfit.Tensors.Core; using Xunit.Abstractions; -using Ops = DevOnBike.Overfit.Ops; namespace DevOnBike.Overfit.Tests.Core.Autograd { diff --git a/Tests/Core/Autograd/RmsNormTests.cs b/Tests/Core/Autograd/RmsNormTests.cs index c2ad1de5..3cca4590 100644 --- a/Tests/Core/Autograd/RmsNormTests.cs +++ b/Tests/Core/Autograd/RmsNormTests.cs @@ -7,7 +7,6 @@ using DevOnBike.Overfit.Tensors; using DevOnBike.Overfit.Tensors.Core; using Xunit.Abstractions; -using Ops = DevOnBike.Overfit.Ops; namespace DevOnBike.Overfit.Tests.Core.Autograd { diff --git a/Tests/Core/Autograd/RopeTests.cs b/Tests/Core/Autograd/RopeTests.cs index f0048e4a..a3033236 100644 --- a/Tests/Core/Autograd/RopeTests.cs +++ b/Tests/Core/Autograd/RopeTests.cs @@ -8,7 +8,6 @@ using DevOnBike.Overfit.Tensors; using DevOnBike.Overfit.Tensors.Core; using Xunit.Abstractions; -using Ops = DevOnBike.Overfit.Ops; namespace DevOnBike.Overfit.Tests.Core.Autograd { diff --git a/Tests/Core/Autograd/SiLUTests.cs b/Tests/Core/Autograd/SiLUTests.cs index d2e03ffc..cde7a7e2 100644 --- a/Tests/Core/Autograd/SiLUTests.cs +++ b/Tests/Core/Autograd/SiLUTests.cs @@ -7,7 +7,6 @@ using DevOnBike.Overfit.Tensors; using DevOnBike.Overfit.Tensors.Core; using Xunit.Abstractions; -using Ops = DevOnBike.Overfit.Ops; namespace DevOnBike.Overfit.Tests.Core.Autograd { diff --git a/Tests/Data/Mnist/MnistAllocBreakdownTests.cs b/Tests/Data/Mnist/MnistAllocBreakdownTests.cs index 238fd4aa..194ee684 100644 --- a/Tests/Data/Mnist/MnistAllocBreakdownTests.cs +++ b/Tests/Data/Mnist/MnistAllocBreakdownTests.cs @@ -5,7 +5,6 @@ using DevOnBike.Overfit.Autograd; using DevOnBike.Overfit.DeepLearning; -using DevOnBike.Overfit.Diagnostics; using DevOnBike.Overfit.Ops; using DevOnBike.Overfit.Optimizers; using DevOnBike.Overfit.Tensors; diff --git a/Tests/Data/Mnist/MnistOneCycleBenchTests.cs b/Tests/Data/Mnist/MnistOneCycleBenchTests.cs index 5a3915e1..f13622d6 100644 --- a/Tests/Data/Mnist/MnistOneCycleBenchTests.cs +++ b/Tests/Data/Mnist/MnistOneCycleBenchTests.cs @@ -11,7 +11,6 @@ using DevOnBike.Overfit.Tensors; using DevOnBike.Overfit.Tensors.Core; using DevOnBike.Overfit.Tests.TestSupport; -using DevOnBike.Overfit.Tests.TestSupport.Helpers; using DevOnBike.Overfit.Training; using Xunit.Abstractions; diff --git a/Tests/DeepLearning/CheckpointParityTests.cs b/Tests/DeepLearning/CheckpointParityTests.cs index 4e6b5df1..94b72aed 100644 --- a/Tests/DeepLearning/CheckpointParityTests.cs +++ b/Tests/DeepLearning/CheckpointParityTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Linq; using DevOnBike.Overfit.Autograd; using DevOnBike.Overfit.DeepLearning; using DevOnBike.Overfit.Maths; diff --git a/Tests/DeepLearning/Diagnostics/CifarCnnBackwardOpProfilerTests.cs b/Tests/DeepLearning/Diagnostics/CifarCnnBackwardOpProfilerTests.cs index f12da0bb..2c01f405 100644 --- a/Tests/DeepLearning/Diagnostics/CifarCnnBackwardOpProfilerTests.cs +++ b/Tests/DeepLearning/Diagnostics/CifarCnnBackwardOpProfilerTests.cs @@ -7,7 +7,6 @@ using System.Reflection; using DevOnBike.Overfit.Autograd; using DevOnBike.Overfit.DeepLearning; -using DevOnBike.Overfit.Ops; using DevOnBike.Overfit.Tensors; using DevOnBike.Overfit.Tensors.Core; using Xunit.Abstractions; diff --git a/Tests/DeepLearning/Diagnostics/CifarCnnForwardLayerProfilerTests.cs b/Tests/DeepLearning/Diagnostics/CifarCnnForwardLayerProfilerTests.cs index 5eaec365..b3d475d3 100644 --- a/Tests/DeepLearning/Diagnostics/CifarCnnForwardLayerProfilerTests.cs +++ b/Tests/DeepLearning/Diagnostics/CifarCnnForwardLayerProfilerTests.cs @@ -6,7 +6,6 @@ using System.Diagnostics; using DevOnBike.Overfit.Autograd; using DevOnBike.Overfit.DeepLearning; -using DevOnBike.Overfit.Ops; using DevOnBike.Overfit.Tensors; using DevOnBike.Overfit.Tensors.Core; using Xunit.Abstractions; diff --git a/Tests/DeepLearning/Diagnostics/MnistCnnBackwardOpProfilerTests.cs b/Tests/DeepLearning/Diagnostics/MnistCnnBackwardOpProfilerTests.cs index 64000de9..ac16b593 100644 --- a/Tests/DeepLearning/Diagnostics/MnistCnnBackwardOpProfilerTests.cs +++ b/Tests/DeepLearning/Diagnostics/MnistCnnBackwardOpProfilerTests.cs @@ -7,7 +7,6 @@ using System.Reflection; using DevOnBike.Overfit.Autograd; using DevOnBike.Overfit.DeepLearning; -using DevOnBike.Overfit.Ops; using DevOnBike.Overfit.Tensors; using DevOnBike.Overfit.Tensors.Core; using Xunit.Abstractions; diff --git a/Tests/DeepLearning/Diagnostics/MnistCnnForwardLayerProfilerTests.cs b/Tests/DeepLearning/Diagnostics/MnistCnnForwardLayerProfilerTests.cs index 5f7843ee..2ddcb0d5 100644 --- a/Tests/DeepLearning/Diagnostics/MnistCnnForwardLayerProfilerTests.cs +++ b/Tests/DeepLearning/Diagnostics/MnistCnnForwardLayerProfilerTests.cs @@ -6,7 +6,6 @@ using System.Diagnostics; using DevOnBike.Overfit.Autograd; using DevOnBike.Overfit.DeepLearning; -using DevOnBike.Overfit.Ops; using DevOnBike.Overfit.Tensors; using DevOnBike.Overfit.Tensors.Core; using Xunit.Abstractions; diff --git a/Tests/DeepLearning/TrainableLlamaBlockTests.cs b/Tests/DeepLearning/TrainableLlamaBlockTests.cs index 78999823..8b9cc8c5 100644 --- a/Tests/DeepLearning/TrainableLlamaBlockTests.cs +++ b/Tests/DeepLearning/TrainableLlamaBlockTests.cs @@ -10,7 +10,6 @@ using DevOnBike.Overfit.Tensors; using DevOnBike.Overfit.Tensors.Core; using Xunit.Abstractions; -using Ops = DevOnBike.Overfit.Ops; namespace DevOnBike.Overfit.Tests.DeepLearning { diff --git a/Tests/DeepLearning/TrainableLlamaModelTests.cs b/Tests/DeepLearning/TrainableLlamaModelTests.cs index 2d2e4aa2..b111b1a3 100644 --- a/Tests/DeepLearning/TrainableLlamaModelTests.cs +++ b/Tests/DeepLearning/TrainableLlamaModelTests.cs @@ -3,12 +3,10 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.IO; using DevOnBike.Overfit.Autograd; using DevOnBike.Overfit.DeepLearning; using DevOnBike.Overfit.LanguageModels.Runtime; using DevOnBike.Overfit.Optimizers; -using DevOnBike.Overfit.Tensors.Core; using Xunit.Abstractions; namespace DevOnBike.Overfit.Tests.DeepLearning diff --git a/Tests/Evolutionary/Algorithms/UninitializedStrategyGuardTests.cs b/Tests/Evolutionary/Algorithms/UninitializedStrategyGuardTests.cs index 44f2fef6..28144c48 100644 --- a/Tests/Evolutionary/Algorithms/UninitializedStrategyGuardTests.cs +++ b/Tests/Evolutionary/Algorithms/UninitializedStrategyGuardTests.cs @@ -5,7 +5,6 @@ using DevOnBike.Overfit.Evolutionary.Storage; using DevOnBike.Overfit.Evolutionary.Strategies; -using DevOnBike.Overfit.Exceptions; namespace DevOnBike.Overfit.Tests.Evolutionary.Algorithms { diff --git a/Tests/Examples/CtcOcrDemoTests.cs b/Tests/Examples/CtcOcrDemoTests.cs index d7c39c2e..c0aa9e7b 100644 --- a/Tests/Examples/CtcOcrDemoTests.cs +++ b/Tests/Examples/CtcOcrDemoTests.cs @@ -3,8 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; -using System.Linq; using DevOnBike.Overfit.Autograd; using DevOnBike.Overfit.DeepLearning; using DevOnBike.Overfit.Optimizers; diff --git a/Tests/LanguageModels/Agents/ReActAgentEndToEndTests.cs b/Tests/LanguageModels/Agents/ReActAgentEndToEndTests.cs index d8e3f657..672161a5 100644 --- a/Tests/LanguageModels/Agents/ReActAgentEndToEndTests.cs +++ b/Tests/LanguageModels/Agents/ReActAgentEndToEndTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; using System.Text.Json; using DevOnBike.Overfit.LanguageModels.Agents; using DevOnBike.Overfit.LanguageModels.Chat; diff --git a/Tests/LanguageModels/Agents/ReActAgentTests.cs b/Tests/LanguageModels/Agents/ReActAgentTests.cs index 6b9b7551..8606ce35 100644 --- a/Tests/LanguageModels/Agents/ReActAgentTests.cs +++ b/Tests/LanguageModels/Agents/ReActAgentTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; using System.Text.Json; using DevOnBike.Overfit.LanguageModels.Agents; using DevOnBike.Overfit.LanguageModels.Tools; diff --git a/Tests/LanguageModels/Chat/HuggingFaceChatModelTests.cs b/Tests/LanguageModels/Chat/HuggingFaceChatModelTests.cs index f7231d81..425475a0 100644 --- a/Tests/LanguageModels/Chat/HuggingFaceChatModelTests.cs +++ b/Tests/LanguageModels/Chat/HuggingFaceChatModelTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.IO; using DevOnBike.Overfit.LanguageModels.Chat; using DevOnBike.Overfit.LanguageModels.Contracts; using DevOnBike.Overfit.Tests.TestSupport; diff --git a/Tests/LanguageModels/Chat/HuggingFaceLlamaModelTests.cs b/Tests/LanguageModels/Chat/HuggingFaceLlamaModelTests.cs index 6ffde5ea..fcb0b8d1 100644 --- a/Tests/LanguageModels/Chat/HuggingFaceLlamaModelTests.cs +++ b/Tests/LanguageModels/Chat/HuggingFaceLlamaModelTests.cs @@ -3,8 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; -using System.IO; using System.Runtime.InteropServices; using DevOnBike.Overfit.LanguageModels.Chat; using DevOnBike.Overfit.LanguageModels.Contracts; diff --git a/Tests/LanguageModels/Chat/QwenChatModelTests.cs b/Tests/LanguageModels/Chat/QwenChatModelTests.cs index e01d6cbf..6df512eb 100644 --- a/Tests/LanguageModels/Chat/QwenChatModelTests.cs +++ b/Tests/LanguageModels/Chat/QwenChatModelTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.IO; using System.Text; using DevOnBike.Overfit.LanguageModels.Chat; using DevOnBike.Overfit.LanguageModels.Contracts; diff --git a/Tests/LanguageModels/Diagnostics/DecodeCostAblationTests.cs b/Tests/LanguageModels/Diagnostics/DecodeCostAblationTests.cs new file mode 100644 index 00000000..d9e20959 --- /dev/null +++ b/Tests/LanguageModels/Diagnostics/DecodeCostAblationTests.cs @@ -0,0 +1,111 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.Diagnostics; +using DevOnBike.Overfit.LanguageModels; +using DevOnBike.Overfit.LanguageModels.Chat; +using DevOnBike.Overfit.LanguageModels.Contracts; +using DevOnBike.Overfit.LanguageModels.Runtime; +using DevOnBike.Overfit.Tests.TestSupport; +using Xunit.Abstractions; + +namespace DevOnBike.Overfit.Tests.LanguageModels.Diagnostics +{ + /// + /// Isolates which of the two TTFT changes cost decode throughput. + /// + /// The observation. Through the server, the full prompt-cache stack drove time to first + /// token from 74.8 ms to 0.6 ms — and simultaneously pushed inter-token latency from 37.20 to 55.20 ms + /// and end-to-end from 1228 to 1711 ms. Counting passes says that should be impossible: the old path ran + /// 33 forward passes for 32 tokens (two before the first emit), the new one runs 32. Less work, more + /// time, so something per-token got slower and the pass count is not the explanation. + /// + /// Two changes landed between the last good measurement and the bad one — emitting the token + /// before the forward pass that follows it, and keeping the end-of-prompt logits. This runs all four + /// combinations in one process, interleaved, because cross-process before/after on this box has + /// already produced a phantom 32% swing on an untouched path. + /// + public sealed class DecodeCostAblationTests + { + private const int Rounds = 3; + private const int NewTokens = 24; + + private readonly ITestOutputHelper _out; + + public DecodeCostAblationTests(ITestOutputHelper output) => _out = output; + + [LongFact] + public void DecodeCost_ByEarlyEmitAndLogitsCache() + { + var path = TestModelPaths.Qwen3B.Q4KmGgufPath; + if (!File.Exists(path)) + { + _out.WriteLine($"missing {path}"); + return; + } + + using var client = OverfitClient.LoadGguf( + path, maxContextLength: 1024, maxNewTokens: NewTokens, sampling: SamplingOptions.Greedy); + + const string Prompt = "Explain, in a short paragraph, why running language models locally " + + "can be useful."; + + // Warm: first turn pays the cold prefill and page-in, which is not what is being compared. + var options = client.Options; + client.Chat.Send(Prompt, in options, onText: null, constraint: null); + + var arms = new (string Label, bool EarlyEmit, bool LogitsCache)[] + { + ("early-emit OFF, logits-cache OFF", false, false), + ("early-emit ON, logits-cache OFF", true, false), + ("early-emit OFF, logits-cache ON", false, true), + ("early-emit ON, logits-cache ON", true, true), + }; + + var best = new double[arms.Length]; + for (var i = 0; i < best.Length; i++) + { + best[i] = double.MaxValue; + } + + // Interleaved: every arm is measured once per round, so a machine drift moves all four together + // instead of favouring whichever ran first. + for (var round = 0; round < Rounds; round++) + { + for (var a = 0; a < arms.Length; a++) + { + ChatSession.DisableEarlyEmit = !arms[a].EarlyEmit; + CachedLlamaSession.DisableLogitsCache = !arms[a].LogitsCache; + + try + { + // Re-send the identical prompt: this is the shape the load driver uses and the only + // one where the logits cache can fire at all. + client.Reset(); + var started = ValueStopwatch.StartNew(); + client.Chat.Send(Prompt, in options, onText: null, constraint: null); + var elapsed = started.GetElapsedTime().TotalMilliseconds; + var generated = client.Chat.LastStats.GeneratedTokens; + + best[a] = Math.Min(best[a], elapsed / Math.Max(1, generated)); + } + finally + { + ChatSession.DisableEarlyEmit = false; + CachedLlamaSession.DisableLogitsCache = false; + } + } + } + + _out.WriteLine($" {"arm",-36}{"ms/token",12}"); + for (var a = 0; a < arms.Length; a++) + { + _out.WriteLine($" {arms[a].Label,-36}{best[a],10:F2}"); + } + + Assert.All(best, b => Assert.True(b > 0)); + } + } +} diff --git a/Tests/LanguageModels/Diagnostics/OrpheusPromptTokenTests.cs b/Tests/LanguageModels/Diagnostics/OrpheusPromptTokenTests.cs index 9633220c..b8dcfb30 100644 --- a/Tests/LanguageModels/Diagnostics/OrpheusPromptTokenTests.cs +++ b/Tests/LanguageModels/Diagnostics/OrpheusPromptTokenTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using DevOnBike.Overfit.LanguageModels.Loading; using DevOnBike.Overfit.LanguageModels.Tokenizers; using Xunit.Abstractions; diff --git a/Tests/LanguageModels/Diagnostics/TinyBlasProjectionHeadroomPhase05Tests.cs b/Tests/LanguageModels/Diagnostics/TinyBlasProjectionHeadroomPhase05Tests.cs index b12034db..469fbca1 100644 --- a/Tests/LanguageModels/Diagnostics/TinyBlasProjectionHeadroomPhase05Tests.cs +++ b/Tests/LanguageModels/Diagnostics/TinyBlasProjectionHeadroomPhase05Tests.cs @@ -6,7 +6,6 @@ using System.Diagnostics; using System.Runtime.Intrinsics; using System.Runtime.Intrinsics.X86; -using System.Threading.Tasks; using DevOnBike.Overfit.LanguageModels.Loading; using DevOnBike.Overfit.LanguageModels.Runtime; using Xunit.Abstractions; diff --git a/Tests/LanguageModels/Embeddings/MiniLmFullLengthTests.cs b/Tests/LanguageModels/Embeddings/MiniLmFullLengthTests.cs index d1f853d7..c6ddc9df 100644 --- a/Tests/LanguageModels/Embeddings/MiniLmFullLengthTests.cs +++ b/Tests/LanguageModels/Embeddings/MiniLmFullLengthTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Linq; using DevOnBike.Overfit.LanguageModels.Embeddings; namespace DevOnBike.Overfit.Tests.LanguageModels.Embeddings diff --git a/Tests/LanguageModels/Loading/BielikSafetensorsParityTests.cs b/Tests/LanguageModels/Loading/BielikSafetensorsParityTests.cs index 6416fa83..7d9bc597 100644 --- a/Tests/LanguageModels/Loading/BielikSafetensorsParityTests.cs +++ b/Tests/LanguageModels/Loading/BielikSafetensorsParityTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Linq; using System.Text.Json; using DevOnBike.Overfit.LanguageModels.Contracts; using DevOnBike.Overfit.LanguageModels.Loading; diff --git a/Tests/LanguageModels/Loading/GgufNestedArrayDepthTests.cs b/Tests/LanguageModels/Loading/GgufNestedArrayDepthTests.cs index 5c880d07..03875e43 100644 --- a/Tests/LanguageModels/Loading/GgufNestedArrayDepthTests.cs +++ b/Tests/LanguageModels/Loading/GgufNestedArrayDepthTests.cs @@ -4,7 +4,6 @@ // For commercial licensing options, contact: devonbike@gmail.com using System.Text; -using DevOnBike.Overfit.Exceptions; using DevOnBike.Overfit.LanguageModels.Loading; namespace DevOnBike.Overfit.Tests.LanguageModels.Loading diff --git a/Tests/LanguageModels/Loading/QLoraGgufBridgeTests.cs b/Tests/LanguageModels/Loading/QLoraGgufBridgeTests.cs index 048ee3ed..028778d9 100644 --- a/Tests/LanguageModels/Loading/QLoraGgufBridgeTests.cs +++ b/Tests/LanguageModels/Loading/QLoraGgufBridgeTests.cs @@ -11,7 +11,6 @@ using DevOnBike.Overfit.Tensors; using DevOnBike.Overfit.Tensors.Core; using Xunit.Abstractions; -using Ops = DevOnBike.Overfit.Ops; namespace DevOnBike.Overfit.Tests.LanguageModels.Loading { diff --git a/Tests/LanguageModels/Loading/QwenGgufQLoraAdapterRoundTripTests.cs b/Tests/LanguageModels/Loading/QwenGgufQLoraAdapterRoundTripTests.cs index 8809c937..aac1beda 100644 --- a/Tests/LanguageModels/Loading/QwenGgufQLoraAdapterRoundTripTests.cs +++ b/Tests/LanguageModels/Loading/QwenGgufQLoraAdapterRoundTripTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.IO; using DevOnBike.Overfit.Autograd; using DevOnBike.Overfit.DeepLearning; using DevOnBike.Overfit.LanguageModels.Runtime; diff --git a/Tests/LanguageModels/Loading/QwenGgufQLoraE2ETests.cs b/Tests/LanguageModels/Loading/QwenGgufQLoraE2ETests.cs index 86191ed5..26b472e9 100644 --- a/Tests/LanguageModels/Loading/QwenGgufQLoraE2ETests.cs +++ b/Tests/LanguageModels/Loading/QwenGgufQLoraE2ETests.cs @@ -11,7 +11,6 @@ using DevOnBike.Overfit.Tensors.Core; using DevOnBike.Overfit.Tests.TestSupport; using Xunit.Abstractions; -using Ops = DevOnBike.Overfit.Ops; namespace DevOnBike.Overfit.Tests.LanguageModels.Loading { diff --git a/Tests/LanguageModels/Loading/QwenGgufTrainingRamTests.cs b/Tests/LanguageModels/Loading/QwenGgufTrainingRamTests.cs index 769b1f19..8bf66db3 100644 --- a/Tests/LanguageModels/Loading/QwenGgufTrainingRamTests.cs +++ b/Tests/LanguageModels/Loading/QwenGgufTrainingRamTests.cs @@ -11,7 +11,6 @@ using DevOnBike.Overfit.Tensors.Core; using DevOnBike.Overfit.Tests.TestSupport; using Xunit.Abstractions; -using Ops = DevOnBike.Overfit.Ops; namespace DevOnBike.Overfit.Tests.LanguageModels.Loading { diff --git a/Tests/LanguageModels/Loading/RepackedSidecarEngineE2ETests.cs b/Tests/LanguageModels/Loading/RepackedSidecarEngineE2ETests.cs index b98395a0..6075bc79 100644 --- a/Tests/LanguageModels/Loading/RepackedSidecarEngineE2ETests.cs +++ b/Tests/LanguageModels/Loading/RepackedSidecarEngineE2ETests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.IO; using DevOnBike.Overfit.LanguageModels.Contracts; using DevOnBike.Overfit.LanguageModels.Loading; using DevOnBike.Overfit.LanguageModels.Runtime; diff --git a/Tests/LanguageModels/Loading/RepackedWeightsFileTests.cs b/Tests/LanguageModels/Loading/RepackedWeightsFileTests.cs index 6fe7f28e..079c371d 100644 --- a/Tests/LanguageModels/Loading/RepackedWeightsFileTests.cs +++ b/Tests/LanguageModels/Loading/RepackedWeightsFileTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.IO; using DevOnBike.Overfit.LanguageModels.Loading; using DevOnBike.Overfit.LanguageModels.Runtime; diff --git a/Tests/LanguageModels/Runtime/EmbeddingsTests.cs b/Tests/LanguageModels/Runtime/EmbeddingsTests.cs index 77a2af34..797b9199 100644 --- a/Tests/LanguageModels/Runtime/EmbeddingsTests.cs +++ b/Tests/LanguageModels/Runtime/EmbeddingsTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using DevOnBike.Overfit.LanguageModels.Contracts; using DevOnBike.Overfit.LanguageModels.Loading; using DevOnBike.Overfit.LanguageModels.Tokenizers; using DevOnBike.Overfit.Tests.TestSupport; diff --git a/Tests/LanguageModels/Runtime/MoeFeedForwardBlockTests.cs b/Tests/LanguageModels/Runtime/MoeFeedForwardBlockTests.cs index b27a4eff..73c27c8c 100644 --- a/Tests/LanguageModels/Runtime/MoeFeedForwardBlockTests.cs +++ b/Tests/LanguageModels/Runtime/MoeFeedForwardBlockTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using DevOnBike.Overfit.DeepLearning; using DevOnBike.Overfit.LanguageModels.Runtime; using DevOnBike.Overfit.Tensors.Core; diff --git a/Tests/LanguageModels/Runtime/Parity/BielikDraftSpeculativeBench.cs b/Tests/LanguageModels/Runtime/Parity/BielikDraftSpeculativeBench.cs index 3fab2c5e..f477bf6a 100644 --- a/Tests/LanguageModels/Runtime/Parity/BielikDraftSpeculativeBench.cs +++ b/Tests/LanguageModels/Runtime/Parity/BielikDraftSpeculativeBench.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; using System.Runtime.InteropServices; using DevOnBike.Overfit.LanguageModels.Contracts; using DevOnBike.Overfit.LanguageModels.Runtime; diff --git a/Tests/LanguageModels/Runtime/Parity/DraftModelSpeculativeBench.cs b/Tests/LanguageModels/Runtime/Parity/DraftModelSpeculativeBench.cs index d9f63972..e9a9b913 100644 --- a/Tests/LanguageModels/Runtime/Parity/DraftModelSpeculativeBench.cs +++ b/Tests/LanguageModels/Runtime/Parity/DraftModelSpeculativeBench.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; using System.Runtime.InteropServices; using DevOnBike.Overfit.LanguageModels.Contracts; using DevOnBike.Overfit.LanguageModels.Loading; diff --git a/Tests/LanguageModels/Runtime/Parity/Q4KBatchedProjectionScalingBench.cs b/Tests/LanguageModels/Runtime/Parity/Q4KBatchedProjectionScalingBench.cs index f377d168..7b8ebb3c 100644 --- a/Tests/LanguageModels/Runtime/Parity/Q4KBatchedProjectionScalingBench.cs +++ b/Tests/LanguageModels/Runtime/Parity/Q4KBatchedProjectionScalingBench.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using DevOnBike.Overfit.LanguageModels.Loading; using DevOnBike.Overfit.LanguageModels.Runtime; using Xunit.Abstractions; diff --git a/Tests/LanguageModels/Runtime/Parity/Q4KWeightStationaryParityTests.cs b/Tests/LanguageModels/Runtime/Parity/Q4KWeightStationaryParityTests.cs index a180920e..890af9d1 100644 --- a/Tests/LanguageModels/Runtime/Parity/Q4KWeightStationaryParityTests.cs +++ b/Tests/LanguageModels/Runtime/Parity/Q4KWeightStationaryParityTests.cs @@ -4,7 +4,6 @@ // For commercial licensing options, contact: devonbike@gmail.com using System.Buffers.Binary; -using DevOnBike.Overfit.LanguageModels.Loading; using DevOnBike.Overfit.LanguageModels.Runtime; namespace DevOnBike.Overfit.Tests.LanguageModels.Runtime.Parity diff --git a/Tests/LanguageModels/Runtime/Parity/SmallModelAgenticProbeTests.cs b/Tests/LanguageModels/Runtime/Parity/SmallModelAgenticProbeTests.cs index b1e294c6..4fdb7b78 100644 --- a/Tests/LanguageModels/Runtime/Parity/SmallModelAgenticProbeTests.cs +++ b/Tests/LanguageModels/Runtime/Parity/SmallModelAgenticProbeTests.cs @@ -4,7 +4,6 @@ // For commercial licensing options, contact: devonbike@gmail.com using DevOnBike.Overfit.LanguageModels.Chat; -using DevOnBike.Overfit.LanguageModels.Constraints; using DevOnBike.Overfit.LanguageModels.Contracts; using DevOnBike.Overfit.LanguageModels.Loading; using DevOnBike.Overfit.LanguageModels.Tokenizers; diff --git a/Tests/LanguageModels/Runtime/Parity/SpeculativeDecodeParityTests.cs b/Tests/LanguageModels/Runtime/Parity/SpeculativeDecodeParityTests.cs index 8f01d61a..d9f9c35f 100644 --- a/Tests/LanguageModels/Runtime/Parity/SpeculativeDecodeParityTests.cs +++ b/Tests/LanguageModels/Runtime/Parity/SpeculativeDecodeParityTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; using DevOnBike.Overfit.LanguageModels.Chat; using DevOnBike.Overfit.LanguageModels.Contracts; using DevOnBike.Overfit.LanguageModels.Runtime; diff --git a/Tests/LanguageModels/Runtime/PromptCacheReuseTests.cs b/Tests/LanguageModels/Runtime/PromptCacheReuseTests.cs index 186647f7..47c83dd5 100644 --- a/Tests/LanguageModels/Runtime/PromptCacheReuseTests.cs +++ b/Tests/LanguageModels/Runtime/PromptCacheReuseTests.cs @@ -193,11 +193,12 @@ public void DivergentPrompt_FallsBackToTheMatchingPrefixOnly() } /// - /// Re-sending the identical prompt is the load-test shape and the degenerate case of the matcher: - /// everything matches, so the implementation must still hold one token back to refresh the logits. + /// Re-sending the identical prompt — a retry, a regenerate, a load test — must cost zero + /// forward passes: the whole prompt is in the cache and the logits it produced were kept, so there + /// is nothing left to derive. Restoring them has to reproduce the recomputed answer exactly. /// [SmallModelFact] - public void IdenticalPrompt_ReusesAllButTheLastToken() + public void IdenticalPrompt_ReusesEverythingAndForwardsNothing() { var path = TestModelPaths.Qwen05B.RequireQ4KmGgufPath(); @@ -216,7 +217,8 @@ public void IdenticalPrompt_ReusesAllButTheLastToken() var actual = new float[session.VocabularySize]; session.GetLastLogits(actual); - Assert.Equal(prompt.Length - 1, reused); + Assert.Equal(prompt.Length, reused); + Assert.Equal(prompt.Length, session.CurrentPosition); var maxDiff = 0f; for (var i = 0; i < expected.Length; i++) @@ -228,6 +230,61 @@ public void IdenticalPrompt_ReusesAllButTheLastToken() Assert.Equal(0f, maxDiff); } + /// + /// The kept logits describe one specific cache length. After the conversation moves on — a reply is + /// generated, then a longer prompt arrives — the zero-forward path must not fire on the stale + /// snapshot; the extended prompt has to produce the same logits a cold prefill would. + /// + [SmallModelFact] + public void ExtendedPromptAfterGeneration_DoesNotReuseStaleLogits() + { + var path = TestModelPaths.Qwen05B.RequireQ4KmGgufPath(); + + using var engine = CachedLlamaInferenceEngine.LoadGguf(path); + var tok = GgufTokenizer.Load(path); + + var headText = "The history of computing began with mechanical calculators and evolved " + + "through vacuum tubes, transistors and integrated circuits into the modern era."; + var turn1 = tok.Encode(headText); + var turn2 = tok.Encode(headText + + " Today, running a language model on a plain desktop processor without any " + + "dedicated accelerator hardware is entirely practical and quite common."); + + float[] reference; + using (var fresh = engine.CreateSession(512)) + { + fresh.Reset(turn2); + reference = new float[fresh.VocabularySize]; + fresh.GetLastLogits(reference); + } + + using var session = engine.CreateSession(512); + session.Reset(turn1); + + // Move the conversation on, so the cache holds prompt + reply while the snapshot still points at + // the end of the prompt. + for (var i = 0; i < 4; i++) + { + session.GenerateNextToken(SamplingOptions.Greedy); + } + + var reused = session.PrefillReusingCache(turn2); + + var actual = new float[session.VocabularySize]; + session.GetLastLogits(actual); + + var maxDiff = 0f; + for (var i = 0; i < reference.Length; i++) + { + maxDiff = Math.Max(maxDiff, Math.Abs(reference[i] - actual[i])); + } + + _out.WriteLine($"reused {reused} of {turn2.Length}, maxAbsLogitDiff = {maxDiff:G6}"); + + Assert.Equal(turn1.Length, reused); + Assert.Equal(0f, maxDiff); + } + /// /// Sliding-window sessions evict from the head, so recorded ids stop matching cache positions. The /// matcher must refuse to reuse rather than attend over shifted K/V. diff --git a/Tests/LanguageModels/Runtime/Q4KDotKernelNeonParityTests.cs b/Tests/LanguageModels/Runtime/Q4KDotKernelNeonParityTests.cs index 8a9cd645..53dd421a 100644 --- a/Tests/LanguageModels/Runtime/Q4KDotKernelNeonParityTests.cs +++ b/Tests/LanguageModels/Runtime/Q4KDotKernelNeonParityTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System; using System.Runtime.Intrinsics.Arm; using DevOnBike.Overfit.LanguageModels.Runtime; diff --git a/Tests/LanguageModels/Runtime/Q6KTiledGemmParityTests.cs b/Tests/LanguageModels/Runtime/Q6KTiledGemmParityTests.cs index eca3f07d..2a5ceb14 100644 --- a/Tests/LanguageModels/Runtime/Q6KTiledGemmParityTests.cs +++ b/Tests/LanguageModels/Runtime/Q6KTiledGemmParityTests.cs @@ -4,7 +4,6 @@ // For commercial licensing options, contact: devonbike@gmail.com using System.Runtime.Intrinsics.X86; -using DevOnBike.Overfit.LanguageModels.Loading; using DevOnBike.Overfit.LanguageModels.Runtime; namespace DevOnBike.Overfit.Tests.LanguageModels.Runtime diff --git a/Tests/LanguageModels/Runtime/Qwen2MoeFeedForwardBlockTests.cs b/Tests/LanguageModels/Runtime/Qwen2MoeFeedForwardBlockTests.cs index 993ee551..b6e7c34e 100644 --- a/Tests/LanguageModels/Runtime/Qwen2MoeFeedForwardBlockTests.cs +++ b/Tests/LanguageModels/Runtime/Qwen2MoeFeedForwardBlockTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using DevOnBike.Overfit.DeepLearning; using DevOnBike.Overfit.LanguageModels.Runtime; using DevOnBike.Overfit.Tensors.Core; diff --git a/Tests/LanguageModels/Skills/Evaluation/OverfitSkillRunnerLongTests.cs b/Tests/LanguageModels/Skills/Evaluation/OverfitSkillRunnerLongTests.cs index 4ba6e178..aef90044 100644 --- a/Tests/LanguageModels/Skills/Evaluation/OverfitSkillRunnerLongTests.cs +++ b/Tests/LanguageModels/Skills/Evaluation/OverfitSkillRunnerLongTests.cs @@ -3,8 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System; -using System.Collections.Generic; using DevOnBike.Overfit.LanguageModels; using DevOnBike.Overfit.LanguageModels.Skills.Evaluation; using DevOnBike.Overfit.Tests.TestSupport; diff --git a/Tests/LanguageModels/Skills/Evaluation/SkillEvaluatorTests.cs b/Tests/LanguageModels/Skills/Evaluation/SkillEvaluatorTests.cs index 93dcb621..006d4f7c 100644 --- a/Tests/LanguageModels/Skills/Evaluation/SkillEvaluatorTests.cs +++ b/Tests/LanguageModels/Skills/Evaluation/SkillEvaluatorTests.cs @@ -3,8 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System; -using System.Collections.Generic; using DevOnBike.Overfit.LanguageModels.Skills.Evaluation; namespace DevOnBike.Overfit.Tests.LanguageModels.Skills.Evaluation diff --git a/Tests/LanguageModels/Skills/Optimization/SkillOptimizerTests.cs b/Tests/LanguageModels/Skills/Optimization/SkillOptimizerTests.cs index 80944506..93cc521d 100644 --- a/Tests/LanguageModels/Skills/Optimization/SkillOptimizerTests.cs +++ b/Tests/LanguageModels/Skills/Optimization/SkillOptimizerTests.cs @@ -3,9 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System; -using System.Collections.Generic; -using System.Linq; using DevOnBike.Overfit.LanguageModels.Skills.Evaluation; using DevOnBike.Overfit.LanguageModels.Skills.Optimization; diff --git a/Tests/LanguageModels/Tokenization/WordPieceTokenizerTests.cs b/Tests/LanguageModels/Tokenization/WordPieceTokenizerTests.cs index a96dc42c..d1ad8258 100644 --- a/Tests/LanguageModels/Tokenization/WordPieceTokenizerTests.cs +++ b/Tests/LanguageModels/Tokenization/WordPieceTokenizerTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; using DevOnBike.Overfit.LanguageModels.Tokenizers; using DevOnBike.Overfit.Tests.TestSupport; diff --git a/Tests/LanguageModels/Whisper/WhisperGgmlLoaderTests.cs b/Tests/LanguageModels/Whisper/WhisperGgmlLoaderTests.cs index f4ce9f83..1115ff6b 100644 --- a/Tests/LanguageModels/Whisper/WhisperGgmlLoaderTests.cs +++ b/Tests/LanguageModels/Whisper/WhisperGgmlLoaderTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.IO; using System.Text; using DevOnBike.Overfit.LanguageModels.Whisper; diff --git a/Tests/Optimizers/AdamCheckpointTests.cs b/Tests/Optimizers/AdamCheckpointTests.cs index 5a87cd87..e5bdae33 100644 --- a/Tests/Optimizers/AdamCheckpointTests.cs +++ b/Tests/Optimizers/AdamCheckpointTests.cs @@ -3,8 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System; -using System.IO; using DevOnBike.Overfit.Autograd; using DevOnBike.Overfit.Optimizers; using DevOnBike.Overfit.Tensors; diff --git a/Tests/Redaction/AllowlistAndEntropyTests.cs b/Tests/Redaction/AllowlistAndEntropyTests.cs index 26052910..d69fe294 100644 --- a/Tests/Redaction/AllowlistAndEntropyTests.cs +++ b/Tests/Redaction/AllowlistAndEntropyTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Linq; using System.Text.RegularExpressions; using DevOnBike.Overfit.Redaction; diff --git a/Tests/Redaction/GatewayConfigTests.cs b/Tests/Redaction/GatewayConfigTests.cs index 21e9de92..bafcddf7 100644 --- a/Tests/Redaction/GatewayConfigTests.cs +++ b/Tests/Redaction/GatewayConfigTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Linq; using DevOnBike.Overfit.Redaction; using DevOnBike.Overfit.Server; diff --git a/Tests/Redaction/RedactionGatewayAuthTests.cs b/Tests/Redaction/RedactionGatewayAuthTests.cs index 3b49b8b4..f9dc39c6 100644 --- a/Tests/Redaction/RedactionGatewayAuthTests.cs +++ b/Tests/Redaction/RedactionGatewayAuthTests.cs @@ -4,7 +4,6 @@ // For commercial licensing options, contact: devonbike@gmail.com using System.Net; -using System.Net.Http; using System.Net.Sockets; using System.Text; using DevOnBike.Overfit.Redaction; diff --git a/Tests/Redaction/RedactionGatewayE2ETests.cs b/Tests/Redaction/RedactionGatewayE2ETests.cs index a399be5f..4add6c27 100644 --- a/Tests/Redaction/RedactionGatewayE2ETests.cs +++ b/Tests/Redaction/RedactionGatewayE2ETests.cs @@ -5,7 +5,6 @@ using System.Diagnostics; using System.Net; -using System.Net.Http; using System.Net.Sockets; using System.Text; using Xunit.Abstractions; diff --git a/Tests/Redaction/RedactionGatewayEndpointsTests.cs b/Tests/Redaction/RedactionGatewayEndpointsTests.cs index b239402e..6fea913b 100644 --- a/Tests/Redaction/RedactionGatewayEndpointsTests.cs +++ b/Tests/Redaction/RedactionGatewayEndpointsTests.cs @@ -4,7 +4,6 @@ // For commercial licensing options, contact: devonbike@gmail.com using System.Net; -using System.Net.Http; using System.Net.Sockets; using System.Text; using DevOnBike.Overfit.Redaction; diff --git a/Tests/Redaction/RedactionGatewayHeaderForwardingTests.cs b/Tests/Redaction/RedactionGatewayHeaderForwardingTests.cs index 0a82e6bc..fbe952b2 100644 --- a/Tests/Redaction/RedactionGatewayHeaderForwardingTests.cs +++ b/Tests/Redaction/RedactionGatewayHeaderForwardingTests.cs @@ -5,7 +5,6 @@ using System.Collections.Specialized; using System.Net; -using System.Net.Http; using System.Net.Sockets; using System.Text; using DevOnBike.Overfit.Redaction; diff --git a/Tests/Redaction/RedactionGatewayResponseScanTests.cs b/Tests/Redaction/RedactionGatewayResponseScanTests.cs index ca78e631..617e4c91 100644 --- a/Tests/Redaction/RedactionGatewayResponseScanTests.cs +++ b/Tests/Redaction/RedactionGatewayResponseScanTests.cs @@ -4,7 +4,6 @@ // For commercial licensing options, contact: devonbike@gmail.com using System.Net; -using System.Net.Http; using System.Net.Sockets; using System.Text; using DevOnBike.Overfit.Redaction; diff --git a/Tests/Redaction/RedactionGatewayStreamingScanTests.cs b/Tests/Redaction/RedactionGatewayStreamingScanTests.cs index 36ed0440..82d724b7 100644 --- a/Tests/Redaction/RedactionGatewayStreamingScanTests.cs +++ b/Tests/Redaction/RedactionGatewayStreamingScanTests.cs @@ -4,7 +4,6 @@ // For commercial licensing options, contact: devonbike@gmail.com using System.Net; -using System.Net.Http; using System.Net.Sockets; using System.Text; using DevOnBike.Overfit.Redaction; diff --git a/Tests/Redaction/RedactionGatewayStreamingTests.cs b/Tests/Redaction/RedactionGatewayStreamingTests.cs index 2f820218..eff00af8 100644 --- a/Tests/Redaction/RedactionGatewayStreamingTests.cs +++ b/Tests/Redaction/RedactionGatewayStreamingTests.cs @@ -4,7 +4,6 @@ // For commercial licensing options, contact: devonbike@gmail.com using System.Net; -using System.Net.Http; using System.Net.Sockets; using System.Text; using DevOnBike.Overfit.Redaction; diff --git a/Tests/Redaction/ResponseScanTests.cs b/Tests/Redaction/ResponseScanTests.cs index b2d8fcef..e6e182d3 100644 --- a/Tests/Redaction/ResponseScanTests.cs +++ b/Tests/Redaction/ResponseScanTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Linq; using DevOnBike.Overfit.Redaction; namespace DevOnBike.Overfit.Tests.Redaction diff --git a/Tests/Redaction/StreamingResponseScannerTests.cs b/Tests/Redaction/StreamingResponseScannerTests.cs index 70028dae..547fd288 100644 --- a/Tests/Redaction/StreamingResponseScannerTests.cs +++ b/Tests/Redaction/StreamingResponseScannerTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Linq; using System.Text; using DevOnBike.Overfit.Redaction; diff --git a/Tests/Serving/ServingLoadReportTests.cs b/Tests/Serving/ServingLoadReportTests.cs index 43b2ab82..d2f7b612 100644 --- a/Tests/Serving/ServingLoadReportTests.cs +++ b/Tests/Serving/ServingLoadReportTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Collections.Generic; using DevOnBike.Overfit.Serving; namespace DevOnBike.Overfit.Tests.Serving diff --git a/Tests/TestSupport/Helpers/SafetensorsTestWriter.cs b/Tests/TestSupport/Helpers/SafetensorsTestWriter.cs index ae36ecb9..2cee9278 100644 --- a/Tests/TestSupport/Helpers/SafetensorsTestWriter.cs +++ b/Tests/TestSupport/Helpers/SafetensorsTestWriter.cs @@ -4,7 +4,6 @@ // For commercial licensing options, contact: devonbike@gmail.com using System.Buffers.Binary; -using System.Collections.Generic; using System.Text; namespace DevOnBike.Overfit.Tests.TestSupport.Helpers diff --git a/Tests/Training/DataParallelSessionTests.cs b/Tests/Training/DataParallelSessionTests.cs index 23c5d295..4986ea33 100644 --- a/Tests/Training/DataParallelSessionTests.cs +++ b/Tests/Training/DataParallelSessionTests.cs @@ -3,10 +3,8 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Linq; using DevOnBike.Overfit.DeepLearning; using DevOnBike.Overfit.Optimizers; -using DevOnBike.Overfit.Parameters; using DevOnBike.Overfit.Training; namespace DevOnBike.Overfit.Tests.Training diff --git a/Tests/Trees/XgboostParityTests.cs b/Tests/Trees/XgboostParityTests.cs index 41236fa4..fe3079a0 100644 --- a/Tests/Trees/XgboostParityTests.cs +++ b/Tests/Trees/XgboostParityTests.cs @@ -3,7 +3,6 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Globalization; using System.Text.Json; using DevOnBike.Overfit.Tests.TestSupport.Helpers; using DevOnBike.Overfit.Trees; diff --git a/docs/overfit-vs-dotllm.md b/docs/overfit-vs-dotllm.md new file mode 100644 index 00000000..6aa19066 --- /dev/null +++ b/docs/overfit-vs-dotllm.md @@ -0,0 +1,113 @@ +# Overfit vs dotLLM — measured + +> **Status:** internal bench note, 2026-07-23. Not linked from the README or `docs/README.md` by design — +> it names a specific competing project and carries caveats that don't belong in launch-facing copy. + +Two pure-C# CPU inference engines, measured through **one load driver over one protocol** (`overfit bench` +against each engine's OpenAI-compatible server) so neither reports its own scorecard. After the prompt-cache +work landed on branch `sauron`, Overfit leads on every axis a chat server is judged on — the caveats that +make the numbers trustworthy are stated as plainly as the wins. + +## Setup + +| | | +|---|---| +| **Model** | Qwen-2.5-3B-Instruct Q4_K_M (`C:\qwen3b\qwen.q4km.gguf`, same file both engines) | +| **Box** | Ryzen 9 9950X3D (Zen 5, 16 physical / 32 logical) | +| **Prompt** | ~25-token chat prompt | +| **Load** | 1 concurrent user, 32 tokens out | +| **Method** | ABAB interleaved, 5 rounds, all servers resident in one wall-clock window, model id verified per port | + +## Verdict + +Ratios are Overfit over dotLLM in its **default configuration** (prompt cache on); `>1` = Overfit ahead. + +| Metric | Overfit | dotLLM (cache on) | Edge | +|---|--:|--:|--:| +| **TTFT** (time to first token) | 0.6 ms | 39 ms | **65×** | +| **ITL** (inter-token latency) | 38.3 ms/tok | 43.0 ms/tok | **1.12×** | +| **E2E** (end to end) | 1190 ms | 1380 ms | **1.16×** | +| **Throughput** | 26.9 tok/s | 23.3 tok/s | **1.15×** | + +## Head to head + +dotLLM shown in both configurations — its default (prompt cache on) and its true cold prefill +(`--no-prompt-cache`), because comparing our prefill to their cache hit would measure a missing feature +rather than kernel quality. + +| Metric | Overfit | dotLLM · cache on | dotLLM · cold prefill | Edge vs default | +|---|--:|--:|--:|--:| +| TTFT | 0.6 ms | 39 ms | 825–992 ms | 65× ahead | +| ITL | 38.3 ms | 43.0 ms | ≈ 38 ms | 1.12× ahead | +| E2E | 1190 ms | 1380 ms | ≈ 2100 ms | 1.16× ahead | +| Throughput | 26.9 tok/s | 23.3 tok/s | ≈ 15 tok/s | 1.15× ahead | + +## What changed — the TTFT collapse + +Time to first token fell three orders of magnitude across one session, each step a separate mechanism, +each measured before the next was built: + +``` +session start ──▶ KV prompt cache ──▶ early token emit ──▶ logits cache + 197 ms 74.8 ms ~40 ms 0.6 ms +``` + +- **KV prompt cache** reuses the key/value state a previous turn already built instead of re-encoding the + shared prefix (`CachedLlamaSession.PrefillReusingCache`, `_cacheTokens` indexed by cache position; + truncation is O(1) bookkeeping). +- **Early token emit** puts each token on the wire *before* the forward pass that prepares the next one — + one whole weight-pass earlier (the `onSampled` hook in `GenerateNextToken` / `GenerateSpeculative`). +- **Logits cache** keeps the end-of-prompt logits, so a re-sent prompt runs **zero** passes + (`_promptLogits`, invalidated on eviction / prefix restore). + +Each was A/B-isolated in one process via env toggles (`OVERFIT_DISABLE_LOGITS_CACHE` / +`OVERFIT_DISABLE_EARLY_EMIT` / `OVERFIT_DISABLE_SPECULATIVE`). The logits cache measured **1.001×** on +decode — free. Speculation on vs off measured **1.002×** — not a factor either way. + +## Who wins where, and why + +**Overfit leads:** + +- **Prefill kernel — ~4.2×.** Cold prefill of a chat prompt: ~197 ms vs their ~825–992 ms. Each weight row + is read once from DRAM and amortised across all prompt rows (the batched projection dispatch). +- **Decode — 1.12×.** Both engines are near the memory-bandwidth ceiling here; the edge is real but small. +- **TTFT on a repeated prompt — 65×.** The full cache stack forwards nothing when the prompt matches. +- **Zero-alloc, no native binary.** Single Native-AOT executable, pure managed C#. + +**dotLLM's ground:** + +- **Prompt caching shipped first.** Its 39 ms default was a cache hit, not a prefill — a feature Overfit + lacked at the start of this session and now matches. +- **Steadier cold prefill under drift.** Its cold path is slower but its across-round variance was tighter + than ours on a loaded box. +- **GPU path exists.** dotLLM ships a CUDA backend; Overfit's public identity is CPU-only. + +## What makes these numbers trustworthy + +- **One box, one model, one prompt shape.** Not a basis for an unqualified "Overfit is faster than dotLLM" + — always name the model, hardware and prompt shape. +- **The 0.6 ms TTFT is the repeated-prompt shape** (retry, regenerate, load test). Real multi-turn chat + appends tokens each turn, taking the tail-prefill path: tens of ms, not sub-millisecond. ITL and E2E are + ordinary decode and hold in every shape. +- **dotLLM's 39 ms is its prompt cache, not its kernel.** Its true cold prefill is 825–992 ms; both are + shown so the comparison is like-for-like. +- **Every ratio is within-run, interleaved.** Cross-process before/after drifts up to ~30% on this box, so + all arms were measured in one window with the untouched arms as canaries; ratios are medians. + +## Retracted during this session — three measurements that were wrong + +Kept here on purpose: the failures are why the surviving numbers are trustworthy. + +- ~~"TTFT 6.2× worse than dotLLM."~~ **Measured Ollama**, not Overfit — port 11434 was already taken and the + load driver silently polled the wrong server. Caught by the model id (`…fp16`) and a 100 ms/token ITL that + only fp16 weights explain. → guard: verify `/v1/models` against the file you launched, use a free port. +- ~~"Decode 1.78× ahead."~~ dotLLM's arm was measured on **cold mmap rounds**; withdrawn once both arms were + warmed in one window. +- ~~"Logits cache regressed ITL 37→55 ms."~~ **Machine drift**, not the change — one round spiked to 55 in + both our arms at once while dotLLM moved too. The in-process ablation had already shown the change was free; + the cross-process comparison that flagged a "regression" was the mistake. + +--- + +*Full suite 1505 / 0 at time of measurement. Cache stack A/B-isolated per change. Numbers from the 5-round +interleave; ratios are medians, never cross-process.* From 0e6f424b167c9c83edc4f9ed6a8d163b040620aa Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Thu, 23 Jul 2026 21:41:23 +0200 Subject: [PATCH 34/37] asp.net --- Sources/Cli/Cli.csproj | 1 + Sources/Cli/Commands.cs | 48 ++-- Sources/Cli/Dockerfile | 6 +- Sources/Cli/Program.cs | 9 +- Sources/Server.AspNet/AspNetResponseSink.cs | 52 ++++ .../Server.AspNet/Endpoints/ChatEndpoints.cs | 44 ++++ .../Endpoints/EmbeddingsEndpoints.cs | 45 ++++ .../Endpoints/EndpointHelpers.cs | 42 ++++ .../Endpoints/ModelsEndpoints.cs | 25 ++ .../Endpoints/SpeechEndpoints.cs | 45 ++++ Sources/Server.AspNet/OverfitAspNetServer.cs | 87 +++++++ Sources/Server.AspNet/Server.AspNet.csproj | 32 +++ .../Services/IOpenAiInferenceService.cs | 44 ++++ .../Services/OverfitInferenceService.cs | 136 +++++++++++ .../Server/OpenAi/ChatCompletionExchange.cs | 191 +++++++++++++++ Sources/Server/OpenAi/ConsoleTraceObserver.cs | 38 +++ Sources/Server/OpenAi/EmbeddingsExchange.cs | 58 +++++ .../Server/OpenAi/HttpListenerResponseSink.cs | 48 ++++ .../Server/OpenAi/IChatExchangeObserver.cs | 38 +++ Sources/Server/OpenAi/IOpenAiResponseSink.cs | 47 ++++ Sources/Server/OpenAi/SpeechExchange.cs | 77 ++++++ Sources/Server/OverfitOpenAiServer.cs | 228 +----------------- 22 files changed, 1103 insertions(+), 238 deletions(-) create mode 100644 Sources/Server.AspNet/AspNetResponseSink.cs create mode 100644 Sources/Server.AspNet/Endpoints/ChatEndpoints.cs create mode 100644 Sources/Server.AspNet/Endpoints/EmbeddingsEndpoints.cs create mode 100644 Sources/Server.AspNet/Endpoints/EndpointHelpers.cs create mode 100644 Sources/Server.AspNet/Endpoints/ModelsEndpoints.cs create mode 100644 Sources/Server.AspNet/Endpoints/SpeechEndpoints.cs create mode 100644 Sources/Server.AspNet/OverfitAspNetServer.cs create mode 100644 Sources/Server.AspNet/Server.AspNet.csproj create mode 100644 Sources/Server.AspNet/Services/IOpenAiInferenceService.cs create mode 100644 Sources/Server.AspNet/Services/OverfitInferenceService.cs create mode 100644 Sources/Server/OpenAi/ChatCompletionExchange.cs create mode 100644 Sources/Server/OpenAi/ConsoleTraceObserver.cs create mode 100644 Sources/Server/OpenAi/EmbeddingsExchange.cs create mode 100644 Sources/Server/OpenAi/HttpListenerResponseSink.cs create mode 100644 Sources/Server/OpenAi/IChatExchangeObserver.cs create mode 100644 Sources/Server/OpenAi/IOpenAiResponseSink.cs create mode 100644 Sources/Server/OpenAi/SpeechExchange.cs diff --git a/Sources/Cli/Cli.csproj b/Sources/Cli/Cli.csproj index 244e2638..b73f681e 100644 --- a/Sources/Cli/Cli.csproj +++ b/Sources/Cli/Cli.csproj @@ -46,6 +46,7 @@ + diff --git a/Sources/Cli/Commands.cs b/Sources/Cli/Commands.cs index 46000db8..fcef18a5 100644 --- a/Sources/Cli/Commands.cs +++ b/Sources/Cli/Commands.cs @@ -18,6 +18,7 @@ using DevOnBike.Overfit.Redaction; using DevOnBike.Overfit.Runtime; using DevOnBike.Overfit.Server; +using DevOnBike.Overfit.Server.AspNet; using DevOnBike.Overfit.Serving; using DevOnBike.Overfit.Trees; @@ -151,7 +152,7 @@ private static int PullEmbedder(string repo) private const string DefaultSystemPrompt = "You are a concise, helpful assistant running locally in pure .NET."; - public static int Serve(string model, string host, int port, string? embedModel, string? ttsModel, string? ttsSnac, int sessions = 1) + public static int Serve(string model, string host, int port, string? embedModel, string? ttsModel, string? ttsSnac, int sessions = 1, bool httpListener = false) { var path = ModelCache.Resolve(model); if (path is null) @@ -257,6 +258,36 @@ public static int Serve(string model, string host, int port, string? embedModel, cts.Cancel(); }; + void PrintBanner(string baseUrl, string hostKind, bool servesExtras) + { + var embedEp = servesExtras && embedder is not null ? " | POST /v1/embeddings" : string.Empty; + var ttsEp = servesExtras && tts is not null ? " | POST /v1/audio/speech" : string.Empty; + Console.WriteLine(); + Console.WriteLine($"Overfit OpenAI-compatible server ({hostKind}) listening on {baseUrl}"); + Console.WriteLine($" model id: {modelName}"); + Console.WriteLine($" endpoints: GET /v1/models | POST /v1/chat/completions (stream + non-stream){embedEp}{ttsEp} | GET /health"); + Console.WriteLine(); + Console.WriteLine($" curl {baseUrl}/v1/chat/completions -H \"Content-Type: application/json\" \\"); + Console.WriteLine($" -d '{{\"model\":\"{modelName}\",\"messages\":[{{\"role\":\"user\",\"content\":\"Hello\"}}]}}'"); + Console.WriteLine(); + Console.WriteLine("Press Ctrl+C to stop."); + } + + // Default: the AOT-ready ASP.NET (Kestrel) host. The dependency-free HttpListener server is kept + // only behind --http-listener as a legacy escape hatch. + if (!httpListener) + { + OverfitAspNetServer.Serve( + pool, modelName, host, port, DefaultSystemPrompt, embedder, tts, + onListening: baseUrl => PrintBanner(baseUrl, "ASP.NET / Kestrel, AOT-ready", servesExtras: true), + cancellationToken: cts.Token); + + embedder?.Dispose(); + tts?.Dispose(); + pool.Dispose(); + return 0; + } + try { OverfitOpenAiServer.Serve( @@ -267,20 +298,7 @@ public static int Serve(string model, string host, int port, string? embedModel, DefaultSystemPrompt, embedder, tts, - onListening: baseUrl => - { - var embedEp = embedder is null ? string.Empty : " | POST /v1/embeddings"; - var ttsEp = tts is null ? string.Empty : " | POST /v1/audio/speech"; - Console.WriteLine(); - Console.WriteLine($"Overfit OpenAI-compatible server listening on {baseUrl}"); - Console.WriteLine($" model id: {modelName}"); - Console.WriteLine($" endpoints: GET /v1/models | POST /v1/chat/completions (stream + non-stream){embedEp}{ttsEp} | GET /health"); - Console.WriteLine(); - Console.WriteLine($" curl {baseUrl}/v1/chat/completions -H \"Content-Type: application/json\" \\"); - Console.WriteLine($" -d '{{\"model\":\"{modelName}\",\"messages\":[{{\"role\":\"user\",\"content\":\"Hello\"}}]}}'"); - Console.WriteLine(); - Console.WriteLine("Press Ctrl+C to stop."); - }, + onListening: baseUrl => PrintBanner(baseUrl, "HttpListener (legacy)", servesExtras: true), cancellationToken: cts.Token); } catch (HttpListenerException ex) diff --git a/Sources/Cli/Dockerfile b/Sources/Cli/Dockerfile index bd7323da..340803e5 100644 --- a/Sources/Cli/Dockerfile +++ b/Sources/Cli/Dockerfile @@ -1,7 +1,9 @@ # Native-AOT build of the `overfit` CLI, exposing the OpenAI-compatible HTTP server (`overfit serve`). # -# Identity: a single self-contained NATIVE binary (no .NET runtime, no ASP.NET) on a chiselled base — -# tiny image, fast cold start. The model is NOT baked in (GGUF files are large + licensed separately): +# Identity: a single self-contained NATIVE binary (no .NET runtime) on a chiselled base — tiny image, +# fast cold start. `overfit serve` runs the AOT-ready ASP.NET (Kestrel + Minimal API) host by default; +# the whole ASP.NET Core graph is Native-AOT compiled into this binary (verified by the aot-guard CI job). +# The model is NOT baked in (GGUF files are large + licensed separately): # provide it at runtime via a mounted volume, e.g. # # docker build -f Sources/Cli/Dockerfile -t overfit . # build context = REPO ROOT diff --git a/Sources/Cli/Program.cs b/Sources/Cli/Program.cs index 687e96a1..65a60a14 100644 --- a/Sources/Cli/Program.cs +++ b/Sources/Cli/Program.cs @@ -99,6 +99,11 @@ + "(serialized, like llama.cpp). N>1 decodes N chats at once at the cost of N× KV-cache RAM.", DefaultValueFactory = _ => 1, }; +var serveHttpListener = new Option("--http-listener") +{ + Description = "Legacy: serve through the dependency-free HttpListener server instead of the default " + + "AOT-ready ASP.NET (Kestrel) host. Kept as an escape hatch; the Kestrel host is the shipping default.", +}; var serveCommand = new Command("serve", "Start an OpenAI-compatible HTTP server for a model.") { serveModel, @@ -108,6 +113,7 @@ serveTtsModel, serveTtsSnac, serveSessions, + serveHttpListener, }; serveCommand.SetAction(parseResult => Commands.Serve( parseResult.GetValue(serveModel)!, @@ -116,7 +122,8 @@ parseResult.GetValue(serveEmbedModel), parseResult.GetValue(serveTtsModel), parseResult.GetValue(serveTtsSnac), - parseResult.GetValue(serveSessions))); + parseResult.GetValue(serveSessions), + parseResult.GetValue(serveHttpListener))); // ── tts: text → speech (WAV), in-process, watermarked. Placeholder engine until the neural backend lands. ── var ttsText = new Option("--text") diff --git a/Sources/Server.AspNet/AspNetResponseSink.cs b/Sources/Server.AspNet/AspNetResponseSink.cs new file mode 100644 index 00000000..0f6c2b12 --- /dev/null +++ b/Sources/Server.AspNet/AspNetResponseSink.cs @@ -0,0 +1,52 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Text; +using DevOnBike.Overfit.Server.OpenAi; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.Http.Features; + +namespace DevOnBike.Overfit.Server.AspNet +{ + /// + /// Adapts an ASP.NET to so the Kestrel host + /// drives the shared unchanged. Writes are synchronous — the + /// exchange streams tokens from a synchronous generate callback — so synchronous body IO is enabled on + /// entering the stream; Kestrel disallows it by default. + /// + internal sealed class AspNetResponseSink : IOpenAiResponseSink + { + private readonly HttpResponse _response; + + public AspNetResponseSink(HttpResponse response) => _response = response; + + public void WriteBody(int statusCode, string contentType, string body) + => WriteBinary(statusCode, contentType, Encoding.UTF8.GetBytes(body)); + + public void WriteBinary(int statusCode, string contentType, byte[] body) + { + _response.StatusCode = statusCode; + _response.ContentType = contentType; + _response.ContentLength = body.Length; + _response.Body.Write(body, 0, body.Length); + } + + public void BeginEventStream() + { + // Synchronous body IO (the exchange writes each token from the decode callback) is enabled by the + // endpoint before the exchange runs; here we only set the SSE headers. + _response.StatusCode = StatusCodes.Status200OK; + _response.ContentType = "text/event-stream"; + _response.Headers.CacheControl = "no-cache"; + } + + public void WriteEvent(string data) + { + var bytes = Encoding.UTF8.GetBytes($"data: {data}\n\n"); + _response.Body.Write(bytes, 0, bytes.Length); + _response.Body.Flush(); + } + } +} diff --git a/Sources/Server.AspNet/Endpoints/ChatEndpoints.cs b/Sources/Server.AspNet/Endpoints/ChatEndpoints.cs new file mode 100644 index 00000000..7415111a --- /dev/null +++ b/Sources/Server.AspNet/Endpoints/ChatEndpoints.cs @@ -0,0 +1,44 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Text.Json; +using DevOnBike.Overfit.Server.AspNet.Services; +using DevOnBike.Overfit.Server.OpenAi; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.Routing; + +namespace DevOnBike.Overfit.Server.AspNet.Endpoints +{ + /// + /// POST /v1/chat/completions (streaming SSE + non-streaming). Thin: reads the body and hands it to + /// the injected ; all session pooling and generation live there. + /// + internal static class ChatEndpoints + { + public static RouteGroupBuilder MapChat(this RouteGroupBuilder v1) + { + v1.MapPost("/chat/completions", async (HttpContext ctx, IOpenAiInferenceService service) => + { + ChatCompletionRequest? req; + try + { + req = await JsonSerializer.DeserializeAsync( + ctx.Request.Body, OpenAiJsonContext.Default.ChatCompletionRequest, ctx.RequestAborted); + } + catch (JsonException ex) + { + EndpointHelpers.WriteError(ctx.Response, StatusCodes.Status400BadRequest, $"invalid JSON body: {ex.Message}"); + return; + } + + EndpointHelpers.EnableSynchronousIO(ctx); + service.CompleteChat(req, new AspNetResponseSink(ctx.Response), ctx.RequestAborted); + }); + + return v1; + } + } +} diff --git a/Sources/Server.AspNet/Endpoints/EmbeddingsEndpoints.cs b/Sources/Server.AspNet/Endpoints/EmbeddingsEndpoints.cs new file mode 100644 index 00000000..adc12b71 --- /dev/null +++ b/Sources/Server.AspNet/Endpoints/EmbeddingsEndpoints.cs @@ -0,0 +1,45 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Text.Json; +using DevOnBike.Overfit.Server.AspNet.Services; +using DevOnBike.Overfit.Server.OpenAi; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.Routing; + +namespace DevOnBike.Overfit.Server.AspNet.Endpoints +{ + /// + /// POST /v1/embeddings — in-process sentence embeddings (nothing leaves the box). Thin: reads the + /// body and delegates to the injected , which returns 501 through the + /// sink when no embedding model was loaded. + /// + internal static class EmbeddingsEndpoints + { + public static RouteGroupBuilder MapEmbeddings(this RouteGroupBuilder v1) + { + v1.MapPost("/embeddings", async (HttpContext ctx, IOpenAiInferenceService service) => + { + EmbeddingsRequest? req; + try + { + req = await JsonSerializer.DeserializeAsync( + ctx.Request.Body, OpenAiJsonContext.Default.EmbeddingsRequest, ctx.RequestAborted); + } + catch (JsonException ex) + { + EndpointHelpers.WriteError(ctx.Response, StatusCodes.Status400BadRequest, $"invalid JSON body: {ex.Message}"); + return; + } + + EndpointHelpers.EnableSynchronousIO(ctx); + service.Embed(req, new AspNetResponseSink(ctx.Response), ctx.RequestAborted); + }); + + return v1; + } + } +} diff --git a/Sources/Server.AspNet/Endpoints/EndpointHelpers.cs b/Sources/Server.AspNet/Endpoints/EndpointHelpers.cs new file mode 100644 index 00000000..58cdaf18 --- /dev/null +++ b/Sources/Server.AspNet/Endpoints/EndpointHelpers.cs @@ -0,0 +1,42 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Text; +using System.Text.Json; +using DevOnBike.Overfit.Server.OpenAi; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.Http.Features; + +namespace DevOnBike.Overfit.Server.AspNet.Endpoints +{ + /// + /// Shared plumbing for the route-group endpoints: enabling synchronous body IO (the shared exchanges + /// write synchronously, and Kestrel disallows it by default) and writing an OpenAI-shaped error before an + /// exchange runs. Serialization stays on the source-gen , so the whole + /// endpoint surface remains reflection-free for Native AOT. + /// + internal static class EndpointHelpers + { + public static void EnableSynchronousIO(HttpContext ctx) + { + var bodyControl = ctx.Features.Get(); + if (bodyControl is not null) + { + bodyControl.AllowSynchronousIO = true; + } + } + + public static void WriteError(HttpResponse response, int status, string message) + { + var body = new OpenAiErrorResponse { Error = new OpenAiError { Message = message } }; + var json = JsonSerializer.Serialize(body, OpenAiJsonContext.Default.OpenAiErrorResponse); + var bytes = Encoding.UTF8.GetBytes(json); + response.StatusCode = status; + response.ContentType = "application/json"; + response.ContentLength = bytes.Length; + response.Body.WriteAsync(bytes).AsTask().GetAwaiter().GetResult(); + } + } +} diff --git a/Sources/Server.AspNet/Endpoints/ModelsEndpoints.cs b/Sources/Server.AspNet/Endpoints/ModelsEndpoints.cs new file mode 100644 index 00000000..3b7ee6d6 --- /dev/null +++ b/Sources/Server.AspNet/Endpoints/ModelsEndpoints.cs @@ -0,0 +1,25 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.Server.AspNet.Services; +using DevOnBike.Overfit.Server.OpenAi; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.Routing; + +namespace DevOnBike.Overfit.Server.AspNet.Endpoints +{ + /// GET /v1/models — the served model's id, so OpenAI clients can discover it. + internal static class ModelsEndpoints + { + public static RouteGroupBuilder MapModels(this RouteGroupBuilder v1) + { + v1.MapGet("/models", (IOpenAiInferenceService service) => + Results.Json(service.ListModels(), OpenAiJsonContext.Default.ModelsResponse)); + + return v1; + } + } +} diff --git a/Sources/Server.AspNet/Endpoints/SpeechEndpoints.cs b/Sources/Server.AspNet/Endpoints/SpeechEndpoints.cs new file mode 100644 index 00000000..3dd4efba --- /dev/null +++ b/Sources/Server.AspNet/Endpoints/SpeechEndpoints.cs @@ -0,0 +1,45 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Text.Json; +using DevOnBike.Overfit.Server.AspNet.Services; +using DevOnBike.Overfit.Server.OpenAi; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.Routing; + +namespace DevOnBike.Overfit.Server.AspNet.Endpoints +{ + /// + /// POST /v1/audio/speech — in-process text-to-speech (WAV / PCM). Thin: reads the body and + /// delegates to the injected , which returns 501 through the sink + /// when no TTS model was loaded. + /// + internal static class SpeechEndpoints + { + public static RouteGroupBuilder MapSpeech(this RouteGroupBuilder v1) + { + v1.MapPost("/audio/speech", async (HttpContext ctx, IOpenAiInferenceService service) => + { + SpeechRequest? req; + try + { + req = await JsonSerializer.DeserializeAsync( + ctx.Request.Body, OpenAiJsonContext.Default.SpeechRequest, ctx.RequestAborted); + } + catch (JsonException ex) + { + EndpointHelpers.WriteError(ctx.Response, StatusCodes.Status400BadRequest, $"invalid JSON body: {ex.Message}"); + return; + } + + EndpointHelpers.EnableSynchronousIO(ctx); + service.Synthesize(req, new AspNetResponseSink(ctx.Response), ctx.RequestAborted); + }); + + return v1; + } + } +} diff --git a/Sources/Server.AspNet/OverfitAspNetServer.cs b/Sources/Server.AspNet/OverfitAspNetServer.cs new file mode 100644 index 00000000..9a3c4393 --- /dev/null +++ b/Sources/Server.AspNet/OverfitAspNetServer.cs @@ -0,0 +1,87 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.Audio.Tts.Orpheus; +using DevOnBike.Overfit.LanguageModels; +using DevOnBike.Overfit.LanguageModels.Embeddings; +using DevOnBike.Overfit.Server.AspNet.Endpoints; +using DevOnBike.Overfit.Server.AspNet.Services; +using DevOnBike.Overfit.Server.OpenAi; +using DevOnBike.Overfit.Serving; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Http; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; + +namespace DevOnBike.Overfit.Server.AspNet +{ + /// + /// The AOT-ready ASP.NET (Kestrel + Minimal API) host for Overfit's OpenAI-compatible server — the server + /// that ships in the NuGet package and the Docker image. Routing goes through the Request Delegate + /// Generator (no reflection), JSON through the source-gen , and the request + /// logic through a DI-resolved , so the whole surface publishes under + /// Native AOT while staying organized like a controller app: a /v1 route group with one endpoint + /// class per resource, all delegating to one service. + /// + public static class OverfitAspNetServer + { + /// + /// Binds Kestrel on : and serves until + /// is cancelled. Blocks the calling thread. Chat rents from + /// (up to pool.Size decode concurrently, HTTP 503 when exhausted); + /// embeddings and TTS are served when / are supplied + /// (501 otherwise). The pool, embedder and TTS engine are owned by the caller. + /// + public static void Serve( + OverfitResourcePool pool, + string modelName, + string host, + int port, + string systemMessage, + SentenceEmbedder? embedder = null, + OrpheusVoiceEngine? tts = null, + Action? onListening = null, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(pool); + + using var service = new OverfitInferenceService(pool, modelName, systemMessage, embedder, tts); + + var builder = WebApplication.CreateSlimBuilder(); + + // The CLI owns the console (it prints the banner via onListening); keep Kestrel's own startup + // logging off the wire so `overfit serve` output stays clean. + builder.Logging.ClearProviders(); + + // Bind and serialize every OpenAI DTO through the source-gen context — the reflection-free path + // Native AOT requires. + builder.Services.ConfigureHttpJsonOptions(options => + options.SerializerOptions.TypeInfoResolverChain.Insert(0, OpenAiJsonContext.Default)); + + builder.Services.AddSingleton(service); + + var app = builder.Build(); + + app.MapGet("/health", () => Results.Text("ok", "text/plain")); + app.MapGet("/", () => Results.Text("ok", "text/plain")); + + // The OpenAI surface as a versioned route group, one endpoint class per resource. + var v1 = app.MapGroup("/v1"); + v1.MapModels(); + v1.MapChat(); + v1.MapEmbeddings(); + v1.MapSpeech(); + + app.Lifetime.ApplicationStarted.Register(() => onListening?.Invoke($"http://{host}:{port}")); + + // Discarding the shutdown Task is intentional (StopAsync is fire-and-forget on cancel); + // `_ =` keeps CS4014 — promoted to error repo-wide — from tripping. + using var reg = cancellationToken.Register(() => _ = app.StopAsync()); + + app.Run($"http://{host}:{port}"); + } + } +} diff --git a/Sources/Server.AspNet/Server.AspNet.csproj b/Sources/Server.AspNet/Server.AspNet.csproj new file mode 100644 index 00000000..9b64e4b9 --- /dev/null +++ b/Sources/Server.AspNet/Server.AspNet.csproj @@ -0,0 +1,32 @@ + + + + + + net10.0 + enable + enable + true + true + DevOnBike.Overfit.Server.AspNet + DevOnBike.Overfit.Server.AspNet + + + + + + + + + + diff --git a/Sources/Server.AspNet/Services/IOpenAiInferenceService.cs b/Sources/Server.AspNet/Services/IOpenAiInferenceService.cs new file mode 100644 index 00000000..d24d1899 --- /dev/null +++ b/Sources/Server.AspNet/Services/IOpenAiInferenceService.cs @@ -0,0 +1,44 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.Server.OpenAi; + +namespace DevOnBike.Overfit.Server.AspNet.Services +{ + /// + /// The inference operations behind the OpenAI-compatible endpoints, resolved from DI so the Minimal-API + /// endpoints stay thin (parse the body, hand it here) and the logic — session pooling, concurrency gating, + /// the shared protocol exchanges, availability of embeddings/TTS — lives in one testable place. + /// + /// Transport-neutral by design: every method writes through an and + /// never touches an ASP.NET type, so the same service is unit-testable against a fake sink and could back + /// a non-Kestrel host unchanged. Methods are synchronous — they drive the synchronous, zero-allocation + /// decode path — and take the request's for the rent/gate waits. + /// + public interface IOpenAiInferenceService + { + /// The served model's id, for GET /v1/models. + ModelsResponse ListModels(); + + /// + /// Runs one chat completion (streaming or not) — rents a session, replays history, generates, restores + /// the baseline system turn — writing the whole response through . Sheds with + /// HTTP 503 through the sink when the pool is exhausted. + /// + void CompleteChat(ChatCompletionRequest? request, IOpenAiResponseSink sink, CancellationToken cancellationToken); + + /// + /// Embeds the request's input in-process. Writes HTTP 501 through when no + /// embedding model was loaded. + /// + void Embed(EmbeddingsRequest? request, IOpenAiResponseSink sink, CancellationToken cancellationToken); + + /// + /// Synthesizes speech (WAV / PCM). Writes HTTP 501 through when no TTS model + /// was loaded. + /// + void Synthesize(SpeechRequest? request, IOpenAiResponseSink sink, CancellationToken cancellationToken); + } +} diff --git a/Sources/Server.AspNet/Services/OverfitInferenceService.cs b/Sources/Server.AspNet/Services/OverfitInferenceService.cs new file mode 100644 index 00000000..26bb8152 --- /dev/null +++ b/Sources/Server.AspNet/Services/OverfitInferenceService.cs @@ -0,0 +1,136 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Text.Json; +using DevOnBike.Overfit.Audio.Tts.Orpheus; +using DevOnBike.Overfit.LanguageModels; +using DevOnBike.Overfit.LanguageModels.Embeddings; +using DevOnBike.Overfit.Runtime; +using DevOnBike.Overfit.Server.OpenAi; +using DevOnBike.Overfit.Serving; + +namespace DevOnBike.Overfit.Server.AspNet.Services +{ + /// + /// The default : holds the session pool plus the optional embedder / + /// TTS engine, serializes access to the single-instance embedder and TTS engine, and drives the shared + /// / / . + /// The pool, embedder and TTS engine are owned by the caller (the CLI disposes them); this service owns + /// only the two gates. + /// + public sealed class OverfitInferenceService : IOpenAiInferenceService, IDisposable + { + private static readonly TimeSpan RentTimeout = TimeSpan.FromSeconds(30); + + private static readonly bool Trace = + Environment.GetEnvironmentVariable(OverfitEnvironment.ServerTrace) == "1"; + + private readonly OverfitResourcePool _pool; + private readonly string _modelName; + private readonly string _systemMessage; + private readonly long _created; + private readonly SentenceEmbedder? _embedder; + private readonly OrpheusVoiceEngine? _tts; + + // A SentenceEmbedder has one scratch arena; the TTS engine is single-instance. Serialize each. + private readonly SemaphoreSlim _embedGate = new(1, 1); + private readonly SemaphoreSlim _ttsGate = new(1, 1); + + public OverfitInferenceService( + OverfitResourcePool pool, + string modelName, + string systemMessage, + SentenceEmbedder? embedder, + OrpheusVoiceEngine? tts) + { + _pool = pool ?? throw new ArgumentNullException(nameof(pool)); + _modelName = modelName; + _systemMessage = systemMessage; + _embedder = embedder; + _tts = tts; + _created = DateTimeOffset.UtcNow.ToUnixTimeSeconds(); + } + + public ModelsResponse ListModels() + => new() { Data = [new ModelInfo { Id = _modelName, Created = _created }] }; + + public void CompleteChat(ChatCompletionRequest? request, IOpenAiResponseSink sink, CancellationToken cancellationToken) + { + OverfitResourcePool.Lease lease; + try + { + if (!_pool.TryRent(RentTimeout, cancellationToken, out lease)) + { + WriteError(sink, 503, $"server busy — all {_pool.Size} sessions in use; retry shortly."); + return; + } + } + catch (OperationCanceledException) + { + WriteError(sink, 503, "server is shutting down."); + return; + } + + using (lease) + { + var observer = Trace ? ConsoleTraceObserver.Instance : null; + ChatCompletionExchange.Handle(request, lease.Value, _modelName, _systemMessage, sink, observer); + } + } + + public void Embed(EmbeddingsRequest? request, IOpenAiResponseSink sink, CancellationToken cancellationToken) + { + if (_embedder is null) + { + WriteError(sink, 501, "embeddings are not served — start with an embedding model " + + "(e.g. 'overfit serve --embed-model ')."); + return; + } + + _embedGate.Wait(cancellationToken); + try + { + EmbeddingsExchange.Handle(request, _embedder, _modelName, sink); + } + finally + { + _embedGate.Release(); + } + } + + public void Synthesize(SpeechRequest? request, IOpenAiResponseSink sink, CancellationToken cancellationToken) + { + if (_tts is null) + { + WriteError(sink, 501, "text-to-speech is not served — start with a TTS model " + + "(e.g. 'overfit serve --tts-model --tts-snac ')."); + return; + } + + _ttsGate.Wait(cancellationToken); + try + { + SpeechExchange.Handle(request, _tts, sink); + } + finally + { + _ttsGate.Release(); + } + } + + public void Dispose() + { + _embedGate.Dispose(); + _ttsGate.Dispose(); + } + + private static void WriteError(IOpenAiResponseSink sink, int status, string message) + { + var body = new OpenAiErrorResponse { Error = new OpenAiError { Message = message } }; + sink.WriteBody(status, "application/json", + JsonSerializer.Serialize(body, OpenAiJsonContext.Default.OpenAiErrorResponse)); + } + } +} diff --git a/Sources/Server/OpenAi/ChatCompletionExchange.cs b/Sources/Server/OpenAi/ChatCompletionExchange.cs new file mode 100644 index 00000000..315acb15 --- /dev/null +++ b/Sources/Server/OpenAi/ChatCompletionExchange.cs @@ -0,0 +1,191 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Text.Json; +using DevOnBike.Overfit.Diagnostics; +using DevOnBike.Overfit.LanguageModels; +using DevOnBike.Overfit.LanguageModels.Contracts; + +namespace DevOnBike.Overfit.Server.OpenAi +{ + /// + /// The one implementation of POST /v1/chat/completions, shared by every host. It owns the whole + /// OpenAI protocol — validating the request, mapping sampling / response-format, replaying history, + /// running the streaming and non-streaming generation, computing finish_reason, and restoring the + /// baseline system turn — and writes exclusively through an , so a host + /// contributes only a byte-output adapter plus an optional . + /// + /// Before this existed the orchestration lived twice (the HttpListener CLI server and the + /// ASP.NET host), and the two drifted the moment either gained a feature the other lacked — the TTFT + /// phase trace was added to one and the prompt-cache reuse count reported by only one. Keeping it in one + /// place is the point. + /// + /// Session lifetime is the caller's. This method neither rents from a pool nor holds a + /// concurrency gate — a session decodes one request at a time, and serializing access to it is the host's + /// job (the CLI rents a pooled client; the ASP.NET host holds a single-flight semaphore). The handler only + /// resets the session afterwards so the next caller starts clean. + /// + public static class ChatCompletionExchange + { + /// + /// Runs one chat-completion exchange to completion, writing the whole response through + /// . may be null (a body that failed to parse) — + /// it is validated here so every host rejects malformed input identically. + /// + public static void Handle( + ChatCompletionRequest? req, + OverfitClient client, + string modelName, + string systemMessage, + IOpenAiResponseSink sink, + IChatExchangeObserver? observer = null) + { + ArgumentNullException.ThrowIfNull(client); + ArgumentNullException.ThrowIfNull(sink); + + if (req is null || req.Messages is not { Count: > 0 }) + { + WriteError(sink, 400, "'messages' is required and must be non-empty."); + return; + } + + var last = req.Messages[^1]; + if (!string.Equals(last.Role, "user", StringComparison.OrdinalIgnoreCase)) + { + WriteError(sink, 400, "the last message must have role 'user'."); + return; + } + + var (sampling, maxTokens) = OpenAiChatMapping.BuildSampling(req); + var options = new GenerationOptions(maxTokens, maxContextLength: 8192, sampling, stopOnEndOfTextToken: true); + var id = "chatcmpl-" + Guid.NewGuid().ToString("N"); + var ts = DateTimeOffset.UtcNow.ToUnixTimeSeconds(); + + ITokenConstraint? constraint; + try + { + constraint = OpenAiChatMapping.BuildResponseFormatConstraint(req.ResponseFormat, client.Tokenizer); + } + catch (JsonException ex) + { + WriteError(sink, 400, $"invalid response_format: {ex.Message}"); + return; + } + + try + { + var replayStarted = observer is null ? default : ValueStopwatch.StartNew(); + OpenAiChatMapping.ReplayHistory(client.Chat, req.Messages); + observer?.OnHistoryReplayed(req.Messages.Count, replayStarted.GetElapsedTime().TotalMilliseconds); + + var userContent = last.Content ?? string.Empty; + + if (!req.Stream) + { + HandleNonStreaming(client, modelName, id, ts, maxTokens, userContent, options, constraint, sink, observer); + return; + } + + HandleStreaming(client, modelName, id, ts, maxTokens, userContent, options, constraint, sink, observer); + } + finally + { + // Restore the baseline system turn so the shared single-tenant session stays clean for the + // next caller. The host owns concurrency; by here it still holds the session exclusively. + client.Reset(); + if (!string.IsNullOrEmpty(systemMessage)) + { + client.AddSystem(systemMessage); + } + } + } + + private static void HandleNonStreaming( + OverfitClient client, string modelName, string id, long ts, int maxTokens, + string userContent, GenerationOptions options, ITokenConstraint? constraint, + IOpenAiResponseSink sink, IChatExchangeObserver? observer) + { + var reply = client.Chat.Send(userContent, in options, onText: null, constraint: constraint); + var stats = client.Chat.LastStats; + + var response = new ChatCompletionResponse + { + Id = id, + Created = ts, + Model = modelName, + Choices = + [ + new ChatChoice + { + Index = 0, + Message = new OpenAiMessage { Role = "assistant", Content = reply }, + FinishReason = stats.GeneratedTokens >= maxTokens ? "length" : "stop", + }, + ], + Usage = new OpenAiUsage + { + PromptTokens = stats.PromptTokens, + CompletionTokens = stats.GeneratedTokens, + TotalTokens = stats.PromptTokens + stats.GeneratedTokens, + }, + }; + + var json = JsonSerializer.Serialize(response, OpenAiJsonContext.Default.ChatCompletionResponse); + sink.WriteBody(200, "application/json", json); + observer?.OnCompleted(streamed: false, stats, client.Chat.CachedPromptTokens); + } + + private static void HandleStreaming( + OverfitClient client, string modelName, string id, long ts, int maxTokens, + string userContent, GenerationOptions options, ITokenConstraint? constraint, + IOpenAiResponseSink sink, IChatExchangeObserver? observer) + { + sink.BeginEventStream(); + WriteChunk(sink, id, ts, modelName, new OpenAiMessage { Role = "assistant" }, finishReason: null); + + var sendStarted = observer is null ? default : ValueStopwatch.StartNew(); + var firstDelta = true; + + client.Chat.Send(userContent, in options, + onText: delta => + { + if (observer is not null && firstDelta) + { + firstDelta = false; + observer.OnFirstToken(sendStarted.GetElapsedTime().TotalMilliseconds); + } + + WriteChunk(sink, id, ts, modelName, new OpenAiMessage { Content = delta }, finishReason: null); + }, + constraint: constraint); + + var stats = client.Chat.LastStats; + var finish = stats.GeneratedTokens >= maxTokens ? "length" : "stop"; + WriteChunk(sink, id, ts, modelName, new OpenAiMessage(), finishReason: finish); + sink.WriteEvent("[DONE]"); + observer?.OnCompleted(streamed: true, stats, client.Chat.CachedPromptTokens); + } + + private static void WriteChunk( + IOpenAiResponseSink sink, string id, long created, string model, OpenAiMessage delta, string? finishReason) + { + var chunk = new ChatCompletionChunk + { + Id = id, + Created = created, + Model = model, + Choices = [new ChatChoice { Index = 0, Delta = delta, FinishReason = finishReason }], + }; + sink.WriteEvent(JsonSerializer.Serialize(chunk, OpenAiJsonContext.Default.ChatCompletionChunk)); + } + + private static void WriteError(IOpenAiResponseSink sink, int status, string message) + { + var body = new OpenAiErrorResponse { Error = new OpenAiError { Message = message } }; + var json = JsonSerializer.Serialize(body, OpenAiJsonContext.Default.OpenAiErrorResponse); + sink.WriteBody(status, "application/json", json); + } + } +} diff --git a/Sources/Server/OpenAi/ConsoleTraceObserver.cs b/Sources/Server/OpenAi/ConsoleTraceObserver.cs new file mode 100644 index 00000000..4320b184 --- /dev/null +++ b/Sources/Server/OpenAi/ConsoleTraceObserver.cs @@ -0,0 +1,38 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.LanguageModels.Contracts; + +namespace DevOnBike.Overfit.Server.OpenAi +{ + /// + /// The opt-in per-request phase trace (OVERFIT_SERVER_TRACE=1) as an + /// : prints replay time, server-side time-to-first-token, and how many + /// prompt tokens the KV cache reused. Split out of the CLI server so the same trace works behind any host + /// that drives . + /// + /// Timings arrive already elapsed, so this never reads a clock; a single shared instance is safe + /// because the server decodes one request at a time. + /// + public sealed class ConsoleTraceObserver : IChatExchangeObserver + { + public static readonly ConsoleTraceObserver Instance = new(); + + public void OnHistoryReplayed(int messageCount, double elapsedMs) + { + Console.WriteLine($"[trace] replay {elapsedMs:F1} ms ({messageCount} message(s))"); + } + + public void OnFirstToken(double elapsedMs) + { + Console.WriteLine($"[trace] first token {elapsedMs:F1} ms"); + } + + public void OnCompleted(bool streamed, GenerationStats stats, int cachedPromptTokens) + { + Console.WriteLine($"[trace] prompt {stats.PromptTokens} tok, {cachedPromptTokens} reused from the KV cache"); + } + } +} diff --git a/Sources/Server/OpenAi/EmbeddingsExchange.cs b/Sources/Server/OpenAi/EmbeddingsExchange.cs new file mode 100644 index 00000000..8f81d096 --- /dev/null +++ b/Sources/Server/OpenAi/EmbeddingsExchange.cs @@ -0,0 +1,58 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Text.Json; +using DevOnBike.Overfit.LanguageModels.Embeddings; + +namespace DevOnBike.Overfit.Server.OpenAi +{ + /// + /// The one implementation of POST /v1/embeddings, shared by every host: validates the input, + /// embeds each string in-process (nothing leaves the box) and writes the response through an + /// . The caller owns concurrency — a has a + /// single scratch arena, so the host serializes calls to it. + /// + public static class EmbeddingsExchange + { + public static void Handle( + EmbeddingsRequest? req, SentenceEmbedder embedder, string modelName, IOpenAiResponseSink sink) + { + ArgumentNullException.ThrowIfNull(embedder); + ArgumentNullException.ThrowIfNull(sink); + + var inputs = req is null ? [] : OpenAiChatMapping.ParseInputs(req.Input); + if (inputs.Count == 0) + { + WriteError(sink, 400, "'input' is required (a string or an array of strings)."); + return; + } + + var data = new List(inputs.Count); + var approxTokens = 0; + for (var i = 0; i < inputs.Count; i++) + { + data.Add(new EmbeddingData { Index = i, Embedding = embedder.Embed(inputs[i]) }); + approxTokens += Math.Max(1, inputs[i].Length / 4); // rough proxy; we don't bill tokens + } + + var response = new EmbeddingsResponse + { + Model = modelName, + Data = data, + Usage = new OpenAiUsage { PromptTokens = approxTokens, TotalTokens = approxTokens }, + }; + + sink.WriteBody(200, "application/json", + JsonSerializer.Serialize(response, OpenAiJsonContext.Default.EmbeddingsResponse)); + } + + private static void WriteError(IOpenAiResponseSink sink, int status, string message) + { + var body = new OpenAiErrorResponse { Error = new OpenAiError { Message = message } }; + sink.WriteBody(status, "application/json", + JsonSerializer.Serialize(body, OpenAiJsonContext.Default.OpenAiErrorResponse)); + } + } +} diff --git a/Sources/Server/OpenAi/HttpListenerResponseSink.cs b/Sources/Server/OpenAi/HttpListenerResponseSink.cs new file mode 100644 index 00000000..f63d2727 --- /dev/null +++ b/Sources/Server/OpenAi/HttpListenerResponseSink.cs @@ -0,0 +1,48 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Net; +using System.Text; + +namespace DevOnBike.Overfit.Server.OpenAi +{ + /// + /// Adapts a raw to so the + /// dependency-free HttpListener server drives the shared + /// exactly as the ASP.NET host does. This is the only place the CLI server touches the wire for chat. + /// + internal sealed class HttpListenerResponseSink : IOpenAiResponseSink + { + private readonly HttpListenerResponse _response; + + public HttpListenerResponseSink(HttpListenerResponse response) => _response = response; + + public void WriteBody(int statusCode, string contentType, string body) + => WriteBinary(statusCode, contentType, Encoding.UTF8.GetBytes(body)); + + public void WriteBinary(int statusCode, string contentType, byte[] body) + { + _response.StatusCode = statusCode; + _response.ContentType = contentType; + _response.ContentLength64 = body.Length; + _response.OutputStream.Write(body, 0, body.Length); + } + + public void BeginEventStream() + { + _response.StatusCode = (int)HttpStatusCode.OK; + _response.ContentType = "text/event-stream"; + _response.Headers["Cache-Control"] = "no-cache"; + _response.SendChunked = true; + } + + public void WriteEvent(string data) + { + var bytes = Encoding.UTF8.GetBytes($"data: {data}\n\n"); + _response.OutputStream.Write(bytes, 0, bytes.Length); + _response.OutputStream.Flush(); + } + } +} diff --git a/Sources/Server/OpenAi/IChatExchangeObserver.cs b/Sources/Server/OpenAi/IChatExchangeObserver.cs new file mode 100644 index 00000000..97476d44 --- /dev/null +++ b/Sources/Server/OpenAi/IChatExchangeObserver.cs @@ -0,0 +1,38 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.LanguageModels.Contracts; + +namespace DevOnBike.Overfit.Server.OpenAi +{ + /// + /// Optional per-exchange hooks so each host can attach what only it cares about — the CLI server prints a + /// phase trace, the ASP.NET host records Prometheus metrics — without either concern leaking into the + /// shared . All methods are no-ops by default; pass null for + /// none. Timings are handed in already elapsed (milliseconds) so the observer never touches a clock. + /// + public interface IChatExchangeObserver + { + /// History replay finished — turns re-applied to the session. + void OnHistoryReplayed(int messageCount, double elapsedMs) + { + } + + /// The first streamed token was produced (streaming path only), + /// after generation began — the server-side component of time-to-first-token. + void OnFirstToken(double elapsedMs) + { + } + + /// + /// The exchange completed. distinguishes the SSE path from the one-shot + /// JSON path; is how many prompt tokens the prompt cache reused + /// instead of re-encoding (0 when nothing matched). + /// + void OnCompleted(bool streamed, GenerationStats stats, int cachedPromptTokens) + { + } + } +} diff --git a/Sources/Server/OpenAi/IOpenAiResponseSink.cs b/Sources/Server/OpenAi/IOpenAiResponseSink.cs new file mode 100644 index 00000000..9b48a5a7 --- /dev/null +++ b/Sources/Server/OpenAi/IOpenAiResponseSink.cs @@ -0,0 +1,47 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +namespace DevOnBike.Overfit.Server.OpenAi +{ + /// + /// The transport-neutral surface the shared chat handler () writes + /// through, so the OpenAI wire protocol — request validation, the streaming SSE shape, finish-reason + /// logic, the response objects — lives once and both hosts (the Native-AOT HttpListener CLI server + /// and the ASP.NET Minimal-API host) supply only a thin adapter over their own response object. + /// + /// Three primitives cover every write the protocol needs: a complete-body response for errors and + /// non-streaming results, the switch into event-stream mode, and one already-serialized SSE frame. The + /// adapter owns nothing but byte output; all serialization stays in the shared handler on the source-gen + /// OpenAiJsonContext, which keeps the whole path allocation-lean and trim/AOT-safe. + /// + public interface IOpenAiResponseSink + { + /// + /// Sends a complete-body response (a validation error, or the non-streaming chat result) and finishes + /// the exchange. Called at most once, and never after . + /// + void WriteBody(int statusCode, string contentType, string body); + + /// + /// Sends a complete binary-body response (synthesized audio) and finishes the exchange. Same one-shot + /// contract as . + /// + void WriteBinary(int statusCode, string contentType, byte[] body); + + /// + /// Switches the response into Server-Sent-Events mode: status 200, text/event-stream, + /// Cache-Control: no-cache, chunked transfer. Any transport-specific streaming setup (e.g. + /// enabling synchronous body writes) belongs here. + /// + void BeginEventStream(); + + /// + /// Writes one SSE frame: the sink adds the data: …\n\n framing and flushes so the client sees + /// the token immediately. is the already-serialized chunk JSON, or the literal + /// [DONE] sentinel. + /// + void WriteEvent(string data); + } +} diff --git a/Sources/Server/OpenAi/SpeechExchange.cs b/Sources/Server/OpenAi/SpeechExchange.cs new file mode 100644 index 00000000..df4e47c3 --- /dev/null +++ b/Sources/Server/OpenAi/SpeechExchange.cs @@ -0,0 +1,77 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Text.Json; +using DevOnBike.Overfit.Audio; +using DevOnBike.Overfit.Audio.Tts; +using DevOnBike.Overfit.Audio.Tts.Orpheus; + +namespace DevOnBike.Overfit.Server.OpenAi +{ + /// + /// The one implementation of POST /v1/audio/speech, shared by every host: validates the request, + /// synthesizes with the single (caller serializes access to it) and + /// writes WAV or raw PCM-16 through an . The WAV/PCM encoders live here + /// so neither host duplicates the audio framing. + /// + public static class SpeechExchange + { + public static void Handle(SpeechRequest? req, OrpheusVoiceEngine tts, IOpenAiResponseSink sink) + { + ArgumentNullException.ThrowIfNull(tts); + ArgumentNullException.ThrowIfNull(sink); + + if (req is null || string.IsNullOrWhiteSpace(req.Input)) + { + WriteError(sink, 400, "'input' is required."); + return; + } + + var format = (req.ResponseFormat ?? "wav").ToLowerInvariant(); + if (format is not ("wav" or "pcm")) + { + WriteError(sink, 400, $"response_format '{req.ResponseFormat}' is not supported; use 'wav' or 'pcm'."); + return; + } + + var voice = string.IsNullOrWhiteSpace(req.Voice) ? OrpheusPrompt.DefaultVoice : req.Voice!; + var audio = tts.Synthesize(req.Input!, voice); + + var isPcm = format == "pcm"; + var contentType = isPcm ? "audio/pcm" : "audio/wav"; + var bytes = isPcm ? ToPcm16Bytes(audio) : ToWavBytes(audio, tts.SampleRate, voice); + + sink.WriteBinary(200, contentType, bytes); + } + + private static byte[] ToWavBytes(float[] audio, int sampleRate, string voice) + { + using var ms = new MemoryStream(); + WavWriter.WriteMono(ms, audio, sampleRate, WavSampleFormat.Pcm16, + SyntheticSpeechMetadata.ForNow(voice).ToInfoComment()); + return ms.ToArray(); + } + + private static byte[] ToPcm16Bytes(float[] samples) + { + var bytes = new byte[samples.Length * 2]; + for (var i = 0; i < samples.Length; i++) + { + var clamped = Math.Clamp(samples[i], -1f, 1f); + var v = (short)MathF.Round(clamped * 32767f); + bytes[i * 2] = (byte)(v & 0xFF); + bytes[(i * 2) + 1] = (byte)((v >> 8) & 0xFF); + } + return bytes; + } + + private static void WriteError(IOpenAiResponseSink sink, int status, string message) + { + var body = new OpenAiErrorResponse { Error = new OpenAiError { Message = message } }; + sink.WriteBody(status, "application/json", + JsonSerializer.Serialize(body, OpenAiJsonContext.Default.OpenAiErrorResponse)); + } + } +} diff --git a/Sources/Server/OverfitOpenAiServer.cs b/Sources/Server/OverfitOpenAiServer.cs index 414d981c..3c05880c 100644 --- a/Sources/Server/OverfitOpenAiServer.cs +++ b/Sources/Server/OverfitOpenAiServer.cs @@ -330,56 +330,7 @@ private static void HandleAudioSpeech(HttpListenerContext ctx, OrpheusVoiceEngin return; } - if (req is null || string.IsNullOrWhiteSpace(req.Input)) - { - TryWriteError(ctx.Response, HttpStatusCode.BadRequest, "'input' is required."); - return; - } - - var format = (req.ResponseFormat ?? "wav").ToLowerInvariant(); - if (format is not ("wav" or "pcm")) - { - TryWriteError(ctx.Response, HttpStatusCode.BadRequest, - $"response_format '{req.ResponseFormat}' is not supported; use 'wav' or 'pcm'."); - return; - } - - var voice = string.IsNullOrWhiteSpace(req.Voice) ? OrpheusPrompt.DefaultVoice : req.Voice!; - var audio = tts.Synthesize(req.Input!, voice); - - // Both outputs are read below, so they must be definitely assigned; split ifs the compiler - // cannot prove exhaustive would not do that. The WAV branch keeps its `using` scope in a block. - var isPcm = format == "pcm"; - var contentType = isPcm ? "audio/pcm" : "audio/wav"; - var bytes = isPcm ? ToPcm16Bytes(audio) : ToWavBytes(audio, tts.SampleRate, voice); - - ctx.Response.StatusCode = (int)HttpStatusCode.OK; - ctx.Response.ContentType = contentType; - ctx.Response.ContentLength64 = bytes.Length; - ctx.Response.OutputStream.Write(bytes, 0, bytes.Length); - } - - /// WAV-encodes the synthesized audio. Split out of the caller so the `using MemoryStream` - /// keeps a scope of its own while the caller stays a single definitely-assigned expression. - private static byte[] ToWavBytes(float[] audio, int sampleRate, string voice) - { - using var ms = new MemoryStream(); - WavWriter.WriteMono(ms, audio, sampleRate, WavSampleFormat.Pcm16, - SyntheticSpeechMetadata.ForNow(voice).ToInfoComment()); - return ms.ToArray(); - } - - private static byte[] ToPcm16Bytes(float[] samples) - { - var bytes = new byte[samples.Length * 2]; - for (var i = 0; i < samples.Length; i++) - { - var clamped = Math.Clamp(samples[i], -1f, 1f); - var v = (short)MathF.Round(clamped * 32767f); - bytes[i * 2] = (byte)(v & 0xFF); - bytes[(i * 2) + 1] = (byte)((v >> 8) & 0xFF); - } - return bytes; + SpeechExchange.Handle(req, tts, new HttpListenerResponseSink(ctx.Response)); } /// Opt-in per-request phase trace (OVERFIT_SERVER_TRACE=1) for TTFT attribution. @@ -399,134 +350,12 @@ private static void HandleChatCompletions(HttpListenerContext ctx, OverfitClient return; } - if (req is null || req.Messages is not { Count: > 0 }) - { - TryWriteError(ctx.Response, HttpStatusCode.BadRequest, "'messages' is required and must be non-empty."); - return; - } - - var last = req.Messages[^1]; - if (!string.Equals(last.Role, "user", StringComparison.OrdinalIgnoreCase)) - { - TryWriteError(ctx.Response, HttpStatusCode.BadRequest, "the last message must have role 'user'."); - return; - } - - var (sampling, maxTokens) = OpenAiChatMapping.BuildSampling(req); - var options = new GenerationOptions(maxTokens, maxContextLength: 8192, sampling, stopOnEndOfTextToken: true); - var id = "chatcmpl-" + Guid.NewGuid().ToString("N"); - var ts = DateTimeOffset.UtcNow.ToUnixTimeSeconds(); - - ITokenConstraint? constraint; - try - { - constraint = OpenAiChatMapping.BuildResponseFormatConstraint(req.ResponseFormat, client.Tokenizer); - } - catch (JsonException ex) - { - TryWriteError(ctx.Response, HttpStatusCode.BadRequest, $"invalid response_format: {ex.Message}"); - return; - } - - try - { - // (A GcLatencyScope.SustainedLowLatency() was tried here and MEASURED to be a no-op — Overfit's - // generation is zero-allocation (0 GC, 0 B over 150 prefill+decode cycles), so there are no gen-2 - // pauses to suppress, while the mode would only trade RAM for nothing. Left off by design; - // GcLatencyScope stays an opt-in primitive for genuinely allocation-heavy host workloads.) - // Opt-in phase timing (OVERFIT_SERVER_TRACE=1). TTFT measured through this server ran ~405 ms - // while the engine's own prefill for the same prompt measured ~150 ms — so most of the latency - // a client feels is NOT the prefill kernel. Rather than guess which of replay, templating or - // the first decode holds it, each phase is timed. - var trace = ServerTrace; - var phase = trace ? ValueStopwatch.StartNew() : default; - - OpenAiChatMapping.ReplayHistory(client.Chat, req.Messages); - - if (trace) - { - Console.WriteLine($"[trace] replay {phase.GetElapsedTime().TotalMilliseconds:F1} ms " - + $"({req.Messages.Count} message(s))"); - } - - var userContent = last.Content ?? string.Empty; - - if (!req.Stream) - { - var reply = client.Chat.Send(userContent, in options, onText: null, constraint: constraint); - var s = client.Chat.LastStats; - - var response = new ChatCompletionResponse - { - Id = id, - Created = ts, - Model = modelName, - Choices = - [ - new ChatChoice - { - Index = 0, - Message = new OpenAiMessage { Role = "assistant", Content = reply }, - FinishReason = s.GeneratedTokens >= maxTokens ? "length" : "stop", - }, - ], - Usage = new OpenAiUsage - { - PromptTokens = s.PromptTokens, - CompletionTokens = s.GeneratedTokens, - TotalTokens = s.PromptTokens + s.GeneratedTokens, - }, - }; - WriteJson(ctx.Response, HttpStatusCode.OK, response, OpenAiJsonContext.Default.ChatCompletionResponse); - return; - } - - // Streaming (SSE). One in-flight request at a time, so the request thread owns the stream. - var resp = ctx.Response; - resp.StatusCode = (int)HttpStatusCode.OK; - resp.ContentType = "text/event-stream"; - resp.Headers["Cache-Control"] = "no-cache"; - resp.SendChunked = true; - - WriteChunk(resp, id, ts, modelName, new OpenAiMessage { Role = "assistant" }, finishReason: null); - - var sendStarted = trace ? ValueStopwatch.StartNew() : default; - var firstDelta = true; - - client.Chat.Send(userContent, in options, - onText: delta => - { - if (trace && firstDelta) - { - firstDelta = false; - Console.WriteLine($"[trace] first token {sendStarted.GetElapsedTime().TotalMilliseconds:F1} ms"); - } - - WriteChunk(resp, id, ts, modelName, new OpenAiMessage { Content = delta }, finishReason: null); - }, - constraint: constraint); - - var streamStats = client.Chat.LastStats; - - if (trace) - { - Console.WriteLine($"[trace] prompt {streamStats.PromptTokens} tok, " - + $"{client.Chat.CachedPromptTokens} reused from the KV cache"); - } - - var streamFinish = streamStats.GeneratedTokens >= maxTokens ? "length" : "stop"; - WriteChunk(resp, id, ts, modelName, new OpenAiMessage(), finishReason: streamFinish); - WriteSseRaw(resp, "[DONE]"); - } - finally - { - // Restore the baseline system turn so the shared single-tenant session stays clean. - client.Reset(); - if (!string.IsNullOrEmpty(systemMessage)) - { - client.AddSystem(systemMessage); - } - } + // Everything past the body parse — validation, sampling, replay, streaming shape, finish-reason, + // system-turn restore — is the shared protocol, run once in ChatCompletionExchange. This host + // supplies only the wire adapter and (opt-in) the phase trace. + var sink = new HttpListenerResponseSink(ctx.Response); + var observer = ServerTrace ? ConsoleTraceObserver.Instance : null; + ChatCompletionExchange.Handle(req, client, modelName, systemMessage, sink, observer); } private static void HandleEmbeddings(HttpListenerContext ctx, SentenceEmbedder embedder, string modelName) @@ -542,48 +371,7 @@ private static void HandleEmbeddings(HttpListenerContext ctx, SentenceEmbedder e return; } - var inputs = req is null ? [] : OpenAiChatMapping.ParseInputs(req.Input); - if (inputs.Count == 0) - { - TryWriteError(ctx.Response, HttpStatusCode.BadRequest, "'input' is required (a string or an array of strings)."); - return; - } - - // In-process, pure .NET embeddings — nothing leaves the box. - var data = new List(inputs.Count); - var approxTokens = 0; - for (var i = 0; i < inputs.Count; i++) - { - data.Add(new EmbeddingData { Index = i, Embedding = embedder.Embed(inputs[i]) }); - approxTokens += Math.Max(1, inputs[i].Length / 4); // rough proxy; we don't bill tokens - } - - var response = new EmbeddingsResponse - { - Model = modelName, - Data = data, - Usage = new OpenAiUsage { PromptTokens = approxTokens, TotalTokens = approxTokens }, - }; - WriteJson(ctx.Response, HttpStatusCode.OK, response, OpenAiJsonContext.Default.EmbeddingsResponse); - } - - private static void WriteChunk(HttpListenerResponse resp, string id, long created, string model, OpenAiMessage delta, string? finishReason) - { - var chunk = new ChatCompletionChunk - { - Id = id, - Created = created, - Model = model, - Choices = [new ChatChoice { Index = 0, Delta = delta, FinishReason = finishReason }], - }; - WriteSseRaw(resp, JsonSerializer.Serialize(chunk, OpenAiJsonContext.Default.ChatCompletionChunk)); - } - - private static void WriteSseRaw(HttpListenerResponse resp, string data) - { - var bytes = Encoding.UTF8.GetBytes($"data: {data}\n\n"); - resp.OutputStream.Write(bytes, 0, bytes.Length); - resp.OutputStream.Flush(); + EmbeddingsExchange.Handle(req, embedder, modelName, new HttpListenerResponseSink(ctx.Response)); } private static void WriteJson(HttpListenerResponse resp, HttpStatusCode status, T body, System.Text.Json.Serialization.Metadata.JsonTypeInfo typeInfo) From 1e7ad3c5a924c1083f2c98aaa3c65ab5b7beea2c Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Thu, 23 Jul 2026 22:57:36 +0200 Subject: [PATCH 35/37] asp.net --- Directory.Packages.props | 1 + Sources/Cli/Commands.cs | 56 +-- Sources/Cli/Program.cs | 9 +- Sources/Mcp/McpServer.cs | 2 +- .../Server.AspNet/Endpoints/DocsEndpoints.cs | 31 ++ .../Endpoints/OverfitOpenAiApi.cs | 56 +++ Sources/Server.AspNet/OverfitAspNetServer.cs | 29 +- .../RedactionGateway.cs | 153 +++--- .../Services/IOpenAiInferenceService.cs | 3 + .../Services/OverfitInferenceService.cs | 30 +- .../Server/OpenAi/HttpListenerResponseSink.cs | 48 -- Sources/Server/OpenAi/IOpenAiResponseSink.cs | 3 +- Sources/Server/OpenAi/OpenApiDocument.cs | 61 +++ Sources/Server/OverfitOpenAiServer.cs | 453 ------------------ .../OverfitAspNetServerIntegrationTests.cs | 273 +++++++++++ Tests/Tests.csproj | 2 + 16 files changed, 561 insertions(+), 649 deletions(-) create mode 100644 Sources/Server.AspNet/Endpoints/DocsEndpoints.cs create mode 100644 Sources/Server.AspNet/Endpoints/OverfitOpenAiApi.cs rename Sources/{Server => Server.AspNet}/RedactionGateway.cs (85%) delete mode 100644 Sources/Server/OpenAi/HttpListenerResponseSink.cs create mode 100644 Sources/Server/OpenAi/OpenApiDocument.cs delete mode 100644 Sources/Server/OverfitOpenAiServer.cs create mode 100644 Tests/Server/OverfitAspNetServerIntegrationTests.cs diff --git a/Directory.Packages.props b/Directory.Packages.props index 4ba073d9..cbd82fa1 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -22,6 +22,7 @@ debt. Bump it only when the SDK's own Roslyn version moves up. --> + diff --git a/Sources/Cli/Commands.cs b/Sources/Cli/Commands.cs index fcef18a5..1eac7e63 100644 --- a/Sources/Cli/Commands.cs +++ b/Sources/Cli/Commands.cs @@ -152,8 +152,12 @@ private static int PullEmbedder(string repo) private const string DefaultSystemPrompt = "You are a concise, helpful assistant running locally in pure .NET."; - public static int Serve(string model, string host, int port, string? embedModel, string? ttsModel, string? ttsSnac, int sessions = 1, bool httpListener = false) + public static int Serve(string model, string host, int port, string? embedModel, string? ttsModel, string? ttsSnac, int sessions = 1) { + // Time from launch to "server is listening" — dominated by the GGUF load, so it is effectively + // the container's cold-start time. Printed in the ready banner. + var startup = ValueStopwatch.StartNew(); + var path = ModelCache.Resolve(model); if (path is null) { @@ -258,54 +262,32 @@ public static int Serve(string model, string host, int port, string? embedModel, cts.Cancel(); }; - void PrintBanner(string baseUrl, string hostKind, bool servesExtras) + void PrintBanner(string baseUrl) { - var embedEp = servesExtras && embedder is not null ? " | POST /v1/embeddings" : string.Empty; - var ttsEp = servesExtras && tts is not null ? " | POST /v1/audio/speech" : string.Empty; + // The server may bind 0.0.0.0 (all interfaces) but you can't *connect* to 0.0.0.0 — show a + // reachable address in the copy-paste examples. + var connectUrl = baseUrl.Replace("0.0.0.0", "127.0.0.1"); + var embedEp = embedder is not null ? " | POST /v1/embeddings" : string.Empty; + var ttsEp = tts is not null ? " | POST /v1/audio/speech" : string.Empty; Console.WriteLine(); - Console.WriteLine($"Overfit OpenAI-compatible server ({hostKind}) listening on {baseUrl}"); + Console.WriteLine($"Overfit OpenAI-compatible server (ASP.NET / Kestrel, AOT) listening on {baseUrl}"); + Console.WriteLine($" ready in: {startup.GetElapsedTime().TotalSeconds:F2} s"); Console.WriteLine($" model id: {modelName}"); - Console.WriteLine($" endpoints: GET /v1/models | POST /v1/chat/completions (stream + non-stream){embedEp}{ttsEp} | GET /health"); + Console.WriteLine($" endpoints: GET /v1/models | POST /v1/chat/completions (stream + non-stream){embedEp}{ttsEp} | GET /health | GET /metrics | GET /docs"); Console.WriteLine(); - Console.WriteLine($" curl {baseUrl}/v1/chat/completions -H \"Content-Type: application/json\" \\"); - Console.WriteLine($" -d '{{\"model\":\"{modelName}\",\"messages\":[{{\"role\":\"user\",\"content\":\"Hello\"}}]}}'"); + Console.WriteLine($" reach it at {connectUrl} (e.g. {connectUrl}/docs , {connectUrl}/metrics)"); + Console.WriteLine($" curl -N {connectUrl}/v1/chat/completions -H \"Content-Type: application/json\" \\"); + Console.WriteLine($" -d '{{\"model\":\"{modelName}\",\"stream\":true,\"messages\":[{{\"role\":\"user\",\"content\":\"Hello\"}}]}}'"); Console.WriteLine(); Console.WriteLine("Press Ctrl+C to stop."); } - // Default: the AOT-ready ASP.NET (Kestrel) host. The dependency-free HttpListener server is kept - // only behind --http-listener as a legacy escape hatch. - if (!httpListener) + try { OverfitAspNetServer.Serve( pool, modelName, host, port, DefaultSystemPrompt, embedder, tts, - onListening: baseUrl => PrintBanner(baseUrl, "ASP.NET / Kestrel, AOT-ready", servesExtras: true), + onListening: PrintBanner, cancellationToken: cts.Token); - - embedder?.Dispose(); - tts?.Dispose(); - pool.Dispose(); - return 0; - } - - try - { - OverfitOpenAiServer.Serve( - pool, - modelName, - host, - port, - DefaultSystemPrompt, - embedder, - tts, - onListening: baseUrl => PrintBanner(baseUrl, "HttpListener (legacy)", servesExtras: true), - cancellationToken: cts.Token); - } - catch (HttpListenerException ex) - { - Console.Error.WriteLine($"Could not bind http://{host}:{port}/ : {ex.Message}"); - Console.Error.WriteLine("The port may be in use, or binding to a non-local host needs elevation / a URL ACL on Windows."); - return 1; } finally { diff --git a/Sources/Cli/Program.cs b/Sources/Cli/Program.cs index 65a60a14..687e96a1 100644 --- a/Sources/Cli/Program.cs +++ b/Sources/Cli/Program.cs @@ -99,11 +99,6 @@ + "(serialized, like llama.cpp). N>1 decodes N chats at once at the cost of N× KV-cache RAM.", DefaultValueFactory = _ => 1, }; -var serveHttpListener = new Option("--http-listener") -{ - Description = "Legacy: serve through the dependency-free HttpListener server instead of the default " - + "AOT-ready ASP.NET (Kestrel) host. Kept as an escape hatch; the Kestrel host is the shipping default.", -}; var serveCommand = new Command("serve", "Start an OpenAI-compatible HTTP server for a model.") { serveModel, @@ -113,7 +108,6 @@ serveTtsModel, serveTtsSnac, serveSessions, - serveHttpListener, }; serveCommand.SetAction(parseResult => Commands.Serve( parseResult.GetValue(serveModel)!, @@ -122,8 +116,7 @@ parseResult.GetValue(serveEmbedModel), parseResult.GetValue(serveTtsModel), parseResult.GetValue(serveTtsSnac), - parseResult.GetValue(serveSessions), - parseResult.GetValue(serveHttpListener))); + parseResult.GetValue(serveSessions))); // ── tts: text → speech (WAV), in-process, watermarked. Placeholder engine until the neural backend lands. ── var ttsText = new Option("--text") diff --git a/Sources/Mcp/McpServer.cs b/Sources/Mcp/McpServer.cs index 0139952b..d4e3fcae 100644 --- a/Sources/Mcp/McpServer.cs +++ b/Sources/Mcp/McpServer.cs @@ -19,7 +19,7 @@ namespace DevOnBike.Overfit.Mcp /// Implements the tools surface of the protocol: initialize (version negotiation + /// capabilities), notifications/initialized, ping, tools/list and /// tools/call. Requests are served strictly one at a time on the caller's thread - /// (single-tenant model session underneath — same stance as OverfitOpenAiServer). + /// (single-tenant model session underneath — same stance as the `overfit serve` host). /// public sealed class McpServer { diff --git a/Sources/Server.AspNet/Endpoints/DocsEndpoints.cs b/Sources/Server.AspNet/Endpoints/DocsEndpoints.cs new file mode 100644 index 00000000..5580c966 --- /dev/null +++ b/Sources/Server.AspNet/Endpoints/DocsEndpoints.cs @@ -0,0 +1,31 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.Server.OpenAi; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.Routing; + +namespace DevOnBike.Overfit.Server.AspNet.Endpoints +{ + /// + /// GET /openapi.yaml (the machine-readable contract, embedded from docs/openapi.yaml) and + /// GET /docs (the Scalar API reference pointed at it). Served from the embedded document so it works + /// from the single AOT binary without the reflection-based runtime OpenAPI generator. + /// + internal static class DocsEndpoints + { + public static WebApplication MapDocs(this WebApplication app) + { + app.MapGet("/openapi.yaml", () => + Results.Text(OpenApiDocument.Yaml(), "application/yaml; charset=utf-8")); + + app.MapGet("/docs", () => + Results.Text(OpenApiDocument.ApiReferenceHtml, "text/html; charset=utf-8")); + + return app; + } + } +} diff --git a/Sources/Server.AspNet/Endpoints/OverfitOpenAiApi.cs b/Sources/Server.AspNet/Endpoints/OverfitOpenAiApi.cs new file mode 100644 index 00000000..77303d04 --- /dev/null +++ b/Sources/Server.AspNet/Endpoints/OverfitOpenAiApi.cs @@ -0,0 +1,56 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.Server.AspNet.Services; +using DevOnBike.Overfit.Server.OpenAi; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Http; +using Microsoft.Extensions.DependencyInjection; + +namespace DevOnBike.Overfit.Server.AspNet.Endpoints +{ + /// + /// The one place the OpenAI-compatible surface is wired up — DI registration and endpoint mapping — so the + /// production host () and integration tests share exactly the same routes, + /// JSON configuration and route-group structure. Tests register a fake + /// and drive the endpoints in-memory, with no model loaded. + /// + public static class OverfitOpenAiApi + { + /// + /// Registers the OpenAI DTO source-gen JSON resolver and the inference service that backs every + /// endpoint. Call before . + /// + public static IServiceCollection AddOverfitOpenAi( + this IServiceCollection services, IOpenAiInferenceService service, ServerMetrics metrics) + { + services.ConfigureHttpJsonOptions(options => + options.SerializerOptions.TypeInfoResolverChain.Insert(0, OpenAiJsonContext.Default)); + services.AddSingleton(service); + services.AddSingleton(metrics); + return services; + } + + /// + /// Maps the full surface: /health, the docs pages, and the versioned /v1 route group with + /// one endpoint class per resource (models, chat, embeddings, speech). + /// + public static WebApplication MapOverfitOpenAiApi(this WebApplication app) + { + app.MapGet("/health", () => Results.Text("ok", "text/plain")); + app.MapGet("/", () => Results.Text("ok", "text/plain")); + app.MapMetrics(); + app.MapDocs(); + + var v1 = app.MapGroup("/v1"); + v1.MapModels(); + v1.MapChat(); + v1.MapEmbeddings(); + v1.MapSpeech(); + + return app; + } + } +} diff --git a/Sources/Server.AspNet/OverfitAspNetServer.cs b/Sources/Server.AspNet/OverfitAspNetServer.cs index 9a3c4393..e2df77c4 100644 --- a/Sources/Server.AspNet/OverfitAspNetServer.cs +++ b/Sources/Server.AspNet/OverfitAspNetServer.cs @@ -48,32 +48,25 @@ public static void Serve( { ArgumentNullException.ThrowIfNull(pool); - using var service = new OverfitInferenceService(pool, modelName, systemMessage, embedder, tts); + using var metrics = new ServerMetrics(); + using var service = new OverfitInferenceService(pool, modelName, systemMessage, embedder, tts, metrics); var builder = WebApplication.CreateSlimBuilder(); - // The CLI owns the console (it prints the banner via onListening); keep Kestrel's own startup - // logging off the wire so `overfit serve` output stays clean. + // ILogger → console via the default Microsoft.Extensions.Logging (no Serilog — AOT-clean, zero + // extra deps), at Information so request/host lifecycle logs are visible on the console. builder.Logging.ClearProviders(); + builder.Logging.AddSimpleConsole(options => options.SingleLine = true); + builder.Logging.SetMinimumLevel(LogLevel.Information); - // Bind and serialize every OpenAI DTO through the source-gen context — the reflection-free path - // Native AOT requires. - builder.Services.ConfigureHttpJsonOptions(options => - options.SerializerOptions.TypeInfoResolverChain.Insert(0, OpenAiJsonContext.Default)); - - builder.Services.AddSingleton(service); + // Source-gen JSON + the inference service + server metrics — the same wiring the integration tests use. + builder.Services.AddOverfitOpenAi(service, metrics); var app = builder.Build(); - app.MapGet("/health", () => Results.Text("ok", "text/plain")); - app.MapGet("/", () => Results.Text("ok", "text/plain")); - - // The OpenAI surface as a versioned route group, one endpoint class per resource. - var v1 = app.MapGroup("/v1"); - v1.MapModels(); - v1.MapChat(); - v1.MapEmbeddings(); - v1.MapSpeech(); + // /health, docs, and the /v1 route group (models / chat / embeddings / speech). Docs are served from + // the AOT-clean embedded document, not a reflection-based runtime generator. + app.MapOverfitOpenAiApi(); app.Lifetime.ApplicationStarted.Register(() => onListening?.Invoke($"http://{host}:{port}")); diff --git a/Sources/Server/RedactionGateway.cs b/Sources/Server.AspNet/RedactionGateway.cs similarity index 85% rename from Sources/Server/RedactionGateway.cs rename to Sources/Server.AspNet/RedactionGateway.cs index 661bc2db..e3700e50 100644 --- a/Sources/Server/RedactionGateway.cs +++ b/Sources/Server.AspNet/RedactionGateway.cs @@ -3,11 +3,14 @@ // DevonBike Overfit is licensed under the GNU AGPLv3. // For commercial licensing options, contact: devonbike@gmail.com -using System.Net; using System.Text; using System.Text.Json; using DevOnBike.Overfit.Redaction; +using DevOnBike.Overfit.Server.AspNet.Endpoints; using DevOnBike.Overfit.Server.OpenAi; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Http; +using Microsoft.Extensions.Logging; namespace DevOnBike.Overfit.Server { @@ -17,16 +20,18 @@ namespace DevOnBike.Overfit.Server /// never see it), restores the placeholders on the way back, and audits every redaction. "Change one base URL." /// /// /v1/chat/completions, streaming (SSE) and non-streaming, restore-on-response, BLOCK policy, - /// JSON-lines audit. Response-side scanning and client authentication are follow-ons. + /// JSON-lines audit. Runs on the AOT-ready Kestrel host (terminal middleware — no routing/reflection), so the + /// gateway ships in the single self-contained binary alongside overfit serve. /// public static class RedactionGateway { /// - /// Binds an HTTP listener on : and proxies chat completions - /// to (e.g. https://api.openai.com/v1), redacting via + /// Binds Kestrel on : and proxies chat completions to + /// (e.g. https://api.openai.com/v1), redacting via /// and auditing via . /// is the gateway-held secret injected as the upstream Authorization — clients authenticate to the - /// gateway, not the upstream. Blocks until the process is stopped. + /// gateway, not the upstream. Blocks until is cancelled (or the process + /// stops). /// public static void Serve( string host, @@ -37,7 +42,8 @@ public static void Serve( IRedactionAuditSink audit, RedactionPolicy policy, IReadOnlyCollection? clientKeys = null, - bool scanResponses = false) + bool scanResponses = false, + CancellationToken cancellationToken = default) { ArgumentException.ThrowIfNullOrEmpty(upstreamBaseUrl); ArgumentNullException.ThrowIfNull(redactor); @@ -52,13 +58,25 @@ public static void Serve( Timeout = TimeSpan.FromSeconds(120) }; - using var listener = new HttpListener(); - listener.Prefixes.Add($"http://{host}:{port}/"); - listener.Start(); + var builder = WebApplication.CreateSlimBuilder(); + builder.Logging.ClearProviders(); + var app = builder.Build(); + + // Terminal middleware handles EVERY request — a transparent proxy needs no routing table, and this + // keeps the whole path reflection-free for Native AOT. + app.Run(ctx => + { + EndpointHelpers.EnableSynchronousIO(ctx); + HandleRequest(ctx, upstream, upstreamApiKey, redactor, audit, policy, auth, scanResponses, http); + return Task.CompletedTask; + }); + + app.Urls.Add($"http://{host}:{port}"); + app.StartAsync(cancellationToken).GetAwaiter().GetResult(); Console.WriteLine($"Redaction gateway listening on http://{host}:{port}"); Console.WriteLine($" → forwarding to {upstream} (outbound PII/secrets redaction, audit on)"); - Console.WriteLine($" point your OpenAI client's base_url here; the real upstream key never leaves the gateway."); + Console.WriteLine(" point your OpenAI client's base_url here; the real upstream key never leaves the gateway."); Console.WriteLine(auth.Enabled ? " client authentication: ON (callers must present a configured gateway key)." : " client authentication: OFF — any caller can reach this gateway. Set gateway keys before exposing it."); @@ -67,30 +85,18 @@ public static void Serve( Console.WriteLine(" response scanning: ON (model-generated secrets/PII masked on non-streaming responses)."); } - // Bound stated in the header (OVERFIT023): the loop runs exactly as long as the listener is up. - // Stopping or disposing it both clear IsListening and make a pending GetContext throw, so no path - // keeps accepting after shutdown. - while (listener.IsListening) + // Block until cancelled; with CancellationToken.None this waits for the lifetime of the process + // (the test harness runs Serve on a background thread and lets it die at process end). + try { - HttpListenerContext ctx; - try - { - ctx = listener.GetContext(); - } - catch (HttpListenerException) - { - break; - } - catch (ObjectDisposedException) - { - break; - } - - // Dispatch each request to the thread pool so a slow (or streaming) call never blocks the next caller. - var captured = ctx; - ThreadPool.QueueUserWorkItem( - _ => HandleRequest(captured, upstream, upstreamApiKey, redactor, audit, policy, auth, scanResponses, http)); + Task.Delay(Timeout.Infinite, cancellationToken).GetAwaiter().GetResult(); + } + catch (OperationCanceledException) + { + // graceful shutdown requested } + + app.StopAsync().GetAwaiter().GetResult(); } /// @@ -169,7 +175,7 @@ public static void RestoreResponse(ChatCompletionResponse response, IReadOnlyLis } private static void HandleRequest( - HttpListenerContext ctx, + HttpContext ctx, string upstream, string? upstreamApiKey, Redactor redactor, @@ -181,20 +187,20 @@ private static void HandleRequest( { try { - var path = ctx.Request.Url?.AbsolutePath ?? string.Empty; - var method = ctx.Request.HttpMethod; + var path = ctx.Request.Path.Value ?? string.Empty; + var method = ctx.Request.Method; // /health is unauthenticated so liveness probes work without a key. if (method == "GET" && path == "/health") { - WriteText(ctx.Response, HttpStatusCode.OK, "ok"); + WriteText(ctx.Response, StatusCodes.Status200OK, "ok"); return; } // Everything that proxies upstream requires a valid gateway client key (when auth is enabled). - if (!auth.IsAuthorized(ctx.Request.Headers["Authorization"])) + if (!auth.IsAuthorized(ctx.Request.Headers.Authorization)) { - WriteText(ctx.Response, HttpStatusCode.Unauthorized, + WriteText(ctx.Response, StatusCodes.Status401Unauthorized, "Unauthorized: present a valid gateway key as 'Authorization: Bearer '. " + "The gateway holds the real upstream key — clients authenticate to the gateway, not upstream."); return; @@ -216,7 +222,7 @@ private static void HandleRequest( { try { - WriteText(ctx.Response, HttpStatusCode.BadGateway, $"gateway error: {ex.Message}"); + WriteText(ctx.Response, StatusCodes.Status502BadGateway, $"gateway error: {ex.Message}"); } catch { @@ -226,7 +232,7 @@ private static void HandleRequest( } private static void HandleChatCompletions( - HttpListenerContext ctx, + HttpContext ctx, string upstream, string? upstreamApiKey, Redactor redactor, @@ -235,10 +241,10 @@ private static void HandleChatCompletions( bool scanResponses, HttpClient http) { - var req = JsonSerializer.Deserialize(ctx.Request.InputStream, OpenAiJsonContext.Default.ChatCompletionRequest); + var req = JsonSerializer.Deserialize(ctx.Request.Body, OpenAiJsonContext.Default.ChatCompletionRequest); if (req is null) { - WriteText(ctx.Response, HttpStatusCode.BadRequest, "invalid request body"); + WriteText(ctx.Response, StatusCodes.Status400BadRequest, "invalid request body"); return; } @@ -331,7 +337,7 @@ private static string ScanResponseBody(string body, Redactor redactor, Redaction /// stream is never fully buffered — chunks are rewritten and forwarded as they arrive. /// private static void StreamResponse( - HttpListenerContext ctx, + HttpContext ctx, HttpRequestMessage upstreamRequest, IReadOnlyList matches, Redactor redactor, @@ -348,11 +354,10 @@ private static void StreamResponse( // Set the SSE-framing headers AFTER forwarding so the gateway's values win over any upstream duplicates. clientResponse.ContentType = "text/event-stream"; clientResponse.Headers["Cache-Control"] = "no-cache"; - clientResponse.SendChunked = true; using var upstreamStream = upstreamResponse.Content.ReadAsStream(); using var reader = new StreamReader(upstreamStream, Encoding.UTF8); - var output = clientResponse.OutputStream; + var output = clientResponse.Body; // Per choice index (usually one, but n>1 is legal): a response scanner (mask model-generated secrets) and // a restorer (re-hydrate the caller's own placeholders). The scanner runs first while the caller's values @@ -397,8 +402,6 @@ private static void StreamResponse( WriteLine(output, string.Empty); output.Flush(); } - - output.Close(); } // Scans (model secrets) then restores (caller placeholders) a single SSE chunk's delta content per choice, @@ -458,7 +461,7 @@ private static string RewriteChunk( // Emits any text the scanners/restorers held back, as a final synthetic chunk per choice, before [DONE]. // Per choice: flush the scanner (mask remaining model secrets) → feed through the restorer → flush it. private static void FlushStreams( - System.IO.Stream output, + Stream output, Dictionary? scanners, Dictionary restorers) { @@ -500,7 +503,7 @@ private static void AuditStreamScanned(IRedactionAuditSink audit, Dictionary private static void HandleGenericProxy( - HttpListenerContext ctx, + HttpContext ctx, string upstream, string? upstreamApiKey, Redactor redactor, @@ -525,8 +528,8 @@ private static void HandleGenericProxy( HttpClient http) { var request = ctx.Request; - var method = request.HttpMethod; - var targetUrl = BuildUpstreamUrl(upstream, request.Url?.AbsolutePath ?? "/") + (request.Url?.Query ?? string.Empty); + var method = request.Method; + var targetUrl = BuildUpstreamUrl(upstream, request.Path.Value ?? "/") + (request.QueryString.Value ?? string.Empty); using var upstreamRequest = new HttpRequestMessage(new HttpMethod(method), targetUrl); @@ -536,7 +539,7 @@ private static void HandleGenericProxy( if (carriesBody) { string body; - using (var reader = new StreamReader(request.InputStream, Encoding.UTF8)) + using (var reader = new StreamReader(request.Body, Encoding.UTF8)) { body = reader.ReadToEnd(); } @@ -617,28 +620,22 @@ private static string BuildUpstreamUrl(string upstream, string path) // Passes the caller's request headers (OpenAI-Beta, OpenAI-Organization/Project, X-*, User-Agent, Accept, …) // through to the upstream so client features keep working — minus the security/framing denylist. The client's // Authorization (its gateway key) is dropped here; the real upstream key is injected separately by the caller. - private static void ForwardRequestHeaders(HttpListenerRequest src, HttpRequestMessage dst) + private static void ForwardRequestHeaders(HttpRequest src, HttpRequestMessage dst) { - var headers = src.Headers; - for (var i = 0; i < headers.Count; i++) + foreach (var header in src.Headers) { - var name = headers.GetKey(i); - if (name is null || NonForwardableRequestHeaders.Contains(name)) + if (NonForwardableRequestHeaders.Contains(header.Key)) { continue; } - var values = headers.GetValues(i); - if (values is not null) - { - dst.Headers.TryAddWithoutValidation(name, values); - } + dst.Headers.TryAddWithoutValidation(header.Key, header.Value.ToArray()); } } // Passes upstream response headers (x-request-id, x-ratelimit-*, openai-*, …) back to the caller so clients // can see rate limits and request ids — minus headers the gateway manages itself. - private static void ForwardResponseHeaders(HttpResponseMessage upstream, HttpListenerResponse client) + private static void ForwardResponseHeaders(HttpResponseMessage upstream, HttpResponse client) { CopyResponseHeaders(upstream.Headers, client); if (upstream.Content is not null) @@ -647,7 +644,7 @@ private static void ForwardResponseHeaders(HttpResponseMessage upstream, HttpLis } } - private static void CopyResponseHeaders(System.Net.Http.Headers.HttpHeaders headers, HttpListenerResponse client) + private static void CopyResponseHeaders(System.Net.Http.Headers.HttpHeaders headers, HttpResponse client) { foreach (var header in headers) { @@ -660,9 +657,9 @@ private static void CopyResponseHeaders(System.Net.Http.Headers.HttpHeaders head { client.Headers[header.Key] = string.Join(", ", header.Value); } - catch (ArgumentException) + catch (InvalidOperationException) { - // Restricted header the HttpListener manages itself — skip it. + // Restricted header Kestrel manages itself — skip it. } } } @@ -686,7 +683,7 @@ private static void AuditRedactions(IRedactionAuditSink audit, IReadOnlyList blockedCategories) + private static void RespondBlocked(HttpContext ctx, IRedactionAuditSink audit, IReadOnlyList blockedCategories) { var blockCounts = new Dictionary(StringComparer.Ordinal); foreach (var category in blockedCategories) @@ -696,34 +693,32 @@ private static void RespondBlocked(HttpListenerContext ctx, IRedactionAuditSink audit.Record(new RedactionAuditRecord( Guid.NewGuid().ToString("N"), DateTimeOffset.UtcNow, blockedCategories.Count, blockCounts)); - WriteText(ctx.Response, HttpStatusCode.Forbidden, + WriteText(ctx.Response, StatusCodes.Status403Forbidden, $"Request refused by the redaction gateway: it contains forbidden category(ies) " + $"[{string.Join(", ", blockedCategories)}] that must not leave the box. Nothing was forwarded."); } - private static void WriteRaw(HttpListenerResponse response, int status, string json) + private static void WriteRaw(HttpResponse response, int status, string json) { WriteRaw(response, status, json, "application/json"); } - private static void WriteRaw(HttpListenerResponse response, int status, string body, string contentType) + private static void WriteRaw(HttpResponse response, int status, string body, string contentType) { var bytes = Encoding.UTF8.GetBytes(body); response.StatusCode = status; response.ContentType = contentType; - response.ContentLength64 = bytes.Length; - response.OutputStream.Write(bytes, 0, bytes.Length); - response.OutputStream.Close(); + response.ContentLength = bytes.Length; + response.Body.Write(bytes, 0, bytes.Length); } - private static void WriteText(HttpListenerResponse response, HttpStatusCode status, string text) + private static void WriteText(HttpResponse response, int status, string text) { var bytes = Encoding.UTF8.GetBytes(text); - response.StatusCode = (int)status; + response.StatusCode = status; response.ContentType = "text/plain"; - response.ContentLength64 = bytes.Length; - response.OutputStream.Write(bytes, 0, bytes.Length); - response.OutputStream.Close(); + response.ContentLength = bytes.Length; + response.Body.Write(bytes, 0, bytes.Length); } } } diff --git a/Sources/Server.AspNet/Services/IOpenAiInferenceService.cs b/Sources/Server.AspNet/Services/IOpenAiInferenceService.cs index d24d1899..baa6a41c 100644 --- a/Sources/Server.AspNet/Services/IOpenAiInferenceService.cs +++ b/Sources/Server.AspNet/Services/IOpenAiInferenceService.cs @@ -22,6 +22,9 @@ public interface IOpenAiInferenceService /// The served model's id, for GET /v1/models. ModelsResponse ListModels(); + /// Current session-pool snapshot for the /metrics gauges (size / active / free / rejected). + PoolStatus PoolStatus { get; } + /// /// Runs one chat completion (streaming or not) — rents a session, replays history, generates, restores /// the baseline system turn — writing the whole response through . Sheds with diff --git a/Sources/Server.AspNet/Services/OverfitInferenceService.cs b/Sources/Server.AspNet/Services/OverfitInferenceService.cs index 26bb8152..d8e8c714 100644 --- a/Sources/Server.AspNet/Services/OverfitInferenceService.cs +++ b/Sources/Server.AspNet/Services/OverfitInferenceService.cs @@ -38,24 +38,47 @@ public sealed class OverfitInferenceService : IOpenAiInferenceService, IDisposab private readonly SemaphoreSlim _embedGate = new(1, 1); private readonly SemaphoreSlim _ttsGate = new(1, 1); + private readonly ServerMetrics _metrics; + private readonly IChatExchangeObserver _chatObserver; + public OverfitInferenceService( OverfitResourcePool pool, string modelName, string systemMessage, SentenceEmbedder? embedder, - OrpheusVoiceEngine? tts) + OrpheusVoiceEngine? tts, + ServerMetrics metrics) { _pool = pool ?? throw new ArgumentNullException(nameof(pool)); _modelName = modelName; _systemMessage = systemMessage; _embedder = embedder; _tts = tts; + _metrics = metrics ?? throw new ArgumentNullException(nameof(metrics)); _created = DateTimeOffset.UtcNow.ToUnixTimeSeconds(); + + // The chat exchange takes ONE observer; metrics always record, the phase trace joins only when + // OVERFIT_SERVER_TRACE=1. + _chatObserver = Trace + ? new CompositeChatObserver(_metrics, ConsoleTraceObserver.Instance) + : _metrics; + + // Publish the live pool gauges on the Meter (this service owns the pool). + _metrics.BindPool(() => PoolStatus); } public ModelsResponse ListModels() => new() { Data = [new ModelInfo { Id = _modelName, Created = _created }] }; + public PoolStatus PoolStatus + { + get + { + var m = _pool.Metrics; + return new PoolStatus(m.Size, m.Active, m.Available, m.TotalRejected, m.PeakActive); + } + } + public void CompleteChat(ChatCompletionRequest? request, IOpenAiResponseSink sink, CancellationToken cancellationToken) { OverfitResourcePool.Lease lease; @@ -75,8 +98,7 @@ public void CompleteChat(ChatCompletionRequest? request, IOpenAiResponseSink sin using (lease) { - var observer = Trace ? ConsoleTraceObserver.Instance : null; - ChatCompletionExchange.Handle(request, lease.Value, _modelName, _systemMessage, sink, observer); + ChatCompletionExchange.Handle(request, lease.Value, _modelName, _systemMessage, sink, _chatObserver); } } @@ -93,6 +115,7 @@ public void Embed(EmbeddingsRequest? request, IOpenAiResponseSink sink, Cancella try { EmbeddingsExchange.Handle(request, _embedder, _modelName, sink); + _metrics.RecordEmbeddingRequest(); } finally { @@ -113,6 +136,7 @@ public void Synthesize(SpeechRequest? request, IOpenAiResponseSink sink, Cancell try { SpeechExchange.Handle(request, _tts, sink); + _metrics.RecordSpeechRequest(); } finally { diff --git a/Sources/Server/OpenAi/HttpListenerResponseSink.cs b/Sources/Server/OpenAi/HttpListenerResponseSink.cs deleted file mode 100644 index f63d2727..00000000 --- a/Sources/Server/OpenAi/HttpListenerResponseSink.cs +++ /dev/null @@ -1,48 +0,0 @@ -// Copyright (c) 2026 DevOnBike. -// This file is part of DevonBike Overfit. -// DevonBike Overfit is licensed under the GNU AGPLv3. -// For commercial licensing options, contact: devonbike@gmail.com - -using System.Net; -using System.Text; - -namespace DevOnBike.Overfit.Server.OpenAi -{ - /// - /// Adapts a raw to so the - /// dependency-free HttpListener server drives the shared - /// exactly as the ASP.NET host does. This is the only place the CLI server touches the wire for chat. - /// - internal sealed class HttpListenerResponseSink : IOpenAiResponseSink - { - private readonly HttpListenerResponse _response; - - public HttpListenerResponseSink(HttpListenerResponse response) => _response = response; - - public void WriteBody(int statusCode, string contentType, string body) - => WriteBinary(statusCode, contentType, Encoding.UTF8.GetBytes(body)); - - public void WriteBinary(int statusCode, string contentType, byte[] body) - { - _response.StatusCode = statusCode; - _response.ContentType = contentType; - _response.ContentLength64 = body.Length; - _response.OutputStream.Write(body, 0, body.Length); - } - - public void BeginEventStream() - { - _response.StatusCode = (int)HttpStatusCode.OK; - _response.ContentType = "text/event-stream"; - _response.Headers["Cache-Control"] = "no-cache"; - _response.SendChunked = true; - } - - public void WriteEvent(string data) - { - var bytes = Encoding.UTF8.GetBytes($"data: {data}\n\n"); - _response.OutputStream.Write(bytes, 0, bytes.Length); - _response.OutputStream.Flush(); - } - } -} diff --git a/Sources/Server/OpenAi/IOpenAiResponseSink.cs b/Sources/Server/OpenAi/IOpenAiResponseSink.cs index 9b48a5a7..693b4153 100644 --- a/Sources/Server/OpenAi/IOpenAiResponseSink.cs +++ b/Sources/Server/OpenAi/IOpenAiResponseSink.cs @@ -8,8 +8,7 @@ namespace DevOnBike.Overfit.Server.OpenAi /// /// The transport-neutral surface the shared chat handler () writes /// through, so the OpenAI wire protocol — request validation, the streaming SSE shape, finish-reason - /// logic, the response objects — lives once and both hosts (the Native-AOT HttpListener CLI server - /// and the ASP.NET Minimal-API host) supply only a thin adapter over their own response object. + /// logic, the response objects — lives once and a host (the AOT ASP.NET Minimal-API server) supplies only a thin adapter over its own response object. /// /// Three primitives cover every write the protocol needs: a complete-body response for errors and /// non-streaming results, the switch into event-stream mode, and one already-serialized SSE frame. The diff --git a/Sources/Server/OpenAi/OpenApiDocument.cs b/Sources/Server/OpenAi/OpenApiDocument.cs new file mode 100644 index 00000000..2e7fa2b5 --- /dev/null +++ b/Sources/Server/OpenAi/OpenApiDocument.cs @@ -0,0 +1,61 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Text; + +namespace DevOnBike.Overfit.Server.OpenAi +{ + /// + /// The self-describing API contract served at GET /openapi.yaml and the interactive API reference + /// (Scalar) at GET /docs. The YAML is embedded (one source of truth = docs/openapi.yaml) so + /// it ships inside the single self-contained binary; the viewer is a self-served HTML page that loads its + /// assets from a CDN rather than bloating the binary — no server-side generator, so the whole docs path + /// stays reflection-free and Native-AOT-clean. Host-agnostic so any host can serve both. + /// + public static class OpenApiDocument + { + private static string? _yaml; + + /// The embedded openapi.yaml contract, read once and cached. + public static string Yaml() + { + if (_yaml is not null) + { + return _yaml; + } + + using var stream = typeof(OpenApiDocument).Assembly.GetManifestResourceStream("openapi.yaml"); + if (stream is null) + { + _yaml = "openapi: 3.0.3\ninfo:\n title: Overfit\n version: '1.0.0'\npaths: {}\n"; + return _yaml; + } + + using var reader = new StreamReader(stream, Encoding.UTF8); + _yaml = reader.ReadToEnd(); + return _yaml; + } + + /// + /// Scalar API-reference viewer, pointed at /openapi.yaml via its standalone CDN bundle — the + /// modern replacement for Swagger UI that the .NET templates moved to. Purely client-side: no + /// server-side OpenAPI generator (which drags MVC.Abstractions and trips IL3053 under the AOT guard). + /// + public const string ApiReferenceHtml = """ + + + + + + Overfit API — Reference + + + + + + + """; + } +} diff --git a/Sources/Server/OverfitOpenAiServer.cs b/Sources/Server/OverfitOpenAiServer.cs deleted file mode 100644 index 3c05880c..00000000 --- a/Sources/Server/OverfitOpenAiServer.cs +++ /dev/null @@ -1,453 +0,0 @@ -// Copyright (c) 2026 DevOnBike. -// This file is part of DevonBike Overfit. -// DevonBike Overfit is licensed under the GNU AGPLv3. -// For commercial licensing options, contact: devonbike@gmail.com - -using System.Net; -using System.Text; -using System.Text.Json; -using DevOnBike.Overfit.Audio; -using DevOnBike.Overfit.Audio.Tts; -using DevOnBike.Overfit.Audio.Tts.Orpheus; -using DevOnBike.Overfit.LanguageModels; -using DevOnBike.Overfit.LanguageModels.Contracts; -using DevOnBike.Overfit.LanguageModels.Embeddings; -using DevOnBike.Overfit.Server.OpenAi; -using DevOnBike.Overfit.Serving; -using DevOnBike.Overfit.Diagnostics; -using DevOnBike.Overfit.Runtime; - -namespace DevOnBike.Overfit.Server -{ - /// - /// A dependency-free, OpenAI-compatible HTTP server over — no ASP.NET Core, so it - /// drops cleanly into the Native-AOT overfit CLI. Exposes /v1/chat/completions (streaming SSE + - /// non-streaming), /v1/models, /v1/embeddings, /v1/audio/speech and /health, plus - /// the self-describing /openapi.yaml (the API contract) and /docs (Swagger UI). Point any OpenAI - /// client at the base URL and only change the model name. Concurrency is bounded by the client pool: the - /// single-client - /// overload serialises requests through one session (like a local llama.cpp server); the - /// - /// overload decodes up to pool.Size chat requests at once and sheds excess load with HTTP 503. - /// - public static class OverfitOpenAiServer - { - // How long a chat request waits for a free session before the server sheds it with HTTP 503. - private const int RentTimeoutSeconds = 30; - - /// - /// Binds an on : and serves - /// requests until is cancelled. Blocks the calling thread. Each - /// request replays its full messages[] and restores the baseline system turn afterwards, so the - /// shared session never accumulates state across calls. - /// - /// A loaded model client; owned by the caller (not disposed here). - /// The id reported by /v1/models and echoed in responses. - /// Bind host. 127.0.0.1/localhost need no elevation; 0.0.0.0/* bind all interfaces (may need a URL ACL / admin on Windows). - /// TCP port. - /// Baseline system prompt restored after every request. - /// Optional in-process sentence embedder. When supplied, /v1/embeddings - /// serves it (pure .NET, no data egress); when null that route returns 501. Owned by the caller. - /// Optional callback invoked once the listener is up, with the base URL. - /// Cancel to stop the server gracefully. - public static void Serve( - OverfitClient client, - string modelName, - string host, - int port, - string systemMessage, - SentenceEmbedder? embedder = null, - OrpheusVoiceEngine? tts = null, - Action? onListening = null, - CancellationToken cancellationToken = default) - { - ArgumentNullException.ThrowIfNull(client); - - // Single caller-owned client → a pool-of-1 that does NOT own it (the caller still disposes it). - // Behaviour is identical to before: one session, requests serialised through the single client. - using var pool = new OverfitResourcePool([client], ownsItems: false); - Serve(pool, modelName, host, port, systemMessage, embedder, tts, onListening, cancellationToken); - } - - /// - /// Multi-session overload: serves requests across a of clients so up - /// to pool.Size chat completions decode concurrently (each client owns its KV cache; the weights are - /// shared via mmap). Requests beyond the pool wait up to a timeout and are otherwise shed with HTTP 503. - /// Embeddings and TTS use single shared engines and are serialised. /health reports pool load. - /// - public static void Serve( - OverfitResourcePool pool, - string modelName, - string host, - int port, - string systemMessage, - SentenceEmbedder? embedder = null, - OrpheusVoiceEngine? tts = null, - Action? onListening = null, - CancellationToken cancellationToken = default) - { - ArgumentNullException.ThrowIfNull(pool); - - var bindHost = host is "0.0.0.0" or "*" or "+" ? "+" : host; - var prefix = $"http://{bindHost}:{port}/"; - - using var listener = new HttpListener(); - listener.Prefixes.Add(prefix); - listener.Start(); - - var displayHost = bindHost == "+" ? "0.0.0.0" : host; - onListening?.Invoke($"http://{displayHost}:{port}"); - - using var stop = cancellationToken.Register(() => - { - try - { - listener.Stop(); - } - catch - { - // listener already torn down — nothing to do. - } - }); - - var state = new ServerState - { - Pool = pool, - ModelName = modelName, - SystemMessage = systemMessage, - Embedder = embedder, - Tts = tts, - Created = DateTimeOffset.UtcNow.ToUnixTimeSeconds(), - EmbedGate = new SemaphoreSlim(1, 1), - TtsGate = new SemaphoreSlim(1, 1), - RentTimeout = TimeSpan.FromSeconds(RentTimeoutSeconds), - StopToken = cancellationToken, - }; - - var inFlight = 0; - while (!cancellationToken.IsCancellationRequested) - { - HttpListenerContext ctx; - try - { - ctx = listener.GetContext(); - } - catch (HttpListenerException) - { - break; // Stop() was called. - } - catch (InvalidOperationException) - { - break; // listener disposed. - } - - // One task per request: up to pool.Size chat decodes run concurrently; the rest wait/shed. - Interlocked.Increment(ref inFlight); - _ = Task.Run(() => - { - try - { - HandleRequest(ctx, state); - } - finally - { - Interlocked.Decrement(ref inFlight); - } - }); - } - - // Drain in-flight requests (bounded) so pooled clients aren't disposed mid-decode by the caller. - for (var i = 0; i < 200 && Volatile.Read(ref inFlight) > 0; i++) - { - Thread.Sleep(50); - } - - state.EmbedGate.Dispose(); - state.TtsGate.Dispose(); - } - - private static void HandleRequest(HttpListenerContext ctx, ServerState s) - { - try - { - var req = ctx.Request; - var path = req.Url?.AbsolutePath ?? "/"; - var method = req.HttpMethod; - - if (method == "GET" && path is "/health" or "/") - { - var m = s.Pool.Metrics; - WriteRaw(ctx.Response, HttpStatusCode.OK, "application/json", - $"{{\"status\":\"ok\",\"sessions\":{{\"size\":{m.Size},\"active\":{m.Active}," - + $"\"available\":{m.Available},\"rented\":{m.TotalRented},\"rejected\":{m.TotalRejected}," - + $"\"peak\":{m.PeakActive}}}}}"); - return; - } - - if (method == "GET" && path == "/openapi.yaml") - { - // The machine-readable contract — import into Swagger UI / Postman / an OpenAPI codegen. - WriteRaw(ctx.Response, HttpStatusCode.OK, "application/yaml; charset=utf-8", OpenApiYaml()); - return; - } - - if (method == "GET" && path is "/docs" or "/docs/") - { - // Swagger UI for this server's /openapi.yaml. The viewer assets load from a CDN, so /docs - // needs internet to render (the API itself stays fully local — no prompt/data leaves). - WriteRaw(ctx.Response, HttpStatusCode.OK, "text/html; charset=utf-8", SwaggerUiHtml); - return; - } - - if (method == "GET" && path == "/v1/models") - { - var models = new ModelsResponse { Data = [new ModelInfo { Id = s.ModelName, Created = s.Created }] }; - WriteJson(ctx.Response, HttpStatusCode.OK, models, OpenAiJsonContext.Default.ModelsResponse); - return; - } - - if (method == "POST" && path == "/v1/chat/completions") - { - // Rent a session for the duration of the decode. Full pool → wait up to RentTimeout, then shed - // load with 503 rather than queue unboundedly. A cancelled wait (server stopping) is also a 503. - OverfitResourcePool.Lease lease; - try - { - if (!s.Pool.TryRent(s.RentTimeout, s.StopToken, out lease)) - { - TryWriteError(ctx.Response, HttpStatusCode.ServiceUnavailable, - $"server busy — all {s.Pool.Size} sessions in use; retry shortly."); - return; - } - } - catch (OperationCanceledException) - { - TryWriteError(ctx.Response, HttpStatusCode.ServiceUnavailable, "server is shutting down."); - return; - } - - using (lease) - { - HandleChatCompletions(ctx, lease.Value, s.ModelName, s.SystemMessage); - } - return; - } - - if (method == "POST" && path == "/v1/embeddings") - { - if (s.Embedder is null) - { - // No embedder loaded — a chat GGUF alone can't serve sentence embeddings. Clear, actionable 501. - TryWriteError(ctx.Response, HttpStatusCode.NotImplemented, - "embeddings are not served — start with an embedding model (e.g. 'overfit serve --embed-model ')."); - return; - } - - // SentenceEmbedder holds a single scratch arena — serialise concurrent embedding calls. - s.EmbedGate.Wait(s.StopToken); - try - { - HandleEmbeddings(ctx, s.Embedder, s.ModelName); - } - finally - { - s.EmbedGate.Release(); - } - return; - } - - if (method == "POST" && path == "/v1/audio/speech") - { - if (s.Tts is null) - { - TryWriteError(ctx.Response, HttpStatusCode.NotImplemented, - "text-to-speech is not served — start with a TTS model (e.g. 'overfit serve " - + "--tts-model --tts-snac ')."); - return; - } - - // Single TTS engine — serialise. - s.TtsGate.Wait(s.StopToken); - try - { - HandleAudioSpeech(ctx, s.Tts); - } - finally - { - s.TtsGate.Release(); - } - return; - } - - TryWriteError(ctx.Response, HttpStatusCode.NotFound, $"no route for {method} {path}"); - } - catch (OperationCanceledException) - { - TryWriteError(ctx.Response, HttpStatusCode.ServiceUnavailable, "server is shutting down."); - } - catch (Exception ex) - { - TryWriteError(ctx.Response, HttpStatusCode.InternalServerError, ex.Message); - } - finally - { - try - { - ctx.Response.Close(); - } - catch - { - // client may have already disconnected (e.g. aborted a stream). - } - } - } - - /// Per-server shared state handed to each request task. - private sealed class ServerState - { - public required OverfitResourcePool Pool; - public required string ModelName; - public required string SystemMessage; - public SentenceEmbedder? Embedder; - public OrpheusVoiceEngine? Tts; - public long Created; - public required SemaphoreSlim EmbedGate; - public required SemaphoreSlim TtsGate; - public TimeSpan RentTimeout; - public CancellationToken StopToken; - } - - private static void HandleAudioSpeech(HttpListenerContext ctx, OrpheusVoiceEngine tts) - { - SpeechRequest? req; - try - { - req = JsonSerializer.Deserialize(ctx.Request.InputStream, OpenAiJsonContext.Default.SpeechRequest); - } - catch (JsonException ex) - { - TryWriteError(ctx.Response, HttpStatusCode.BadRequest, $"invalid JSON body: {ex.Message}"); - return; - } - - SpeechExchange.Handle(req, tts, new HttpListenerResponseSink(ctx.Response)); - } - - /// Opt-in per-request phase trace (OVERFIT_SERVER_TRACE=1) for TTFT attribution. - private static readonly bool ServerTrace = - Environment.GetEnvironmentVariable(OverfitEnvironment.ServerTrace) == "1"; - - private static void HandleChatCompletions(HttpListenerContext ctx, OverfitClient client, string modelName, string systemMessage) - { - ChatCompletionRequest? req; - try - { - req = JsonSerializer.Deserialize(ctx.Request.InputStream, OpenAiJsonContext.Default.ChatCompletionRequest); - } - catch (JsonException ex) - { - TryWriteError(ctx.Response, HttpStatusCode.BadRequest, $"invalid JSON body: {ex.Message}"); - return; - } - - // Everything past the body parse — validation, sampling, replay, streaming shape, finish-reason, - // system-turn restore — is the shared protocol, run once in ChatCompletionExchange. This host - // supplies only the wire adapter and (opt-in) the phase trace. - var sink = new HttpListenerResponseSink(ctx.Response); - var observer = ServerTrace ? ConsoleTraceObserver.Instance : null; - ChatCompletionExchange.Handle(req, client, modelName, systemMessage, sink, observer); - } - - private static void HandleEmbeddings(HttpListenerContext ctx, SentenceEmbedder embedder, string modelName) - { - EmbeddingsRequest? req; - try - { - req = JsonSerializer.Deserialize(ctx.Request.InputStream, OpenAiJsonContext.Default.EmbeddingsRequest); - } - catch (JsonException ex) - { - TryWriteError(ctx.Response, HttpStatusCode.BadRequest, $"invalid JSON body: {ex.Message}"); - return; - } - - EmbeddingsExchange.Handle(req, embedder, modelName, new HttpListenerResponseSink(ctx.Response)); - } - - private static void WriteJson(HttpListenerResponse resp, HttpStatusCode status, T body, System.Text.Json.Serialization.Metadata.JsonTypeInfo typeInfo) - { - var json = JsonSerializer.Serialize(body, typeInfo); - WriteRaw(resp, status, "application/json", json); - } - - private static void WriteRaw(HttpListenerResponse resp, HttpStatusCode status, string contentType, string body) - { - var bytes = Encoding.UTF8.GetBytes(body); - resp.StatusCode = (int)status; - resp.ContentType = contentType; - resp.ContentLength64 = bytes.Length; - resp.OutputStream.Write(bytes, 0, bytes.Length); - } - - private static void TryWriteError(HttpListenerResponse resp, HttpStatusCode status, string message) - { - try - { - var body = new OpenAiErrorResponse { Error = new OpenAiError { Message = message } }; - var json = JsonSerializer.Serialize(body, OpenAiJsonContext.Default.OpenAiErrorResponse); - WriteRaw(resp, status, "application/json", json); - } - catch - { - // headers already sent (e.g. mid-stream) — can't change the status now. - } - } - - private static string? _openApiYaml; - - /// The embedded openapi.yaml contract, read once and cached. The server handles one - /// request at a time (single-threaded accept loop), so a lock-free lazy init is safe here. - private static string OpenApiYaml() - { - // Returning from each branch rather than falling through to a shared `return`: the field is - // nullable, and the two assignments above a common exit are not enough for the compiler to prove - // it was set (CS8603). With `else` banned, an early return per branch is the honest shape. - if (_openApiYaml is not null) - { - return _openApiYaml; - } - - using var stream = typeof(OverfitOpenAiServer).Assembly.GetManifestResourceStream("openapi.yaml"); - if (stream is null) - { - _openApiYaml = "openapi: 3.0.3\ninfo:\n title: Overfit\n version: '1.0.0'\npaths: {}\n"; - return _openApiYaml; - } - - using var reader = new StreamReader(stream, Encoding.UTF8); - _openApiYaml = reader.ReadToEnd(); - return _openApiYaml; - } - - // Swagger UI viewer for /openapi.yaml. Lean by design: the UI bundle loads from a CDN instead of - // bloating the single self-contained binary with ~1.5 MB of assets. Same-origin spec fetch + "try it out". - private const string SwaggerUiHtml = """ - - - - - - Overfit API — Swagger UI - - - -
- - - - - """; - } -} diff --git a/Tests/Server/OverfitAspNetServerIntegrationTests.cs b/Tests/Server/OverfitAspNetServerIntegrationTests.cs new file mode 100644 index 00000000..1903f8cb --- /dev/null +++ b/Tests/Server/OverfitAspNetServerIntegrationTests.cs @@ -0,0 +1,273 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Net; +using System.Net.Http.Json; +using System.Text; +using System.Text.Json; +using DevOnBike.Overfit.Server.AspNet.Endpoints; +using DevOnBike.Overfit.Server.AspNet.Services; +using DevOnBike.Overfit.Server.OpenAi; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.TestHost; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; + +namespace DevOnBike.Overfit.Tests.Server +{ + /// + /// Integration tests for the AOT-ready ASP.NET host, driven through Microsoft's in-memory + /// (Microsoft.AspNetCore.TestHost) — a real request pipeline, no sockets, + /// no model. The engine is faked behind (the reason the logic sits + /// behind an interface), so these exercise the whole HTTP surface — routing, the source-gen JSON binding, + /// the SSE framing, status codes, the docs endpoints — and run anywhere, including GitHub CI where no GGUF + /// exists. The real-model path is covered separately by [SmallModelFact]/[LongFact] tests. + /// + public sealed class OverfitAspNetServerIntegrationTests + { + private static async Task<(WebApplication App, HttpClient Client)> StartAsync(IOpenAiInferenceService service) + { + var builder = WebApplication.CreateSlimBuilder(); + builder.WebHost.UseTestServer(); + builder.Logging.ClearProviders(); + builder.Services.AddOverfitOpenAi(service, new ServerMetrics()); + + var app = builder.Build(); + app.MapOverfitOpenAiApi(); + await app.StartAsync(); + return (app, app.GetTestClient()); + } + + [Fact] + public async Task Health_ReturnsOk() + { + var (app, client) = await StartAsync(new FakeInferenceService()); + await using (app) + { + var resp = await client.GetAsync("/health"); + Assert.Equal(HttpStatusCode.OK, resp.StatusCode); + Assert.Equal("ok", await resp.Content.ReadAsStringAsync()); + } + } + + [Fact] + public async Task Models_ReportsTheServedModelId() + { + var (app, client) = await StartAsync(new FakeInferenceService()); + await using (app) + { + var doc = await client.GetFromJsonAsync("/v1/models"); + Assert.Equal("fake-model", doc.GetProperty("data")[0].GetProperty("id").GetString()); + } + } + + [Fact] + public async Task ChatCompletions_NonStreaming_ReturnsAssistantContent() + { + var (app, client) = await StartAsync(new FakeInferenceService()); + await using (app) + { + var resp = await client.PostAsync("/v1/chat/completions", JsonBody( + """{"model":"m","stream":false,"messages":[{"role":"user","content":"hi"}]}""")); + + Assert.Equal(HttpStatusCode.OK, resp.StatusCode); + var doc = JsonDocument.Parse(await resp.Content.ReadAsStringAsync()); + var msg = doc.RootElement.GetProperty("choices")[0].GetProperty("message"); + Assert.Equal("assistant", msg.GetProperty("role").GetString()); + Assert.Equal("pong", msg.GetProperty("content").GetString()); + } + } + + [Fact] + public async Task ChatCompletions_Streaming_EmitsSseFramesAndDone() + { + var (app, client) = await StartAsync(new FakeInferenceService()); + await using (app) + { + var resp = await client.PostAsync("/v1/chat/completions", JsonBody( + """{"model":"m","stream":true,"messages":[{"role":"user","content":"hi"}]}""")); + + Assert.Equal(HttpStatusCode.OK, resp.StatusCode); + Assert.Equal("text/event-stream", resp.Content.Headers.ContentType?.MediaType); + + var body = await resp.Content.ReadAsStringAsync(); + var frames = body.Split("\n\n", StringSplitOptions.RemoveEmptyEntries); + Assert.Contains("\"content\":\"po\"", body); + Assert.Contains("\"content\":\"ng\"", body); + Assert.EndsWith("data: [DONE]", frames[^1].Trim()); + } + } + + [Fact] + public async Task ChatCompletions_MalformedJson_Returns400() + { + var (app, client) = await StartAsync(new FakeInferenceService()); + await using (app) + { + var resp = await client.PostAsync("/v1/chat/completions", + new StringContent("{ this is not json", Encoding.UTF8, "application/json")); + + Assert.Equal(HttpStatusCode.BadRequest, resp.StatusCode); + var doc = JsonDocument.Parse(await resp.Content.ReadAsStringAsync()); + Assert.Contains("invalid JSON", doc.RootElement.GetProperty("error").GetProperty("message").GetString()); + } + } + + [Fact] + public async Task Embeddings_ReturnsVectorFromTheService() + { + var (app, client) = await StartAsync(new FakeInferenceService()); + await using (app) + { + var resp = await client.PostAsync("/v1/embeddings", JsonBody( + """{"model":"m","input":"hello"}""")); + + Assert.Equal(HttpStatusCode.OK, resp.StatusCode); + var doc = JsonDocument.Parse(await resp.Content.ReadAsStringAsync()); + var vector = doc.RootElement.GetProperty("data")[0].GetProperty("embedding"); + Assert.Equal(3, vector.GetArrayLength()); + } + } + + [Fact] + public async Task Embeddings_WhenServiceHasNoModel_Returns501() + { + var (app, client) = await StartAsync(new FakeInferenceService { EmbeddingsAvailable = false }); + await using (app) + { + var resp = await client.PostAsync("/v1/embeddings", JsonBody("""{"input":"hello"}""")); + Assert.Equal(HttpStatusCode.NotImplemented, resp.StatusCode); + } + } + + [Fact] + public async Task Docs_ServeOpenApiYamlAndApiReference() + { + var (app, client) = await StartAsync(new FakeInferenceService()); + await using (app) + { + var yaml = await client.GetAsync("/openapi.yaml"); + Assert.Equal(HttpStatusCode.OK, yaml.StatusCode); + Assert.Contains("openapi", await yaml.Content.ReadAsStringAsync()); + + var docs = await client.GetAsync("/docs"); + Assert.Equal(HttpStatusCode.OK, docs.StatusCode); + Assert.Equal("text/html", docs.Content.Headers.ContentType?.MediaType); + } + } + + [Fact] + public async Task Metrics_ExposePrometheusProcessMetrics() + { + var (app, client) = await StartAsync(new FakeInferenceService()); + await using (app) + { + var resp = await client.GetAsync("/metrics"); + Assert.Equal(HttpStatusCode.OK, resp.StatusCode); + Assert.StartsWith("text/plain", resp.Content.Headers.ContentType?.MediaType ?? ""); + + var body = await resp.Content.ReadAsStringAsync(); + // Prometheus exposition format: HELP/TYPE lines + the memory + CPU series the user asked for. + Assert.Contains("# TYPE process_resident_memory_bytes gauge", body); + Assert.Contains("# TYPE process_cpu_seconds_total counter", body); + Assert.Contains("dotnet_gc_collections_total{generation=\"0\"}", body); + + // Server metrics: requests, tokens (rate() -> tokens/s), and live session-pool gauges. + Assert.Contains("# TYPE overfit_chat_requests_total counter", body); + Assert.Contains("# TYPE overfit_generated_tokens_total counter", body); + Assert.Contains("overfit_pool_active_sessions ", body); + Assert.Contains("overfit_pool_size ", body); + + // The resident-memory value must parse as a positive number (the working set is never zero). + var line = Array.Find(body.Split('\n'), l => l.StartsWith("process_resident_memory_bytes ", StringComparison.Ordinal)); + Assert.NotNull(line); + Assert.True(long.Parse(line!.Split(' ')[1]) > 0); + } + } + + private static StringContent JsonBody(string json) => new(json, Encoding.UTF8, "application/json"); + + /// + /// A deterministic stand-in for the real engine: it writes canned responses through the sink so the + /// HTTP surface can be exercised with no model. Streaming splits "pong" across two SSE chunks so the + /// framing is genuinely tested. + /// + private sealed class FakeInferenceService : IOpenAiInferenceService + { + public bool EmbeddingsAvailable { get; init; } = true; + + public PoolStatus PoolStatus => new(Size: 4, Active: 0, Available: 4, RejectedTotal: 0, PeakActive: 1); + + public ModelsResponse ListModels() + => new() { Data = [new ModelInfo { Id = "fake-model", Created = 0 }] }; + + public void CompleteChat(ChatCompletionRequest? request, IOpenAiResponseSink sink, CancellationToken cancellationToken) + { + if (request?.Stream == true) + { + sink.BeginEventStream(); + sink.WriteEvent(Chunk("po")); + sink.WriteEvent(Chunk("ng")); + sink.WriteEvent("[DONE]"); + return; + } + + var response = new ChatCompletionResponse + { + Id = "chatcmpl-fake", + Model = "fake-model", + Choices = + [ + new ChatChoice + { + Index = 0, + Message = new OpenAiMessage { Role = "assistant", Content = "pong" }, + FinishReason = "stop", + }, + ], + Usage = new OpenAiUsage { PromptTokens = 1, CompletionTokens = 1, TotalTokens = 2 }, + }; + sink.WriteBody(200, "application/json", + JsonSerializer.Serialize(response, OpenAiJsonContext.Default.ChatCompletionResponse)); + } + + public void Embed(EmbeddingsRequest? request, IOpenAiResponseSink sink, CancellationToken cancellationToken) + { + if (!EmbeddingsAvailable) + { + sink.WriteBody(501, "application/json", + JsonSerializer.Serialize( + new OpenAiErrorResponse { Error = new OpenAiError { Message = "no embedding model" } }, + OpenAiJsonContext.Default.OpenAiErrorResponse)); + return; + } + + var response = new EmbeddingsResponse + { + Model = "fake-model", + Data = [new EmbeddingData { Index = 0, Embedding = [0.1f, 0.2f, 0.3f] }], + Usage = new OpenAiUsage { PromptTokens = 1, TotalTokens = 1 }, + }; + sink.WriteBody(200, "application/json", + JsonSerializer.Serialize(response, OpenAiJsonContext.Default.EmbeddingsResponse)); + } + + public void Synthesize(SpeechRequest? request, IOpenAiResponseSink sink, CancellationToken cancellationToken) + => sink.WriteBinary(200, "audio/wav", [0x52, 0x49, 0x46, 0x46]); + + private static string Chunk(string content) + { + var chunk = new ChatCompletionChunk + { + Id = "chatcmpl-fake", + Model = "fake-model", + Choices = [new ChatChoice { Index = 0, Delta = new OpenAiMessage { Content = content } }], + }; + return JsonSerializer.Serialize(chunk, OpenAiJsonContext.Default.ChatCompletionChunk); + } + } + } +} diff --git a/Tests/Tests.csproj b/Tests/Tests.csproj index e8b3e158..3a302238 100644 --- a/Tests/Tests.csproj +++ b/Tests/Tests.csproj @@ -14,6 +14,7 @@ + @@ -32,6 +33,7 @@ + From fff1f0997c6b036b939cc37806ed52dddfef1613 Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Thu, 23 Jul 2026 23:05:12 +0200 Subject: [PATCH 36/37] asp.net --- .../Endpoints/MetricsEndpoints.cs | 119 ++++++++++++++++++ .../Services/CompositeChatObserver.cs | 45 +++++++ .../Services/LatencyHistogram.cs | 80 ++++++++++++ .../Services/OverfitInferenceService.cs | 3 + Sources/Server.AspNet/Services/PoolStatus.cs | 13 ++ .../Server.AspNet/Services/ServerMetrics.cs | 117 +++++++++++++++++ .../OverfitAspNetServerIntegrationTests.cs | 6 + 7 files changed, 383 insertions(+) create mode 100644 Sources/Server.AspNet/Endpoints/MetricsEndpoints.cs create mode 100644 Sources/Server.AspNet/Services/CompositeChatObserver.cs create mode 100644 Sources/Server.AspNet/Services/LatencyHistogram.cs create mode 100644 Sources/Server.AspNet/Services/PoolStatus.cs create mode 100644 Sources/Server.AspNet/Services/ServerMetrics.cs diff --git a/Sources/Server.AspNet/Endpoints/MetricsEndpoints.cs b/Sources/Server.AspNet/Endpoints/MetricsEndpoints.cs new file mode 100644 index 00000000..89bd6a67 --- /dev/null +++ b/Sources/Server.AspNet/Endpoints/MetricsEndpoints.cs @@ -0,0 +1,119 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Diagnostics; +using System.Globalization; +using System.Text; +using DevOnBike.Overfit.Server.AspNet.Services; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.Routing; + +namespace DevOnBike.Overfit.Server.AspNet.Endpoints +{ + /// + /// GET /metrics — process metrics in the Prometheus text exposition format, hand-rolled so the + /// whole path stays reflection-free and Native-AOT-clean (the OpenTelemetry Prometheus exporter drags + /// reflection-heavy dependencies that would break the AOT guard, the same trap as the OpenAPI generator). + /// + /// The headline gauge is process_resident_memory_bytes (the working set): with the model + /// memory-mapped it grows as weight pages are touched, so it is the honest "how much RAM the server holds + /// with the GGUF loaded" number. CPU is exposed the Prometheus way — process_cpu_seconds_total as a + /// counter, from which a dashboard derives utilisation via rate(). + /// + internal static class MetricsEndpoints + { + private static readonly double StartUnixSeconds = + new DateTimeOffset(Process.GetCurrentProcess().StartTime.ToUniversalTime()).ToUnixTimeMilliseconds() / 1000.0; + + public static WebApplication MapMetrics(this WebApplication app) + { + app.MapGet("/metrics", (ServerMetrics metrics, IOpenAiInferenceService service) => + Results.Text(Render(metrics, service.PoolStatus), "text/plain; version=0.0.4; charset=utf-8")); + + return app; + } + + private static string Render(ServerMetrics metrics, PoolStatus pool) + { + using var process = Process.GetCurrentProcess(); + var sb = new StringBuilder(2048); + + // ── Server metrics: requests, tokens (rate() -> tokens/s), and live session-pool load. ── + Counter(sb, "overfit_chat_requests_total", "Completed chat-completion requests.", metrics.ChatRequests); + Counter(sb, "overfit_embedding_requests_total", "Completed embedding requests.", metrics.EmbeddingRequests); + Counter(sb, "overfit_speech_requests_total", "Completed text-to-speech requests.", metrics.SpeechRequests); + Counter(sb, "overfit_prompt_tokens_total", "Prompt tokens processed across all chat requests.", metrics.PromptTokens); + Counter(sb, "overfit_generated_tokens_total", + "Tokens generated across all chat requests (rate() gives tokens/second).", metrics.GeneratedTokens); + + Gauge(sb, "overfit_pool_size", "Total sessions in the pool (max concurrent decodes).", pool.Size); + Gauge(sb, "overfit_pool_active_sessions", "Sessions currently decoding a request.", pool.Active); + Gauge(sb, "overfit_pool_available_sessions", "Sessions free to serve a request right now.", pool.Available); + Gauge(sb, "overfit_pool_peak_active_sessions", "High-water mark of concurrent active sessions.", pool.PeakActive); + Counter(sb, "overfit_pool_rejected_total", "Requests shed with HTTP 503 because the pool was full.", pool.RejectedTotal); + + metrics.Ttft.Write(sb, "overfit_chat_ttft", "Server-side time to first streamed token, in seconds."); + metrics.ResponseTime.Write(sb, "overfit_chat_response_time", "Chat completion wall-clock time, in seconds."); + + Gauge(sb, "process_resident_memory_bytes", + "Resident set size (working set) in bytes — includes paged-in mmap'd model weights.", + process.WorkingSet64); + Gauge(sb, "process_private_memory_bytes", + "Private (committed) memory in bytes.", process.PrivateMemorySize64); + Gauge(sb, "process_virtual_memory_bytes", + "Virtual address space in bytes (includes the mmap'd model, mostly not resident).", + process.VirtualMemorySize64); + + Counter(sb, "process_cpu_seconds_total", + "Total user + system CPU time consumed by the process, in seconds.", + process.TotalProcessorTime.TotalSeconds); + Gauge(sb, "process_start_time_seconds", + "Process start time since the unix epoch, in seconds.", StartUnixSeconds); + Gauge(sb, "process_num_threads", "Number of OS threads.", process.Threads.Count); + + Gauge(sb, "dotnet_total_memory_bytes", + "Managed GC heap memory currently allocated, in bytes.", GC.GetTotalMemory(forceFullCollection: false)); + var gc = GC.GetGCMemoryInfo(); + Gauge(sb, "dotnet_gc_heap_size_bytes", "GC heap size after the last collection, in bytes.", gc.HeapSizeBytes); + Gauge(sb, "dotnet_gc_committed_bytes", "Committed GC memory, in bytes.", gc.TotalCommittedBytes); + + sb.Append("# HELP dotnet_gc_collections_total Number of GC collections, by generation.\n"); + sb.Append("# TYPE dotnet_gc_collections_total counter\n"); + for (var generation = 0; generation <= GC.MaxGeneration; generation++) + { + sb.Append("dotnet_gc_collections_total{generation=\"") + .Append(generation) + .Append("\"} ") + .Append(GC.CollectionCount(generation).ToString(CultureInfo.InvariantCulture)) + .Append('\n'); + } + + return sb.ToString(); + } + + private static void Gauge(StringBuilder sb, string name, string help, long value) + => Metric(sb, name, help, "gauge", value.ToString(CultureInfo.InvariantCulture)); + + private static void Gauge(StringBuilder sb, string name, string help, double value) + => Metric(sb, name, help, "gauge", Format(value)); + + private static void Counter(StringBuilder sb, string name, string help, double value) + => Metric(sb, name, help, "counter", Format(value)); + + private static void Counter(StringBuilder sb, string name, string help, long value) + => Metric(sb, name, help, "counter", value.ToString(CultureInfo.InvariantCulture)); + + private static void Metric(StringBuilder sb, string name, string help, string type, string value) + { + sb.Append("# HELP ").Append(name).Append(' ').Append(help).Append('\n'); + sb.Append("# TYPE ").Append(name).Append(' ').Append(type).Append('\n'); + sb.Append(name).Append(' ').Append(value).Append('\n'); + } + + // Prometheus wants a plain decimal (no thousands separators, no scientific notation). + private static string Format(double value) => value.ToString("0.######", CultureInfo.InvariantCulture); + } +} diff --git a/Sources/Server.AspNet/Services/CompositeChatObserver.cs b/Sources/Server.AspNet/Services/CompositeChatObserver.cs new file mode 100644 index 00000000..284265fa --- /dev/null +++ b/Sources/Server.AspNet/Services/CompositeChatObserver.cs @@ -0,0 +1,45 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using DevOnBike.Overfit.LanguageModels.Contracts; +using DevOnBike.Overfit.Server.OpenAi; + +namespace DevOnBike.Overfit.Server.AspNet.Services +{ + /// + /// Fans one chat exchange's observer callbacks out to several observers — used to run metric recording and + /// the opt-in phase trace together, since the exchange takes a single observer. + /// + internal sealed class CompositeChatObserver : IChatExchangeObserver + { + private readonly IChatExchangeObserver[] _observers; + + public CompositeChatObserver(params IChatExchangeObserver[] observers) => _observers = observers; + + public void OnHistoryReplayed(int messageCount, double elapsedMs) + { + foreach (var observer in _observers) + { + observer.OnHistoryReplayed(messageCount, elapsedMs); + } + } + + public void OnFirstToken(double elapsedMs) + { + foreach (var observer in _observers) + { + observer.OnFirstToken(elapsedMs); + } + } + + public void OnCompleted(bool streamed, GenerationStats stats, int cachedPromptTokens) + { + foreach (var observer in _observers) + { + observer.OnCompleted(streamed, stats, cachedPromptTokens); + } + } + } +} diff --git a/Sources/Server.AspNet/Services/LatencyHistogram.cs b/Sources/Server.AspNet/Services/LatencyHistogram.cs new file mode 100644 index 00000000..3911a5aa --- /dev/null +++ b/Sources/Server.AspNet/Services/LatencyHistogram.cs @@ -0,0 +1,80 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Globalization; +using System.Text; + +namespace DevOnBike.Overfit.Server.AspNet.Services +{ + /// + /// A fixed-bucket latency histogram accumulated for the Prometheus /metrics endpoint. The Meter's + /// own Histogram<double> records the same samples for OpenTelemetry / dotnet-counters, but it + /// does not expose its accumulated buckets for scraping — so this keeps the cumulative counts, sum and + /// count that Prometheus's histogram format needs. Buckets are chosen for LLM latencies (tens of ms to + /// several seconds). Recording is under a short lock — it happens once per request, off the decode path. + /// + internal sealed class LatencyHistogram + { + // Upper bounds in seconds (le = less-than-or-equal), ascending. + private static readonly double[] Bounds = + [0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]; + + private readonly long[] _bucketCounts = new long[Bounds.Length + 1]; // +1 = the +Inf overflow bucket + private readonly object _lock = new(); + private long _count; + private double _sum; + + public void Record(double seconds) + { + var index = 0; + while (index < Bounds.Length && seconds > Bounds[index]) + { + index++; + } + + lock (_lock) + { + _bucketCounts[index]++; + _count++; + _sum += seconds; + } + } + + /// Writes the histogram in Prometheus exposition format; gains the + /// _seconds unit suffix (e.g. overfit_chat_ttft → overfit_chat_ttft_seconds). + public void Write(StringBuilder sb, string name, string help) + { + long[] snapshot; + long count; + double sum; + lock (_lock) + { + snapshot = (long[])_bucketCounts.Clone(); + count = _count; + sum = _sum; + } + + sb.Append("# HELP ").Append(name).Append("_seconds ").Append(help).Append('\n'); + sb.Append("# TYPE ").Append(name).Append("_seconds histogram\n"); + + var cumulative = 0L; + for (var i = 0; i < Bounds.Length; i++) + { + cumulative += snapshot[i]; + sb.Append(name).Append("_seconds_bucket{le=\"") + .Append(Bounds[i].ToString("0.###", CultureInfo.InvariantCulture)) + .Append("\"} ").Append(cumulative.ToString(CultureInfo.InvariantCulture)).Append('\n'); + } + + cumulative += snapshot[Bounds.Length]; + sb.Append(name).Append("_seconds_bucket{le=\"+Inf\"} ") + .Append(cumulative.ToString(CultureInfo.InvariantCulture)).Append('\n'); + sb.Append(name).Append("_seconds_sum ") + .Append(sum.ToString("0.######", CultureInfo.InvariantCulture)).Append('\n'); + sb.Append(name).Append("_seconds_count ") + .Append(count.ToString(CultureInfo.InvariantCulture)).Append('\n'); + } + } +} diff --git a/Sources/Server.AspNet/Services/OverfitInferenceService.cs b/Sources/Server.AspNet/Services/OverfitInferenceService.cs index d8e8c714..9418fef8 100644 --- a/Sources/Server.AspNet/Services/OverfitInferenceService.cs +++ b/Sources/Server.AspNet/Services/OverfitInferenceService.cs @@ -5,6 +5,7 @@ using System.Text.Json; using DevOnBike.Overfit.Audio.Tts.Orpheus; +using DevOnBike.Overfit.Diagnostics; using DevOnBike.Overfit.LanguageModels; using DevOnBike.Overfit.LanguageModels.Embeddings; using DevOnBike.Overfit.Runtime; @@ -96,10 +97,12 @@ public void CompleteChat(ChatCompletionRequest? request, IOpenAiResponseSink sin return; } + var started = ValueStopwatch.StartNew(); using (lease) { ChatCompletionExchange.Handle(request, lease.Value, _modelName, _systemMessage, sink, _chatObserver); } + _metrics.RecordResponseTime(started.GetElapsedTime().TotalSeconds); } public void Embed(EmbeddingsRequest? request, IOpenAiResponseSink sink, CancellationToken cancellationToken) diff --git a/Sources/Server.AspNet/Services/PoolStatus.cs b/Sources/Server.AspNet/Services/PoolStatus.cs new file mode 100644 index 00000000..dfe422c4 --- /dev/null +++ b/Sources/Server.AspNet/Services/PoolStatus.cs @@ -0,0 +1,13 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +namespace DevOnBike.Overfit.Server.AspNet.Services +{ + /// + /// A point-in-time snapshot of the session pool, for the /metrics gauges: how many sessions exist, + /// how many are decoding right now, how many are free, and how many requests have been shed with 503. + /// + public readonly record struct PoolStatus(int Size, int Active, int Available, long RejectedTotal, int PeakActive); +} diff --git a/Sources/Server.AspNet/Services/ServerMetrics.cs b/Sources/Server.AspNet/Services/ServerMetrics.cs new file mode 100644 index 00000000..1ec6dbf3 --- /dev/null +++ b/Sources/Server.AspNet/Services/ServerMetrics.cs @@ -0,0 +1,117 @@ +// Copyright (c) 2026 DevOnBike. +// This file is part of DevonBike Overfit. +// DevonBike Overfit is licensed under the GNU AGPLv3. +// For commercial licensing options, contact: devonbike@gmail.com + +using System.Diagnostics.Metrics; +using DevOnBike.Overfit.LanguageModels.Contracts; +using DevOnBike.Overfit.Server.OpenAi; + +namespace DevOnBike.Overfit.Server.AspNet.Services +{ + /// + /// Collects the server's request / token / session-pool metrics through the idiomatic .NET metrics API — + /// a with observable instruments — so the same numbers are visible to + /// dotnet-counters, an OpenTelemetry pipeline, or any other System.Diagnostics.Metrics + /// consumer, not only the built-in Prometheus endpoint. + /// + /// Why the Meter, but not the OpenTelemetry Prometheus exporter. The instrumentation side + /// (, observable counters/gauges) is part of the framework and Native-AOT-clean. The + /// usual Prometheus exporter is not — it drags the same reflection-heavy dependencies that trip the + /// AOT guard (the trap the OpenAPI generator hit). So the values are published to the Meter for tooling, + /// and exposed to Prometheus by the hand-rolled /metrics endpoint, which reads the counters below + /// directly. + /// + /// It doubles as an : the shared chat exchange calls + /// at the end of every completion, which is exactly where the token counts are + /// known. Counters are interlocked — several pooled sessions decode concurrently. + /// + public sealed class ServerMetrics : IChatExchangeObserver, IDisposable + { + private readonly Meter _meter = new("DevOnBike.Overfit.Server", "1.0.0"); + + private long _chatRequests; + private long _embeddingRequests; + private long _speechRequests; + private long _promptTokens; + private long _generatedTokens; + + private readonly Histogram _ttftMeter; + private readonly Histogram _responseMeter; + + // Prometheus-side accumulation of the same samples (the Meter histograms don't expose their buckets). + internal LatencyHistogram Ttft { get; } = new(); + internal LatencyHistogram ResponseTime { get; } = new(); + + public ServerMetrics() + { + _ttftMeter = _meter.CreateHistogram("overfit.chat.ttft", + unit: "s", description: "Server-side time to first token (streaming)."); + _responseMeter = _meter.CreateHistogram("overfit.chat.response_time", + unit: "s", description: "Chat completion wall-clock time."); + + // Observable counters read the interlocked totals on demand — no double-bookkeeping, and the + // hand-rolled /metrics endpoint reads the same fields. + _meter.CreateObservableCounter("overfit.chat.requests", () => ChatRequests, + unit: "{request}", description: "Completed chat-completion requests."); + _meter.CreateObservableCounter("overfit.embedding.requests", () => EmbeddingRequests, + unit: "{request}", description: "Completed embedding requests."); + _meter.CreateObservableCounter("overfit.speech.requests", () => SpeechRequests, + unit: "{request}", description: "Completed text-to-speech requests."); + _meter.CreateObservableCounter("overfit.chat.prompt_tokens", () => PromptTokens, + unit: "{token}", description: "Prompt tokens processed across all chat requests."); + _meter.CreateObservableCounter("overfit.chat.generated_tokens", () => GeneratedTokens, + unit: "{token}", description: "Tokens generated across all chat requests."); + } + + public long ChatRequests => Interlocked.Read(ref _chatRequests); + public long EmbeddingRequests => Interlocked.Read(ref _embeddingRequests); + public long SpeechRequests => Interlocked.Read(ref _speechRequests); + public long PromptTokens => Interlocked.Read(ref _promptTokens); + public long GeneratedTokens => Interlocked.Read(ref _generatedTokens); + + /// + /// Registers the live session-pool gauges on the Meter. Called once by the service, which owns the + /// pool; the same snapshot backs the /metrics pool gauges. + /// + public void BindPool(Func poolStatus) + { + ArgumentNullException.ThrowIfNull(poolStatus); + _meter.CreateObservableGauge("overfit.pool.size", () => poolStatus().Size, + description: "Total sessions in the pool (max concurrent decodes)."); + _meter.CreateObservableGauge("overfit.pool.active_sessions", () => poolStatus().Active, + description: "Sessions currently decoding a request."); + _meter.CreateObservableGauge("overfit.pool.available_sessions", () => poolStatus().Available, + description: "Sessions free to serve a request right now."); + } + + public void RecordEmbeddingRequest() => Interlocked.Increment(ref _embeddingRequests); + + public void RecordSpeechRequest() => Interlocked.Increment(ref _speechRequests); + + /// Records the wall-clock time of one chat completion (called by the service). + public void RecordResponseTime(double seconds) + { + _responseMeter.Record(seconds); + ResponseTime.Record(seconds); + } + + /// Server-side time to first streamed token (called by the chat exchange for streaming requests). + public void OnFirstToken(double elapsedMs) + { + var seconds = elapsedMs / 1000.0; + _ttftMeter.Record(seconds); + Ttft.Record(seconds); + } + + /// Records one completed chat request and its token counts (called by the chat exchange). + public void OnCompleted(bool streamed, GenerationStats stats, int cachedPromptTokens) + { + Interlocked.Increment(ref _chatRequests); + Interlocked.Add(ref _promptTokens, stats.PromptTokens); + Interlocked.Add(ref _generatedTokens, stats.GeneratedTokens); + } + + public void Dispose() => _meter.Dispose(); + } +} diff --git a/Tests/Server/OverfitAspNetServerIntegrationTests.cs b/Tests/Server/OverfitAspNetServerIntegrationTests.cs index 1903f8cb..7b35aaa5 100644 --- a/Tests/Server/OverfitAspNetServerIntegrationTests.cs +++ b/Tests/Server/OverfitAspNetServerIntegrationTests.cs @@ -181,6 +181,12 @@ public async Task Metrics_ExposePrometheusProcessMetrics() Assert.Contains("overfit_pool_active_sessions ", body); Assert.Contains("overfit_pool_size ", body); + // Latency histograms (TTFT + response time) in Prometheus histogram format. + Assert.Contains("# TYPE overfit_chat_ttft_seconds histogram", body); + Assert.Contains("overfit_chat_ttft_seconds_bucket{le=\"+Inf\"}", body); + Assert.Contains("# TYPE overfit_chat_response_time_seconds histogram", body); + Assert.Contains("overfit_chat_response_time_seconds_count", body); + // The resident-memory value must parse as a positive number (the working set is never zero). var line = Array.Find(body.Split('\n'), l => l.StartsWith("process_resident_memory_bytes ", StringComparison.Ordinal)); Assert.NotNull(line); From be01d36fa96b4f5711e884f398da84e2ce52d72d Mon Sep 17 00:00:00 2001 From: DevOnBike Date: Thu, 23 Jul 2026 23:13:48 +0200 Subject: [PATCH 37/37] asp.net --- docs/aiops-canary-blueprint.md | 191 +++++++++++++++++++++++++++++++++ 1 file changed, 191 insertions(+) create mode 100644 docs/aiops-canary-blueprint.md diff --git a/docs/aiops-canary-blueprint.md b/docs/aiops-canary-blueprint.md new file mode 100644 index 00000000..741cad97 --- /dev/null +++ b/docs/aiops-canary-blueprint.md @@ -0,0 +1,191 @@ +# Overfit AIOps — Automated Canary Analysis (design & strategy blueprint) + +> **Status:** internal design/strategy note, 2026-07-23. Not linked from the README — it names a competing +> system (Kayenta) and carries market/strategy judgements that don't belong in launch-facing copy. This is a +> *candidate* product direction, not shipped work. The engineering corrections in §3 are the load-bearing part: +> the original vision's statistics (Z-score) must be replaced before any of this is credible. + +An **Edge AIOps** engine that runs 100% on-premise inside the customer's Kubernetes cluster and performs +**automated canary analysis** in real time — judging whether a new app version is stable versus the old one, +without any telemetry leaving the firewall. It is not a traffic manager; it is the **"brain"** (a webhook) +that controllers like **Argo Rollouts** consult at each step of a rollout. + +--- + +## 1. Concept + +At each rollout step (5% → 20% → 50% traffic), Argo Rollouts calls the engine. The engine compares the new +version (canary) against the old (baseline) and returns a binary verdict — continue or roll back. Nothing +leaves the box: it reads metrics from the in-cluster Prometheus and decides locally. + +**Why it fits Overfit's DNA.** Pure .NET, single self-contained Native-AOT binary, zero data egress, +air-gapped-capable, no separate analysis service to deploy. It is a Prometheus *consumer*; Overfit's inference +server is a Prometheus *exporter* — same ecosystem, same identity, different product. + +--- + +## 2. Architecture — the "two brains" (keep them separate) + +Two cooperating but **independent** mechanisms. **Ship brain 2 first; brain 1 is a separate product.** + +### Brain 2 — the Canary Sniper (relative A/B analysis) — the sellable core +Fires only during a rollout. Ignores long-term history. + +- **Normalise on the fly.** Reduce both versions to a *unit cost* (e.g. CPU-seconds per HTTP request) so a + version serving 5% of traffic is comparable to one serving 95%. +- **No absolute thresholds.** It never judges raw values; it asks whether the canary's normalised cost is + worse than the baseline's, under a live statistical test (see §3). +- **Event correlation.** If latency rises in the canary **but rises identically in the baseline** (a shared + DB hiccup), it is classified as an *infrastructure* problem, not new-code regression — and the rollout is + allowed to continue. This is the single most important behaviour: it is what makes automated rollback + usable instead of a false-alarm generator. + +### Brain 1 — the Global Guardian (long-term anomaly detection) — a *separate* SKU +Runs in the background on stable production (100% traffic). + +- **Cold start (day 1).** Before it knows the daily cycle, it leans on hard "physics of IT" heuristics — + a sudden vertical RAM climb, an avalanche of 5xx — via rate-of-change (derivative) analysis. Frame this + honestly: it is *threshold-on-derivative*, not "AI". +- **Adaptation.** Over a 3–4 week sliding window it learns seasonality (morning peaks, nightly batch, weekend + dips) to suppress false alarms. Implement as univariate time-series anomaly detection: a seasonal-naive + baseline (same hour last week) with **robust z on the MAD** (median absolute deviation — resistant to + outliers, unlike mean/std), or STL decomposition + residual thresholding. + +> **Recommendation:** do NOT bundle these into one product for v1. The Sniper (brain 2) has a crisp buyer and +> a crisp integration (Argo). The Guardian (brain 1) is a different, standalone anomaly product. Combining +> them dilutes focus. + +--- + +## 3. The statistical engine — the corrected core (this is the moat) + +The original vision specified **Z-score / standard deviation**. That is the wrong tool and must be replaced — +Z-score assumes a normal distribution, and latency / request-cost are heavy-tailed (log-normal). A naive +Z-score on p99 latency fires constantly. The credible engine has four layers: + +### 3a. Test — Mann-Whitney U (Wilcoxon rank-sum), not Z-score +Rank all requests from baseline and canary together, sum the ranks, compute U. It tests "is the canary +stochastically worse than the baseline" **without assuming any distribution** — exactly right for heavy-tailed +latency. This is what Netflix's Kayenta uses. + +### 3b. Gate on effect size, not just the p-value +With large N, a microscopic difference becomes "statistically significant" (p < 0.05) and you get rollbacks +from nothing. Require an **effect size** threshold too — Cliff's delta, or the common-language effect size +("probability a random canary request is worse than a random baseline request"). Rule: roll back only when +**significant AND effect > threshold**. This is what tames false alarms under high traffic. + +### 3c. Control-vs-experiment design (the correlation insight, formalised) +Do not compare the canary to fixed history. Compare it to a **baseline running simultaneously under the same +conditions**. Infrastructure hiccups hit both, so the relative comparison cancels them. This is §2's +"event correlation" made rigorous — and it is why it works where absolute thresholds don't. + +### 3d. Sequential testing (the honest answer to "too few samples at 5% traffic") +At 5% traffic, samples arrive over time. Naively re-running the test every minute is p-hacking — repeated +peeking inflates the false-positive rate. Use a **sequential test** (always-valid p-values / mixture SPRT) so +the engine decides the moment there is enough evidence, without corrupting significance by peeking. + +### 3e. Multiple comparisons +Testing CPU, latency, errors, memory each at 5% gives ≈19% chance of an unlucky rollback across four metrics. +Correct for it (Bonferroni / Benjamini-Hochberg) or fold the metrics into **one aggregate score** (Kayenta's +approach). + +--- + +## 4. Integration — Argo Rollouts `AnalysisTemplate` + +The engine is exposed as a webhook metric provider. Argo Rollouts' `AnalysisTemplate` calls it each step and +acts on the verdict: + +- `HTTP 200 OK` / `Successful` → proceed to the next traffic step. +- `HTTP 406 Not Acceptable` / `Failed` → abort and roll back. +- `Inconclusive` → hold (not enough evidence yet — pairs with the sequential test in §3d). +- `Error` → surfaced, does not silently pass. + +No separate control plane, no Spinnaker — one webhook in the cluster. + +--- + +## 5. Configuration — opinionated, three risk profiles + +Eliminates the "hell of sliders": one choice from three tolerance profiles. **Reframe the profiles in terms of +effect size + power, not "σ"** — a σ multiplier on log-normal data does not mean what the label implies. + +| Profile | Intent | Use | +|---|---|---| +| **Paranoid** | Catch the smallest regression (thread-pool starvation, µs stalls) | FinTech, payments | +| **Balanced** | Tolerate K8s network noise, react hard to leaks / EF problems | E-commerce, B2B | +| **Loose** | Ignore mild perf regressions, intervene only on hard failure | Internal tooling | + +--- + +## 6. What to cut — over-engineering + +The vision proposes pinned/unmanaged memory (POH) for Prometheus vectors. **Canary analysis pulls kilobytes, +not gigabytes** — a handful of metrics × a few hundred points. LOH pressure is a non-problem here; this is +LLM-scale memory discipline applied where the data fits in L2. Keep SIMD if it helps, but do not sell +"unmanaged heap" for a workload that fits in cache — a technical buyer will see through it. + +--- + +## 7. Competition & positioning + +- **Kayenta** (Netflix/Spinnaker) — the reference, but heavy (needs Spinnaker or a standalone deployment, JVM). +- **Flagger** — built-in canary analysis, but threshold-based, no statistical rigour. +- **Argo Rollouts** metric providers (Prometheus, Datadog…) — thresholds/templates, **no statistical testing**. + +**The gap:** statistically-rigorous canary analysis, air-gapped, **single native binary**, plugged into Argo as +a webhook. The moat is the statistics (Mann-Whitney + effect size + sequential) **plus** deployment simplicity. +Pitch: **"Kayenta without the Spinnaker footprint — air-gapped, zero-egress, one binary."** + +--- + +## 8. Go-to-market + +Ranked by strength: + +1. **DevOps Multiplier (OEM licensing) — strongest.** Annual, unlimited licence to software houses / MSPs. It + becomes their internal cost-cutter: kills alert fatigue, lets the same team run twice as many client + clusters. Recurring revenue, clear ROI. +2. **Air-gapped Kubernetes vault.** CRD images sold to secrecy-bound institutions (medical, defence, large law + firms). High CapEx sale for an AI tool that *physically cannot* leak logs to a public cloud. Real market, + but long cycles and heavy compliance. +3. **"Hitman" premium audits.** Plug the engine into a troubled cluster for a week, surface low-level anomalies + (async misuse, retained-object leaks), invoice once. This is **consulting, not product** — a cash-flow + bridge, dangerous as a strategy (sells your time, doesn't scale). + +Aligns with the AGPL-open / commercial-licence moat: an autonomous black box supports asymmetric sale. + +--- + +## 9. MVP — the thinnest thing that proves it + +1. An Argo `AnalysisTemplate` webhook. +2. **Mann-Whitney U on one metric** (CPU cost per request), control-vs-experiment. +3. Run it against a **recorded, real canary** from one actual deployment. + +If it correctly separates a bad deploy from a DB hiccup on real data, there is a product. No two brains, no +three profiles, no POH — those are the superstructure. + +--- + +## 10. The real question — strategic focus (not technical) + +This is a **different product, market and buyer** than the inference engine. Overfit is "pure .NET CPU LLM/DL +inference". A canary analyser is a **DevOps/SRE** tool — a different buyer (platform teams, not ML/app devs), a +different sales motion, different marketing. It shares Overfit's DNA (.NET, zero-alloc, air-gapped) but it is a +**strategic fork**: is Overfit an inference engine, or an AIOps company? + +For a small/solo project, doing both dilutes focus. This is not "don't do it" — it is "decide deliberately". +The blueprint is **shippable faster than the ML anomaly track** (a statistical test, not a transformer), so as +a *fast, standalone* product it defends itself. But treated as part of Overfit, it risks leaving neither +product with enough attention. + +**Verdict:** architecture sound; the statistical core must swap Z-score → Mann-Whitney + effect size + +control-vs-experiment + sequential testing (without which it is a toy). Ship the Sniper alone, cut the Guardian +and POH. Narrow market, but the gap is real and the "air-gapped Kayenta-lite" position is defensible. The +decisive question is focus, not code. + +--- + +*Related: `docs/gp-anomaly-baseline.md` (a different, model-based anomaly track). The two share the "anomaly" +word but nothing else — this one is a statistical canary analyser, that one is ML.*