GeomReg

Geometric regulation and correlation dimension tracking for autoregressive text generation in C++

About

Large language models can fall into mode collapse during long-horizon generation — producing repetitive, looping, or increasingly bland text.

• Correlation dimension tracking — an online geometric diagnostic that detects mode collapse earlier and more robustly than token-level proxies like entropy or Distinct-n.
• Reinforced Mode Regulation (RMR) — a lightweight inference-time intervention that identifies and dampens self-reinforcing directions in the Transformer value cache, preventing geometric collapse without degrading text quality.

Dependencies

• C++17 or later
• Eigen — header-only linear algebra
• llama.cpp — built separately, linked as a library

Building

RMR requires a patched llama.cpp build that exposes mutable Transformer value-cache rows. A plain upstream llama.cpp checkout is enough for basic generation only; --rmr will not work unless the GeomReg V-cache patch is applied before building libllama.

git clone https://github.com/Linear-Fox-Labs/GeomReg
cd GeomReg

mkdir -p requirements
git clone --branch 3.4.0 --depth 1 https://gitlab.com/libeigen/eigen.git requirements/eigen
git clone https://github.com/ggerganov/llama.cpp requirements/llama.cpp

# Apply the GeomReg V-cache patch to requirements/llama.cpp before building.
# CPU-only for testing.
cmake -B requirements/llama.cpp/build-rmr-cpu \
      -S requirements/llama.cpp \
      -DCMAKE_BUILD_TYPE=Release \
      -DGGML_METAL=OFF \
      -DGGML_CCACHE=OFF
cmake --build requirements/llama.cpp/build-rmr-cpu \
      --config Release \
      --target llama \
      -j$(sysctl -n hw.logicalcpu)

cmake -B build-rmr-cpu \
      -S . \
      -DLLAMA_DIR=requirements/llama.cpp \
      -DEIGEN_INCLUDE_DIR=requirements/eigen \
      -DLLAMA_LIB=requirements/llama.cpp/build-rmr-cpu/bin/libllama.dylib
cmake --build build-rmr-cpu -j$(sysctl -n hw.logicalcpu)

Quick sanity check:

build-rmr-cpu/geom-reg --help | grep -A6 RMR
build-rmr-cpu/geom-reg \
  --model /path/to/model.gguf \
  --prompt "Say something about life" \
  --n-gen 20 \
  --temp 0.5 \
  --rmr

Authors

• Nathan Fargo at Linear Fox Labs. @x | @github

Contributing

Contributions are welcome! Please open an issue or submit a pull request with any improvements, bug fixes, or new features.

References

• Du, X. & Tanaka-Ishii, K. (2026). Escaping Mode Collapse in LLM Generation via Geometric Regulation. arXiv:2605.00435.

• Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2020). The Curious Case of Neural Text Degeneration. ICLR 2020.

• Pipis, C., Garg, S., Kontonis, V., Shrivastava, V., Krishnamurthy, A., & Papailiopoulos, D. (2025). Wait, Wait, Wait... Why Do Reasoning Models Loop? arXiv:2512.12895.

• Belitsky, M. et al. (2025). KV Cache Steering for Controlling Frozen LLMs. arXiv:2507.08799.