Skip to content

REF-050.md

Latest commit

 

History

History
74 lines (59 loc) · 2.96 KB

REF-050.md

File metadata and controls

74 lines (59 loc) · 2.96 KB
Raw
id REF-050
workflow github
github_issue emulebb/emulebb#103
title Tune compression and hash buffers with bounded memory
status OPEN
priority Minor
category refactor
labels
compression
hashing
zlib
memory
performance
post-0.7.3
milestone post-0.7.3
created 2026-05-27
source senior C++ performance and I/O review of network and filesystem hot paths

Workflow status is tracked in GitHub: emulebb/emulebb#103. This local document is retained as an engineering spec/evidence record.

REF-050 - Tune compression and hash buffers with bounded memory

Summary

Compression and hashing hot paths still allocate or buffer conservatively. The safe improvement direction is bounded reuse and measurement, not unbounded RAM use or protocol-visible default changes.

Current Evidence

  • Packet::PackPacket uses Z_BEST_COMPRESSION for live packed traffic, including Kad/source/file-list callers.
  • Decompression paths allocate fresh buffers and can retry-grow per packet.
  • CKnownFile::CreateHash reads with a 64 KiB stack buffer, while nearby file hashing configures a smaller stdio buffer.
  • Larger read buffers could improve sequential hashing, but AICH boundary handling must remain exact.

Intended Shape

  1. Add low-overhead timing/count counters for compression time, input size, output size, and fragmentation-sensitive packet sizes.
  2. Replace repeated decompression allocations with thread-owned or scoped bounded scratch buffers that have explicit maximum retained memory.
  3. Benchmark Z_BEST_COMPRESSION, Z_DEFAULT_COMPRESSION, and a size-aware policy on a corpus of Kad/source/file-list packets before changing defaults.
  4. Increase hash read buffering conservatively, for example a heap-backed 128 KiB buffer, only with MD4/AICH golden coverage.
  5. Defer multi-megabyte buffers or retained global caches until profiling shows a clear benefit and memory caps are explicit.

Scope Constraints

  • Do not change packet formats, opcode semantics, or zlib framing.
  • Do not change the compression level default without corpus benchmarks and fragmentation analysis.
  • Do not retain unbounded per-peer or global scratch memory.
  • Do not change AICH or ED2K hash results; boundary behavior must be golden tested.

Acceptance Criteria

  • Compression counters expose per-path timing and size behavior.
  • Decompression uses bounded reusable scratch storage with explicit thread ownership or scope ownership.
  • Compression-level benchmarks compare CPU, size, and fragmentation risk.
  • Hash read-buffer changes pass MD4/AICH golden files across boundary sizes.
  • Memory-retention tests prove scratch buffers stay within configured caps.

Validation

  • python -m emule_workspace validate
  • compressed-packet fuzz and malformed-packet tests
  • Kad burst receive benchmark
  • MD4/AICH golden hash tests across boundary sizes
  • x64 Debug and Release app builds before commit