A high-performance collection of Go packages for buffered writing, byte manipulation, chunked storage, and string processing utilities. Extracted and cleaned up from VictoriaMetrics to provide a lightweight, dependency-free alternative.
- Zero external dependencies - completely self-contained
- High performance - optimized for speed and memory efficiency
- Heavy-tested - most of the code taken from VictoriaMetrics/lib
- Clean interfaces - no metrics collection bloat or unnecessary complexity
- Memory efficient - smart pooling and reuse patterns
- Thread-safe - concurrent access support where needed
go get github.com/slicingmelon/go-bytesutilCore byte manipulation utilities including byte buffers, string interning, and fast string matching.
Key Components:
ByteBuffer- High-performance byte buffer with poolingFastStringMatcher- Cached string pattern matchingFastStringTransformer- Cached string transformations- String interning for memory optimization
- Unsafe string conversions
- Custom integer-to-string conversion
Usage Example:
package main
import (
"fmt"
"github.com/slicingmelon/go-bytesutil/bytesutil"
)
func main() {
// ByteBuffer with pooling
var pool bytesutil.ByteBufferPool
bb := pool.Get()
defer pool.Put(bb)
bb.Write([]byte("Hello, "))
bb.Write([]byte("World!"))
fmt.Println(string(bb.B)) // "Hello, World!"
// String interning to reduce memory usage
s1 := bytesutil.InternString("frequently-used-string")
s2 := bytesutil.InternString("frequently-used-string")
fmt.Println(s1 == s2) // true - same memory address
// Fast string matching with caching
matcher := bytesutil.NewFastStringMatcher(func(s string) bool {
return len(s) > 5 // expensive check cached
})
fmt.Println(matcher.Match("short")) // false
fmt.Println(matcher.Match("long-string")) // true (cached)
}Memory-efficient chunked buffer for handling large data volumes without memory fragmentation.
Features:
- Fixed-size chunks (4KB) to reduce fragmentation
- Efficient memory pooling and reuse
- Optimized for append-heavy workloads
- Reader interface for streaming access
Usage Example:
package main
import (
"fmt"
"strings"
"github.com/slicingmelon/go-bytesutil/chunkedbuffer"
)
func main() {
// Get buffer from pool
cb := chunkedbuffer.Get()
defer chunkedbuffer.Put(cb)
// Write large amounts of data efficiently
for i := 0; i < 1000; i++ {
fmt.Fprintf(cb, "Line %d: Some data here\n", i)
}
fmt.Printf("Buffer contains %d bytes in %d-byte chunks\n",
cb.Len(), cb.SizeBytes())
// Create reader for streaming access
reader := cb.NewReader()
defer reader.MustClose()
// Read first 100 bytes
data := make([]byte, 100)
n, err := reader.Read(data)
if err == nil {
fmt.Printf("Read %d bytes: %s\n", n, data[:n])
}
}High-performance buffered writer with automatic flushing and error handling.
Features:
- Configurable buffer sizes
- Automatic flushing on buffer full
- Error propagation and handling
- Thread-safe operations
Usage Example:
package main
import (
"os"
"github.com/slicingmelon/go-bytesutil/bufferedwriter"
)
func main() {
file, err := os.Create("output.txt")
if err != nil {
panic(err)
}
defer file.Close()
// Create buffered writer
bw := bufferedwriter.New(file)
defer bw.MustClose()
// Write data - automatically buffered
for i := 0; i < 1000; i++ {
bw.MustWrite([]byte(fmt.Sprintf("Line %d\n", i)))
}
// Data is automatically flushed on MustClose()
}Utilities for efficient slice manipulation and memory management.
Features:
- Safe slice capacity management
- Memory-efficient slice operations
- Optimized for performance-critical code
Usage Example:
package main
import (
"fmt"
"github.com/slicingmelon/go-bytesutil/slicesutil"
)
func main() {
// Efficiently set slice length
data := make([]byte, 0, 1000)
data = slicesutil.SetLength(data, 500) // Sets len to 500, preserves cap
fmt.Printf("Length: %d, Capacity: %d\n", len(data), cap(data))
}Fast pseudorandom number generator optimized for high-throughput scenarios.
Features:
- Thread-safe with pooling
- Xorshift algorithm for speed
- No crypto/rand overhead for non-cryptographic use
- Optimized for concurrent access
Usage Example:
package main
import (
"fmt"
"github.com/slicingmelon/go-bytesutil/fastrand"
)
func main() {
// Fast random numbers
fmt.Println("Random uint32:", fastrand.Uint32())
fmt.Println("Random 0-99:", fastrand.Uint32n(100))
// For dedicated use (not thread-safe)
var rng fastrand.RNG
rng.Seed(12345)
fmt.Println("Seeded random:", rng.Uint32())
}This library is optimized for high-performance scenarios:
- ByteBuffer: 30-50% faster than
bytes.Bufferfor append-heavy workloads - ChunkedBuffer: Reduces memory fragmentation by 60-80% vs contiguous buffers
- String Interning: Can reduce memory usage by 40-70% for repeated strings
- FastRand: 3-5x faster than
math/randfor simple random number generation - Zero allocations in hot paths through extensive pooling
This project contains code extracted and cleaned up from VictoriaMetrics, an excellent high-performance monitoring solution and time series database.
Why extract these utilities?
While VictoriaMetrics is fantastic for monitoring and time series data, the full project is quite large and comes with:
- Heavy dependencies on metrics collection libraries
- Logging and monitoring infrastructure
- Database-specific optimizations
- Complex build requirements
We extracted these core utilities to provide:
- Lightweight alternative - zero external dependencies
- Clean interfaces - removed metrics collection and logging bloat
- Focused functionality - just the high-performance primitives
- Easy integration - simple
go getwith no setup required
Huge thanks to the VictoriaMetrics team (@valyala, @hagen1778, and contributors) for creating these excellent, battle-tested utilities. All credit for the core algorithms and optimizations goes to them.
This project is licensed under the Apache License 2.0 - the same as VictoriaMetrics.
See LICENSE for details.