diff --git a/CHANGELOG.md b/CHANGELOG.md index ab62038..f79ab46 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,8 @@ All notable changes to this project will be documented in this file. - Added transparent `.gz` and `.zst` reads for persisted file APIs when the corresponding compression feature is enabled. +- Added a non-public binary log record format and reader research prototype; + no production binary backend is enabled yet. ## [v1.0.2] - 2026-09-19 diff --git a/docs/adr/0008-binary-log-record-format-research.md b/docs/adr/0008-binary-log-record-format-research.md new file mode 100644 index 0000000..e17d7cb --- /dev/null +++ b/docs/adr/0008-binary-log-record-format-research.md @@ -0,0 +1,57 @@ +# ADR 0008: Binary log record format research prototype + +- Status: Accepted +- Date: 2026-09-19 + +## Context + +Binary logging is a possible future backend, but committing to a wire format +before its compatibility and corruption behavior are explicit would make later +readers difficult to evolve. The existing MDBX value format is storage-specific +and is not a suitable public file contract. + +## Decision + +The research prototype uses a framed, big-endian record format with the magic +`LGBR`, a one-byte format version, one-byte flags, reserved bytes, and a +32-bit payload length. Version 1 stores the stable `LogRecordSnapshot` fields: +session id, timestamp, sequence, level, payload id, source line, and three +length-prefixed UTF-8 strings for message, file, and function. + +The prototype reader is deliberately outside the installed library API. It +rejects unknown versions and flags, malformed or truncated frames, invalid log +levels, oversized fields, and trailing payload bytes. A stream is decoded one +length-delimited frame at a time, so a future reader can skip or recover around +individual records without relying on native struct layout. + +This is a format and reader experiment only. No production backend, file +extension, persistence policy, compression, or compatibility promise is added +until the format is reviewed against real workloads and migration requirements. + +## Consequences + +The prototype provides deterministic bytes and explicit failure modes that can +be reviewed and fuzzed independently of a backend. Big-endian fields and +length-prefixed strings avoid ABI and host-endian coupling. The current +whole-record decoder has bounded allocations and keeps all decoded strings +owning. + +The format carries no schema negotiation beyond its version and flags, and it +does not yet encode MDC/NDC context, arbitrary attributes, or formatter state. +Those omissions are intentional until the production record contract is chosen. + +## Alternatives considered + +- Reusing the MDBX value serializer would couple a file format to one storage + backend and its migration history. +- Dumping native C++ structs would expose padding, endianness, enum width, and + ABI details. +- Adopting an external serialization library before defining the record contract + would add dependency and schema commitments prematurely. + +## References + +- `tests/binary_log_record_codec.hpp` +- `tests/binary_log_record_codec_test.cpp` +- `docs/future-plans.md` + diff --git a/docs/adr/README.md b/docs/adr/README.md index c58a45f..0882ba3 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -30,3 +30,4 @@ explains **why the current boundary or trade-off exists**. - [0005 — Benchmark evidence and comparison methodology](0005-benchmark-methodology.md) - [0006 — Explicit capability for concurrent dispatch](0006-concurrent-dispatch-capability.md) - [0007 — Transparent compressed-file reads](0007-transparent-compressed-file-reads.md) +- [0008 — Binary log record format research prototype](0008-binary-log-record-format-research.md) diff --git a/docs/future-plans.md b/docs/future-plans.md index 125f40f..65408b5 100644 --- a/docs/future-plans.md +++ b/docs/future-plans.md @@ -42,9 +42,9 @@ Legend: - [x] **Release 1.0.2** — changelog/release notes, package overlays, non-`-dev` documentation, annotated tag, GitHub Release, and published Pages site are complete. -- [ ] **Binary logging research** — choose a versioned binary record format, - define compatibility/versioning rules, and prototype a reader before adding - a production backend. +- [x] **Binary logging research** — a framed, versioned record format and + bounded reader prototype with compatibility/corruption rules are documented + in ADR 0008. No production backend or persistence contract is committed yet. - [x] **Transparent compressed-file reads** — `read_log_file()` and `read_log_files()` now decompress `.gz`/`.zst` entries when the corresponding feature is enabled; disabled-feature and malformed-input paths remain diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 7475f16..3492593 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -12,6 +12,7 @@ else() backpressure_ordering_spsc_test.cpp backpressure_ordering_test.cpp backpressure_policy_test.cpp + binary_log_record_codec_test.cpp backend_shutdown_terminal_test.cpp compiled_level_runtime_caveat_test.cpp compiled_level_test.cpp diff --git a/tests/binary_log_record_codec.hpp b/tests/binary_log_record_codec.hpp new file mode 100644 index 0000000..866c1e3 --- /dev/null +++ b/tests/binary_log_record_codec.hpp @@ -0,0 +1,266 @@ +#pragma once +#ifndef LOGIT_CPP_TEST_BINARY_LOG_RECORD_CODEC_HPP_INCLUDED +#define LOGIT_CPP_TEST_BINARY_LOG_RECORD_CODEC_HPP_INCLUDED + +#include + +#include +#include +#include +#include +#include +#include + +namespace logit_binary_research { + +enum class DecodeError { + None, + Truncated, + BadMagic, + UnsupportedVersion, + UnsupportedFlags, + InvalidFrame, + InvalidField +}; + +struct DecodeResult { + bool ok = false; + DecodeError error = DecodeError::None; + std::size_t consumed = 0; + logit::LogRecordSnapshot record; +}; + +struct StreamResult { + bool ok = false; + DecodeError error = DecodeError::None; + std::size_t error_offset = 0; + std::vector records; +}; + +namespace detail { + +static const std::size_t kHeaderSize = 12; +static const std::size_t kMaxFrameSize = 64u * 1024u * 1024u; +static const std::size_t kMaxFieldSize = 16u * 1024u * 1024u; + +inline void append_u8(std::vector& out, uint8_t value) { + out.push_back(value); +} + +inline void append_u16(std::vector& out, uint16_t value) { + out.push_back(static_cast((value >> 8) & 0xFFu)); + out.push_back(static_cast(value & 0xFFu)); +} + +inline void append_u32(std::vector& out, uint32_t value) { + for (int shift = 24; shift >= 0; shift -= 8) { + out.push_back(static_cast((value >> shift) & 0xFFu)); + } +} + +inline void append_u64(std::vector& out, uint64_t value) { + for (int shift = 56; shift >= 0; shift -= 8) { + out.push_back(static_cast((value >> shift) & 0xFFu)); + } +} + +inline void append_string(std::vector& out, const std::string& value) { + if (value.size() > kMaxFieldSize || + value.size() > static_cast((std::numeric_limits::max)())) { + throw std::length_error("binary log record field is too large"); + } + append_u32(out, static_cast(value.size())); + out.insert(out.end(), value.begin(), value.end()); +} + +class Cursor { +public: + Cursor(const uint8_t* data, std::size_t size) : m_data(data), m_size(size) {} + + bool read_u8(uint8_t& value) { + if (!take(1)) return false; + value = m_data[m_offset++]; + return true; + } + + bool read_u32(uint32_t& value) { + if (!take(4)) return false; + value = 0; + for (int i = 0; i < 4; ++i) { + value = (value << 8) | static_cast(m_data[m_offset++]); + } + return true; + } + + bool read_u64(uint64_t& value) { + if (!take(8)) return false; + value = 0; + for (int i = 0; i < 8; ++i) { + value = (value << 8) | static_cast(m_data[m_offset++]); + } + return true; + } + + bool read_string(std::string& value) { + uint32_t size = 0; + if (!read_u32(size) || size > kMaxFieldSize || !take(size)) return false; + value.assign(reinterpret_cast(m_data + m_offset), size); + m_offset += size; + return true; + } + + std::size_t remaining() const { return m_size - m_offset; } + +private: + const uint8_t* m_data; + std::size_t m_size; + std::size_t m_offset = 0; + + bool take(std::size_t size) const { + return size <= m_size - m_offset; + } +}; + +inline DecodeResult decode_payload(const uint8_t* data, std::size_t size) { + DecodeResult result; + Cursor cursor(data, size); + uint64_t session_id = 0; + uint64_t timestamp_bits = 0; + uint32_t sequence = 0; + uint8_t level = 0; + uint8_t reserved[3] = {0, 0, 0}; + uint64_t payload_id = 0; + uint32_t line_bits = 0; + if (!cursor.read_u64(session_id) || + !cursor.read_u64(timestamp_bits) || + !cursor.read_u32(sequence) || + !cursor.read_u8(level) || + !cursor.read_u8(reserved[0]) || + !cursor.read_u8(reserved[1]) || + !cursor.read_u8(reserved[2]) || + !cursor.read_u64(payload_id) || + !cursor.read_u32(line_bits)) { + result.error = DecodeError::Truncated; + return result; + } + if (level >= 6 || reserved[0] != 0 || reserved[1] != 0 || reserved[2] != 0) { + result.error = DecodeError::InvalidField; + return result; + } + if (!cursor.read_string(result.record.message) || + !cursor.read_string(result.record.file) || + !cursor.read_string(result.record.function) || + cursor.remaining() != 0) { + result.error = DecodeError::InvalidField; + return result; + } + + result.record.session_id = session_id; + result.record.timestamp_ms = static_cast(timestamp_bits); + result.record.sequence = sequence; + result.record.level = static_cast(level); + result.record.payload_id = payload_id; + result.record.line = static_cast(line_bits); + result.ok = true; + result.error = DecodeError::None; + return result; +} + +} // namespace detail + +inline std::vector encode(const logit::LogRecordSnapshot& record) { + std::vector payload; + payload.reserve(64 + record.message.size() + record.file.size() + record.function.size()); + detail::append_u64(payload, record.session_id); + detail::append_u64(payload, static_cast(record.timestamp_ms)); + detail::append_u32(payload, record.sequence); + detail::append_u8(payload, static_cast(record.level)); + detail::append_u8(payload, 0); + detail::append_u8(payload, 0); + detail::append_u8(payload, 0); + detail::append_u64(payload, record.payload_id); + detail::append_u32(payload, static_cast(record.line)); + detail::append_string(payload, record.message); + detail::append_string(payload, record.file); + detail::append_string(payload, record.function); + if (payload.size() > detail::kMaxFrameSize || + payload.size() > static_cast((std::numeric_limits::max)())) { + throw std::length_error("binary log record frame is too large"); + } + + std::vector frame; + frame.reserve(detail::kHeaderSize + payload.size()); + frame.push_back('L'); + frame.push_back('G'); + frame.push_back('B'); + frame.push_back('R'); + detail::append_u8(frame, 1); + detail::append_u8(frame, 0); + detail::append_u16(frame, 0); + detail::append_u32(frame, static_cast(payload.size())); + frame.insert(frame.end(), payload.begin(), payload.end()); + return frame; +} + +inline DecodeResult decode_one(const uint8_t* data, std::size_t size) { + DecodeResult result; + if (data == nullptr && size != 0) { + result.error = DecodeError::Truncated; + return result; + } + if (size < detail::kHeaderSize) { + result.error = DecodeError::Truncated; + return result; + } + if (data[0] != 'L' || data[1] != 'G' || data[2] != 'B' || data[3] != 'R') { + result.error = DecodeError::BadMagic; + return result; + } + if (data[4] != 1) { + result.error = DecodeError::UnsupportedVersion; + return result; + } + if (data[5] != 0 || data[6] != 0 || data[7] != 0) { + result.error = DecodeError::UnsupportedFlags; + return result; + } + const uint32_t frame_size = + (static_cast(data[8]) << 24) | + (static_cast(data[9]) << 16) | + (static_cast(data[10]) << 8) | + static_cast(data[11]); + if (frame_size == 0 || frame_size > detail::kMaxFrameSize) { + result.error = DecodeError::InvalidFrame; + return result; + } + if (static_cast(frame_size) > size - detail::kHeaderSize) { + result.error = DecodeError::Truncated; + return result; + } + + result = detail::decode_payload(data + detail::kHeaderSize, frame_size); + if (result.ok) result.consumed = detail::kHeaderSize + frame_size; + return result; +} + +inline StreamResult decode_stream(const std::vector& bytes) { + StreamResult result; + std::size_t offset = 0; + while (offset < bytes.size()) { + const DecodeResult one = decode_one(bytes.data() + offset, bytes.size() - offset); + if (!one.ok) { + result.error = one.error; + result.error_offset = offset; + return result; + } + result.records.push_back(one.record); + offset += one.consumed; + } + result.ok = true; + result.error = DecodeError::None; + return result; +} + +} // namespace logit_binary_research + +#endif // LOGIT_CPP_TEST_BINARY_LOG_RECORD_CODEC_HPP_INCLUDED diff --git a/tests/binary_log_record_codec_test.cpp b/tests/binary_log_record_codec_test.cpp new file mode 100644 index 0000000..8281c61 --- /dev/null +++ b/tests/binary_log_record_codec_test.cpp @@ -0,0 +1,93 @@ +#include "binary_log_record_codec.hpp" + +#include +#include +#include + +namespace { + +logit::LogRecordSnapshot make_record() { + logit::LogRecordSnapshot record; + record.session_id = 42; + record.timestamp_ms = -123456789; + record.sequence = 7; + record.level = logit::LogLevel::LOG_LVL_WARN; + record.message = "binary message"; + record.payload_id = 99; + record.file = "source.cpp"; + record.function = "emit"; + record.line = 314; + return record; +} + +void assert_same(const logit::LogRecordSnapshot& lhs, const logit::LogRecordSnapshot& rhs) { + assert(lhs.session_id == rhs.session_id); + assert(lhs.timestamp_ms == rhs.timestamp_ms); + assert(lhs.sequence == rhs.sequence); + assert(lhs.level == rhs.level); + assert(lhs.message == rhs.message); + assert(lhs.payload_id == rhs.payload_id); + assert(lhs.file == rhs.file); + assert(lhs.function == rhs.function); + assert(lhs.line == rhs.line); +} + +} // namespace + +int main() { + const logit::LogRecordSnapshot first = make_record(); + const std::vector first_bytes = logit_binary_research::encode(first); + const logit_binary_research::DecodeResult decoded = + logit_binary_research::decode_one(first_bytes.data(), first_bytes.size()); + assert(decoded.ok); + assert(decoded.consumed == first_bytes.size()); + assert_same(first, decoded.record); + + logit::LogRecordSnapshot second = first; + second.sequence = 8; + second.message = "second binary message"; + const std::vector second_bytes = logit_binary_research::encode(second); + std::vector stream = first_bytes; + stream.insert(stream.end(), second_bytes.begin(), second_bytes.end()); + const logit_binary_research::StreamResult decoded_stream = + logit_binary_research::decode_stream(stream); + assert(decoded_stream.ok); + assert(decoded_stream.records.size() == 2); + assert_same(first, decoded_stream.records[0]); + assert_same(second, decoded_stream.records[1]); + + std::vector truncated(first_bytes.begin(), first_bytes.end() - 1); + assert(logit_binary_research::decode_one(truncated.data(), truncated.size()).error == + logit_binary_research::DecodeError::Truncated); + + std::vector bad_magic = first_bytes; + bad_magic[0] = 'X'; + assert(logit_binary_research::decode_one(bad_magic.data(), bad_magic.size()).error == + logit_binary_research::DecodeError::BadMagic); + + std::vector future_version = first_bytes; + future_version[4] = 2; + assert(logit_binary_research::decode_one(future_version.data(), future_version.size()).error == + logit_binary_research::DecodeError::UnsupportedVersion); + + std::vector unsupported_flags = first_bytes; + unsupported_flags[5] = 1; + assert(logit_binary_research::decode_one(unsupported_flags.data(), unsupported_flags.size()).error == + logit_binary_research::DecodeError::UnsupportedFlags); + + std::vector oversized_frame = first_bytes; + oversized_frame[8] = 0xFF; + oversized_frame[9] = 0xFF; + oversized_frame[10] = 0xFF; + oversized_frame[11] = 0xFF; + assert(logit_binary_research::decode_one(oversized_frame.data(), oversized_frame.size()).error == + logit_binary_research::DecodeError::InvalidFrame); + + stream.push_back(0); + const logit_binary_research::StreamResult trailing = + logit_binary_research::decode_stream(stream); + assert(!trailing.ok); + assert(trailing.error == logit_binary_research::DecodeError::Truncated); + assert(trailing.error_offset == first_bytes.size() + second_bytes.size()); + return 0; +}