Change Buffer #1453
Draft
Change Buffer #1453
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BenchmarksComparisonBenchmark execution time: 2026-02-10 20:31:01 Comparing candidate commit a472e65 in PR branch Found 10 performance improvements and 5 performance regressions! Performance is the same for 42 metrics, 2 unstable metrics. scenario:benching deserializing traces from msgpack to their internal representation
scenario:credit_card/is_card_number/ 378282246310005
scenario:credit_card/is_card_number/378282246310005
scenario:credit_card/is_card_number/x371413321323331
scenario:credit_card/is_card_number_no_luhn/x371413321323331
scenario:normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...
scenario:normalization/normalize_name/normalize_name/bad-name
scenario:normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...
scenario:normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters
CandidateCandidate benchmark detailsGroup 1
Group 2
Group 3
Group 4
Group 5
Group 6
Group 7
Group 8
Group 9
Group 10
Group 11
Group 12
Group 13
Group 14
Group 15
Group 16
Group 17
Group 18
Group 19
BaselineOmitted due to size. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #1453 +/- ##
==========================================
- Coverage 71.29% 70.93% -0.36%
==========================================
Files 416 418 +2
Lines 66872 67233 +361
==========================================
+ Hits 47676 47694 +18
- Misses 19196 19539 +343
🚀 New features to boost your workflow:
|
bengl
force-pushed
the
bengl/change-buffer
branch
from
ae7b0d1 to
4c76607
Compare
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
ekump
reviewed
Jan 21, 2026
| trace_span_counts: HashMap<u128, usize>, | ||
| tracer_service: T, | ||
| tracer_language: T, | ||
| pid: f64, |
Contributor
Author
There was a problem hiding this comment.
Not in trace exporter's config 🤷
Contributor
There was a problem hiding this comment.
Is this what you're referring to? Because we send it as a metric, which has to be f64?
Would it be better to do the conversion to f64 internally and not expose that implementation detail in the public API? Going from u32 to f64 should be cheap in Rust.
This is non-blocking. It's not a hill I need to die on.
ekump
reviewed
Jan 21, 2026
|
|
||
| impl From<u64> for OpCode { | ||
| fn from(val: u64) -> Self { | ||
| unsafe { std::mem::transmute(val) } |
Contributor
There was a problem hiding this comment.
This is UB if someone sends an invalid opcode. Could you use try_from instead?
Something like...
impl TryFrom<u64> for OpCode {
type Error = anyhow::Error;
fn try_from(val: u64) -> Result<Self, Self::Error> {
match val {
0 => Ok(OpCode::Create),
...
_ => Err(anyhow!("invalid opcode: {}", val))
}
}
}You'll need to update BufferedOperation::from_buf and flush_change_buffer to handle the Result, but both of those functions return Results anyway, so that shouldn't be a problem.
Contributor
Author
There was a problem hiding this comment.
I'd like to keep the transmute and just bounds-check it in the try_from. SGTY?
Contributor
There was a problem hiding this comment.
Is there any reason to do bound-check + transmute rather than exhaust matching?
Contributor
Author
There was a problem hiding this comment.
The main reason is to avoid having all the OpCodes listed out in two places. Maybe I can macro that away? 🤔
Contributor
There was a problem hiding this comment.
Maybe using a crate like num_enum or num_traits?
ekump
reviewed
Jan 21, 2026
| use crate::span::{Span, SpanText}; | ||
|
|
||
| #[derive(Clone, Copy)] | ||
| pub struct ChangeBuffer(*const u8); |
Contributor
There was a problem hiding this comment.
Instead of a raw pointer, can the buffer be &mut [u8]? If we can do that, I think you can move the unsafe code to the FFI layer / caller where it better belongs. The caller would need to do something like
let slice = unsafe { std::slice::from_raw_parts(ptr, len) };
let change_buffer = ChangeBuffer::new(slice); If we can do that, then we won't need the unsafe code in get_num_raw where every read has the potential for UB if we get the pointer math wrong. I think you'd also be able to get rid of the unsafe Send and Sync definitions.
Contributor
Author
There was a problem hiding this comment.
The problem is lifetimes. In napi.rs, the Buffer object we get from Node.js will have a lifetime as long as the function call. That said, the informal promise is that the Buffer will persist on the JS side forever, as good as 'static. Raw pointers allowed me to get around that.
I think I can take it a step further and convert that back to a slice with new ownership so that we can avoid all the unsafe. I'll give it a try.
ekump
reviewed
Jan 21, 2026
| .insert(T::from_static_str("_dd.rule_psr"), rule); | ||
| } | ||
| if let Some(rule) = trace.sampling_limit_decision { | ||
| span.metrics.insert(T::from_static_str("_dd.;_psr"), rule); |
hoolioh
reviewed
Jan 27, 2026
|
|
||
| impl From<u64> for OpCode { | ||
| fn from(val: u64) -> Self { | ||
| unsafe { std::mem::transmute(val) } |
Contributor
There was a problem hiding this comment.
Is there any reason to do bound-check + transmute rather than exhaust matching?
Contributor
|
This is more of a general comment. Since trace-utils is too crowded and we talked about refactoring it. Would it make sense to place this implmentation in a new crate? I don't say that I has to happen now. Maybe in a later PR when we split trace-utils. |
hoolioh
reviewed
Jan 27, 2026
hoolioh
reviewed
Jan 27, 2026
📚 Documentation Check Results📦
|
🔒 Cargo Deny Results📦
|
bengl
force-pushed
the
bengl/change-buffer
branch
from
5d9d55b to
210da31
Compare
hoolioh
approved these changes
Feb 2, 2026
| @@ -0,0 +1,397 @@ | |||
| use std::collections::HashMap; | |||
|
|
|||
| use anyhow::{anyhow, bail, Result}; | |||
Contributor
There was a problem hiding this comment.
Personal opinion about anyhow. I think it's good for either prototyping things or using it in standalone binaries. On the other hand using it in libraries has its downsides, the main one being bubbling errors up to the caller, hence difficulting the SDKs consuming our libraries to do proper error handling. That said, I think that is not a problem right now and we can talk about it in subsequent PRs.
Contributor
There was a problem hiding this comment.
non-blocking: +1. I don't think anyhow should be used in the public API for a library. But I'm ok with a TODO comment and taking care of it in the future.
hoolioh
approved these changes
Feb 2, 2026
|
|
||
| impl From<u64> for OpCode { | ||
| fn from(val: u64) -> Self { | ||
| unsafe { std::mem::transmute(val) } |
Contributor
There was a problem hiding this comment.
Maybe using a crate like num_enum or num_traits?
ekump
reviewed
Feb 3, 2026
| let slice = self.as_slice(); | ||
| let bytes = slice.get(*index..*index + size) | ||
| .ok_or(anyhow!("read_u64 out of bounds: offset={}, len={}", *index, self.len))?; | ||
| let array: [u8; 8] = bytes.try_into() |
Contributor
There was a problem hiding this comment.
This method is generic over T. What happens if T isn't 8 bytes, like u32 or u128?
Do you need to convert bytes into array here? I don't think we lose any error fidelity because I don't think it's possible for try_into to fail here? We have the size, and slice.get() uses size to get exactly the correct bytes.
You could just pass bytes directly with: Ok(T::from_bytes(bytes)).
ekump
reviewed
Feb 3, 2026
| pub fn flush_change_buffer(&mut self) -> Result<()> { | ||
| let mut index = 0; | ||
| let mut count = self.change_buffer.read::<u64>(&mut index)?; | ||
| index += 8; |
Contributor
There was a problem hiding this comment.
I might be misunderstanding, but doesn't the index get incremented in read? Won't this double increment the index?
Assuming I'm setting it up correctly this test would expose the double increment and the size issue in Read.
#[cfg(test)]
mod tests {
use super::*;
use libdd_tinybytes::BytesString;
#[test]
fn test_flush_change_buffer_single_create() {
// Buffer: [count=1, opcode=Create(0), span_id, trace_id, parent_id]
let mut buf = vec![];
buf.extend_from_slice(&1u64.to_le_bytes()); // count = 1
buf.extend_from_slice(&0u64.to_le_bytes()); // opcode = Create
buf.extend_from_slice(&123u64.to_le_bytes()); // span_id
buf.extend_from_slice(&456u128.to_le_bytes()); // trace_id
buf.extend_from_slice(&0u64.to_le_bytes()); // parent_id
let change_buffer =
unsafe { ChangeBuffer::from_raw_parts(buf.as_ptr(), buf.len()) };
let mut state = ChangeBufferState::<BytesString>::new(
change_buffer,
BytesString::from_static("service"),
BytesString::from_static("rust"),
1234.0,
);
state.flush_change_buffer().unwrap();
let span = state.get_span(&123).unwrap();
assert_eq!(span.span_id, 123);
assert_eq!(span.trace_id, 456);
}
}Also, I just realized the change buffer has no unit tests.
ekump
reviewed
Feb 3, 2026
|
|
||
| pub mod span; | ||
|
|
||
| pub mod change_buffer; |
Contributor
There was a problem hiding this comment.
non-blocking: Should we put the change_buffer behind an optional feature for now while it's still experimental?
ekump
approved these changes
Feb 9, 2026
Clippy Allow Annotation ReportComparing clippy allow annotations between branches:
Summary by Rule
Annotation Counts by File
Annotation Stats by Crate
About This ReportThis report tracks Clippy allow annotations for specific rules, showing how they've changed in this PR. Decreasing the number of these annotations generally improves code quality. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
A brief description of the change being made with this pull request.
Motivation
What inspired you to submit this pull request?
Additional Notes
Anything else we should know when reviewing?
How to test the change?
Describe here in detail how the change can be validated.