Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
541a0fe
Persist custom and OOTB log fields in state
snowp Sep 9, 2026
1bb9a75
Simplify virtual log field resolution
snowp Sep 9, 2026
a74c400
Protect virtual field ownership
snowp Sep 9, 2026
1cd6c7c
Satisfy state scope lint
snowp Sep 9, 2026
e312dd1
Match nightly formatting
snowp Sep 9, 2026
c3f17b9
Clarify initial field capacity handling
snowp Sep 9, 2026
3b72029
Address state field review feedback
snowp Sep 9, 2026
7a89d38
Fix virtual field test data
snowp Sep 9, 2026
a703c13
Reconcile initial state-backed fields
snowp Sep 10, 2026
51a4709
Use prior state for previous-process logs
snowp Sep 10, 2026
8223cd8
Document prior global field fallback
snowp Sep 10, 2026
b72909e
Address state field review feedback
snowp Sep 10, 2026
d67c815
Simplify virtual field state handling
snowp Sep 10, 2026
ac7a16f
Fix state field CI failures
snowp Sep 10, 2026
706c05c
Simplify virtual log field persistence
snowp Sep 10, 2026
43353dc
Fix state field test compilation
snowp Sep 10, 2026
72804f2
Satisfy state field clippy lint
snowp Sep 10, 2026
0a23574
Require state admission for log fields
snowp Sep 10, 2026
87f9357
Skip unsupported multi-tag state values
snowp Sep 10, 2026
44513ff
Document graceful state downgrade
snowp Sep 10, 2026
3cfae25
Fix rebased log field tests
snowp Sep 10, 2026
5872c0d
Fix log field ownership handling
snowp Sep 10, 2026
bfa3c66
Honor state fields in generated logs
snowp Sep 10, 2026
cf1c202
Simplify state-backed log field replay
snowp Sep 10, 2026
c7bb7a2
Deflake stats index rotation test
snowp Sep 10, 2026
075fcdc
Document state-backed filter transforms
snowp Sep 10, 2026
ceaafef
Retain state after clear capacity failures
snowp Sep 10, 2026
99386e8
Refine state-backed log field replay
snowp Sep 11, 2026
8b162c2
Share previous-run state snapshot
snowp Sep 11, 2026
91f6a61
Preserve legacy previous-run inline fields
snowp Sep 11, 2026
c1972c6
Document state field filter semantics
snowp Sep 14, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions bd-api/src/api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -587,6 +587,8 @@ impl Api {
StateScope::FEATURE_FLAG => Some(bd_state::Scope::FeatureFlagExposure),
StateScope::GLOBAL_STATE => Some(bd_state::Scope::GlobalState),
StateScope::SYSTEM => Some(bd_state::Scope::System),
// Log-field scopes are owned by the SDK metadata collector. Server-pushed state updates
// must not overwrite their virtual matcher/extractor view independently of live metadata.
StateScope::CUSTOM_FIELDS | StateScope::OOTB_FIELDS | StateScope::UNSPECIFIED => None,
}
}
Expand Down
7 changes: 7 additions & 0 deletions bd-api/src/api_test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ use bd_proto::protos::client::api::{
};
use bd_proto::protos::logging::payload::LogType;
use bd_proto::protos::logging::payload::data::Data_type;
use bd_proto::protos::state::scope::StateScope;
use bd_proto::protos::workflow::workflow::workflow::action::action_flush_buffers;
use bd_runtime::runtime::{ConfigLoader, FeatureFlag};
use bd_state::StateReader;
Expand Down Expand Up @@ -725,6 +726,12 @@ fn make_server_pushed_feature_flag_update(flag: &str, value: &str) -> ClientStat
update
}

#[test]
fn server_state_updates_exclude_sdk_owned_log_field_scopes() {
assert_eq!(Api::map_state_scope(StateScope::CUSTOM_FIELDS.into()), None);
assert_eq!(Api::map_state_scope(StateScope::OOTB_FIELDS.into()), None);
}

fn make_server_pushed_feature_flag_clear(flag: &str) -> ClientStateUpdate {
let mut update = ClientStateUpdate::new();
update.set_clear_client_state(client_state_update::ClearClientState {
Expand Down
4 changes: 2 additions & 2 deletions bd-crash-handler/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ pub trait CrashReportHook: Send + Sync {
#[derive(Clone)]
pub struct Monitor {
report_directory: PathBuf,
previous_run_state: bd_versioned_kv::ScopedMaps,
previous_run_state: Arc<bd_versioned_kv::ScopedMaps>,
state: bd_state::Store,
artifact_client: Arc<dyn bd_artifact_upload::Client>,

Expand All @@ -158,7 +158,7 @@ impl Monitor {
session: Arc<bd_session::Strategy>,
init_lifecycle: &InitLifecycleState,
state: bd_state::Store,
previous_run_state: bd_versioned_kv::ScopedMaps,
previous_run_state: Arc<bd_versioned_kv::ScopedMaps>,
emit_log: impl Fn(CrashLog) -> anyhow::Result<()> + Send + Sync + 'static,
crash_report_hook: Option<Arc<dyn CrashReportHook>>,
) -> Self {
Expand Down
2 changes: 1 addition & 1 deletion bd-crash-handler/src/monitor_test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -322,7 +322,7 @@ impl Setup {
session,
&InitLifecycleState::new(),
(*state).clone(),
previous_run_state,
Arc::new(previous_run_state),
emit_log,
crash_report_hook,
);
Expand Down
11 changes: 11 additions & 0 deletions bd-log-filter/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,17 @@ impl FilterChain {
(Self { filters }, failures_count)
}

/// Applies matching filters and their inline-log transforms.
///
/// Matchers may read state-backed custom and OOTB fields through `state`, but transforms never
/// materialize, redact, replace, or remove those values. Later state-aware consumers resolve
/// the same state again, so transforms apply only to fields concretely present on `log`.
Comment thread
snowp marked this conversation as resolved.
Comment on lines +93 to +95

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this a problem in the future when we don't dual write? We have to be able to redact/transform what winds up in state also. Maybe this PR is too bug but is there any reason to not just do this now?

IIRC even today we do transforms before matching so how is this any different?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah part of this is that it was getting too complex since you end up having to create a new map that holds all the re-written fields from state that needs to be plumbed through in addition to the state reader which holds the current feature flags.

This is all a bit messy since if we want to rewrite state in a way that is reflected in timeline the state transformation needs to be consistently applied in the snapshot as well, but the current Filter logic allows different transformations for different logs. I figured for now we'd keep this simple and figure out how we want this to work for state separately. I can update the comment to be clearer on the path forward, it's something I need to figure out

///
/// TODO(snowp): Before eliding these fields from logs, preserve filter semantics with a per-log
/// state overlay. The overlay must carry rewritten and removed state values through every
/// downstream consumer, including buffer/workflow matching and tailing, without mutating the
/// global state. Snapshot uploads need the corresponding resolved values, not the current
/// state, because a snapshot can outlive the filter configuration that transformed its log.
pub fn process(&self, log: &mut Log, state: &dyn bd_state::StateReader) {
for filter in &self.filters {
let fields_ref = FieldsRef::new(&log.fields, &log.matching_fields);
Expand Down
Loading
Loading