-
Notifications
You must be signed in to change notification settings - Fork 3
Persist custom and OOTB log fields in state #633
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
541a0fe
1bb9a75
a74c400
1cd6c7c
e312dd1
c3f17b9
3b72029
7a89d38
a703c13
51a4709
8223cd8
b72909e
d67c815
ac7a16f
706c05c
43353dc
72804f2
0a23574
87f9357
44513ff
3cfae25
5872c0d
bfa3c66
cf1c202
c7bb7a2
075fcdc
ceaafef
99386e8
8b162c2
91f6a61
c1972c6
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -88,6 +88,17 @@ impl FilterChain { | |
| (Self { filters }, failures_count) | ||
| } | ||
|
|
||
| /// Applies matching filters and their inline-log transforms. | ||
| /// | ||
| /// Matchers may read state-backed custom and OOTB fields through `state`, but transforms never | ||
| /// materialize, redact, replace, or remove those values. Later state-aware consumers resolve | ||
| /// the same state again, so transforms apply only to fields concretely present on `log`. | ||
|
Comment on lines
+93
to
+95
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Isn't this a problem in the future when we don't dual write? We have to be able to redact/transform what winds up in state also. Maybe this PR is too bug but is there any reason to not just do this now? IIRC even today we do transforms before matching so how is this any different?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah part of this is that it was getting too complex since you end up having to create a new map that holds all the re-written fields from state that needs to be plumbed through in addition to the state reader which holds the current feature flags. This is all a bit messy since if we want to rewrite state in a way that is reflected in timeline the state transformation needs to be consistently applied in the snapshot as well, but the current Filter logic allows different transformations for different logs. I figured for now we'd keep this simple and figure out how we want this to work for state separately. I can update the comment to be clearer on the path forward, it's something I need to figure out |
||
| /// | ||
| /// TODO(snowp): Before eliding these fields from logs, preserve filter semantics with a per-log | ||
| /// state overlay. The overlay must carry rewritten and removed state values through every | ||
| /// downstream consumer, including buffer/workflow matching and tailing, without mutating the | ||
| /// global state. Snapshot uploads need the corresponding resolved values, not the current | ||
| /// state, because a snapshot can outlive the filter configuration that transformed its log. | ||
| pub fn process(&self, log: &mut Log, state: &dyn bd_state::StateReader) { | ||
| for filter in &self.filters { | ||
| let fields_ref = FieldsRef::new(&log.fields, &log.matching_fields); | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.