Make large MI measurement logs more compact#102
Open
okt-viacheslav wants to merge 5 commits into
Open
Conversation
Allow users to log measurements in non-standard units without letting TE perform any unit convertions. This is primarily useful for graphs, e.g. this allows the user to log and view data in micro- or nanoseconds instead of having them converted to seconds, which hurts readability. Signed-off-by: Viacheslav Galaktionov <viacheslav.galaktionov@oktet.tech>
This functionality will be needed in later patches. Signed-off-by: Viacheslav Galaktionov <viacheslav.galaktionov@oktet.tech>
ol-andrewr
requested changes
Feb 19, 2026
Provide test with a memory-efficient way to log large sets of measured values that share the same properties. Store these values in a separate field called "values" and introduce a new aggregation type called "series". Signed-off-by: Viacheslav Galaktionov <viacheslav.galaktionov@oktet.tech>
Parse the new "values" field from MI measurement JSON and extract the values. Make sure the measured values are shown correctly in HTML and text logs. Signed-off-by: Viacheslav Galaktionov <viacheslav.galaktionov@oktet.tech>
The modifier can take up a significant amount of disk space when the number of values is large. Signed-off-by: Viacheslav Galaktionov <viacheslav.galaktionov@oktet.tech>
8597309 to
e605d9c
Compare
ol-andrewr
approved these changes
Mar 1, 2026
This was referenced Mar 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MI measurements are logged as separate JSON objects, each duplicating the measurement aggregation type, units and multiplier. This makes each measurement take much more space than a single floating point value.
Introduce a new "series" aggregation type that allows tests to log a series of measurements that share their properties in one JSON object.
For example, the following sequence
turns into one object:
Our test suite provides test iterations that can log 30000 such measurements, see below for the final log sizes before these patches, after compacting MI log messages, and after removing 1.0 multipliers from HTML logs: