Skip to content

Make large MI measurement logs more compact#102

Open
okt-viacheslav wants to merge 5 commits into
oktetlabs:mainfrom
okt-viacheslav:mi_log_compact
Open

Make large MI measurement logs more compact#102
okt-viacheslav wants to merge 5 commits into
oktetlabs:mainfrom
okt-viacheslav:mi_log_compact

Conversation

@okt-viacheslav

Copy link
Copy Markdown
Contributor

MI measurements are logged as separate JSON objects, each duplicating the measurement aggregation type, units and multiplier. This makes each measurement take much more space than a single floating point value.

Introduce a new "series" aggregation type that allows tests to log a series of measurements that share their properties in one JSON object.

For example, the following sequence

{"aggr":"single","value":0.00202225,"base_units":"seconds","multiplier":"1"}
{"aggr":"single","value":0.00303048,"base_units":"seconds","multiplier":"1"}
{"aggr":"single","value":0.00403947, "base_units":"seconds","multiplier":"1"}

turns into one object:

{"aggr":"series","values":[0.00202225, 0.00303048, 0.00403947],"base_units":"seconds","multiplier":"1"}

Our test suite provides test iterations that can log 30000 such measurements, see below for the final log sizes before these patches, after compacting MI log messages, and after removing 1.0 multipliers from HTML logs:

~/run-old$ du -sh tmp_raw_log log.txt html
854M    tmp_raw_log
395M    log.txt
1.3G    html

~/run-compact$ du -sh tmp_raw_log log.txt html
292M    tmp_raw_log
398M    log.txt
768M    html

run-compact-html$ du -sh tmp_raw_log log.txt html
287M    tmp_raw_log
392M    log.txt
352M    html

okt-viacheslav and others added 2 commits February 4, 2026 18:24
Allow users to log measurements in non-standard units without letting TE
perform any unit convertions. This is primarily useful for graphs, e.g. this
allows the user to log and view data in micro- or nanoseconds instead of having
them converted to seconds, which hurts readability.

Signed-off-by: Viacheslav Galaktionov <viacheslav.galaktionov@oktet.tech>
This functionality will be needed in later patches.

Signed-off-by: Viacheslav Galaktionov <viacheslav.galaktionov@oktet.tech>
Comment thread lib/tools/te_mi_log.c
Comment thread lib/tools/te_mi_log.c Outdated
Comment thread tools/rgt/rgt-format/xml2html-multi/xml2html-multi.c Outdated
Comment thread tools/rgt/rgt-format/xml2html-multi/xml2html-multi.c
Comment thread tools/rgt/rgt-format/xml2text/xml2text.c Outdated
Comment thread tools/rgt/rgt-format/xml2text/xml2text.c Outdated
Provide test with a memory-efficient way to log large sets of measured values
that share the same properties. Store these values in a separate field called
"values" and introduce a new aggregation type called "series".

Signed-off-by: Viacheslav Galaktionov <viacheslav.galaktionov@oktet.tech>
Parse the new "values" field from MI measurement JSON and extract the values.
Make sure the measured values are shown correctly in HTML and text logs.

Signed-off-by: Viacheslav Galaktionov <viacheslav.galaktionov@oktet.tech>
The modifier can take up a significant amount of disk space when the number of
values is large.

Signed-off-by: Viacheslav Galaktionov <viacheslav.galaktionov@oktet.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants