Get low-frequency Waveform data from HL7 to Interchange by jeremyestein · Pull Request #172 · SAFEHR-data/emap

jeremyestein · 2026-08-07T15:16:52Z

Start at Hl7ParseAndQueue.java if you want to see the main part of the change first.

Process low frequency waveform data, eg. settings, from HL7 to Emap interchange messages. These are scalar values reported at a lower frequency (typically 3Hz) and can measure anything, but are not actually waveforms themselves.

Move much of WaveformMessage into abstract base class WaveformBaseMessage and add another subclass WaveformLowFreqMessage for low frequency messages. Kept original name for compatibility with downstream waveform controller.
emap core processor intentionally ignores LF messages as this is out of scope for this phase of the waveform project
synthetic generator emits LF messages alongside HF ones, for testing outside of production
LF interchange messages are sent to the publisher as-is, unlike HF ones which go through collation first. De-duping, if necessary, will have to be done in the waveform controller
Anonymised real HL7 test messages settings*.hl7 and parse tests cover numeric + string + categorical mapping

Fully anonymised: - Messages IDs made sequential - All timestamps shifted by same amount - Location changed to another bed - Numerical values changed slightly Also extract the "value type" out of CSV metadata.

create these messages for non-Waveform variables. Not passing them into the collator just yet.

github-actions · 2026-08-07T15:17:13Z

PR checklist

Default guide for a PR (if multiple PRs for the work, only keep one version of it and link to it on the other PRs)

From the UCLH data science desktop, a validation run has been set off
Check that content that reveals Epic IP (eg. Clarity/Caboodle queries) has not been checked into a public repo.
load times
in UCL teams has been populated with the run information
During the run, glowroot has been checked for any queries which are taking a substantial proportion of the
total processing time. This can be useful to identify indexes that are required.
After the run, look for any unexpected errors in the etl_per_message_logging table, the error_search.sql file
on the shared drive can be used for this \\sharefs6\UCLH6\EMAP\Shared\EmapSqlScripts\devops\error_search.sql.
Create an issue if you find an unexpected exception and is not related to the changes you've made, otherwise
fix them!
After the run, populate the end time in
load times
Let Aasiyah know about the completed validation and give her information on the changes and where to start
with the validation
Check validation report and give any feedback to Aasiyah if there are any changes needed on her side,
iterate on getting the validation to match at least 99% (validation and emap code).

generator

list

message, and fix the tests

thompson318

Looks logical and the workflow makes sense. A few comments / questions.
General comment. I wonder if by keeping "waveform" messages for backward compatibility with what we've already done on the controller we're storing up problems for the future. Would it be better to rename then "waveformHighFreqMessage" to be clear about the two different message types? It would make a lot of work for us for marginal gain so perhaps not.

thompson318 · 2026-08-13T10:42:37Z

+    record ExpectedWaveformMessage(String variableId, String sourceValue, String mappedUnits, Double numericValue, String stringValue) {
+        public void assertIsEqual(WaveformLowFreqMessage actualMessage) {
+            assertEquals(variableId, actualMessage.getSourceVariableId());
+            assertEquals(sourceValue, actualMessage.getSourceValue().get());


Suggested change

assertEquals(sourceValue, actualMessage.getSourceValue().get());

assertEquals(sourceValue, actualMessage.getSourceValue().get());

assertEquals(mappedUnits, actualMessage.getMappedUnits().get());

Why aren't we checking the mapped units? Would the above suggestion work?

Well spotted. It's actualMessage.getUnit(), we don't even send the source unit ID. Fixed now.

thompson318 · 2026-08-13T11:00:36Z

Lovely flow chart, makes it clear what's supposed to happen.

thompson318 · 2026-08-13T11:06:31Z

+        logger.trace("HL7 message generated {} Waveform messages ({} collatable, {} not), sending for collation",
+                msgs.size(), waveformMessages.size(), lfMessages.size());
+        for (var m: lfMessages) {
+            waveformOperations.sendMessage(m);


What's the rationale for not sending low frequency messages to a collator?

Low frequency messages don't have a fixed sampling frequency in the same way that HF ones do, so I decided we should store them as one row (CSV row or emap-star row) per data point, as opposed to arrays that we already use for the HF data.

In practice, they seem to be sent 3 times a second (haven't checked intervals for exactness), but my take was that this is probably configurable and not an inherent property of the source data.

But yes, you could treat these exactly like an HF data source, and reduce them down to a start time and an array, from which you have to work out the time of any particular point by knowing its position in the array and the sampling frequency (start_time + idx / samp_freq).

And maybe 3/sec will create an annoyingly long CSV/parquet file, and we'll wish we used arrays.

Another option, if we're not sure the intervals are spot on, is to squash them into periods with a start and end date where the value is the same (good for things like ventilation mode that rarely change, but maybe bad for other things?), as we talked about in one of the meetings this week. But this would be extra work.

Maybe we should talk about this more?

jeremyestein · 2026-08-13T14:23:37Z

I wonder if by keeping "waveform" messages for backward compatibility with what we've already done on the controller we're storing up problems for the future. Would it be better to rename then "waveformHighFreqMessage" to be clear about the two different message types? It would make a lot of work for us for marginal gain so perhaps not.

It wouldn't be that much work to rename. And I'm currently working on the waveform controller code anyway (which, as it happens, doesn't check the message type anyway). So it would be possible to change it if you think it's better.

thompson318 · 2026-08-13T14:36:05Z

I wonder if by keeping "waveform" messages for backward compatibility with what we've already done on the controller we're storing up problems for the future. Would it be better to rename then "waveformHighFreqMessage" to be clear about the two different message types? It would make a lot of work for us for marginal gain so perhaps not.

It wouldn't be that much work to rename. And I'm currently working on the waveform controller code anyway (which, as it happens, doesn't check the message type anyway). So it would be possible to change it if you think it's better.

If it's not much work then I suggest renaming it. I'm trying to think from the perspective of someone in a couple of years time who get's asked to implement a new type of waveform message, and may struggle to understand the inheritance structure as is.

values too.

jeremyestein and others added 5 commits August 5, 2026 16:58

Add example settings/variables HL7 messages.

a487cf1

Fully anonymised: - Messages IDs made sequential - All timestamps shifted by same amount - Location changed to another bed - Numerical values changed slightly Also extract the "value type" out of CSV metadata.

Add HL7 unit codes from manual via Claire

d0b5973

Create a new interchange message type for low-freq Waveform data, and

d48c500

create these messages for non-Waveform variables. Not passing them into the collator just yet.

Add HL7->Waveform flowchart

285e316

Auto-render PlantUML diagrams (1 file(s))

ec0b46a

jeremyestein and others added 9 commits August 8, 2026 18:48

Send LF messages directly to the publisher; do not collate them.

bc358ad

Add generation of low frequency waveform messages in the synthetic

2f81502

generator

Remove trailing comma in CSV header

22b3f98

Merge branch 'develop' into jeremy/settings-variables

c6265c6

Be more specific about when units must exist and clarify docs re allow

4d6bfdf

list

Test having two values for a variable with different units in the same

ce4f0b3

message, and fix the tests

Add test case for string HL7 type (and fix code)

a60843b

Add more HL7 error cases and fix discovered bug

28ed7de

Auto-render PlantUML diagrams (1 file(s))

73bcc9c

jeremyestein marked this pull request as ready for review August 12, 2026 14:33

jeremyestein and others added 3 commits August 12, 2026 16:08

Add type check to flowchart

dc0babd

Auto-render PlantUML diagrams (1 file(s))

84dc5b7

Skip OBX if number is malformed

6243947

jeremyestein requested review from skeating and thompson318 August 12, 2026 15:54

thompson318 reviewed Aug 13, 2026

View reviewed changes

jeremyestein and others added 6 commits August 13, 2026 16:22

Timestamps were not being advanced in synthetic LF data.

27660be

Unit code was not being substituted

19c044c

Was not checking units in the interchange messages. Fix the expected

e792ee8

values too.

Rename WaveformMessage to WaveformHighFreqMessage for consistency

9f1427a

Missed a couple of renames

336b099

Auto-render PlantUML diagrams (1 file(s))

967edb1

Navigation Menu

Uh oh!

Get low-frequency Waveform data from HL7 to Interchange - #172

Get low-frequency Waveform data from HL7 to Interchange#172
jeremyestein wants to merge 23 commits into
developfrom
jeremy/settings-variables

jeremyestein commented Aug 7, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 7, 2026

Uh oh!

thompson318 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thompson318 Aug 13, 2026

Uh oh!

jeremyestein Aug 13, 2026

Uh oh!

thompson318 Aug 13, 2026

Uh oh!

thompson318 Aug 13, 2026

Uh oh!

jeremyestein Aug 13, 2026

Uh oh!

jeremyestein commented Aug 13, 2026

Uh oh!

thompson318 commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	assertEquals(sourceValue, actualMessage.getSourceValue().get());
	assertEquals(sourceValue, actualMessage.getSourceValue().get());
	assertEquals(mappedUnits, actualMessage.getMappedUnits().get());

Uh oh!

Conversation

jeremyestein commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 7, 2026

PR checklist

Uh oh!

thompson318 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thompson318 Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

jeremyestein Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

thompson318 Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

thompson318 Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

jeremyestein Aug 13, 2026

Choose a reason for hiding this comment

Uh oh!

jeremyestein commented Aug 13, 2026

Uh oh!

thompson318 commented Aug 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jeremyestein commented Aug 7, 2026 •

edited

Loading