From 38e3e295acca5b34ebd89014cb1eaca9024fbf97 Mon Sep 17 00:00:00 2001 From: Stephen Thompson Date: Wed, 9 Jul 2025 14:30:05 +0100 Subject: [PATCH 1/6] Updated source locations --- docs/dev/intellij.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/dev/intellij.md b/docs/dev/intellij.md index 4718604c6..d33f2af94 100644 --- a/docs/dev/intellij.md +++ b/docs/dev/intellij.md @@ -24,8 +24,8 @@ These setup instructions are aimed at developing in [IntelliJ IDEA](https://www. ```bash cd ~/projects/EMAP - git clone https://github.com/UCLH-DHCT/emap.git - git clone https://github.com/inform-health-informatics/hoover.git + git clone https://github.com/SAFEHR-data/emap.git + git clone https://github.com/SAFEHR-data/hoover.git ``` From d3e87d064b2712541bd1edffd90934d277c5cf4e Mon Sep 17 00:00:00 2001 From: Stephen Thompson Date: Mon, 21 Jul 2025 14:12:15 +0100 Subject: [PATCH 2/6] Fix typo. in glossary --- docs/Glossary.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Glossary.md b/docs/Glossary.md index eabe35873..2f24cc8fc 100644 --- a/docs/Glossary.md +++ b/docs/Glossary.md @@ -56,7 +56,7 @@ - **CSN** - The Contact Serial Number, assigned to a patient on arrival at the hospital, is related to a particular visit. Within [EPIC}(#epic) many interactons are related to the patient via the CSN. Note though that not every patient is assigned + The Contact Serial Number, assigned to a patient on arrival at the hospital, is related to a particular visit. Within [EPIC](#epic) many interactons are related to the patient via the CSN. Note though that not every patient is assigned a CSN. Information relevant to a patient is available through the CSN or [MRN](#mrn) of a patient. - **Continuous integration** From 216459c09fcfe184f73dd58c0c81a72e6c0a8496 Mon Sep 17 00:00:00 2001 From: Stephen Thompson Date: Mon, 21 Jul 2025 14:18:48 +0100 Subject: [PATCH 3/6] More specific naming of plugins --- docs/dev/intellij.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/dev/intellij.md b/docs/dev/intellij.md index d33f2af94..ae73b12d7 100644 --- a/docs/dev/intellij.md +++ b/docs/dev/intellij.md @@ -89,9 +89,9 @@ These setup instructions are aimed at developing in [IntelliJ IDEA](https://www. 8.
- Add lombok and checkstyle plugins + Add Lombok and CheckStyle-IDEA plugins - Go to File > Settings > search for plugins, search lombok and checkstyle and install them + Go to File > Settings > search for plugins, search Lombok and CheckStyle-IDEA and install them
9.
From 0082a200a08daee463d952af4007300bca20093d Mon Sep 17 00:00:00 2001 From: Stephen Thompson Date: Mon, 21 Jul 2025 14:27:27 +0100 Subject: [PATCH 4/6] Adds a link to the further information on RabbitMQ. --- docs/technical_overview/Technical_overview_of_EMAP.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/technical_overview/Technical_overview_of_EMAP.md b/docs/technical_overview/Technical_overview_of_EMAP.md index 1e304dd10..252be94d2 100644 --- a/docs/technical_overview/Technical_overview_of_EMAP.md +++ b/docs/technical_overview/Technical_overview_of_EMAP.md @@ -1,6 +1,6 @@ # Technical Overview EMAP -Note a Glossary of terms used is provided as a separate document. +Note a [Glossary](../Glossary.md) of terms used is provided as a separate document. ## Introduction @@ -110,7 +110,8 @@ involving data sources or destinations. ### RabbitMQ server -While there is more information on RabbitMQ in section "Technologies used", it should be mentioned here that once +There is more information on RabbitMQ in the [Technologies used](technologies_used/RabbitMQ.md) section +It should be mentioned here that once messages have been converted into the interchange format, they are batched and sent to the appropriate queue managed by the RabbitMQ server. Each queue has a maximum number of messages that are allowed, and the services publishing to the queues implement an exponential backoff policy to limit the amount of disk From 35e5afceb19f8f3da8f3c560eca24c2c3cb67160 Mon Sep 17 00:00:00 2001 From: Stephen Thompson Date: Mon, 21 Jul 2025 14:41:13 +0100 Subject: [PATCH 5/6] Fixes links in onbaording --- docs/dev/onboarding.md | 10 +++++----- docs/technical_overview/technologies_used/RabbitMQ.md | 4 ++-- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/dev/onboarding.md b/docs/dev/onboarding.md index f2a8e66f7..5ca0d6d4e 100644 --- a/docs/dev/onboarding.md +++ b/docs/dev/onboarding.md @@ -9,17 +9,17 @@ There are currently two data sources for EMAP: - HL7 data - Persisted in - the [Immutable Data Store](https://github.com/inform-health-informatics/emap_documentation/blob/main/technical_overview/Technical_overview_of_EMAP.md#immutable-data-store-ids) + the [Immutable Data Store](/docs/technical_overview/Technical_overview_of_EMAP.md#immutable-data-store-ids) (IDS), from a copy of specific HL7 message streams - The IDS is read by - the [HL7 reader](https://github.com/inform-health-informatics/emap_documentation/blob/main/technical_overview/Technical_overview_of_EMAP.md#hl7-reader), + the [HL7 reader](/docs/technical_overview/Technical_overview_of_EMAP.md#hl7-reader), (defined in the [hl7-reader](https://github.com/UCLH-DHCT/emap/tree/main/hl7-reader) module) converting the HL7 message into a source-agnostic format (interchange message, defined in the [emap-interchange](https://github.com/UCLH-DHCT/emap/tree/main/emap-interchange) module) and published to a rabbitMQ queue for processing by the core processor. - Hospital database polling - - The [Hoover](https://github.com/inform-health-informatics/emap_documentation/blob/main/technical_overview/Technical_overview_of_EMAP.md#hoover) + The [Hoover](/docs/technical_overview/Technical_overview_of_EMAP.md#hoover) (defined in the [hoover](https://github.com/UCLH-DHCT/hoover) repository) service polls hospital databases (Clarity and Caboodle) for data that has changed since the last poll. It converts the query outputs into the interchange message and publishes these to a rabbitMQ queue for processing by @@ -27,11 +27,11 @@ There are currently two data sources for EMAP: We can't make the Hoover repository public because the SQL queries contain the intellectual property of the hospital patient record system, EPIC. -The [core processor](https://github.com/inform-health-informatics/emap_documentation/blob/main/technical_overview/Technical_overview_of_EMAP.md#the-eventprocessor) +The [core processor](/docs/technical_overview/Technical_overview_of_EMAP.md#the-eventprocessor) (defined in the [core](https://github.com/UCLH-DHCT/emap/tree/main/core) module) is responsible for processing the interchange messages and updating -the [emap database](https://github.com/inform-health-informatics/emap_documentation/blob/main/technical_overview/Technical_overview_of_EMAP.md#star-schema) +the [emap database](/docs/technical_overview/Technical_overview_of_EMAP.md#star-schema) (defined in the [emap-star](https://github.com/UCLH-DHCT/emap/tree/main/emap-star) module). The core processor compares what is already known in the EMAP database, with the data in the interchange message and diff --git a/docs/technical_overview/technologies_used/RabbitMQ.md b/docs/technical_overview/technologies_used/RabbitMQ.md index bd52999a0..91a6b3683 100644 --- a/docs/technical_overview/technologies_used/RabbitMQ.md +++ b/docs/technical_overview/technologies_used/RabbitMQ.md @@ -61,7 +61,7 @@ Each pubisher writes to a queue which is published to the core processor and sen -EMAP is run as a collection of docker containers, each providing a single service such as RabbitMQ. We take the latest RabbitMQ management image from dockerhub and configure it as required. This involves setting ports, authentication and details such as maximum queue size and waiting times. As the RabbitMQ component sits in the middle of the pipeline it is necessary to spin up a version of RabbitMQ even if only testing part of the full pipeline. We configure to 1M hl7 messages and 100K hoover messages to be queued at any point to avoid loss of disk space when a large amount of messages are being processed. In general this is only likely to apply when we are starting off a run from scratch. The normal day to day running of EMAP should not encounter memory problems. +EMAP is run as a collection of docker containers, each providing a single service such as RabbitMQ. We take the latest RabbitMQ management image from dockerhub and configure it as required. This involves setting ports, authentication and details such as maximum queue size and waiting times. As the RabbitMQ component sits in the middle of the pipeline it is necessary to spin up a version of RabbitMQ even if only testing part of the full pipeline. We configure to [100k hl7 messages](https://github.com/SAFEHR-data/emap/blob/f2edcbc0aa8a9c0c87b1f01892ba69e67b9ec228/hl7-reader/src/main/resources/application.properties#L28) and [1M hoover messages](https://github.com/SAFEHR-data/hoover/blob/3f0e196f71dd57f09a810e61fa6e1c637cd9b423/src/main/resources/application.properties#L19) to be queued at any point to avoid loss of disk space when a large amount of messages are being processed. In general this is only likely to apply when we are starting off a run from scratch. The normal day to day running of EMAP should not encounter memory problems. @@ -83,4 +83,4 @@ All our applications use callbacks to write a progress update to the database. T The RabbitMQ can accept a message from the receiver that states that it could not process the message for some reason, usually it has missing information or is known that we cannot process it. At this point RabbitMQ does not resend the message. -**Roma and Stef disagree on this. Stef thinks we do send a NACK but Roma thinks we send an ACK which also avoids requeueing** \ No newline at end of file +**Roma and Stef disagree on this. Stef thinks we do send a NACK but Roma thinks we send an ACK which also avoids requeueing** From 2c4d155af4670ecf30141e1f642aeab84bead145 Mon Sep 17 00:00:00 2001 From: Stephen Thompson Date: Mon, 21 Jul 2025 15:44:51 +0100 Subject: [PATCH 6/6] Updating more links --- docs/dev/onboarding.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/dev/onboarding.md b/docs/dev/onboarding.md index 5ca0d6d4e..fa24eb9c5 100644 --- a/docs/dev/onboarding.md +++ b/docs/dev/onboarding.md @@ -230,9 +230,9 @@ required before changes should impact the running codebase. If this is an entirely new data type with no effect on existing data, then feature flags can be used to disable the processing of the messages in production. For a change to an existing data type or to release into production, then follow -the [validation SOP](https://github.com/UCLH-DHCT/internal_emap_documentation/blob/main/SOP/validation_run.md). +the [validation SOP](https://github.com/SAFEHR-data/internal_emap_documentation/blob/main/SOP/validation_run.md). ### Deployment Deployment is carried out using the emap-setup tool, follow -the [release procedure SOP](https://github.com/UCLH-DHCT/internal_emap_documentation/blob/main/SOP/release_procedure.md) +the [release procedure SOP](https://github.com/SAFEHR-data/internal_emap_documentation/blob/main/SOP/release_procedure.md).