Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/Glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@

- <a id="csn">**CSN**</a>

The Contact Serial Number, assigned to a patient on arrival at the hospital, is related to a particular visit. Within [EPIC}(#epic) many interactons are related to the patient via the CSN. Note though that not every patient is assigned
The Contact Serial Number, assigned to a patient on arrival at the hospital, is related to a particular visit. Within [EPIC](#epic) many interactons are related to the patient via the CSN. Note though that not every patient is assigned
a CSN. Information relevant to a patient is available through the CSN or [MRN](#mrn) of a patient.

- <a id="ci">**Continuous integration**</a>
Expand Down
8 changes: 4 additions & 4 deletions docs/dev/intellij.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ These setup instructions are aimed at developing in [IntelliJ IDEA](https://www.

```bash
cd ~/projects/EMAP
git clone https://github.com/UCLH-DHCT/emap.git
git clone https://github.com/inform-health-informatics/hoover.git
git clone https://github.com/SAFEHR-data/emap.git
git clone https://github.com/SAFEHR-data/hoover.git
```

</details>
Expand Down Expand Up @@ -89,9 +89,9 @@ These setup instructions are aimed at developing in [IntelliJ IDEA](https://www.
</details>

8. <details>
<summary>Add lombok and checkstyle plugins</summary>
<summary>Add Lombok and CheckStyle-IDEA plugins</summary>

Go to <b>File > Settings > search for plugins</b>, search lombok and checkstyle and install them
Go to <b>File > Settings > search for plugins</b>, search Lombok and CheckStyle-IDEA and install them
</details>

9. <details>
Expand Down
14 changes: 7 additions & 7 deletions docs/dev/onboarding.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,29 +9,29 @@ There are currently two data sources for EMAP:

- HL7 data
- Persisted in
the [Immutable Data Store](https://github.com/inform-health-informatics/emap_documentation/blob/main/technical_overview/Technical_overview_of_EMAP.md#immutable-data-store-ids)
the [Immutable Data Store](/docs/technical_overview/Technical_overview_of_EMAP.md#immutable-data-store-ids)
(IDS), from a copy of specific HL7 message streams
- The IDS is read by
the [HL7 reader](https://github.com/inform-health-informatics/emap_documentation/blob/main/technical_overview/Technical_overview_of_EMAP.md#hl7-reader),
the [HL7 reader](/docs/technical_overview/Technical_overview_of_EMAP.md#hl7-reader),
(defined in the [hl7-reader](https://github.com/UCLH-DHCT/emap/tree/main/hl7-reader) module)
converting the HL7 message into a source-agnostic format (interchange message, defined in
the [emap-interchange](https://github.com/UCLH-DHCT/emap/tree/main/emap-interchange) module)
and published to a rabbitMQ queue for processing by the core processor.
- Hospital database polling
-
The [Hoover](https://github.com/inform-health-informatics/emap_documentation/blob/main/technical_overview/Technical_overview_of_EMAP.md#hoover)
The [Hoover](/docs/technical_overview/Technical_overview_of_EMAP.md#hoover)
(defined in the [hoover](https://github.com/UCLH-DHCT/hoover) repository)
service polls hospital databases (Clarity and Caboodle) for data that has changed since the last poll.
It converts the query outputs into the interchange message and publishes these to a rabbitMQ queue for processing by
the core processor.
We can't make the Hoover repository public because the SQL queries contain the intellectual property of the hospital
patient record system, EPIC.

The [core processor](https://github.com/inform-health-informatics/emap_documentation/blob/main/technical_overview/Technical_overview_of_EMAP.md#the-eventprocessor)
The [core processor](/docs/technical_overview/Technical_overview_of_EMAP.md#the-eventprocessor)
(defined in the [core](https://github.com/UCLH-DHCT/emap/tree/main/core) module) is responsible for processing the
interchange messages and
updating
the [emap database](https://github.com/inform-health-informatics/emap_documentation/blob/main/technical_overview/Technical_overview_of_EMAP.md#star-schema)
the [emap database](/docs/technical_overview/Technical_overview_of_EMAP.md#star-schema)
(defined in the [emap-star](https://github.com/UCLH-DHCT/emap/tree/main/emap-star) module).

The core processor compares what is already known in the EMAP database, with the data in the interchange message and
Expand Down Expand Up @@ -230,9 +230,9 @@ required before changes should impact the running codebase.
If this is an entirely new data type with no effect on existing data, then feature flags can be used to disable the
processing of the messages in production.
For a change to an existing data type or to release into production, then follow
the [validation SOP](https://github.com/UCLH-DHCT/internal_emap_documentation/blob/main/SOP/validation_run.md).
the [validation SOP](https://github.com/SAFEHR-data/internal_emap_documentation/blob/main/SOP/validation_run.md).

### Deployment

Deployment is carried out using the emap-setup tool, follow
the [release procedure SOP](https://github.com/UCLH-DHCT/internal_emap_documentation/blob/main/SOP/release_procedure.md)
the [release procedure SOP](https://github.com/SAFEHR-data/internal_emap_documentation/blob/main/SOP/release_procedure.md).
5 changes: 3 additions & 2 deletions docs/technical_overview/Technical_overview_of_EMAP.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Technical Overview EMAP

Note a Glossary of terms used is provided as a separate document.
Note a [Glossary](../Glossary.md) of terms used is provided as a separate document.

## Introduction

Expand Down Expand Up @@ -110,7 +110,8 @@ involving data sources or destinations.

### RabbitMQ server

While there is more information on RabbitMQ in section "Technologies used", it should be mentioned here that once
There is more information on RabbitMQ in the [Technologies used](technologies_used/RabbitMQ.md) section
It should be mentioned here that once
messages have been converted into the interchange format, they are batched and sent to the appropriate queue
managed by the RabbitMQ server. Each queue has a maximum number of messages that are allowed, and the services
publishing to the queues implement an exponential backoff policy to limit the amount of disk
Expand Down
4 changes: 2 additions & 2 deletions docs/technical_overview/technologies_used/RabbitMQ.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ Each pubisher writes to a queue which is published to the core processor and sen



EMAP is run as a collection of docker containers, each providing a single service such as RabbitMQ. We take the latest RabbitMQ management image from dockerhub and configure it as required. This involves setting ports, authentication and details such as maximum queue size and waiting times. As the RabbitMQ component sits in the middle of the pipeline it is necessary to spin up a version of RabbitMQ even if only testing part of the full pipeline. We configure to 1M hl7 messages and 100K hoover messages to be queued at any point to avoid loss of disk space when a large amount of messages are being processed. In general this is only likely to apply when we are starting off a run from scratch. The normal day to day running of EMAP should not encounter memory problems.
EMAP is run as a collection of docker containers, each providing a single service such as RabbitMQ. We take the latest RabbitMQ management image from dockerhub and configure it as required. This involves setting ports, authentication and details such as maximum queue size and waiting times. As the RabbitMQ component sits in the middle of the pipeline it is necessary to spin up a version of RabbitMQ even if only testing part of the full pipeline. We configure to [100k hl7 messages](https://github.com/SAFEHR-data/emap/blob/f2edcbc0aa8a9c0c87b1f01892ba69e67b9ec228/hl7-reader/src/main/resources/application.properties#L28) and [1M hoover messages](https://github.com/SAFEHR-data/hoover/blob/3f0e196f71dd57f09a810e61fa6e1c637cd9b423/src/main/resources/application.properties#L19) to be queued at any point to avoid loss of disk space when a large amount of messages are being processed. In general this is only likely to apply when we are starting off a run from scratch. The normal day to day running of EMAP should not encounter memory problems.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be worth double checking this. I may have misunderstood how the configuration is done.




Expand All @@ -83,4 +83,4 @@ All our applications use callbacks to write a progress update to the database. T


The RabbitMQ can accept a message from the receiver that states that it could not process the message for some reason, usually it has missing information or is known that we cannot process it. At this point RabbitMQ does not resend the message.
**Roma and Stef disagree on this. Stef thinks we do send a NACK but Roma thinks we send an ACK which also avoids requeueing**
**Roma and Stef disagree on this. Stef thinks we do send a NACK but Roma thinks we send an ACK which also avoids requeueing**
Loading