From 93dc6ba9e30ffa06d899d9223311d63fc5d8da22 Mon Sep 17 00:00:00 2001 From: Sarah Keating Date: Wed, 27 Nov 2024 15:55:03 +0000 Subject: [PATCH 01/16] update main read me --- README.md | 94 +++++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 78 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index e9a30860d..bf779e8d8 100644 --- a/README.md +++ b/README.md @@ -6,13 +6,19 @@ PIXL Image eXtraction Laboratory `PIXL` is a system for extracting, linking and de-identifying DICOM imaging data, structured EHR data and free-text data from radiology reports at UCLH. -Please see the [rolling-skeleton]([https://github.com/SAFEHR-data/the-rolling-skeleton=](https://github.com/SAFEHR-data/the-rolling-skeleton/blob/main/docs/design/100-day-design.md)) for more details. -PIXL is intended run on one of the [GAE (General Application Environments)](https://github.com/SAFEHR-data/Book-of-FlowEHR/blob/main/glossary.md#gaes)s and comprises -several services orchestrated by [Docker Compose](https://docs.docker.com/compose/). +It comprises several services orchestrated by [Docker Compose](https://docs.docker.com/compose/). + +
UCLH SPECIFIC + +PIXL is intended run on one of the [GAE (General Application Environments)](https://github.com/SAFEHR-data/Book-of-FlowEHR/blob/main/glossary.md#gaes)s. To get access to the GAE, [see the documentation on Slab](https://uclh.slab.com/posts/gae-access-7hkddxap). -Please request access to Slab and add further details in a [new blank issue](https://github.com/SAFEHR-data/PIXL/issues/new). + +Please request access to Slab and add further details in a [new blank issue](https://github.com/SAFEHR-data/PIXL/issues/new). + +
+ ## Installation @@ -67,7 +73,7 @@ destination. Provides helper functions for de-identifying DICOM data -### PostgreSQL +### [PostgreSQL](.postgres/README.md) RDBMS which stores DICOM metadata, application data and anonymised patient record data. @@ -79,7 +85,7 @@ HTTP API to export files (parquet and DICOM) from UCLH to endpoints. HTTP API to process messages from the `imaging` queue and populate the raw orthanc instance with images from PACS/VNA. -## Setup `PIXL` in GAE +## Setup `PIXL`
Click here to expand steps and configurations @@ -202,7 +208,7 @@ These variables can be set in the `.env` file. For testing, they can be set in the `test/.secrets.env` file. For dev purposes find the `pixl-dev-secrets.env` note on LastPass for the necessary values. -If an Azure Keyvault hasn't been set up yet, follow [these instructions](./docs/setup/azure-keyvault.md). +At UCLH if an Azure Keyvault hasn't been set up yet, follow [these instructions](./docs/setup/azure-keyvault.md). A second Azure Keyvault is used to store hashing keys and salts for the `hasher` service. This kevyault is configured with the following environment variables: @@ -218,7 +224,7 @@ See the [hasher documentation](./hasher/README.md) for more information.
-## Run `PIXL` in GAE +## Run `PIXL`
Click here to view detailed steps @@ -278,6 +284,9 @@ test/resources/omop/public /*.parquet ### OMOP ES extract dir (input to PIXL) +>[!NOTE] +> OMOP ES is the tool used to extract Electronic Health Records that may be linked to images. + EXTRACT_DIR is the directory passed to `pixl populate` as the input `PARQUET_PATH` argument. ``` @@ -288,8 +297,8 @@ EXTRACT_DIR/public /*.parquet ### PIXL Export dir (PIXL intermediate) -The directory where PIXL will copy the public OMOP extract files (which now contain -the radiology reports) to. +The directory where PIXL will copy the public OMOP extract files and the radiology reports. + These files will subsequently be uploaded to the `parquet` destination specified in the [project config](#3-configure-a-new-project). @@ -310,10 +319,63 @@ FTPROOT/PROJECT_SLUG/EXTRACT_DATETIME/parquet/radiology/radiology.parquet ..............................................omop/public/*.parquet ``` -## :octocat: Cloning repository -* Generate your SSH keys as suggested [here](https://docs.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent) -* Clone the repository by typing (or copying) the following lines in a terminal -``` -git clone git@github.com:SAFEHR-data/PIXL.git -``` +## 'PIXL' Directory Contents + +
+ + +

Subdirectories with links to the relevant README

+ + + + +[bin](./bin/README.md) + +[cli](./cli/README.md) + +[docker](./docker/README.md) + +[docs](./docs/README.md) + +[hasher](./hasher/README.md) + +[orthanc](./orthanc/README.md) + +[pixl_core](./pixl_core/README.md) + +[pixl_dcmd](./pixl_dcmd/README.md) + +[pixl_export](./pixl_export/README.md) + +[pixl_imaging](./pixl_imaging/README.md) + +[postgres](./postgres/README.md) + +[projects](./projects/README.md) + +[pytest-pixl](./pytest-pixl/README.md) + +[schemas](./schemas/README.md) + +[scripts](./scripts/README.md) + +[test](./test/README.md) +
+
+ + +### Files + + + +| **Configuration** | **User docs** | **Housekeeping** | +| :--- | :--- | :--- | +| .env.sample | CODE_OF_CONDUCT.md | .renovaterc.json5 | +| .pre-commit-config.yaml | CONTRIBUTING.md | codecov.yml | +| docker-compose.yml | LICENSE | | +| mypy.ini | NOTICE | | +| pytest.ini | README.md | | +| ruff.toml | | | +| template_config.yaml | | | +
\ No newline at end of file From edbf6dcc1dfc6678568305bfaea8e98a7d8a2454 Mon Sep 17 00:00:00 2001 From: Sarah Keating Date: Tue, 24 Dec 2024 10:18:07 +0000 Subject: [PATCH 02/16] update main read me (#529) --- README.md | 94 +++++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 78 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index e9a30860d..bf779e8d8 100644 --- a/README.md +++ b/README.md @@ -6,13 +6,19 @@ PIXL Image eXtraction Laboratory `PIXL` is a system for extracting, linking and de-identifying DICOM imaging data, structured EHR data and free-text data from radiology reports at UCLH. -Please see the [rolling-skeleton]([https://github.com/SAFEHR-data/the-rolling-skeleton=](https://github.com/SAFEHR-data/the-rolling-skeleton/blob/main/docs/design/100-day-design.md)) for more details. -PIXL is intended run on one of the [GAE (General Application Environments)](https://github.com/SAFEHR-data/Book-of-FlowEHR/blob/main/glossary.md#gaes)s and comprises -several services orchestrated by [Docker Compose](https://docs.docker.com/compose/). +It comprises several services orchestrated by [Docker Compose](https://docs.docker.com/compose/). + +
UCLH SPECIFIC + +PIXL is intended run on one of the [GAE (General Application Environments)](https://github.com/SAFEHR-data/Book-of-FlowEHR/blob/main/glossary.md#gaes)s. To get access to the GAE, [see the documentation on Slab](https://uclh.slab.com/posts/gae-access-7hkddxap). -Please request access to Slab and add further details in a [new blank issue](https://github.com/SAFEHR-data/PIXL/issues/new). + +Please request access to Slab and add further details in a [new blank issue](https://github.com/SAFEHR-data/PIXL/issues/new). + +
+ ## Installation @@ -67,7 +73,7 @@ destination. Provides helper functions for de-identifying DICOM data -### PostgreSQL +### [PostgreSQL](.postgres/README.md) RDBMS which stores DICOM metadata, application data and anonymised patient record data. @@ -79,7 +85,7 @@ HTTP API to export files (parquet and DICOM) from UCLH to endpoints. HTTP API to process messages from the `imaging` queue and populate the raw orthanc instance with images from PACS/VNA. -## Setup `PIXL` in GAE +## Setup `PIXL`
Click here to expand steps and configurations @@ -202,7 +208,7 @@ These variables can be set in the `.env` file. For testing, they can be set in the `test/.secrets.env` file. For dev purposes find the `pixl-dev-secrets.env` note on LastPass for the necessary values. -If an Azure Keyvault hasn't been set up yet, follow [these instructions](./docs/setup/azure-keyvault.md). +At UCLH if an Azure Keyvault hasn't been set up yet, follow [these instructions](./docs/setup/azure-keyvault.md). A second Azure Keyvault is used to store hashing keys and salts for the `hasher` service. This kevyault is configured with the following environment variables: @@ -218,7 +224,7 @@ See the [hasher documentation](./hasher/README.md) for more information.
-## Run `PIXL` in GAE +## Run `PIXL`
Click here to view detailed steps @@ -278,6 +284,9 @@ test/resources/omop/public /*.parquet ### OMOP ES extract dir (input to PIXL) +>[!NOTE] +> OMOP ES is the tool used to extract Electronic Health Records that may be linked to images. + EXTRACT_DIR is the directory passed to `pixl populate` as the input `PARQUET_PATH` argument. ``` @@ -288,8 +297,8 @@ EXTRACT_DIR/public /*.parquet ### PIXL Export dir (PIXL intermediate) -The directory where PIXL will copy the public OMOP extract files (which now contain -the radiology reports) to. +The directory where PIXL will copy the public OMOP extract files and the radiology reports. + These files will subsequently be uploaded to the `parquet` destination specified in the [project config](#3-configure-a-new-project). @@ -310,10 +319,63 @@ FTPROOT/PROJECT_SLUG/EXTRACT_DATETIME/parquet/radiology/radiology.parquet ..............................................omop/public/*.parquet ``` -## :octocat: Cloning repository -* Generate your SSH keys as suggested [here](https://docs.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent) -* Clone the repository by typing (or copying) the following lines in a terminal -``` -git clone git@github.com:SAFEHR-data/PIXL.git -``` +## 'PIXL' Directory Contents + +
+ + +

Subdirectories with links to the relevant README

+ + + + +[bin](./bin/README.md) + +[cli](./cli/README.md) + +[docker](./docker/README.md) + +[docs](./docs/README.md) + +[hasher](./hasher/README.md) + +[orthanc](./orthanc/README.md) + +[pixl_core](./pixl_core/README.md) + +[pixl_dcmd](./pixl_dcmd/README.md) + +[pixl_export](./pixl_export/README.md) + +[pixl_imaging](./pixl_imaging/README.md) + +[postgres](./postgres/README.md) + +[projects](./projects/README.md) + +[pytest-pixl](./pytest-pixl/README.md) + +[schemas](./schemas/README.md) + +[scripts](./scripts/README.md) + +[test](./test/README.md) +
+
+ + +### Files + + + +| **Configuration** | **User docs** | **Housekeeping** | +| :--- | :--- | :--- | +| .env.sample | CODE_OF_CONDUCT.md | .renovaterc.json5 | +| .pre-commit-config.yaml | CONTRIBUTING.md | codecov.yml | +| docker-compose.yml | LICENSE | | +| mypy.ini | NOTICE | | +| pytest.ini | README.md | | +| ruff.toml | | | +| template_config.yaml | | | +
\ No newline at end of file From 65c7b5f92a3657c34636c2567dea58fa8eea35a8 Mon Sep 17 00:00:00 2001 From: Sarah Keating Date: Tue, 24 Dec 2024 14:39:32 +0000 Subject: [PATCH 03/16] made read mes consistent --- bin/README.md | 20 +++++++++++++++++++- bin/linters/README.md | 14 ++++++++++++-- 2 files changed, 31 insertions(+), 3 deletions(-) diff --git a/bin/README.md b/bin/README.md index eedf89797..13c2e1543 100644 --- a/bin/README.md +++ b/bin/README.md @@ -1,6 +1,24 @@ ## 'PIXL/bin' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [linters](./linters/README.md) +
+ +
+ +

Files

+ + + +| **User docs** | +| :--- | +README.md + +
+ diff --git a/bin/linters/README.md b/bin/linters/README.md index a03874a75..84698fd12 100644 --- a/bin/linters/README.md +++ b/bin/linters/README.md @@ -1,6 +1,16 @@ +A directory that contains the files used for linting. + ## 'PIXL/bin/linters' Directory Contents -### Files +
+ +

Files

+ + + +| **Code** | **User docs** | +| :--- | :--- | +| check_headers_exist.sh | README.md | -check_headers_exist.sh +
From 3dcc4fbeca26070384039e5813746fc580d3b235 Mon Sep 17 00:00:00 2001 From: Sarah Keating Date: Tue, 24 Dec 2024 15:09:09 +0000 Subject: [PATCH 04/16] Made the read mes more consistent --- cli/src/README.md | 22 +++++++++++++++++++++- cli/src/pixl_cli/README.md | 28 ++++++++++++++++------------ cli/tests/README.md | 38 +++++++++++++++++++++----------------- 3 files changed, 58 insertions(+), 30 deletions(-) diff --git a/cli/src/README.md b/cli/src/README.md index 807a572b5..c1c1c9029 100644 --- a/cli/src/README.md +++ b/cli/src/README.md @@ -1,6 +1,26 @@ +This directory contains the source files for the PIXL commands line interface. + ## 'PIXL/cli/src' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [pixl_cli](./pixl_cli/README.md) +
+ +
+ +

Files

+ + + +| **User docs** | +| :--- | +README.md + +
+ diff --git a/cli/src/pixl_cli/README.md b/cli/src/pixl_cli/README.md index 11f286869..685a139c5 100644 --- a/cli/src/pixl_cli/README.md +++ b/cli/src/pixl_cli/README.md @@ -1,18 +1,22 @@ -## 'PIXL/cli/src/pixl_cli' Directory Contents - -### Files - -main.py +This directory contains the files necessary to create a command line instance of PIXL. -_config.py - -_database.py +## 'PIXL/cli/src/pixl_cli' Directory Contents -_docker_commands.py +
+ +

Files

-_io.py + -_message_processing.py +| **Code** | **User docs** | +| :--- | :--- | +| main.py | README.md | +| _config.py | | +| _database.py | | +| _docker_commands.py | | +| _io.py | | +| _message_processing.py | | +| __init__.py | | -__init__.py +
diff --git a/cli/tests/README.md b/cli/tests/README.md index 85977d651..2d25bff75 100644 --- a/cli/tests/README.md +++ b/cli/tests/README.md @@ -1,5 +1,10 @@ +# PIXL cli tests -Remove the db container and associated data +This directory contains the code for the tests of the PIXL command line interface. + + + +In order to remove the db container and associated data after the tests have been run use the following command: ```bash docker container rm pixl-test-db -v -f @@ -7,23 +12,22 @@ docker container rm pixl-test-db -v -f ## 'PIXL/cli/tests' Directory Contents -### Files - -conftest.py - -README.md - -test_check_env.py - -test_database.py - -test_docker_commands.py - -test_io.py +
+ +

Files

-test_messages_from_files.py + -test_message_processing.py +| **Code** | **User docs** | +| :--- | :--- | +| conftest.py | README.md | +| test_check_env.py | | +| test_database.py | | +| test_docker_commands.py | | +| test_io.py | | +| test_messages_from_files.py | | +| test_message_processing.py | | +| test_populate.py | | -test_populate.py +
From a48ec93e46d7fac3c552685e1bde3e3128ca5737 Mon Sep 17 00:00:00 2001 From: Sarah Keating Date: Fri, 27 Dec 2024 15:05:32 +0000 Subject: [PATCH 05/16] made read mes more consistent --- cli/README.md | 22 ++++++++++++++++++---- docker/README.md | 20 ++++++++++++++++---- docker/export-api/README.md | 12 ++++++++++-- docker/hasher-api/README.md | 12 ++++++++++-- docker/imaging-api/README.md | 12 ++++++++++-- docker/orthanc-anon/README.md | 12 ++++++++++-- docker/orthanc-raw/README.md | 12 ++++++++++-- docker/postgres/README.md | 12 ++++++++++-- hasher/README.md | 24 ++++++++++++++++++++---- hasher/src/README.md | 20 +++++++++++++++++++- hasher/src/hasher/README.md | 17 +++++++++++------ hasher/tests/README.md | 17 +++++++++++------ 12 files changed, 155 insertions(+), 37 deletions(-) diff --git a/cli/README.md b/cli/README.md index bbbddeef6..4a1888c48 100644 --- a/cli/README.md +++ b/cli/README.md @@ -145,15 +145,29 @@ pytest -vs tests/test_docker_commands.py #e.g., for particular tests ## 'PIXL/cli' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [src](./src/README.md) [tests](./tests/README.md) -### Files +
+ +
+ +

Files

+ + + +| **Configuration** | **User docs** | +| :--- | :--- | +| pyproject.toml | README.md | + +
-pyproject.toml -README.md diff --git a/docker/README.md b/docker/README.md index d423d8cb6..083e669bc 100644 --- a/docker/README.md +++ b/docker/README.md @@ -1,6 +1,10 @@ ## 'PIXL/docker' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [export-api](./export-api/README.md) @@ -14,9 +18,17 @@ [postgres](./postgres/README.md) -### Files +
+ +
+ +

Files

+ + -.dockerignore +| **Configuration** | **User docs** | **Housekeeping** | +| :--- | :--- | :--- | +| common.env | README.md | .dockerignore | -common.env +
diff --git a/docker/export-api/README.md b/docker/export-api/README.md index f12d22846..5890f1ee2 100644 --- a/docker/export-api/README.md +++ b/docker/export-api/README.md @@ -1,6 +1,14 @@ ## 'PIXL/docker/export-api' Directory Contents -### Files +
+ +

Files

-Dockerfile + + +| **Configuration** | **User docs** | +| :--- | :--- | +| Dockerfile | README.md | + +
diff --git a/docker/hasher-api/README.md b/docker/hasher-api/README.md index f54e14972..7500b7a27 100644 --- a/docker/hasher-api/README.md +++ b/docker/hasher-api/README.md @@ -1,6 +1,14 @@ ## 'PIXL/docker/hasher-api' Directory Contents -### Files +
+ +

Files

-Dockerfile + + +| **Configuration** | **User docs** | +| :--- | :--- | +| Dockerfile | README.md | + +
diff --git a/docker/imaging-api/README.md b/docker/imaging-api/README.md index 400bee965..72ff585b6 100644 --- a/docker/imaging-api/README.md +++ b/docker/imaging-api/README.md @@ -1,6 +1,14 @@ ## 'PIXL/docker/imaging-api' Directory Contents -### Files +
+ +

Files

-Dockerfile + + +| **Configuration** | **User docs** | +| :--- | :--- | +| Dockerfile | README.md | + +
diff --git a/docker/orthanc-anon/README.md b/docker/orthanc-anon/README.md index b1f2a60bc..d2b913295 100644 --- a/docker/orthanc-anon/README.md +++ b/docker/orthanc-anon/README.md @@ -1,6 +1,14 @@ ## 'PIXL/docker/orthanc-anon' Directory Contents -### Files +
+ +

Files

-Dockerfile + + +| **Configuration** | **User docs** | +| :--- | :--- | +| Dockerfile | README.md | + +
diff --git a/docker/orthanc-raw/README.md b/docker/orthanc-raw/README.md index a137fb98a..e949c46ad 100644 --- a/docker/orthanc-raw/README.md +++ b/docker/orthanc-raw/README.md @@ -1,6 +1,14 @@ ## 'PIXL/docker/orthanc-raw' Directory Contents -### Files +
+ +

Files

-Dockerfile + + +| **Configuration** | **User docs** | +| :--- | :--- | +| Dockerfile | README.md | + +
diff --git a/docker/postgres/README.md b/docker/postgres/README.md index f43084e01..7135418ac 100644 --- a/docker/postgres/README.md +++ b/docker/postgres/README.md @@ -1,6 +1,14 @@ ## 'PIXL/docker/postgres' Directory Contents -### Files +
+ +

Files

-Dockerfile + + +| **Configuration** | **User docs** | +| :--- | :--- | +| Dockerfile | README.md | + +
diff --git a/hasher/README.md b/hasher/README.md index b9055be2c..faea710d2 100644 --- a/hasher/README.md +++ b/hasher/README.md @@ -61,6 +61,8 @@ See the [Azure Key vault setup](../docs/setup/azure-keyvault.md) documentation f Save the credentials in `.secrets.env` and a LastPass `Hasher API secrets` note. +SK QUESTION: is the reference to Last Pass something that is specific to us or is it a dependency somebody else would need. actually I assume the whole Azure thing is rather how we have chosen to do that rather than a necessity for somebody i.e. they might use a different system for storing their hashes + ``` HASHER_API_AZ_CLIENT_ID= HASHER_API_AZ_CLIENT_PASSWORD= @@ -70,15 +72,29 @@ HASHER_API_AZ_KEY_VAULT_NAME= ## 'PIXL/hasher' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [src](./src/README.md) [tests](./tests/README.md) -### Files +
+ +
+ +

Files

+ + + +| **Configuration** | **User docs** | +| :--- | :--- | +| pyproject.toml | README.md | + +
-pyproject.toml -README.md diff --git a/hasher/src/README.md b/hasher/src/README.md index 99a56a701..0de380702 100644 --- a/hasher/src/README.md +++ b/hasher/src/README.md @@ -1,6 +1,24 @@ ## 'PIXL/hasher/src' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [hasher](./hasher/README.md) +
+ +
+ +

Files

+ + + +| **User docs** | +| :--- | +| README.md | + +
+ diff --git a/hasher/src/hasher/README.md b/hasher/src/hasher/README.md index 40b064a8d..736088286 100644 --- a/hasher/src/hasher/README.md +++ b/hasher/src/hasher/README.md @@ -1,12 +1,17 @@ ## 'PIXL/hasher/src/hasher' Directory Contents -### Files +
+ +

Files

-endpoints.py + -hashing.py +| **Code** | **User docs** | +| :--- | :--- | +| endpoints.py | README.md | +| hashing.py | | +| main.py | | +| __init__.py | | -main.py - -__init__.py +
diff --git a/hasher/tests/README.md b/hasher/tests/README.md index 1a161940e..9f492bde3 100644 --- a/hasher/tests/README.md +++ b/hasher/tests/README.md @@ -1,12 +1,17 @@ ## 'PIXL/hasher/tests' Directory Contents -### Files +
+ +

Files

-conftest.py + -test_endpoints.py +| **Code** | **User docs** | +| :--- | :--- | +| conftest.py | README.md | +| test_endpoints.py | | +| test_hashing.py | | +| __init__.py | | -test_hashing.py - -__init__.py +
From 7c777c91a726d25b0bd831f00f22a6f1bf690437 Mon Sep 17 00:00:00 2001 From: Sarah Keating Date: Fri, 3 Jan 2025 15:29:40 +0000 Subject: [PATCH 06/16] making readmes consistent --- orthanc/README.md | 21 +++++++++-- orthanc/assets/README.md | 14 +++++-- orthanc/orthanc-anon/README.md | 41 +++++++++++++++------ orthanc/orthanc-anon/config/README.md | 13 +++++-- orthanc/orthanc-anon/docs/README.md | 21 +++++++++-- orthanc/orthanc-anon/docs/assets/README.md | 12 +++++- orthanc/orthanc-anon/plugin/README.md | 13 +++++-- orthanc/orthanc-raw/README.md | 24 +++++++++--- orthanc/orthanc-raw/config/README.md | 15 ++++++-- orthanc/orthanc-raw/plugin/README.md | 12 +++++- pixl_core/src/README.md | 20 +++++++++- pixl_core/src/core/README.md | 23 +++++++++--- pixl_core/src/core/db/README.md | 14 +++++-- pixl_core/src/core/patient_queue/README.md | 20 ++++++---- pixl_core/src/core/project_config/README.md | 17 ++++++--- pixl_core/src/core/rest_api/README.md | 13 +++++-- pixl_core/src/core/token_buffer/README.md | 14 +++++-- pixl_core/src/core/uploader/README.md | 29 ++++++++------- pixl_core/tests/README.md | 22 ++++++++--- pixl_core/tests/data/README.md | 14 +++++-- pixl_core/tests/patient_queue/README.md | 14 +++++-- pixl_core/tests/project_config/README.md | 13 +++++-- pixl_core/tests/token_buffer/README.md | 12 +++++- pixl_core/tests/uploader/README.md | 17 ++++++--- 24 files changed, 318 insertions(+), 110 deletions(-) diff --git a/orthanc/README.md b/orthanc/README.md index 674d7d272..d6ea61701 100644 --- a/orthanc/README.md +++ b/orthanc/README.md @@ -12,7 +12,11 @@ functionality . ## 'PIXL/orthanc' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [assets](./assets/README.md) @@ -20,7 +24,18 @@ functionality . [orthanc-raw](./orthanc-raw/README.md) -### Files +
+ +
+ +

Files

+ + + +| **User docs** | +| :--- | +| README.md | + +
-README.md diff --git a/orthanc/assets/README.md b/orthanc/assets/README.md index e6468fad6..4ce46f006 100644 --- a/orthanc/assets/README.md +++ b/orthanc/assets/README.md @@ -1,10 +1,16 @@ ## 'PIXL/orthanc/assets' Directory Contents -### Files +
+ +

Files

-orthanc-anon-az-dicom.png + -orthanc-anon-web.png +| **Images** | **User docs** | +| :--- | :--- | +| orthanc-anon-az-dicom.png | README.md | +| orthanc-anon-web.png | | +| orthanc-raw-web.png | | -orthanc-raw-web.png +
diff --git a/orthanc/orthanc-anon/README.md b/orthanc/orthanc-anon/README.md index 29417db44..a1ee16f54 100644 --- a/orthanc/orthanc-anon/README.md +++ b/orthanc/orthanc-anon/README.md @@ -1,7 +1,7 @@ # Orthanc Anon _The Orthanc instance responsible for anonymising DICOM data from PACS/VNA and forwarding the images -to their final destination (currently the UCL Data Safe Haven)._ +to their final destination (at UCL/UCLH this is currently the UCL Data Safe Haven). ## Setup @@ -24,8 +24,8 @@ The following assumptions are made: ### Configuration -- The Docker image is based on `orthancteam/orthanc`. -- Configuration is driven through customised JSON config. files stored in the [config](./config/) +- The Docker image is based on `orthancteam/orthanc`. <--- This is unclear where is this? +- Configuration is driven through customised JSON config. files stored in the [orthanc-anon/config](./config/) directory. - The files are populated with values from environment variables and injected into the container as secrets. Orthanc interprets all `.json` files in the `/run/secrets` mount as config. files. @@ -74,7 +74,7 @@ Start the instance via Docker compose. ### Step 3 -If you have chosen to expose the portas, you should now be able to navigate the web interface at `http://localhost:`, supply the chosen credentials and will be presented with the Orthanc web interface: +If you have chosen to expose the ports, you should now be able to navigate the web interface at `http://localhost:`, supply the chosen credentials and will be presented with the Orthanc web interface: ![Orthanc Raw Web interface](../assets/orthanc-anon-web.png) ### Step 4 @@ -104,19 +104,36 @@ start. ## References -- [Cheat sheet of the REST API](https://book.orthanc-server.com/users/rest-cheatsheet.html) +- [Cheat sheet for the REST API](https://book.orthanc-server.com/users/rest-cheatsheet.html) -## 'orthanc' Directory Contents -### Subdirectories +## 'PIXL/orthanc/orthanc-anon' Directory Contents -[assets](./assets/README.md) +
+ +

Subdirectories with links to the relevant README

-[orthanc-anon](./orthanc-anon/README.md) + -[orthanc-raw](./orthanc-raw/README.md) +[config](./config/README.md) + +[docs](./docs/README.md) + +[plugin](./plugin/README.md) + +
+ +
+ +

Files

+ + + +| **User docs** | +| :--- | +| README.md | + +
-### Files -README.md diff --git a/orthanc/orthanc-anon/config/README.md b/orthanc/orthanc-anon/config/README.md index ef5de4dce..02fdbac03 100644 --- a/orthanc/orthanc-anon/config/README.md +++ b/orthanc/orthanc-anon/config/README.md @@ -1,8 +1,15 @@ ## 'PIXL/orthanc/orthanc-anon/config' Directory Contents -### Files +
+ +

Files

-dicom.json + -orthanc.json +| **Configuration** | **User docs** | +| :--- | :--- | +| dicom.json | README.md | +| orthanc.json | | + +
diff --git a/orthanc/orthanc-anon/docs/README.md b/orthanc/orthanc-anon/docs/README.md index 8a6fa25bf..594e155de 100644 --- a/orthanc/orthanc-anon/docs/README.md +++ b/orthanc/orthanc-anon/docs/README.md @@ -1,10 +1,25 @@ ## 'PIXL/orthanc/orthanc-anon/docs' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [assets](./assets/README.md) -### Files +
+ +
+ +

Files

+ + + +| **User docs** | +| :--- | +| DicomServiceViaAAD.md | +| README.md | -DicomServiceViaAAD.md +
diff --git a/orthanc/orthanc-anon/docs/assets/README.md b/orthanc/orthanc-anon/docs/assets/README.md index 484b4f3e3..9dc439e65 100644 --- a/orthanc/orthanc-anon/docs/assets/README.md +++ b/orthanc/orthanc-anon/docs/assets/README.md @@ -1,6 +1,14 @@ ## 'PIXL/orthanc/orthanc-anon/docs/assets' Directory Contents -### Files +
+ +

Files

-DicomServiceAuthority.png + + +| **Images** | **User docs** | +| :--- | :--- | +| DicomServiceAuthority.png | README.md | + +
diff --git a/orthanc/orthanc-anon/plugin/README.md b/orthanc/orthanc-anon/plugin/README.md index 6f5c0eba0..51a17ee9e 100644 --- a/orthanc/orthanc-anon/plugin/README.md +++ b/orthanc/orthanc-anon/plugin/README.md @@ -1,8 +1,15 @@ ## 'PIXL/orthanc/orthanc-anon/plugin' Directory Contents -### Files +
+ +

Files

-download_dicom_spec.py + -pixl.py +| **Code** | **User docs** | +| :--- | :--- | +| download_dicom_spec.py | README.md | +| pixl.py | | + +
diff --git a/orthanc/orthanc-raw/README.md b/orthanc/orthanc-raw/README.md index 2e695cc3a..e95ea819d 100644 --- a/orthanc/orthanc-raw/README.md +++ b/orthanc/orthanc-raw/README.md @@ -22,7 +22,7 @@ available shortly when the service is started). ### Configuration -- The Docker image is a deployment of `orthancteam/orthanc` with some extra configuration +- The Docker image is a deployment of `orthancteam/orthanc` with some extra configuration <--- is orthancteam/orthanc supposed to point to somewhere in the tree - `ORTHANC_RAW_MAXIMUM_STORAGE_SIZE` to limit the storage size - `ORTHANC_RAW_JOB_HISTORY_SIZE` has been increased so that while there is concurrent processing, the job should always exist for being able to query its status @@ -109,17 +109,31 @@ team. ## References - - [Cheat sheet of the REST API](https://book.orthanc-server.com/users/rest-cheatsheet.html) + - [Cheat sheet for the REST API](https://book.orthanc-server.com/users/rest-cheatsheet.html) ## 'PIXL/orthanc/orthanc-raw' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [config](./config/README.md) [plugin](./plugin/README.md) -### Files +
+ +
+ +

Files

+ + + +| **User docs** | +| :--- | +| README.md | -README.md +
diff --git a/orthanc/orthanc-raw/config/README.md b/orthanc/orthanc-raw/config/README.md index 6d345e63e..2933d2b62 100644 --- a/orthanc/orthanc-raw/config/README.md +++ b/orthanc/orthanc-raw/config/README.md @@ -1,10 +1,17 @@ ## 'PIXL/orthanc/orthanc-raw/config' Directory Contents -### Files +
+ +

Files

-dicom.json + -orthanc.json +| **Configuration** | **User docs** | +| :--- | :--- | +| dicom.json | README.md | +| orthanc.json | | +| postgres.json | | -postgres.json + +
diff --git a/orthanc/orthanc-raw/plugin/README.md b/orthanc/orthanc-raw/plugin/README.md index bfebf0781..13b57c61e 100644 --- a/orthanc/orthanc-raw/plugin/README.md +++ b/orthanc/orthanc-raw/plugin/README.md @@ -1,6 +1,14 @@ ## 'PIXL/orthanc/orthanc-raw/plugin' Directory Contents -### Files +
+ +

Files

-pixl.py + + +| **Code** | **User docs** | +| :--- | :--- | +| pixl.py | README.md | + +
diff --git a/pixl_core/src/README.md b/pixl_core/src/README.md index 56763ccf2..849af740e 100644 --- a/pixl_core/src/README.md +++ b/pixl_core/src/README.md @@ -1,6 +1,24 @@ ## 'PIXL/pixl_core/src' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [core](./core/README.md) +
+ +
+ +

Files

+ + + +| **User docs** | +| :--- | +| README.md | + +
+ diff --git a/pixl_core/src/core/README.md b/pixl_core/src/core/README.md index c450b32c7..a16c0b96d 100644 --- a/pixl_core/src/core/README.md +++ b/pixl_core/src/core/README.md @@ -1,6 +1,10 @@ ## 'PIXL/pixl_core/src/core' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [db](./db/README.md) @@ -14,13 +18,20 @@ [uploader](./uploader/README.md) -### Files +
-dicom_tags.py +
+ +

Files

-exceptions.py + -exports.py +| **Code** | **User docs** | +| :--- | :--- | +| dicom_tags.py | README.md | +| exceptions.py | | +| exports.py | | +| __init__.py | | -__init__.py +
diff --git a/pixl_core/src/core/db/README.md b/pixl_core/src/core/db/README.md index 2b3e187dd..a59fe00ef 100644 --- a/pixl_core/src/core/db/README.md +++ b/pixl_core/src/core/db/README.md @@ -1,10 +1,16 @@ ## 'PIXL/pixl_core/src/core/db' Directory Contents -### Files +
+ +

Files

-models.py + -queries.py +| **Code** | **User docs** | +| :--- | :--- | +| models.py | README.md | +| queries.py | | +| __init__.py | | -__init__.py +
diff --git a/pixl_core/src/core/patient_queue/README.md b/pixl_core/src/core/patient_queue/README.md index f53cead1f..3ac2b3449 100644 --- a/pixl_core/src/core/patient_queue/README.md +++ b/pixl_core/src/core/patient_queue/README.md @@ -1,14 +1,18 @@ ## 'PIXL/pixl_core/src/core/patient_queue' Directory Contents -### Files +
+ +

Files

-message.py + -producer.py +| **Code** | **User docs** | +| :--- | :--- | +| message.py | README.md | +| producer.py | | +| subscriber.py | | +| _base.py | | +| __init__.py | | -subscriber.py - -_base.py - -__init__.py +
diff --git a/pixl_core/src/core/project_config/README.md b/pixl_core/src/core/project_config/README.md index 4a8232810..529758a4f 100644 --- a/pixl_core/src/core/project_config/README.md +++ b/pixl_core/src/core/project_config/README.md @@ -1,12 +1,17 @@ ## 'PIXL/pixl_core/src/core/project_config' Directory Contents -### Files +
+ +

Files

-pixl_config_model.py + -secrets.py +| **Code** | **User docs** | +| :--- | :--- | +| pixl_config_model.py | README.md | +| secrets.py | | +| tag_operations.py | | +| __init__.py | | -tag_operations.py - -__init__.py +
diff --git a/pixl_core/src/core/rest_api/README.md b/pixl_core/src/core/rest_api/README.md index 2d481757b..8dd17e3ad 100644 --- a/pixl_core/src/core/rest_api/README.md +++ b/pixl_core/src/core/rest_api/README.md @@ -1,8 +1,15 @@ ## 'PIXL/pixl_core/src/core/rest_api' Directory Contents -### Files +
+ +

Files

-router.py + -__init__.py +| **Code** | **User docs** | +| :--- | :--- | +| router.py | README.md | +| __init__.py | | + +
diff --git a/pixl_core/src/core/token_buffer/README.md b/pixl_core/src/core/token_buffer/README.md index d25fd7b64..8b5d341b2 100644 --- a/pixl_core/src/core/token_buffer/README.md +++ b/pixl_core/src/core/token_buffer/README.md @@ -1,10 +1,16 @@ ## 'PIXL/pixl_core/src/core/token_buffer' Directory Contents -### Files +
+ +

Files

-models.py + -tokens.py +| **Code** | **User docs** | +| :--- | :--- | +| models.py | README.md | +| tokens.py | | +| __init__.py | | -__init__.py +
diff --git a/pixl_core/src/core/uploader/README.md b/pixl_core/src/core/uploader/README.md index 1a0e6a444..09a39793d 100644 --- a/pixl_core/src/core/uploader/README.md +++ b/pixl_core/src/core/uploader/README.md @@ -1,16 +1,19 @@ ## 'PIXL/pixl_core/src/core/uploader' Directory Contents -### Files - -base.py - -_dicomweb.py - -_ftps.py - -_orthanc.py - -_xnat.py - -__init__.py +
+ +

Files

+ + + +| **Code** | **User docs** | +| :--- | :--- | +| base.py | README.md | +| _dicomweb.py | | +| _ftps.py | | +| _orthanc.py | | +| _xnat.py | | +| __init__.py | | + +
diff --git a/pixl_core/tests/README.md b/pixl_core/tests/README.md index 621d1b3c7..94ae3ea10 100644 --- a/pixl_core/tests/README.md +++ b/pixl_core/tests/README.md @@ -1,6 +1,10 @@ ## 'PIXL/pixl_core/tests' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [data](./data/README.md) @@ -12,11 +16,19 @@ [uploader](./uploader/README.md) -### Files +
+ +
+ +

Files

-conftest.py + -docker-compose.yml +| **Code** | **User docs** | +| :--- | :--- | +| conftest.py | README.md | +| docker-compose.yml | | +| test_exports.py | | -test_exports.py +
diff --git a/pixl_core/tests/data/README.md b/pixl_core/tests/data/README.md index 065c917e7..8c3111207 100644 --- a/pixl_core/tests/data/README.md +++ b/pixl_core/tests/data/README.md @@ -1,10 +1,16 @@ ## 'PIXL/pixl_core/tests/data' Directory Contents -### Files +
+ +

Files

-dicom_series.zip + -public.zip +| **Data** | **User docs** | +| :--- | :--- | +| dicom_series.zip | README.md | +| public.zip | | +| xnat_project.xml | | -xnat_project.xml +
diff --git a/pixl_core/tests/patient_queue/README.md b/pixl_core/tests/patient_queue/README.md index 895a4d9fc..02a555e07 100644 --- a/pixl_core/tests/patient_queue/README.md +++ b/pixl_core/tests/patient_queue/README.md @@ -1,10 +1,16 @@ ## 'PIXL/pixl_core/tests/patient_queue' Directory Contents -### Files +
+ +

Files

-test_message.py + -test_producer.py +| **Code** | **User docs** | +| :--- | :--- | +| test_message.py | README.md | +| test_producer.py | | +| test_subscriber.py | | -test_subscriber.py +
diff --git a/pixl_core/tests/project_config/README.md b/pixl_core/tests/project_config/README.md index eb889f4e8..c5751ce88 100644 --- a/pixl_core/tests/project_config/README.md +++ b/pixl_core/tests/project_config/README.md @@ -1,8 +1,15 @@ ## 'PIXL/pixl_core/tests/project_config' Directory Contents -### Files +
+ +

Files

-test_project_config.py + -test_secrets.py +| **Code** | **User docs** | +| :--- | :--- | +| test_project_config.py | README.md | +| test_secrets.py | | + +
diff --git a/pixl_core/tests/token_buffer/README.md b/pixl_core/tests/token_buffer/README.md index 259ef6888..58e597e77 100644 --- a/pixl_core/tests/token_buffer/README.md +++ b/pixl_core/tests/token_buffer/README.md @@ -1,6 +1,14 @@ ## 'PIXL/pixl_core/tests/token_buffer' Directory Contents -### Files +
+ +

Files

-test_tokens.py + + +| **Code** | **User docs** | +| :--- | :--- | +| test_tokens.py | README.md | + +
diff --git a/pixl_core/tests/uploader/README.md b/pixl_core/tests/uploader/README.md index e960568d5..a20250f05 100644 --- a/pixl_core/tests/uploader/README.md +++ b/pixl_core/tests/uploader/README.md @@ -1,12 +1,17 @@ ## 'PIXL/pixl_core/tests/uploader' Directory Contents -### Files +
+ +

Files

-test_base.py + -test_dicomweb.py +| **Code** | **User docs** | +| :--- | :--- | +| test_base.py | README.md | +| test_dicomweb.py | | +| test_ftps.py | | +| test_xnat.py | | -test_ftps.py - -test_xnat.py +
From 4109db48e1440ee2557004f51b3128047b0dad06 Mon Sep 17 00:00:00 2001 From: Sarah Keating Date: Fri, 3 Jan 2025 15:41:52 +0000 Subject: [PATCH 07/16] making readmes consistent --- pixl_dcmd/src/README.md | 20 ++++++++++++- pixl_dcmd/src/pixl_dcmd/README.md | 32 +++++++++++++-------- pixl_dcmd/src/pixl_dcmd/resources/README.md | 12 ++++++-- pixl_dcmd/tests/README.md | 29 ++++++++++--------- 4 files changed, 65 insertions(+), 28 deletions(-) diff --git a/pixl_dcmd/src/README.md b/pixl_dcmd/src/README.md index 2ba0cd0e1..bd4b95bbb 100644 --- a/pixl_dcmd/src/README.md +++ b/pixl_dcmd/src/README.md @@ -1,6 +1,24 @@ ## 'PIXL/pixl_dcmd/src' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [pixl_dcmd](./pixl_dcmd/README.md) +
+ +
+ +

Files

+ + + +| **User docs** | +| :--- | +| README.md | + +
+ diff --git a/pixl_dcmd/src/pixl_dcmd/README.md b/pixl_dcmd/src/pixl_dcmd/README.md index e94fbfd07..312dfdca5 100644 --- a/pixl_dcmd/src/pixl_dcmd/README.md +++ b/pixl_dcmd/src/pixl_dcmd/README.md @@ -1,22 +1,30 @@ ## 'PIXL/pixl_dcmd/src/pixl_dcmd' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

-[resources](./resources/README.md) - -### Files + -main.py - -tagrecording.py +[resources](./resources/README.md) -_database.py +
-_dicom_helpers.py +
+ +

Files

-_tag_schemes.py + -_version.py +| **Code** | **User docs** | +| :--- | :--- | +| main.py | README.md | +| tagrecording.py | | +| _database.py | | +| _dicom_helpers.py | | +| _tag_schemes.py | | +| _version.py | | +| __init__.py | | -__init__.py +
diff --git a/pixl_dcmd/src/pixl_dcmd/resources/README.md b/pixl_dcmd/src/pixl_dcmd/resources/README.md index b6d19ecc6..2a9a5e9df 100644 --- a/pixl_dcmd/src/pixl_dcmd/resources/README.md +++ b/pixl_dcmd/src/pixl_dcmd/resources/README.md @@ -1,6 +1,14 @@ ## 'PIXL/pixl_dcmd/src/pixl_dcmd/resources' Directory Contents -### Files +
+ +

Files

-recorded-headers.yml + + +| **Configuration** | **User docs** | +| :--- | :--- | +| recorded-headers.yml | README.md | + +
diff --git a/pixl_dcmd/tests/README.md b/pixl_dcmd/tests/README.md index 9f05da066..5447c3203 100644 --- a/pixl_dcmd/tests/README.md +++ b/pixl_dcmd/tests/README.md @@ -1,16 +1,19 @@ ## 'PIXL/pixl_dcmd/tests' Directory Contents -### Files - -conftest.py - -test_database.py - -test_dicom_validator.py - -test_main.py - -test_tagrecording.py - -test_tag_schemes.py +
+ +

Files

+ + + +| **Code** | **User docs** | +| :--- | :--- | +| conftest.py | README.md | +| test_database.py | | +| test_dicom_validator.py | | +| test_main.py | | +| test_tagrecording.py | | +| test_tag_schemes.py | | + +
From 08b4ca5d0c86a4f2ca73ce48506522dbed31af94 Mon Sep 17 00:00:00 2001 From: Sarah Keating Date: Fri, 3 Jan 2025 15:49:48 +0000 Subject: [PATCH 08/16] making readmes consistent --- pixl_export/src/README.md | 20 +++++++++++++++++++- pixl_export/src/pixl_export/README.md | 17 +++++++++++------ pixl_export/tests/README.md | 13 +++++++++---- 3 files changed, 39 insertions(+), 11 deletions(-) diff --git a/pixl_export/src/README.md b/pixl_export/src/README.md index 06ab4efea..3e4e55566 100644 --- a/pixl_export/src/README.md +++ b/pixl_export/src/README.md @@ -1,6 +1,24 @@ ## 'PIXL/pixl_export/src' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [pixl_export](./pixl_export/README.md) +
+ +
+ +

Files

+ + + +| **User docs** | +| :--- | +| README.md | + +
+ diff --git a/pixl_export/src/pixl_export/README.md b/pixl_export/src/pixl_export/README.md index c21da40db..a40140515 100644 --- a/pixl_export/src/pixl_export/README.md +++ b/pixl_export/src/pixl_export/README.md @@ -1,12 +1,17 @@ ## 'PIXL/pixl_export/src/pixl_export' Directory Contents -### Files +
+ +

Files

-main.py + -_databases.py +| **Code** | **User docs** | +| :--- | :--- | +| main.py | README.md | +| _databases.py | | +| _queries.py | | +| __init__.py | | -_queries.py - -__init__.py +
diff --git a/pixl_export/tests/README.md b/pixl_export/tests/README.md index 607a81078..e308b1232 100644 --- a/pixl_export/tests/README.md +++ b/pixl_export/tests/README.md @@ -1,10 +1,15 @@ ## 'PIXL/pixl_export/tests' Directory Contents -### Files +
+ +

Files

-conftest.py + -docker-compose.yml +| **Configuration** | **Code** | **User docs** | +| :--- | :--- | :--- | +| docker-compose.yml | conftest.py | README.md | +| | test_app.py | | -test_app.py +
From 474894e10f99ef8c3f7fe8e498eeb118f337bf68 Mon Sep 17 00:00:00 2001 From: Sarah Keating Date: Mon, 6 Jan 2025 10:41:39 +0000 Subject: [PATCH 09/16] making readmes consistent --- pixl_core/README.md | 33 +++++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/pixl_core/README.md b/pixl_core/README.md index 5b9f784fb..9d5c943a8 100644 --- a/pixl_core/README.md +++ b/pixl_core/README.md @@ -15,7 +15,7 @@ Specifically, it defines: - Pydantic models for [project configuration](./src/core/project_config/pixl_config_model.py) - [Secrets management](./src/core/project_config/secrets.py) via an Azure Key Vault. - Handling of [uploads over FTPS](./src/core/uploader/_ftps.py), used to transfer images and parquet files - to the DSH (Data Safe Haven) + to the relevant FTPS server (at UCL this is the DSH (Data Safe Haven)) - [Uploading DICOM files to a DICOMWeb server](./src/core/uploader/_dicomweb.py) - [Uploading DICOM files to XNAT](./src/core/uploader/_xnat.py) @@ -69,7 +69,7 @@ and export of stable DICOM to orthanc-anon while still pulling from the VNA. ### OMOP ES files -Public parquet exports from OMOP ES that should be transferred outside the hospital are copied to +Public parquet exports from OMOP ES that may be transferred outside the hospital are copied to the `exports` directory at the repository base. Within this directory each project has a directory, with all extracts stored in `all_extracts` and @@ -99,7 +99,7 @@ for convenience `latest` is a symlink to the most recent extract. ## Project configuration The `project_config` module provides the functionality to handle -[project configurations](../README.md#configure-a-new-project). +[project configurations](../README.md#configure-a-new-project). <== SK comment I'm not sure this goes to exactly the right place OR the name in the # is misleading ### Design @@ -115,7 +115,7 @@ such as the `FTPSUploader` implement the actual upload functionality. The creden uploading are queried from an **Azure Keyvault** instance (implemented in `core.project_config.secrets`), for which the setup instructions are in the [top-level README](../README.md#project-secrets) -When an extract is ready to be published to the DSH, the PIXL pipeline will upload the **Public** +When an extract is ready to be published to the relevant FTPS server (UCL DSH), the PIXL pipeline will upload the **Public** and **Radiology** [_parquet_ files](../docs/file_types/parquet_files.md) to the `` directory where the DICOM datasets are stored (see the directory structure below). The uploading is controlled by `upload_parquet_files` in [`upload.py`](./src/core/upload.py) which takes a `ParquetExport` @@ -124,7 +124,7 @@ by the `export-patient-data` API endpoint defined in the [Export API](../pixl_export/src/pixl_export/main.py), which in turn is called by the `export_patient_data` command in the [PIXL CLI](../cli/README.md). -Once the parquet files have been uploaded to the DSH, the directory structure will look like this: +Once the parquet files have been uploaded, the directory structure will look like this: ```sh @@ -220,7 +220,7 @@ with existing session and series data: ### XNAT testing setup -For unit testing, we use [`xnat4tests`](https://github.com/Australian-Imaging-Service/xnat4tests) to spin up an XNAT +For unit testing, we use the publicly available [`xnat4tests`](https://github.com/Australian-Imaging-Service/xnat4tests) to spin up an XNAT instance in a Docker container. Secrets are not used for these unit testing. Instead, the following environment variables are used to configure XNAT for testing: @@ -237,15 +237,28 @@ in the `XNAT_USER_NAME` and `XNAT_PASSWORD` environment variables. ## 'PIXL/pixl_core' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [src](./src/README.md) [tests](./tests/README.md) -### Files +
+ +
+ +

Files

+ + + +| **Configuration** | **User docs** | +| :--- | :--- | +| pyproject.toml | README.md | -pyproject.toml +
-README.md From b0360ea01599df32065129639b734d0f284a3480 Mon Sep 17 00:00:00 2001 From: Sarah Keating Date: Tue, 7 Jan 2025 12:52:17 +0000 Subject: [PATCH 10/16] Making readmes more consistent --- pixl_dcmd/README.md | 23 ++++++++++++++++++----- pixl_export/README.md | 27 ++++++++++++++++++++------- 2 files changed, 38 insertions(+), 12 deletions(-) diff --git a/pixl_dcmd/README.md b/pixl_dcmd/README.md index a614e865d..e06dd80d4 100644 --- a/pixl_dcmd/README.md +++ b/pixl_dcmd/README.md @@ -10,7 +10,7 @@ For external users, the `pixl_dcmd` package provides the following functionality and deletes any tags not mentioned in the tag scheme. The dataset is updated in place. - There is also an option to synchronise to the PIXL database, external users can avoid this to just run the allow-list and applying the tag scheme. - - Will throw a `PixlSkipInstanceError` for any series based on the project config file. Specifically, an error + - Will throw a `PixlSkipInstanceError` for any series based on the project config file. {SK: this sentence doesn't quite make sense to me} Specifically, an error will be thrown if: - the series description matches any series in `series_filters` (usually to remove localiser series) - the modality of the DICOM is not in `modalities` @@ -58,18 +58,31 @@ and optionally a `manufacturer_overrides`. If a `manufacturer_overrides` is defined, it will be used to override the `base` tags, if the manufacturer of the DICOM file matches the manufacturer in the `manufacturer_overrides`. Any tags in the `manufacturer_overrides` that are not in the `base` will be added to the scheme as well. +[SK: Between this and the previous read me I'm not sure this is totally clear, is it possible to have a full example of the yml file linked so that base and a manufacturer_overrides make a bit more sense ] ## 'PIXL/pixl_dcmd' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [src](./src/README.md) [tests](./tests/README.md) -### Files +
+ +
+ +

Files

+ + -pyproject.toml +| **Configuration** | **User docs** | +| :--- | :--- | +| pyproject.toml | README.md | -README.md +
diff --git a/pixl_export/README.md b/pixl_export/README.md index 89481df38..9c7bc85c5 100644 --- a/pixl_export/README.md +++ b/pixl_export/README.md @@ -3,7 +3,7 @@ The Export API provides HTTP endpoints to control the copying of EHR data from the OMOP extract to its destination (eg. FTPS). It also uploads DICOM data to its destination after it has been processed by the Imaging API and orthanc(s). -It no longer accepts messages from rabbitmq. +It no longer accepts messages from rabbitmq. ## Installation @@ -38,23 +38,36 @@ pytest ## Usage -Usage should be from the CLI driver, which calls the HTTP endpoints. +Usage should be from the [CLI driver](../cli/README.md), which calls the HTTP endpoints. ## Notes -- The height/weight/GCS value is extracted only within a 24 h time window +- The height/weight/GCS value is extracted only within a 24 h time window ## 'PIXL/pixl_export' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [src](./src/README.md) [tests](./tests/README.md) -### Files +
+ +
+ +

Files

+ + + +| **Configuration** | **User docs** | +| :--- | :--- | +| pyproject.toml | README.md | -pyproject.toml +
-README.md From 2babe4ddc65234953006d8aa8f50966b9775d641 Mon Sep 17 00:00:00 2001 From: Sarah Keating Date: Fri, 10 Jan 2025 10:04:04 +0000 Subject: [PATCH 11/16] Making readmes consistent --- pixl_imaging/README.md | 28 +++++++++++++--- pixl_imaging/alembic/README.md | 32 ++++++++++++------- pixl_imaging/alembic/versions/README.md | 17 ++++++---- pixl_imaging/scripts/README.md | 12 +++++-- pixl_imaging/src/README.md | 20 +++++++++++- pixl_imaging/src/pixl_imaging/README.md | 17 ++++++---- pixl_imaging/tests/README.md | 21 +++++++++--- .../tests/orthanc_raw_config/README.md | 13 ++++++-- 8 files changed, 121 insertions(+), 39 deletions(-) diff --git a/pixl_imaging/README.md b/pixl_imaging/README.md index 26463b37c..1b2b48e15 100644 --- a/pixl_imaging/README.md +++ b/pixl_imaging/README.md @@ -3,6 +3,8 @@ The PIXL imaging API processes messages created by the [CLI](../cli/README.md) and sent to imaging queues to query images from a dicom server and transfer them to the [`orthanc-raw` instance](../orthanc/orthanc-raw/README.md). +
+## UCLH Specific The imaging API has two queues: - `imaging-primary`, for querying the VNA @@ -20,7 +22,7 @@ If the study has be identified in VNA or PACS, a query to `orthanc-raw` is made exists locally. If it does exist locally, a check is made to ensure all instances exist locally and any missing instances are retrieved. If the study does not exist locally, the entire study is retrieved from the archive. -Once the study and all its instances are in `orthanc-raw`, the study is sent to `orthanc-anon` via a C-STORE +Once the study and all its instances are in `orthanc-raw`, the study is sent to `orthanc-anon` via a C-STORE operation. >[!NOTE] @@ -29,6 +31,8 @@ operation. > query will be retrieved and sent to Orthanc Anon for anonymisation. In Orthanc Anon, the studies will be combined > into a single study as they share the same MRN and Accession Number. +
+ ## Installation ```bash @@ -63,7 +67,11 @@ The `SKIP_ALEMBIC` environmental variable is used to control whether migrations ## 'PIXL/pixl_imaging' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [alembic](./alembic/README.md) @@ -73,9 +81,19 @@ The `SKIP_ALEMBIC` environmental variable is used to control whether migrations [tests](./tests/README.md) -### Files +
+ +
+ +

Files

+ + + +| **Configuration** | **User docs** | +| :--- | :--- | +| pyproject.toml | README.md | + -pyproject.toml +
-README.md diff --git a/pixl_imaging/alembic/README.md b/pixl_imaging/alembic/README.md index d7054661f..4fefece03 100644 --- a/pixl_imaging/alembic/README.md +++ b/pixl_imaging/alembic/README.md @@ -10,9 +10,9 @@ as the entry point is [migrate_and_run.sh](../scripts/migrate_and_run.sh). ## Creating a new migration after editing the database model -For convenience, the [autogenerate-migration.sh](autogenerate-migration.sh) has been made. +For convenience, the [autogenerate-migration.sh](autogenerate-migration.sh) script has been created. -Which you can run giving a name for the migration like this: +It can be run giving a name for the migration like this: ```shell cd alembic @@ -22,7 +22,7 @@ cd alembic - This creates a postgres container - Runs the existing migrations - Checks for differences between the SQLAlchemy [models](../../pixl_core/src/core/db/models.py) - and creates a new migration in [versions](versions) + and creates a new migration in the [versions](versions) directory - Takes down the postgres container There's a couple of manual steps: @@ -32,21 +32,31 @@ There's a couple of manual steps: ## 'PIXL/pixl_imaging/alembic' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [versions](./versions/README.md) -### Files +
-alembic.ini +
+ +

Files

-autogenerate-migration.sh + -env.py +| **Configuration** | **User docs** | +| :--- | :--- | +| alembic.ini | README.md | +| autogenerate-migration.sh | | +| env.py | | +| migrations.env | | +| script.py.mako | | -migrations.env +
-README.md -script.py.mako diff --git a/pixl_imaging/alembic/versions/README.md b/pixl_imaging/alembic/versions/README.md index ea043622a..67da1fc1d 100644 --- a/pixl_imaging/alembic/versions/README.md +++ b/pixl_imaging/alembic/versions/README.md @@ -1,12 +1,17 @@ ## 'PIXL/pixl_imaging/alembic/versions' Directory Contents -### Files +
+ +

Files

-83dcb3812628_add_study_uid_column_to_image_table.py + -bcaef54e2bfe_create_extract_and_image_tables.py +| **Code** | **User docs** | +| :--- | :--- | +| 83dcb3812628_add_study_uid_column_to_image_table.py | README.md | +| bcaef54e2bfe_create_extract_and_image_tables.py | | +| cb5ee12a6e20_replace_hashed_id_with_pseudo_study_uid.py | | +| d947cc715eb1_add_pseudo_patient_id_column_to_image_table.py | | -cb5ee12a6e20_replace_hashed_id_with_pseudo_study_uid.py - -d947cc715eb1_add_pseudo_patient_id_column_to_image_table.py +
diff --git a/pixl_imaging/scripts/README.md b/pixl_imaging/scripts/README.md index afd8a5659..ae1e0718a 100644 --- a/pixl_imaging/scripts/README.md +++ b/pixl_imaging/scripts/README.md @@ -1,6 +1,14 @@ ## 'PIXL/pixl_imaging/scripts' Directory Contents -### Files +
+ +

Files

-migrate_and_run.sh + + +| **Code** | **User docs** | +| :--- | :--- | +| migrate_and_run.sh | README.md | + +
diff --git a/pixl_imaging/src/README.md b/pixl_imaging/src/README.md index a8f1b4ac5..d56a9acbc 100644 --- a/pixl_imaging/src/README.md +++ b/pixl_imaging/src/README.md @@ -1,6 +1,24 @@ ## 'PIXL/pixl_imaging/src' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [pixl_imaging](./pixl_imaging/README.md) +
+ +
+ +

Files

+ + + +| **User docs** | +| :--- | +| README.md | + +
+ diff --git a/pixl_imaging/src/pixl_imaging/README.md b/pixl_imaging/src/pixl_imaging/README.md index 8250982db..dbe9a22e4 100644 --- a/pixl_imaging/src/pixl_imaging/README.md +++ b/pixl_imaging/src/pixl_imaging/README.md @@ -1,12 +1,17 @@ ## 'PIXL/pixl_imaging/src/pixl_imaging' Directory Contents -### Files +
+ +

Files

-main.py + -_orthanc.py +| **Code** | **User docs** | +| :--- | :--- | +| main.py | README.md | +| _orthanc.py | | +| _processing.py | | +| __init__.py | | -_processing.py - -__init__.py +
diff --git a/pixl_imaging/tests/README.md b/pixl_imaging/tests/README.md index 8903dd81c..98447e8ec 100644 --- a/pixl_imaging/tests/README.md +++ b/pixl_imaging/tests/README.md @@ -1,14 +1,25 @@ ## 'PIXL/pixl_imaging/tests' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [orthanc_raw_config](./orthanc_raw_config/README.md) -### Files +
+ +
+ +

Files

-conftest.py + -docker-compose.yml +| **Configuration** | **Code** | **User docs** | +| :--- | :--- | :--- | +| docker-compose.yml | conftest.py | README.md | +| | test_imaging_processing.py | | -test_imaging_processing.py +
diff --git a/pixl_imaging/tests/orthanc_raw_config/README.md b/pixl_imaging/tests/orthanc_raw_config/README.md index 3b95d29ca..3deb7691a 100644 --- a/pixl_imaging/tests/orthanc_raw_config/README.md +++ b/pixl_imaging/tests/orthanc_raw_config/README.md @@ -1,8 +1,15 @@ ## 'PIXL/pixl_imaging/tests/orthanc_raw_config' Directory Contents -### Files +
+ +

Files

-dicom.json + -orthanc.json +| **Configuration** | **User docs** | +| :--- | :--- | +| dicom.json | README.md | +| orthanc.json | | + +
From 415761f47a23d5d0f04d89cb4abdaa12fe8d409d Mon Sep 17 00:00:00 2001 From: Sarah Keating Date: Fri, 10 Jan 2025 10:22:35 +0000 Subject: [PATCH 12/16] Making readmes consistent --- postgres/README.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/postgres/README.md b/postgres/README.md index e532b9469..f17a85177 100644 --- a/postgres/README.md +++ b/postgres/README.md @@ -8,11 +8,17 @@ See [/pixl_imaging/alembic](../pixl_imaging/alembic) for how these are defined ## 'PIXL/postgres' Directory Contents -### Files +
+ +

Files

-pixl-db_init.sh + + +| **Configuration** | **Code** | **User docs** | +| :--- | :--- | :--- | +| postgres.conf | pixl-db_init.sh | README.md | + +
-postgres.conf -README.md From 57f13b9f43ef434f536525aa988a9049f6d40c95 Mon Sep 17 00:00:00 2001 From: Sarah Keating Date: Fri, 10 Jan 2025 11:37:43 +0000 Subject: [PATCH 13/16] Making readmes consistent --- projects/README.md | 20 ++++++++++- projects/configs/README.md | 32 ++++++++++------- projects/configs/tag-operations/README.md | 35 +++++++++++-------- .../manufacturer-overrides/README.md | 17 ++++++--- 4 files changed, 73 insertions(+), 31 deletions(-) diff --git a/projects/README.md b/projects/README.md index dcfdcd654..f393a0d79 100644 --- a/projects/README.md +++ b/projects/README.md @@ -1,8 +1,26 @@ ## 'PIXL/projects' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [configs](./configs/README.md) [exports](./exports/README.md) +
+ +
+ +

Files

+ + + +| **User docs** | +| :--- | +| README.md | + +
+ diff --git a/projects/configs/README.md b/projects/configs/README.md index 351680b9b..15420eb30 100644 --- a/projects/configs/README.md +++ b/projects/configs/README.md @@ -1,22 +1,30 @@ ## 'PIXL/projects/configs' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

-[tag-operations](./tag-operations/README.md) - -### Files + -ms-pinpoint.yaml - -test-external-user.yaml +[tag-operations](./tag-operations/README.md) -test-extract-uclh-omop-cdm-dicomweb.yaml +
-test-extract-uclh-omop-cdm-xnat.yaml +
+ +

Files

-test-extract-uclh-omop-cdm.yaml + -uclh-nasogastric-tube-project-ngt-only-full-dataset.yaml +| **Configuration** | **User docs** | +| :--- | :--- | +| ms-pinpoint.yaml | README.md | +| test-external-user.yaml | | +| test-extract-uclh-omop-cdm-dicomweb.yaml | | +| test-extract-uclh-omop-cdm-xnat.yaml | | +| test-extract-uclh-omop-cdm.yaml | | +| uclh-nasogastric-tube-project-ngt-only-full-dataset.yaml | | +| uclh-prostate-mri-external-dataset.yaml | | -uclh-prostate-mri-external-dataset.yaml +
diff --git a/projects/configs/tag-operations/README.md b/projects/configs/tag-operations/README.md index 83c53c681..d97782d7a 100644 --- a/projects/configs/tag-operations/README.md +++ b/projects/configs/tag-operations/README.md @@ -1,24 +1,31 @@ ## 'PIXL/projects/configs/tag-operations' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

-[manufacturer-overrides](./manufacturer-overrides/README.md) - -### Files - -base.yaml + -diffusion-weighted-mri.yaml - -mri.yaml +[manufacturer-overrides](./manufacturer-overrides/README.md) -ms-pinpoint.yaml +
-test-external-user.yaml +
+ +

Files

-test-extract-uclh-omop-cdm.yaml + -uclh-nasogastric-tube-project-ngt-only-full-dataset.yaml +| **Configuration** | **User docs** | +| :--- | :--- | +| base.yaml | README.md | +| diffusion-weighted-mri.yaml | | +| mri.yaml | | +| ms-pinpoint.yaml | | +| test-external-user.yaml | | +| test-extract-uclh-omop-cdm.yaml | | +| uclh-nasogastric-tube-project-ngt-only-full-dataset.yaml | | +| xray.yaml | | -xray.yaml +
diff --git a/projects/configs/tag-operations/manufacturer-overrides/README.md b/projects/configs/tag-operations/manufacturer-overrides/README.md index b77c531e3..d0c3127b0 100644 --- a/projects/configs/tag-operations/manufacturer-overrides/README.md +++ b/projects/configs/tag-operations/manufacturer-overrides/README.md @@ -1,8 +1,17 @@ -## 'manufacturer-overrides' Directory Contents +## 'PIXL/projects/configs/tag-operations/manufacturer-overrides' Directory Contents -### Files +
+ +

Files

+ + + +| **Configuration** | **User docs** | +| :--- | :--- | +| mri-diffusion.yaml | README.md | +| mri.yaml | | + +
-mri-diffusion.yaml -mri.yaml From e00eeaed5ff3160f2012ec999e9cf5ee9f5a2f35 Mon Sep 17 00:00:00 2001 From: Sarah Keating Date: Fri, 10 Jan 2025 15:27:01 +0000 Subject: [PATCH 14/16] Making readmes consistent --- pixl_imaging/README.md | 6 +--- pytest-pixl/README.md | 28 +++++++++++++++---- pytest-pixl/scripts/README.md | 12 ++++++-- pytest-pixl/src/README.md | 20 ++++++++++++- pytest-pixl/src/pytest_pixl/README.md | 26 +++++++++++------ pytest-pixl/src/pytest_pixl/data/README.md | 21 +++++++++++--- .../pytest_pixl/data/omop-resources/README.md | 28 +++++++++++++------ .../data/omop-resources/omop/README.md | 22 ++++++++++++--- .../omop-resources/omop/private/README.md | 14 +++++++--- .../data/omop-resources/omop/public/README.md | 14 ++++++++-- pytest-pixl/src/resources/README.md | 20 ++++++++++++- pytest-pixl/src/resources/ssl/README.md | 13 +++++++-- pytest-pixl/tests/README.md | 21 +++++++++++--- .../tests/samples_for_fixture_tests/README.md | 20 ++++++++++++- .../test_ftpserver_fixture/README.md | 12 ++++++-- 15 files changed, 220 insertions(+), 57 deletions(-) diff --git a/pixl_imaging/README.md b/pixl_imaging/README.md index 1b2b48e15..0b0f6dbf9 100644 --- a/pixl_imaging/README.md +++ b/pixl_imaging/README.md @@ -3,8 +3,6 @@ The PIXL imaging API processes messages created by the [CLI](../cli/README.md) and sent to imaging queues to query images from a dicom server and transfer them to the [`orthanc-raw` instance](../orthanc/orthanc-raw/README.md). -
-## UCLH Specific The imaging API has two queues: - `imaging-primary`, for querying the VNA @@ -29,9 +27,7 @@ operation. > When querying the archives, if we do not know the `StudyInstanceUID` we will query by MRN and Accession Number. > This may result in multiple studies being found in the archives. In this instance, all studies returned by the > query will be retrieved and sent to Orthanc Anon for anonymisation. In Orthanc Anon, the studies will be combined -> into a single study as they share the same MRN and Accession Number. - -
+> into a single study as they share the same MRN and Accession Number. ## Installation diff --git a/pytest-pixl/README.md b/pytest-pixl/README.md index 30c8c4c21..2a17fcff4 100644 --- a/pytest-pixl/README.md +++ b/pytest-pixl/README.md @@ -27,6 +27,8 @@ the following environment variables: ## Available testing utilities +Functions from 'PIXL/pytest-pixl/src/pytest_pixl': + - `dicom.write_volume`: write a volume of MRI DICOMs for testing - `dicom.generate_dicom_dataset`: generate a DICOM dataset for testing @@ -46,7 +48,7 @@ Currently we only handle the following dictionary keys - `window_width` - `pixel_data` -This is useful for example when generating DICOM datasets for a volume of slices. +This is useful when generating DICOM datasets for a volume of slices. See for example `dicom.write_volume()`. In addition to the tags dictionary, `generate_dicom_dataset()` has a `**kwargs` parameter that @@ -64,7 +66,7 @@ and `private_tags` is a list of `tuple`s with the following format: [(tag_id, VR, value), ...] ``` -where `tag` can be a `str`, `int` or `Tuple[int, int]`, `VR` is a `str` and `value` is a `str`. +where `tag_id` can be a `str`, `int` or `Tuple[int, int]`, `VR` is a `str` and `value` is a `str`. Note that this requires the [VR](https://dicom.nema.org/dicom/2013/output/chtml/part05/sect_6.2.html) of the tag to be known. @@ -104,7 +106,11 @@ with the details implemented in [`pytest_pixl.dicom._create_default_json`](./src ## 'PIXL/pytest-pixl' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [scripts](./scripts/README.md) @@ -112,9 +118,19 @@ with the details implemented in [`pytest_pixl.dicom._create_default_json`](./src [tests](./tests/README.md) -### Files +
+ +
+ +

Files

+ + + +| **Configuration** | **User docs** | +| :--- | :--- | +| pyproject.toml | README.md | + +
-pyproject.toml -README.md diff --git a/pytest-pixl/scripts/README.md b/pytest-pixl/scripts/README.md index 65085443d..c9259c4e5 100644 --- a/pytest-pixl/scripts/README.md +++ b/pytest-pixl/scripts/README.md @@ -1,6 +1,14 @@ ## 'PIXL/pytest-pixl/scripts' Directory Contents -### Files +
+ +

Files

-create_default_dicom_tags_json.py + + +| **Code** | **User docs** | +| :--- | :--- | +| create_default_dicom_tags_json.py | README.md | + +
diff --git a/pytest-pixl/src/README.md b/pytest-pixl/src/README.md index 138dffa26..9a38c50e5 100644 --- a/pytest-pixl/src/README.md +++ b/pytest-pixl/src/README.md @@ -1,8 +1,26 @@ ## 'PIXL/pytest-pixl/src' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [pytest_pixl](./pytest_pixl/README.md) [resources](./resources/README.md) +
+ +
+ +

Files

+ + + +| **User docs** | +| :--- | +| README.md | + +
+ diff --git a/pytest-pixl/src/pytest_pixl/README.md b/pytest-pixl/src/pytest_pixl/README.md index 98744d18c..0e5186840 100644 --- a/pytest-pixl/src/pytest_pixl/README.md +++ b/pytest-pixl/src/pytest_pixl/README.md @@ -1,18 +1,28 @@ ## 'PIXL/pytest-pixl/src/pytest_pixl' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

-[data](./data/README.md) + -### Files +[data](./data/README.md) -dicom.py +
-ftpserver.py +
+ +

Files

-helpers.py + -plugin.py +| **Code** | **User docs** | +| :--- | :--- | +| dicom.py | README.md | +| ftpserver.py | | +| helpers.py | | +| plugin.py | | +| __init__.py | | -__init__.py +
diff --git a/pytest-pixl/src/pytest_pixl/data/README.md b/pytest-pixl/src/pytest_pixl/data/README.md index cfa58b0bb..09b8b2a13 100644 --- a/pytest-pixl/src/pytest_pixl/data/README.md +++ b/pytest-pixl/src/pytest_pixl/data/README.md @@ -1,12 +1,25 @@ ## 'PIXL/pytest-pixl/src/pytest_pixl/data' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [omop-resources](./omop-resources/README.md) -### Files +
+ +
+ +

Files

+ + -default_dicom_tags.json +| **Configuration** | **User docs** | +| :--- | :--- | +| default_dicom_tags.json | README.md | +| volume_dicom_variables.json | | -volume_dicom_variables.json +
diff --git a/pytest-pixl/src/pytest_pixl/data/omop-resources/README.md b/pytest-pixl/src/pytest_pixl/data/omop-resources/README.md index 7a457c566..780ec047f 100644 --- a/pytest-pixl/src/pytest_pixl/data/omop-resources/README.md +++ b/pytest-pixl/src/pytest_pixl/data/omop-resources/README.md @@ -1,18 +1,28 @@ -## 'omop-resources' Directory Contents +## 'PIXL/pytest-pixl/src/pytest_pixl/data/omop-resources' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

-[omop](./omop/README.md) + -### Files +[omop](./omop/README.md) -batch_input.csv +
-duplicate_input.csv +
+ +

Files

-multiple_projects.csv + -participant_id.csv +| **Data** | **User docs** | +| :--- | :--- | +| batch_input.csv | README.md | +| duplicate_input.csv | | +| multiple_projects.csv | | +| participant_id.csv | | +| test.csv | | -test.csv +
diff --git a/pytest-pixl/src/pytest_pixl/data/omop-resources/omop/README.md b/pytest-pixl/src/pytest_pixl/data/omop-resources/omop/README.md index e5d45c492..328a0fc7c 100644 --- a/pytest-pixl/src/pytest_pixl/data/omop-resources/omop/README.md +++ b/pytest-pixl/src/pytest_pixl/data/omop-resources/omop/README.md @@ -1,12 +1,26 @@ -## 'omop' Directory Contents +## 'PIXL/pytest-pixl/src/pytest_pixl/data/omop-resources/omop' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [private](./private/README.md) [public](./public/README.md) -### Files +
+ +
+ +

Files

+ + + +| **Configuration** | **User docs** | +| :--- | :--- | +| extract_summary.json | README.md | -extract_summary.json +
diff --git a/pytest-pixl/src/pytest_pixl/data/omop-resources/omop/private/README.md b/pytest-pixl/src/pytest_pixl/data/omop-resources/omop/private/README.md index 2e47873ed..81c4746bc 100644 --- a/pytest-pixl/src/pytest_pixl/data/omop-resources/omop/private/README.md +++ b/pytest-pixl/src/pytest_pixl/data/omop-resources/omop/private/README.md @@ -1,8 +1,14 @@ -## 'private' Directory Contents +## 'PIXL/pytest-pixl/src/pytest_pixl/data/omop-resources/omop/private' Directory Contents -### Files +
+ +

Files

-PERSON_LINKS.parquet + -PROCEDURE_OCCURRENCE_LINKS.parquet +| **Data** | **User docs** | +| :--- | :--- | +| PERSON_LINKS.parquet | README.md | +| PROCEDURE_OCCURRENCE_LINKS.parquet | | +
diff --git a/pytest-pixl/src/pytest_pixl/data/omop-resources/omop/public/README.md b/pytest-pixl/src/pytest_pixl/data/omop-resources/omop/public/README.md index d35e10141..d446f6902 100644 --- a/pytest-pixl/src/pytest_pixl/data/omop-resources/omop/public/README.md +++ b/pytest-pixl/src/pytest_pixl/data/omop-resources/omop/public/README.md @@ -1,6 +1,14 @@ -## 'public' Directory Contents +## 'PIXL/pytest-pixl/src/pytest_pixl/data/omop-resources/omop/public' Directory Contents -### Files +
+ +

Files

-PROCEDURE_OCCURRENCE.parquet + + +| **Data** | **User docs** | +| :--- | :--- | +| PROCEDURE_OCCURRENCE.parquet | README.md | + +
diff --git a/pytest-pixl/src/resources/README.md b/pytest-pixl/src/resources/README.md index 8c118dc40..591013d8e 100644 --- a/pytest-pixl/src/resources/README.md +++ b/pytest-pixl/src/resources/README.md @@ -1,6 +1,24 @@ ## 'PIXL/pytest-pixl/src/resources' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [ssl](./ssl/README.md) +
+ +
+ +

Files

+ + + +| **User docs** | +| :--- | +| README.md | + +
+ diff --git a/pytest-pixl/src/resources/ssl/README.md b/pytest-pixl/src/resources/ssl/README.md index 20c619682..d6d2ec754 100644 --- a/pytest-pixl/src/resources/ssl/README.md +++ b/pytest-pixl/src/resources/ssl/README.md @@ -1,8 +1,15 @@ ## 'PIXL/pytest-pixl/src/resources/ssl' Directory Contents -### Files +
+ +

Files

-localhost.crt + -localhost.key +| **Keys** | **User docs** | +| :--- | :--- | +| localhost.crt | README.md | +| localhost.key | | + +
diff --git a/pytest-pixl/tests/README.md b/pytest-pixl/tests/README.md index d7c2900b2..f4b67339b 100644 --- a/pytest-pixl/tests/README.md +++ b/pytest-pixl/tests/README.md @@ -1,12 +1,25 @@ ## 'PIXL/pytest-pixl/tests' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [samples_for_fixture_tests](./samples_for_fixture_tests/README.md) -### Files +
+ +
+ +

Files

+ + -conftest.py +| **Code** | **User docs** | +| :--- | :--- | +| conftest.py | README.md | +| test_ftpserver_fixture.py | | -test_ftpserver_fixture.py +
diff --git a/pytest-pixl/tests/samples_for_fixture_tests/README.md b/pytest-pixl/tests/samples_for_fixture_tests/README.md index ee9d21902..7ff4031d4 100644 --- a/pytest-pixl/tests/samples_for_fixture_tests/README.md +++ b/pytest-pixl/tests/samples_for_fixture_tests/README.md @@ -1,6 +1,24 @@ ## 'PIXL/pytest-pixl/tests/samples_for_fixture_tests' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [test_ftpserver_fixture](./test_ftpserver_fixture/README.md) +
+ +
+ +

Files

+ + + +| **User docs** | +| :--- | +| README.md | + +
+ diff --git a/pytest-pixl/tests/samples_for_fixture_tests/test_ftpserver_fixture/README.md b/pytest-pixl/tests/samples_for_fixture_tests/test_ftpserver_fixture/README.md index 06effc923..ab97db7b0 100644 --- a/pytest-pixl/tests/samples_for_fixture_tests/test_ftpserver_fixture/README.md +++ b/pytest-pixl/tests/samples_for_fixture_tests/test_ftpserver_fixture/README.md @@ -1,6 +1,14 @@ ## 'PIXL/pytest-pixl/tests/samples_for_fixture_tests/test_ftpserver_fixture' Directory Contents -### Files +
+ +

Files

-test_ftpserver_login.py + + +| **Code** | **User docs** | +| :--- | :--- | +| test_ftpserver_login.py | README.md | + +
From 35c34b269ac66aac84798ed4c85c1ba7404f2b24 Mon Sep 17 00:00:00 2001 From: Sarah Keating Date: Fri, 10 Jan 2025 15:59:48 +0000 Subject: [PATCH 15/16] Making readmes consistent --- schemas/README.md | 12 +++++- scripts/README.md | 17 +++++--- test/README.md | 40 ++++++++++--------- test/dicomweb_config/README.md | 13 ++++-- test/resources/README.md | 21 ++++++++-- test/resources/omop-dicomweb/README.md | 22 ++++++++-- .../resources/omop-dicomweb/private/README.md | 16 ++++++-- test/resources/omop-dicomweb/public/README.md | 14 +++++-- test/resources/omop/README.md | 20 ++++++++-- test/resources/omop/private/README.md | 13 ++++-- test/resources/omop/public/README.md | 12 +++++- test/vna_config/README.md | 13 ++++-- 12 files changed, 156 insertions(+), 57 deletions(-) diff --git a/schemas/README.md b/schemas/README.md index 432a9cc7b..0e34f82ee 100644 --- a/schemas/README.md +++ b/schemas/README.md @@ -1,6 +1,14 @@ ## 'PIXL/schemas' Directory Contents -### Files +
+ +

Files

-github-issue-forms.json + + +| **Configuration** | **User docs** | +| :--- | :--- | +| github-issue-forms.json | README.md | + +
diff --git a/scripts/README.md b/scripts/README.md index 26ff995a3..992fc993e 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,12 +1,17 @@ ## 'PIXL/scripts' Directory Contents -### Files +
+ +

Files

-cmove_all_studies.py + -delete_oldest_n_studies.py +| **Code** | **User docs** | +| :--- | :--- | +| cmove_all_studies.py | README.md | +| delete_oldest_n_studies.py | | +| filter_cohort_for_those_present_in_raw.py | | +| list_newest_n_studies.py | | -filter_cohort_for_those_present_in_raw.py - -list_newest_n_studies.py +
diff --git a/test/README.md b/test/README.md index b1c25278f..20495378b 100644 --- a/test/README.md +++ b/test/README.md @@ -39,9 +39,9 @@ For CI, there is also another subcommand to run pytest, reporting coverage ## The `pytest-pixl` plugin -We provide a [`pytest` plugin](../pytest-pixl/README.md) with shared functionality for PIXL system +We provide a [`pytest` plugin](../pytest-pixl/README.md) with shared functionality for the PIXL system and unit tests. This includes an `ftp_server` fixture to spin up a lightweight FTP server, -to mock the FTP server used by the Data Safe Haven. +to mock an FTP server being used to deposit the data. ## File organisation @@ -51,7 +51,7 @@ to mock the FTP server used by the Data Safe Haven. ### Scripts -`./scripts` contains bash and Python scripts to check the individual components of the system test. +`../scripts` contains bash and Python scripts to check the individual components of the system test. ### Resources @@ -72,7 +72,11 @@ but configured to upload to a [DICOMweb server](#dicomweb-config) ## 'PIXL/test' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [dicomweb_config](./dicomweb_config/README.md) @@ -80,23 +84,21 @@ but configured to upload to a [DICOMweb server](#dicomweb-config) [vna_config](./vna_config/README.md) -### Files - -.env - -.secrets.env.sample - -conftest.py - -docker-compose.yml - -README.md +
-run-system-test.sh +
+ +

Files

-system_test.py + -test_parquet_exports.py +|**Configuration** | **Code** | **User docs** | +| :--- | :--- | :--- | +| .env | conftest.py | README.md | +| .secrets.env.sample | run-system-test.sh | | +| docker-compose.yml | system_test.py | | +| | test_parquet_exports.py | | +| | utils.py | | -utils.py +
diff --git a/test/dicomweb_config/README.md b/test/dicomweb_config/README.md index 41729879a..23d069559 100644 --- a/test/dicomweb_config/README.md +++ b/test/dicomweb_config/README.md @@ -1,10 +1,15 @@ ## 'PIXL/test/dicomweb_config' Directory Contents -### Files +
+ +

Files

-.env + -dicom.json +| **Configuration** | **User docs** | +| :--- | :--- | +| dicom.json | README.md | +| orthanc.json | | -orthanc.json +
diff --git a/test/resources/README.md b/test/resources/README.md index 132dcb141..f633fe31e 100644 --- a/test/resources/README.md +++ b/test/resources/README.md @@ -1,14 +1,27 @@ ## 'PIXL/test/resources' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [omop](./omop/README.md) [omop-dicomweb](./omop-dicomweb/README.md) -### Files +
+ +
+ +

Files

+ + -Dicom1.dcm +| **Dicom** | **User docs** | +| :--- | :--- | +| Dicom1.dcm | README.md | +| Dicom2.dcm | | -Dicom2.dcm +
diff --git a/test/resources/omop-dicomweb/README.md b/test/resources/omop-dicomweb/README.md index 0847fb646..1d1788ac7 100644 --- a/test/resources/omop-dicomweb/README.md +++ b/test/resources/omop-dicomweb/README.md @@ -1,12 +1,26 @@ -## 'omop-dicomweb' Directory Contents +## 'PIXL/test/resources/omop-dicomweb' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [private](./private/README.md) [public](./public/README.md) -### Files +
+ +
+ +

Files

+ + + +| **Configuration** | **User docs** | +| :--- | :--- | +| extract_summary.json | README.md | -extract_summary.json +
diff --git a/test/resources/omop-dicomweb/private/README.md b/test/resources/omop-dicomweb/private/README.md index 2e47873ed..92299c635 100644 --- a/test/resources/omop-dicomweb/private/README.md +++ b/test/resources/omop-dicomweb/private/README.md @@ -1,8 +1,16 @@ -## 'private' Directory Contents +## 'PIXL/test/resources/omop-dicomweb/private' Directory Contents -### Files +
+ +

Files

-PERSON_LINKS.parquet + + +| **Data** | **User docs** | +| :--- | :--- | +| PERSON_LINKS.parquet | README.md | +| PROCEDURE_OCCURRENCE_LINKS.parquet | | + +
-PROCEDURE_OCCURRENCE_LINKS.parquet diff --git a/test/resources/omop-dicomweb/public/README.md b/test/resources/omop-dicomweb/public/README.md index d35e10141..98de24d26 100644 --- a/test/resources/omop-dicomweb/public/README.md +++ b/test/resources/omop-dicomweb/public/README.md @@ -1,6 +1,14 @@ -## 'public' Directory Contents +## 'PIXL/test/resources/omop-dicomweb/public' Directory Contents -### Files +
+ +

Files

-PROCEDURE_OCCURRENCE.parquet + + +| **Data** | **User docs** | +| :--- | :--- | +| PROCEDURE_OCCURRENCE.parquet | README.md | + +
diff --git a/test/resources/omop/README.md b/test/resources/omop/README.md index f8f0010b4..228f6840a 100644 --- a/test/resources/omop/README.md +++ b/test/resources/omop/README.md @@ -1,12 +1,26 @@ ## 'PIXL/test/resources/omop' Directory Contents -### Subdirectories +
+ +

Subdirectories with links to the relevant README

+ + [private](./private/README.md) [public](./public/README.md) -### Files +
+ +
+ +

Files

+ + + +| **Configuration** | **User docs** | +| :--- | :--- | +| extract_summary.json | README.md | -extract_summary.json +
diff --git a/test/resources/omop/private/README.md b/test/resources/omop/private/README.md index d0082ec64..969599330 100644 --- a/test/resources/omop/private/README.md +++ b/test/resources/omop/private/README.md @@ -1,8 +1,15 @@ ## 'PIXL/test/resources/omop/private' Directory Contents -### Files +
+ +

Files

-PERSON_LINKS.parquet + -PROCEDURE_OCCURRENCE_LINKS.parquet +| **Data** | **User docs** | +| :--- | :--- | +| PERSON_LINKS.parquet | README.md | +| PROCEDURE_OCCURRENCE_LINKS.parquet | | + +
diff --git a/test/resources/omop/public/README.md b/test/resources/omop/public/README.md index be9075cd4..e10168fbd 100644 --- a/test/resources/omop/public/README.md +++ b/test/resources/omop/public/README.md @@ -1,6 +1,14 @@ ## 'PIXL/test/resources/omop/public' Directory Contents -### Files +
+ +

Files

-PROCEDURE_OCCURRENCE.parquet + + +| **Data** | **User docs** | +| :--- | :--- | +| PROCEDURE_OCCURRENCE.parquet | README.md | + +
diff --git a/test/vna_config/README.md b/test/vna_config/README.md index ee4cd9d44..d5f0491e9 100644 --- a/test/vna_config/README.md +++ b/test/vna_config/README.md @@ -1,8 +1,15 @@ ## 'PIXL/test/vna_config' Directory Contents -### Files +
+ +

Files

-dicom.json + -orthanc.json +| **Configuration** | **User docs** | +| :--- | :--- | +| dicom.json | README.md | +| orthanc.json | | + +
From 89b071066c2ad9e615a3755823bbf531544df18b Mon Sep 17 00:00:00 2001 From: Sarah Keating Date: Wed, 4 Jun 2025 16:16:16 +0100 Subject: [PATCH 16/16] Apply suggestions from code review Co-authored-by: Tom Roberts --- orthanc/orthanc-anon/README.md | 6 +++--- orthanc/orthanc-raw/README.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/orthanc/orthanc-anon/README.md b/orthanc/orthanc-anon/README.md index d43f9ffbe..258e5789a 100644 --- a/orthanc/orthanc-anon/README.md +++ b/orthanc/orthanc-anon/README.md @@ -1,6 +1,6 @@ # Orthanc Anon -_The Orthanc instance responsible for anonymising DICOM data from PACS/VNA and forwarding the images +Orthanc Anon is responsible for anonymising DICOM data from PACS/VNA and forwarding the images to their final destination (at UCL/UCLH this is currently the UCL Data Safe Haven). ## Setup @@ -24,8 +24,8 @@ The following assumptions are made: ### Configuration -- The Docker image is based on `orthancteam/orthanc`. <--- This is unclear where is this? -- Configuration is driven through customised JSON config. files stored in the [orthanc-anon/config](./config/) +- Orthanc Anon is based on the Docker image: [orthancteam/orthanc](https://hub.docker.com/r/orthancteam/orthanc). +- Orthanc Anon is configured via a customised JSON config files stored in the [orthanc-anon/config](./config/) directory. - The files are populated with values from environment variables and injected into the container as secrets. Orthanc interprets all `.json` files in the `/run/secrets` mount as config. files. diff --git a/orthanc/orthanc-raw/README.md b/orthanc/orthanc-raw/README.md index 9be91f488..7a76f9d6d 100644 --- a/orthanc/orthanc-raw/README.md +++ b/orthanc/orthanc-raw/README.md @@ -22,7 +22,7 @@ available shortly when the service is started). ### Configuration -- The Docker image is a deployment of `orthancteam/orthanc` with some extra configuration <--- is orthancteam/orthanc supposed to point to somewhere in the tree +- The Orthanc Raw instance is a deployment of the [orthancteam/orthanc](https://hub.docker.com/r/orthancteam/orthanc) container with the following additional configuration: - `ORTHANC_RAW_MAXIMUM_STORAGE_SIZE` to limit the storage size - `ORTHANC_RAW_JOB_HISTORY_SIZE` has been increased so that while there is concurrent processing, the job should always exist for being able to query its status