Staging ---> Main

.gitignore

@@ -59,6 +59,20 @@
 !/tests/*/resources/**/
 !/tests/*/resources/**/MOCK*
 
+!/demo
+!/demo/README.md
+!/demo/example*
+!/demo/example*/data
+!/demo/example*/data/**/mock*/
+!/demo/example*/data/**/*.csv
+!/demo/example*/data/**/*.md
+!/demo/example*/data/**/*.txt
+!/demo/example*/data/**/*.log
+!/demo/example*/README.md
+!/demo/example*/requests.yaml
+!/demo/img
+!/demo/img/*.png
+
 # ----------------------------------------------------------------
 # FORCE REMOVE
 # ----------------------------------------------------------------

README.md

@@ -1,9 +1,11 @@
+[![Rabbit MQ](https://img.shields.io/badge/-RabbitMQ-FF6600?style=flat&logo=rabbitmq&logoColor=white)](https://www.rabbitmq.com)
 [![Python version: 3.14](https://img.shields.io/badge/python%20version-3.14-1464b4.svg)](https://www.python.org)
 [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 
 [![qa manual:main](https://github.com/raj-open/example-rabbit-mq/actions/workflows/manual.yaml/badge.svg?branch=main)](https://github.com/raj-open/example-rabbit-mq/actions/workflows/manual.yaml)
 [![qa manual:staging](https://github.com/raj-open/example-rabbit-mq/actions/workflows/manual.yaml/badge.svg?branch=staging)](https://github.com/raj-open/example-rabbit-mq/actions/workflows/manual.yaml)
 
+[![qa auto:main](https://github.com/raj-open/example-rabbit-mq/actions/workflows/auto.yaml/badge.svg?branch=main)](https://github.com/raj-open/example-rabbit-mq/actions/workflows/auto.yaml)
 [![qa auto:staging](https://github.com/raj-open/example-rabbit-mq/actions/workflows/auto.yaml/badge.svg?branch=staging)](https://github.com/raj-open/example-rabbit-mq/actions/workflows/auto.yaml)
 [![qa auto:current](https://github.com/raj-open/example-rabbit-mq/actions/workflows/auto.yaml/badge.svg)](https://github.com/raj-open/example-rabbit-mq/actions/workflows/auto.yaml)
 
@@ -189,8 +191,8 @@ Fill in `setup/requests.yaml` as follows:
 ```yaml
 label: 'Mock example'
 
-# apply some generous limits
 options:
+  # skip-empty: true # false (default) => includes empty files; true => skips them
   max-depth: 100 # limits depth of folder structure
   max-items: 1_000_000 # limits number of items that can be logged
   max-duration: 00:05:00 # limits maximum computation time
@@ -234,3 +236,7 @@ data:
 
         The file reference in this body can of course be a json
         and located anywhere on your system.
+
+## Demos ##
+
+Some simple example cases can be found in the [demo](demo) folder.

demo/README.md

@@ -0,0 +1,32 @@
+# Demo #
+
+Each folder in this directory contains an example filesystem with requests config.
+
+The README file in each folder explains how to run tool against the case
+and the expected outcome.
+
+## General usage ##
+
+For ease of use one can also run the demos as follows:
+
+1. Open 2 terminals
+
+2. In terminal 1 start the queue
+
+    ```bash
+    just docker-start-queue
+    ``
+
+3. In terminal 2 run the command
+
+    ```bash
+    just demo {name}
+    ````
+
+    where `{name}` is the name of the subfolder, e.g. `"example-case-1"`.
+
+## Results ##
+
+The current demos should result in the following:
+
+![Results](./img/results-demo.png)

demo/example-case-1/README.md

@@ -0,0 +1,30 @@
+# Example - Case 1 #
+
+## Description ##
+
+- An nearly flat filesystem with a nested folder containing empty files.
+- `skip-empty` set to `true`
+
+## Execution ##
+
+Run as follows
+
+1. Open 2 terminals
+
+2. In terminal 1 start the queue
+
+    ```bash
+    just docker-start-queue
+    ```
+
+3. In terminal 2 run the command
+
+    ```bash
+    just run SEARCH-FS --requests 'demo/example-case-1/requests.yaml'
+    ````
+
+## Expected results ##
+
+- Should yield 4 messages in the queue.
+- Only the folders at lowest level should be logged.
+- The nested subfolder containing empty logs should be skipped.

demo/example-case-1/data/change.log

@@ -0,0 +1,3 @@
+2025-10-01 added customers.csv
+2025-10-08 added orders.csv
+2025-10-09 filled in orders.csv

demo/example-case-1/data/customers.csv

@@ -0,0 +1,5 @@
+"Id";"Company";"Surname";"Name"
+102;"ACME";"Hargrave";"Julianne"
+103;"Google";"Schorche";"Iain"
+227;;"Cruz";"David"
+489;"Amazon";;

demo/example-case-1/data/notes.md

@@ -0,0 +1,10 @@
+# Notes #
+
+Folder structure
+
+```text
+├── customers.csv
+├── orders.csv
+├── change.log
+└── notes.md
+```

demo/example-case-1/data/orders.csv

@@ -0,0 +1,6 @@
+"Id";"Customer.Id";"Article";"Qty";"Cost";"TotalCost";"Status";"Checked"
+0;382;"A19038012";10;0.50;50.00;COMPLETED;true
+1;103;"B18031017";8;4.57;36.56;COMPLETED;true
+8;;"A63278732";;10.17;;ERROR;false
+9;;"A63278732";20;10.17;203.4;CANCELLED;false
+13;489;"Z20009991";3;0.10;0.30;PENDING;true

demo/example-case-1/requests.yaml

@@ -0,0 +1,12 @@
+label: 'Demo Case 1'
+
+options:
+  skip-empty: true
+  max-depth: 100
+  max-items: 10_000_000
+  max-duration: 00:30:00
+
+data:
+  inputs:
+    location: OS
+    path: 'demo/example-case-1/data'

demo/example-case-2/README.md

@@ -0,0 +1,30 @@
+# Example - Case 2 #
+
+## Description ##
+
+- An nearly flat file system with a nested folder containing empty files.
+- `skip-empty` set to `false`
+
+## Execution ##
+
+Run as follows
+
+1. Open 2 terminals
+
+2. In terminal 1 start the queue
+
+    ```bash
+    just docker-start-queue
+    ```
+
+3. In terminal 2 run the command
+
+    ```bash
+    just run SEARCH-FS --requests 'demo/example-case-2/requests.yaml'
+    ````
+
+## Expected results ##
+
+- Should yield 6 messages in the queue.
+- The folders at lowest level should be logged.
+- The nested subfolder containing empty logs should _not_ be skipped.

demo/example-case-2/data/change.log

@@ -0,0 +1,3 @@
+2025-10-01 added customers.csv
+2025-10-08 added orders.csv
+2025-10-09 filled in orders.csv

demo/example-case-2/data/customers.csv

@@ -0,0 +1,5 @@
+"Id";"Company";"Surname";"Name"
+102;"ACME";"Hargrave";"Julianne"
+103;"Google";"Schorche";"Iain"
+227;;"Cruz";"David"
+489;"Amazon";;

demo/example-case-2/data/notes.md

@@ -0,0 +1,10 @@
+# Notes #
+
+Folder structure
+
+```text
+├── customers.csv
+├── orders.csv
+├── change.log
+└── notes.md
+```

demo/example-case-2/data/orders.csv

@@ -0,0 +1,6 @@
+"Id";"Customer.Id";"Article";"Qty";"Cost";"TotalCost";"Status";"Checked"
+0;382;"A19038012";10;0.50;50.00;COMPLETED;true
+1;103;"B18031017";8;4.57;36.56;COMPLETED;true
+8;;"A63278732";;10.17;;ERROR;false
+9;;"A63278732";20;10.17;203.4;CANCELLED;false
+13;489;"Z20009991";3;0.10;0.30;PENDING;true

demo/example-case-2/requests.yaml

@@ -0,0 +1,12 @@
+label: 'Demo Case 2'
+
+options:
+  skip-empty: false
+  max-depth: 100
+  max-items: 10_000_000
+  max-duration: 00:30:00
+
+data:
+  inputs:
+    location: OS
+    path: 'demo/example-case-2/data'

demo/example-case-3/README.md

@@ -0,0 +1,30 @@
+# Example - Case 3 #
+
+## Description ##
+
+- A nested filesystem with some empty files
+- `skip-empty` set to `true`
+
+## Execution ##
+
+Run as follows
+
+1. Open 2 terminals
+
+2. In terminal 1 start the queue
+
+    ```bash
+    just docker-start-queue
+    ```
+
+3. In terminal 2 run the command
+
+    ```bash
+    just run SEARCH-FS --requests 'demo/example-case-3/requests.yaml'
+    ```
+
+## Expected results ##
+
+- Should yield 14 messages in the queue.
+- All files at all levels should be covered.
+- The nested subfolder containing empty logs should be skipped.

demo/example-case-3/data/change.log

@@ -0,0 +1,3 @@
+2025-10-01 added customers.csv
+2025-10-08 added orders.csv
+2025-10-09 filled in orders.csv

demo/example-case-3/data/customers.csv

@@ -0,0 +1,5 @@
+"Id";"Company";"Surname";"Name"
+102;"ACME";"Hargrave";"Julianne"
+103;"Google";"Schorche";"Iain"
+227;;"Cruz";"David"
+489;"Amazon";;

demo/example-case-3/data/mock_articles/articles.csv

@@ -0,0 +1,5 @@
+"Article";"Cost"
+"A19038012";0.50
+"A63278732";10.17
+"B18031017";4.57
+"Z20009991";0.10

demo/example-case-3/data/mock_articles/mock_A19038012/description.txt

@@ -0,0 +1,3 @@
+Labore atque perferendis adipisci cumque voluptas non ducimus minus hic repellat, sint dolorem architecto alias molestiae quos delectus perferendis optio enim sapiente. Temporibus quibusdam accusamus fugiat perspiciatis, labore reprehenderit incidunt ut mollitia numquam ea libero cumque rerum, atque ab possimus.
+
+Est tenetur blanditiis, esse ipsum architecto eligendi ducimus, delectus eius veniam soluta itaque libero amet ipsum dolore, ea doloremque facilis reprehenderit delectus ipsum error corporis eveniet eos quaerat asperiores, consequuntur pariatur porro dolore at. Aspernatur amet perferendis porro odio nobis eius, blanditiis veniam debitis velit deserunt vero cum neque explicabo omnis.

demo/example-case-3/data/mock_articles/mock_A19038012/licence.txt

@@ -0,0 +1,5 @@
+Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+
+Dicta non delectus nisi reprehenderit, odit a incidunt eveniet ratione aspernatur distinctio beatae? Accusamus itaque deserunt perferendis ratione nihil quia odit excepturi numquam, provident voluptatum a perspiciatis accusantium mollitia est culpa possimus cumque tenetur quidem.
+
+Nihil harum ipsum, blanditiis iure impedit molestiae cumque ratione nihil doloribus eius facilis eum deleniti, ab corporis deserunt, fugit fuga officiis et blanditiis? Provident quos sed exercitationem iure necessitatibus rem, deserunt obcaecati vitae, eligendi quo id reiciendis iste asperiores autem, molestias cum ex beatae. Pariatur modi iste reiciendis sequi ipsam ipsa ducimus possimus iure, eius similique dolores amet officiis ipsum consectetur odio, totam placeat consequatur ipsam quae accusantium libero? Qui illo hic perferendis, adipisci ut ullam distinctio voluptatem a eaque error debitis praesentium atque culpa, hic iusto pariatur non blanditiis, aliquid commodi libero inventore vel laborum, aut quos reiciendis labore non nisi a harum rem facere?

demo/example-case-3/data/mock_articles/mock_A63278732/description.txt

@@ -0,0 +1,3 @@
+Labore atque perferendis adipisci cumque voluptas non ducimus minus hic repellat, sint dolorem architecto alias molestiae quos delectus perferendis optio enim sapiente. Temporibus quibusdam accusamus fugiat perspiciatis, labore reprehenderit incidunt ut mollitia numquam ea libero cumque rerum, atque ab possimus.
+
+Est tenetur blanditiis, esse ipsum architecto eligendi ducimus, delectus eius veniam soluta itaque libero amet ipsum dolore, ea doloremque facilis reprehenderit delectus ipsum error corporis eveniet eos quaerat asperiores, consequuntur pariatur porro dolore at. Aspernatur amet perferendis porro odio nobis eius, blanditiis veniam debitis velit deserunt vero cum neque explicabo omnis.

demo/example-case-3/data/mock_articles/mock_A63278732/licence.txt

@@ -0,0 +1,5 @@
+Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+
+Dicta non delectus nisi reprehenderit, odit a incidunt eveniet ratione aspernatur distinctio beatae? Accusamus itaque deserunt perferendis ratione nihil quia odit excepturi numquam, provident voluptatum a perspiciatis accusantium mollitia est culpa possimus cumque tenetur quidem.
+
+Nihil harum ipsum, blanditiis iure impedit molestiae cumque ratione nihil doloribus eius facilis eum deleniti, ab corporis deserunt, fugit fuga officiis et blanditiis? Provident quos sed exercitationem iure necessitatibus rem, deserunt obcaecati vitae, eligendi quo id reiciendis iste asperiores autem, molestias cum ex beatae. Pariatur modi iste reiciendis sequi ipsam ipsa ducimus possimus iure, eius similique dolores amet officiis ipsum consectetur odio, totam placeat consequatur ipsam quae accusantium libero? Qui illo hic perferendis, adipisci ut ullam distinctio voluptatem a eaque error debitis praesentium atque culpa, hic iusto pariatur non blanditiis, aliquid commodi libero inventore vel laborum, aut quos reiciendis labore non nisi a harum rem facere?

demo/example-case-3/data/mock_articles/mock_B18031017/description.txt

@@ -0,0 +1,3 @@
+Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+
+Fuga illo veritatis repellendus, iusto assumenda distinctio illo repellat facere mollitia saepe veritatis dignissimos nemo, adipisci nobis iure sequi ipsum laborum delectus deserunt cupiditate doloremque beatae esse, consequatur neque corporis totam assumenda labore eum voluptatibus facilis voluptates eius. Quos alias obcaecati, a atque excepturi. Atque dolore accusantium distinctio accusamus nisi voluptates voluptatem unde, animi illum dicta nam? Necessitatibus quidem cum perspiciatis quisquam esse aut reprehenderit quod facere dolores, debitis perferendis quibusdam repellendus necessitatibus eaque cupiditate sequi excepturi, eligendi sint illum.

demo/example-case-3/data/mock_articles/mock_B18031017/licence.txt

@@ -0,0 +1,5 @@
+Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+
+Maxime dolor suscipit explicabo, cum molestias quibusdam esse quam nulla enim iure, doloremque cum numquam debitis molestias suscipit dolorem, reiciendis at id quia culpa repudiandae debitis deserunt quisquam iste nobis necessitatibus, accusantium eveniet ea? Optio itaque voluptatum similique, doloribus pariatur magni, nostrum facere harum, tenetur eos corrupti? Veritatis tenetur doloremque veniam, iste repellat libero nemo, quia aliquam ipsa quo ex, quo iusto maiores quos, quae culpa facere sint nobis nihil blanditiis necessitatibus? Quidem ratione officiis nisi minima maiores odit autem, pariatur consequatur architecto assumenda aperiam qui, ad iure natus aspernatur ratione quisquam eos esse vel.
+
+Ab fugiat sequi praesentium minus voluptates excepturi similique quam minima soluta, deserunt voluptatem quibusdam necessitatibus reiciendis ratione saepe, corrupti delectus soluta, eum ad deserunt sed tempora magni ex? Vel voluptatibus omnis libero consequatur impedit maiores recusandae corporis nemo, possimus velit voluptatem commodi sapiente voluptatibus facilis quo assumenda, consectetur aperiam quis architecto, quidem consectetur modi aliquid optio excepturi nulla voluptate quo quisquam accusantium? Quo eaque laborum quisquam incidunt quasi fugit sit eos, explicabo eveniet quia magnam obcaecati ipsa voluptates iusto. Qui placeat ut saepe, illum debitis a eum repellendus expedita consequatur?

demo/example-case-3/data/mock_articles/mock_Z20009991/description.txt

@@ -0,0 +1,3 @@
+Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+
+Voluptate vel animi necessitatibus cumque, tenetur dolorem nobis. Iusto non ratione, adipisci asperiores impedit ullam autem quos velit nulla voluptatibus, velit cupiditate sed perspiciatis quisquam illo architecto saepe dignissimos at pariatur doloribus, odit debitis vitae quod illo voluptate dolorum incidunt cumque dolor, nihil rem suscipit? Doloribus perspiciatis fugit, pariatur optio at delectus quis aliquid tempore odio quas error corrupti, sed amet laudantium dolorem numquam maiores. Amet odit id, quasi aspernatur dolorem illo fugit aliquam laudantium perferendis quibusdam numquam recusandae adipisci, dolores hic accusamus necessitatibus delectus dolor dicta?

demo/example-case-3/data/mock_articles/mock_Z20009991/licence.txt

@@ -0,0 +1,5 @@
+Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
+
+Cum excepturi dignissimos mollitia eveniet voluptas nostrum ea harum voluptatum minus, earum consectetur distinctio recusandae iusto autem placeat quod obcaecati quos ullam dolorum? Neque dolore minus modi adipisci cupiditate facilis eligendi quidem iste, qui mollitia magnam voluptate fugiat iure, id doloremque ab?
+
+Maxime autem reprehenderit ipsum, quibusdam iste nulla odio nam et quis, iusto asperiores sit? Reiciendis magnam libero saepe rerum nostrum maxime quae laboriosam, quo nulla fugit quam ex quisquam? Quod deserunt veritatis, provident impedit adipisci in porro dolorem ducimus est ipsa quas sunt, voluptatum rerum nihil nostrum eius in, necessitatibus ipsa maiores ut modi iusto rem alias quos odio est velit. Exercitationem enim aliquam aspernatur, dolore odit assumenda doloremque ut rerum atque dolorum?

demo/example-case-3/data/mock_companies/companies.csv

@@ -0,0 +1,6 @@
+"Company"
+"Amazon"
+"ACME"
+"GCHQ"
+"Google"
+"Nandos"

demo/example-case-3/data/notes.md

@@ -0,0 +1,10 @@
+# Notes #
+
+Folder structure
+
+```text
+├── customers.csv
+├── orders.csv
+├── change.log
+└── notes.md
+```

demo/example-case-3/data/orders.csv

@@ -0,0 +1,6 @@
+"Id";"Customer.Id";"Article";"Qty";"Cost";"TotalCost";"Status";"Checked"
+0;382;"A19038012";10;0.50;50.00;COMPLETED;true
+1;103;"B18031017";8;4.57;36.56;COMPLETED;true
+8;;"A63278732";;10.17;;ERROR;false
+9;;"A63278732";20;10.17;203.4;CANCELLED;false
+13;489;"Z20009991";3;0.10;0.30;PENDING;true

demo/example-case-3/requests.yaml

@@ -0,0 +1,12 @@
+label: 'Demo Case 3'
+
+options:
+  skip-empty: true
+  max-depth: 100
+  max-items: 10_000_000
+  max-duration: 00:30:00
+
+data:
+  inputs:
+    location: OS
+    path: 'demo/example-case-3/data'

demo/example-case-4/README.md

@@ -0,0 +1,30 @@
+# Example - Case 4 #
+
+## Description ##
+
+- A nested filesystem with some empty files
+- `skip-empty` set to `false`
+
+## Execution ##
+
+Run as follows
+
+1. Open 2 terminals
+
+2. In terminal 1 start the queue
+
+    ```bash
+    just docker-start-queue
+    ```
+
+3. In terminal 2 run the command
+
+    ```bash
+    just run SEARCH-FS --requests 'demo/example-case-4/requests.yaml'
+    ````
+
+## Expected results ##
+
+- Should yield 26 messages in the queue.
+- All files at all levels should be covered.
+- The nested subfolder containing empty logs should _not_ be skipped.

demo/example-case-4/data/change.log

@@ -0,0 +1,3 @@
+2025-10-01 added customers.csv
+2025-10-08 added orders.csv
+2025-10-09 filled in orders.csv

demo/example-case-4/data/customers.csv

@@ -0,0 +1,5 @@
+"Id";"Company";"Surname";"Name"
+102;"ACME";"Hargrave";"Julianne"
+103;"Google";"Schorche";"Iain"
+227;;"Cruz";"David"
+489;"Amazon";;

demo/example-case-4/data/mock_articles/articles.csv

@@ -0,0 +1,5 @@
+"Article";"Cost"
+"A19038012";0.50
+"A63278732";10.17
+"B18031017";4.57
+"Z20009991";0.10