Fix reproducibility, statistical validity, and code hygiene

.github/workflows/ci.yml

@@ -25,12 +25,14 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install -r requirements.txt
-          pip install -r requirements-dev.txt
+          pip install -e ".[dev]"
 
       - name: Validate committed data and notebooks
         run: pytest tests/ -q
 
+      - name: Verify group summaries regenerate + reconcile
+        run: python pipeline/build_group_summaries.py
+
   security:
     name: Dependency audit (pip-audit)
     runs-on: ubuntu-latest
@@ -68,8 +70,9 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install -r requirements.txt
-          pip install -r requirements-dev.txt
+          pip install -e ".[dev]"
 
       - name: Execute notebooks
-        run: pytest --nbmake --nbmake-timeout=900 case-study group individual
+        # --nbmake-kernel=python3 overrides any embedded kernel name so a clean
+        # checkout runs regardless of the author's local kernel.
+        run: pytest --nbmake --nbmake-kernel=python3 --nbmake-timeout=900 case-study group individual

.gitignore

@@ -31,3 +31,7 @@ htmlcov/
 *.HEIC
 
 .claude/
+
+# Generated by pipeline/build_group_summaries.py
+data/group_results/reconstructed/
+mms.egg-info/

.python-version

@@ -1 +1 @@
-3.9
+3.11

CITATION.cff

@@ -15,4 +15,4 @@ keywords:
   - psychometric-testing
   - multimodal-analysis
   - computational-neuropsychology
-license: MIT
+license: CC-BY-4.0

DATA_ETHICS.md

@@ -30,10 +30,12 @@ penalty.
   details, dates of birth, or device identifiers).
 - Participants are referred to only by non-reversible pseudonymous codes
   (e.g. `01`, `02`).
-- Recording timestamps are retained for time-series analysis. They carry no
-  location or identity information and, combined only with pseudonymous codes,
-  present low re-identification risk; they can be shifted to relative session
-  time on request.
+- Recording timestamps are retained for time-series analysis. To minimise the
+  residual re-identification risk of absolute appointment dates/times, run
+  [`scripts/deidentify_timestamps.py`](scripts/deidentify_timestamps.py)
+  (`--apply`) to shift each session to a relative origin (`2000-01-01`),
+  preserving within-session alignment while removing the wall-clock date/time.
+  The tool is dry-run by default.
 - If you believe any released field could re-identify a participant, please open
   an issue and it will be removed.
 

DATA_LICENSE.md

@@ -0,0 +1,28 @@
+# Data license
+
+The repository's [`LICENSE`](LICENSE) (MIT) governs the **code** (notebooks, the
+`mms/` package, scripts). MIT is a software license — it speaks of "the
+Software" and grants rights to sell/sublicense — so it does not cleanly govern a
+dataset. The **data** under [`data/`](data/) and the derived figures are
+therefore released separately under:
+
+## Creative Commons Attribution 4.0 International (CC-BY-4.0)
+
+SPDX: `CC-BY-4.0` · Full text: https://creativecommons.org/licenses/by/4.0/legalcode
+
+You are free to share and adapt the data for any purpose, including
+commercially, provided you give appropriate credit (cite via
+[`CITATION.cff`](CITATION.cff)).
+
+### Additional condition — no re-identification
+
+Because this is pseudonymised **special-category health data** from human
+participants (see [`DATA_ETHICS.md`](DATA_ETHICS.md)), one binding condition is
+added on top of CC-BY-4.0:
+
+> You must **not** attempt to re-identify any participant, nor use the data to
+> make decisions about any individual. Report suspected re-identification risks
+> by opening an issue.
+
+This condition reflects the participants' consent terms and does not otherwise
+restrict the CC-BY-4.0 grant.

DATA_PROVENANCE.md

@@ -0,0 +1,79 @@
+# Data Provenance & Reproducibility
+
+This document states, honestly, **what in this repository can be regenerated
+from committed data and what cannot** — the question a reviewer or reuser asks
+first. It also documents the raw→processed→summary pipeline and known data
+issues. It complements [DATA_ETHICS.md](DATA_ETHICS.md) (consent, GDPR,
+de-identification).
+
+## TL;DR reproducibility status
+
+| Layer | Regenerable from this repo? | How |
+|-------|-----------------------------|-----|
+| Headline reliability numbers (ICC) | ✅ Yes | `mms.stats.icc1` on `data/group_results/*.csv` — reproduces the README's 0.22 / 0.45 / 0.61 **and** adds the 95% CIs |
+| Case-study participant's summary row (= **P02**) | ✅ Yes, exactly | `python pipeline/build_group_summaries.py` |
+| Group summary rows for the other 9 participants | ❌ No | Their raw streams were not released (privacy) — see below |
+| Case-study per-session figures (HR, HRV, fixation, clustering) | ✅ Yes | Run the `case-study/` notebooks (data present) |
+
+## The layers
+
+```
+data/case-study/raw/*.txt          # sensor exports (semicolon-delimited), one participant
+        │  (parsing, column rename, derived fixation/duration columns)
+        ▼
+data/case-study/processed/*.csv    # tidy per-session streams: hr_0N, ibi_0N, sed_fix_0N
+        │  (per-session summary statistics — see mms/)
+        ▼
+data/group_results/*.csv           # one row per participant, one column per session
+```
+
+- **raw → processed**: the raw `.txt` files are semicolon-delimited and use
+  dotted names (`gazeDir.x`); the processed `.csv` files are comma-delimited,
+  rename gaze columns, and add derived `fixation`, `fixation_id`, `duration`,
+  and `gaze_diff` columns. Only the case-study participant's raw files are
+  present, so only that participant's processed layer is regenerable end-to-end.
+- **processed → summary**: `pipeline/build_group_summaries.py` computes SDNN,
+  pupil-diameter STD and response-duration STD per session. It reproduces the
+  committed **P02** row exactly (verified to < 1e-6; see
+  `data/group_results/reconstructed/MANIFEST.json`).
+
+## Why only one participant is regenerable
+
+The case-study streams correspond to participant **P02** in the group tables.
+This was verified, not assumed: the case-study IBI, pupil and duration data
+reproduce the committed P02 row for all three metrics **exactly**. The raw
+streams for P01 and P03–P10 were **not released** — a deliberate,
+privacy-legitimate minimisation choice (only pseudonymised summaries are shared;
+see DATA_ETHICS.md). Consequently their summary rows are provided *as released
+values* and cannot be recomputed from this repository. `build_group_summaries.py`
+does **not** invent them.
+
+## Two recipes: original vs improved
+
+`build_group_summaries.py` reports both, so the choice is explicit:
+
+- **Original recipe** (reproduces the committed values): `ibi.std()` and
+  `pupil.std()` with **no artifact filtering**. This is faithful to how the
+  committed summaries were made, but it lets dropped beats and blink artifacts
+  inflate variance — e.g. P02's Session-2 pupil STD of **1.12** is an artifact
+  spike, ~4× the other sessions.
+- **Improved recipe** (recommended going forward): `mms.hrv.sdnn` applies a
+  300–2000 ms normal-to-normal filter; `mms.fixation.pupil_std` gates on the
+  `pupilQ` quality flag. These are more defensible and remove the S2 spike.
+
+## Known data-integrity issues
+
+- **P01 HRV SDNN, Session 02 == Session 03 (both 65.39).** These two values are
+  byte-identical, a suspected copy-paste artifact, and the archived technical
+  report lists a different P01 Session-3 value. Because P01's raw data is not in
+  the repo, **the correct value cannot be recovered here**, so it has been
+  *flagged and left unmodified* rather than guessed. The data owner should
+  restore it from the original source. `tests/test_data_integrity.py` now fails
+  loudly if an undocumented duplicate like this appears in a group summary.
+
+## `_modified` psychometric files
+
+`data/case-study/psychometric/*_modified.csv` drop the `Type` column, add a
+parsed `datetime` column, and reformat timestamps relative to the originals.
+They are analysis conveniences derived from the canonical
+`Psychometric_Test_Results_0N.csv` files; the originals are authoritative.

README.md

@@ -49,7 +49,15 @@ Longitudinal study where 10 adults completed standardized psychology tests acros
 
 ## Key findings
 
-How stable are these patterns within a person across sessions? A test–retest reliability check (ICC(1), n = 10, 3 sessions) on the committed summaries gives ICC = 0.22 for HRV SDNN, 0.45 for pupil-dilation variability, and 0.61 for response-duration variability — poor for the physiological measures, moderate at best. So the data do **not** support a strong "stable individual traits" reading: with only 10 participants, these measures look closer to session-to-session fluctuation than to reliable traits. Any stability claim should be read as tentative and underpowered.
+How stable are these patterns within a person across sessions? A test–retest reliability check (ICC(1), n = 10, 3 sessions) on the committed summaries gives:
+
+| Measure | ICC(1) | 95% CI |
+|---------|:------:|:------:|
+| HRV SDNN | 0.22 | **[−0.13, 0.66]** |
+| Pupil-dilation variability | 0.45 | [0.07, 0.79] |
+| Response-duration variability | 0.61 | [0.25, 0.87] |
+
+The confidence intervals are the real story: for HRV the interval **crosses zero**, meaning the data are equally consistent with negative and moderate reliability — i.e. essentially uninformative at n = 10. So the data do **not** support a strong "stable individual traits" reading; these measures look closer to session-to-session fluctuation than to reliable traits. Any stability claim should be read as tentative and underpowered. These numbers are fully reproducible — `mms.stats.icc1` recomputes them (point estimate and CI) from `data/group_results/`.
 
 ![Correlation matrix of HRV SDNN and Pupil Dilation STD across sessions](images/correlation_heatmap_with_values_final.png)
 
@@ -63,6 +71,42 @@ How stable are these patterns within a person across sessions? A test–retest r
 |:---:|:---:|
 | ![PCA K-Means clusters](images/pca_kmeans_clusters.png) | ![Silhouette score vs number of clusters](images/silhouette_score.png) |
 
+## Reproduce this analysis
+
+```bash
+git clone https://github.com/urme-b/Multimodal-Multisensor
+cd Multimodal-Multisensor
+
+python -m venv .venv && source .venv/bin/activate   # Python >= 3.11
+pip install -e ".[dev]"                             # installs deps + the shared `mms` package
+
+# Regenerate the group-level summaries from committed data + reconciliation report
+python pipeline/build_group_summaries.py
+
+# Reproduce the headline reliability numbers (ICC + 95% CIs)
+python -c "import mms, pandas as pd; \
+m=mms.io.load_group_summary('HRV_SDNN')[['Session 01','Session 02','Session 03']].to_numpy(float); \
+print(mms.stats.icc1(m))"
+
+# Explore the notebooks
+jupyter lab
+```
+
+**Suggested notebook order** (each folder is independent):
+`case-study/preprocess_raw_to_csv → build_hrv → 1_hr / 1_hrv / 2_fixation → 3_clustering`,
+then `group/group_analysis → group_correlation_matrix → group_correlation_heatmap`.
+
+Shared loaders and metrics live in the [`mms/`](mms/) package (`mms.io`,
+`mms.hrv`, `mms.stats`, `mms.fixation`) so notebooks import tested code instead
+of re-implementing it. Reproducibility scope and known data issues are documented
+in **[DATA_PROVENANCE.md](DATA_PROVENANCE.md)**.
+
+## Reports & publications
+
+- [Multimodal Multisensor Technical Report.pdf](Multimodal%20Multisensor%20Technical%20Report.pdf) — methods and results write-up
+- [Thesis Report.pdf](Thesis%20Report.pdf) — full thesis
+- [CYPSY_Poster.pdf](CYPSY_Poster.pdf) — CyberPsychology conference poster
+
 ## Tech Stack
 
 Python · Jupyter · pandas · NumPy · SciPy · Matplotlib · Seaborn · scikit-learn
@@ -79,4 +123,5 @@ IoT · Machine Learning · Multimodal · Neurophysiological · Multi-Sensors ·
 
 ## License
 
-[MIT](LICENSE)
+- **Code** (notebooks, `mms/`, scripts) — [MIT](LICENSE)
+- **Data** (`data/`) and figures — [CC-BY-4.0 with a no-re-identification term](DATA_LICENSE.md) (special-category health data)