EMA EPAR Document Index

This folder contains a local retrieval of the European Medicines Agency (EMA) JSON feed for documents related to centrally authorised medicines.

Source

• EMA JSON data overview: https://www.ema.europa.eu/en/about-us/about-website/download-website-data-json-data-format
• EPAR documents feed: https://www.ema.europa.eu/en/documents/report/documents-output-epar_documents_json-report_en.json

EMA states that these JSON files are updated twice a day, at 06:00 and 18:00 Amsterdam time.

Files

• data/raw/ema_epar_documents.json: raw EMA JSON feed.
• data/processed/epars_by_year.json: all records grouped by first_published_date year.
• data/processed/years/YYYY.json: one JSON file per publication year.
• data/processed/epars_flat.csv: flat index with one row per document.
• data/processed/epar_counts_by_year.csv: annual record counts.
• data/processed/epar_counts_by_type.csv: document-type counts.
• data/processed/summary.json: source metadata and aggregate counts.
• reports/epar_statistics_summary.html: reusable source for the statistics report.
• reports/epar_statistics_summary.pdf: rendered statistics report.
• reports/epar_category_analysis.html: reusable source for the category analysis report.
• reports/epar_category_analysis.pdf: rendered category analysis report.

Refresh

Run:

python3 scripts/retrieve_epars.py

To rebuild outputs from the existing raw JSON without downloading again:

python3 scripts/retrieve_epars.py --no-download

The script retrieves document metadata and URLs. It does not bulk-download the PDF/DOCX documents themselves.

Monthly automated reports

Use the monthly runner when you want one command that fetches the latest EMA JSON, rebuilds the processed data, regenerates both standalone PDF/HTML reports, and creates shareable month-prefixed copies:

python3 scripts/generate_monthly_report.py

The runner writes:

• latest report copies:
  • reports/epar_category_analysis.pdf
  • reports/epar_category_analysis.html
  • reports/epar_statistics_summary.pdf
  • reports/epar_statistics_summary.html
• month-prefixed archive copies:
  • reports/YYYY/MM-ema_epar_documents.json
  • reports/YYYY/MM-epar_category_analysis.pdf
  • reports/YYYY/MM-epar_category_analysis.html
  • reports/YYYY/MM-epar_statistics_summary.pdf
  • reports/YYYY/MM-epar_statistics_summary.html
  • reports/YYYY/MM-epar_processed_summary.json
  • reports/YYYY/MM-manifest.json

By default, the timestamp in archived filenames is the EMA source-feed timestamp, and the year/month archive folder is based on that timestamp. Use --timestamp-mode generated to use local generation time instead. If the EMA source timestamp is unchanged, rerunning the command overwrites the same monthly archive files.

Useful commands:

python3 scripts/generate_monthly_report.py --help
python3 scripts/generate_monthly_report.py --no-download
python3 scripts/generate_monthly_report.py --require-download
python3 scripts/generate_monthly_report.py --archive-root reports

The runner attempts to download the latest EMA feed by default. If EMA returns a transient error such as rate limiting and an existing raw snapshot is available, the runner falls back to that snapshot and records the download status in the manifest. Use --require-download when a reused snapshot should be treated as a hard failure.

Example cron entry for the first day of each month at 07:15 local time:

15 7 1 * * cd /Users/alboul/tmp/epar && /usr/bin/env python3 scripts/generate_monthly_report.py >> logs/monthly_report.log 2>&1

GitHub Actions automation

This repository includes .github/workflows/monthly-report.yml. The workflow:

• runs on the first day of each month at 07:15 UTC and can also be started manually with Actions > Monthly EMA EPAR reports > Run workflow;
• installs Chrome and Poppler;
• runs scripts/generate_monthly_report.py --require-download, so scheduled reports fail rather than silently using stale data;
• validates both generated PDFs;
• commits the monthly reports/YYYY/MM-* outputs back to the repository;
• uploads the same report files as workflow artifacts.

No LLM or private API key is required. The narrative text is generated from deterministic Python templates and rule-based category profiles committed in scripts/build_category_analysis.py; the statistics are calculated directly from EMA's public JSON feed.

For GitHub Actions to push the generated monthly files, keep repository Settings > Actions > General > Workflow permissions set to Read and write permissions. The workflow also declares contents: write.

GitLab CI automation

This repository includes .gitlab-ci.yml with a monthly_epar_reports job. The job:

• installs Chromium and Poppler in a python:3.12-slim container;
• runs scripts/generate_monthly_report.py --require-download, so scheduled CI reports fail rather than silently using stale data;
• validates that the generated PDF exists and contains the expected report and traceability text;
• stores the latest reports and reports/YYYY/MM-* monthly reports as GitLab artifacts.

To run it monthly in GitLab:

1. Push this repository to GitLab.
2. Open Build > Pipeline schedules.
3. Create a new schedule on the default branch.
4. Use a cron expression such as 15 7 1 * * for the first day of each month at 07:15 in the schedule timezone.
5. Leave REPORT_TIMESTAMP_MODE=source unless you want filenames based on CI generation time; then set REPORT_TIMESTAMP_MODE=generated.
6. Save the schedule. The generated PDF, HTML and manifest will be available in the scheduled pipeline artifacts.

You can also run the CI job manually from Build > Pipelines > Run pipeline.

Statistics report

Run:

python3 scripts/build_stats_report.py

This rebuilds the HTML source and PDF report from the processed count files.

Category analysis and focused repositories

Run:

python3 scripts/build_category_analysis.py

This creates a standalone category-analysis HTML/PDF report with embedded figures, statistics, recommendations, source URLs and traceability metadata. It also creates local generated repositories for assessment-report and variation-report records under data/processed/focused/. These focused CSV/JSON files are generated data and are intentionally excluded from git.