TRAC -- Time-Use Research Activity Collector

TRAC is a web-based research software for time-use research: users can report what they did during one or more days by selecting activities and placing them on one or more timelines per day. E.g., depending on the study, there may be one timeline for 'Primary Activity', and another one for 'Secondary Activity', allowing users to report things like listening to music while riding on the subway.

The frontend is based on github.com/andreifoldes/o-timeusediary by Andrei Tamas Foldes et al. but heavily adapted, and the backend was written from scratch.

When using the software in this repo, please also cite Andrei Tamas Foldes' paper Time use diary design for our times - an overview, presenting a Click-and-Drag Diary Instrument (CaDDI) for online application.

Installation Instructions

TRAC consists of three components that must be set up together:

1. PostgreSQL database — stores participant data and study definitions
2. Python/FastAPI backend — serves the REST API and the admin interface
3. Frontend — a static JavaScript/HTML/CSS app served by any web server

Prerequisites

• A PostgreSQL server (any recent version)
• Python 3.10+ and uv for the backend
• A web server (e.g., nginx, Apache, Caddy) to serve the frontend static files and optionally act as a reverse proxy in front of the backend
• For production: a domain name and TLS/HTTPS (strongly recommended — see Security below)

1. Database Setup

Create a dedicated PostgreSQL database and user for the application. The database schema is created automatically by the backend on first startup. You need to provide the connection details in the backend configuration (see next step).

2. Backend Configuration

The backend is configured via a .env file placed in the directory from which the backend process is started. Copy backend/.env.example to backend/.env and adjust the values:

# Database connection
TUD_DATABASE_USER=<db_user>
TUD_DATABASE_PASSWORD=<db_password>     # use a strong password
TUD_DATABASE_HOST=<db_host>             # usually 'localhost'
TUD_DATABASE_PORT=5432
TUD_DATABASE_NAME=<db_name>
TUD_DATABASE_URL=postgresql://<db_user>:<db_password>@<db_host>:5432/<db_name>

# CORS: list all origins (scheme + host + port) from which the frontend will be served
TUD_ALLOWED_ORIGINS='["https://your.domain.example.com"]'

# Admin interface credentials — use strong, unique values
TUD_API_ADMIN_USERNAME=<admin_username>
TUD_API_ADMIN_PASSWORD=<admin_password>

# Root path: set this if the backend API is served under a sub-path via a reverse proxy
# e.g., TUD_ROOTPATH=/tud_backend  when proxied at https://your.domain.example.com/tud_backend/
TUD_ROOTPATH=/

Install the backend into a virtual environment and start it with a WSGI server such as uvicorn (development) or gunicorn (production).

Backend startup workflow (recommended for development, CI, and production):

1. Run schema migrations explicitly.
2. Import studies explicitly when needed.
3. Start backend (startup does not perform schema/data bootstrap).

Minimal examples:

cd backend/

# 1) Prepare runtime state explicitly
uv run tud db upgrade
uv run tud studies import --config studies_config.json
# or import multiple files (for example one file per study)
uv run tud studies import --config studies_config_a.json --config studies_config_b.json

# 2) Start backend (development)
uv run gunicorn --reload -c ../deployment/gunicorn_conf.dev.py o_timeusediary_backend.api:app

# 3) Start backend (production-style)
uv run gunicorn -c ../deployment/gunicorn_conf.py o_timeusediary_backend.api:app

Schema management is migration-first: run uv run tud db upgrade explicitly during deployment/startup scripts.

You can also inspect the current DB migration revision via:

cd backend/
uv run tud db current

Important: Large Request Handling for non-root users in local development

TRAC supports exporting and importing study configurations with embedded activity definitions, which can result in large HTTP POST request bodies (typically 10-50 KB depending on the number of activities and languages supported).

For nginx, the size can become so large that it writes requests to a temp dir instead of relying on in-memory handling. This is no problem normally, as a dir for that should be configured by default, but for the dev scripts, where you run nginx without root priviledges as your own user, that directory is not usable. You will therefor need to:

• nginx: Configure a client_body_temp_path directive in your nginx configuration to a directory where the nginx process has write permissions. See the developer documentation for details.
• Apache: Ensure the LimitRequestBody directive is set high enough (default 10 MB should be sufficient).
• Other web servers: Verify that large POST body handling is configured appropriately for your setup.

If you encounter HTTP 413 (Payload Too Large) or 500 errors when importing study configurations, the root cause is typically insufficient request body handling configuration in your web server.

As mentioned before, this should NOT be need for production.

3. Study Configuration

Studies are defined in one or more studies_config files (for example backend/studies_config.json). Each entry specifies the study name, supported languages, the days to cover, participant handling (open or invite-only via allow_unlisted_participants), and references to one or more activity list files (backend/activities_*.json). You can import one file or multiple files via admin endpoints or the CLI command shown above.

Each study uses name_short as its technical identifier. This short name is important because it is used by the frontend configuration and in participant invitation links via the study_name URL parameter.

TRAC also supports study-level internationalization. In studies_config.json, each study defines a default_language and a list of supported_languages. Study texts such as introductions, end messages, and day labels can be provided per language. Activity lists can also be language-specific via activities_json_files, which maps language codes to separate activities_*.json files. On the frontend, the language is chosen in this order: lang URL parameter if present, otherwise the browser language if supported, otherwise the study's default language.

Participants access the app via an invitation link containing their unique ID. For open studies any visitor is assigned an ID automatically.

Participant Invitation Links

TRAC identifies participants primarily through the pid URL parameter together with the target study in study_name. A minimal invitation link therefore looks like this:

https://your.domain.example.com/report/index.html?study_name=default&pid=PARTICIPANT_ID

All invitation URL parameters are listed below. Parameters are read from the query string of index.html and preserved across page navigations (instructions, consent, tasks, thank-you) automatically.

Parameter	Required	Description
study_name	Yes	The study short name (name_short in studies_config.json). Identifies which study configuration to load.
pid	Yes¹	The participant identifier. For invite-only studies this must match a pre-assigned participant; for open studies any value is accepted and a new participant record is created if needed.
lang	No	Language override as an ISO 639-1 two-letter code (e.g., en, de, sv). When omitted, the browser language is used if supported by the study; otherwise the study's default_language is used.
template_user	No	Participant ID of another user whose timeline entries should be copied as a starting point when the participant first opens the diary. Useful when a parent enters similar data for siblings, for example.
return_url	No	A fully URL-encoded absolute URL to which the participant is redirected after completing the study (shown as a link on the thank-you page). Must be properly encoded (e.g., https%3A%2F%2Fexample.org%2Ffinish).

¹ pid is required for invite-only studies. For open studies (allow_unlisted_participants: true) a missing pid is replaced with a randomly generated, fresh ID automatically.

Examples:

# Select study and participant
https://your.domain.example.com/report/index.html?study_name=default&pid=c303282d

# Also force the language shown in the frontend
https://your.domain.example.com/report/index.html?study_name=default&pid=c303282d&lang=sv

# Use another participant as a template for first-time initialization
https://your.domain.example.com/report/index.html?study_name=study1&pid=c303282d&template_user=a5sf35gh

# Return to an external system after completion
https://your.domain.example.com/report/index.html?study_name=default&pid=c303282d&return_url=https%3A%2F%2Fexample.org%2Ffinish%3Ftoken%3Dabc123

External Tasks (external integrations)

Studies may include external_tasks entries in backend/studies_config.json to describe external systems participants should visit (for example, external surveys or payment forms). The current configuration format used in this repository is:

• task_key: short identifier used by the app
• task_level: positive integer used to define task hierarchy: If a task has task_level N it means it can only be started once all taks with level less than N have been completed by the user. Set all to 1 if you need to hierarchy.
• name, description: localized objects (e.g. { "en": "..." })
• confirmation_type: must be "callback" (callback means the study expects a confirmation on return)
• outbound_tokens: array of token definitions; each has a name and a by_participant map of participant id → token string
• outbound_url: URL template containing placeholders which will be substituted per-participant. Common placeholders: {participant_id}, {study_name}, {task_key}, and token placeholders matching the outbound_tokens name (for example {pay_token})
• hmac_secret_reference: (optional) name of a shared secret in TUD_EXTERNAL_TASK_HMAC_SECRETS for HMAC-signed callbacks

Example (excerpt from backend/studies_config.json used in the pilot studies):

{
    "task_key": "payment_info",
    "task_level": 2,
    "name": { "de": "Bankdaten eingeben" },
    "description": { "de": "Geben Sie Ihre Bankdaten ein..." },
    "confirmation_type": "callback",
    "outbound_tokens": [
        {
            "name": "pay_token",
            "by_participant": {
                "bernd": "pay-bernd-123434214",
                "sophia": "pay-sophia-987654321"
            }
        }
    ],
    "outbound_url": "https://survey.example.org/f/153222?pid={participant_id}&study_name={study_name}&task={task_key}&token={pay_token}"
}

How it works at runtime

• When rendering task links the backend substitutes the placeholders and returns external_tasks entries (with continuation_url / outbound_url already expanded per participant) in the study-config API response.

• The frontend pages/tasks.html is now responsible for handling return/confirmation flows from external providers. When an external provider redirects participants back, it should send the following query parameters to the tasks page:

  • callback_task_key: the task_key of the task being confirmed
  • callback_token: the token that was assigned to this participant for that task

  Example return URL a provider should redirect to after completion:

  https://your.domain.example.com/report/pages/tasks.html?study_name=default&pid=bernd&callback_task_key=payment_info&callback_token=pay-bernd-123434214
  

• On page load pages/tasks.html reads callback_task_key and callback_token and POSTs a confirmation JSON payload to the backend endpoint:

  POST /api/studies/{study_name}/participants/{participant_id}/external-tasks/confirm

  with body { "task_key": "<task_key>", "assigned_token": "<token>" }.

• pages/tasks.html also supports an optional return_url parameter. If present it is persisted in localStorage and appended to outbound continuation links so the external provider (or the user when they come back) can be forwarded to an external finish URL after the internal task flow completes.

Notes

• The confirmation responsibility moved from thank-you.html to pages/tasks.html — receivers and integrators should redirect back to pages/tasks.html (see example above).
• Token names in outbound_tokens must match the placeholders used in outbound_url so the backend can substitute the correct per-participant token.

HMAC-Signed Callbacks (Optional Per-Task)

To prevent participants from forging callback confirmations, individual external tasks can opt into HMAC-signed return URLs. This requires a shared secret between TRAC and the remote system's backend.

Configuration

1. Add the shared secret to .env:

   TUD_EXTERNAL_TASK_HMAC_SECRETS='{"survey_hub_v1":"a1b2c3d4e5f6..."}'
   

   Generate a secret with python3 -c "import secrets; print(secrets.token_hex(32))".

2. Reference it in the task definition in studies_config.json:

   {
     "task_key": "depression_survey",
     ...
     "hmac_secret_reference": "survey_hub_v1"
   }

Remote system contract

After the participant completes the task, the remote system must compute:

message    = "study_name|participant_id|task_key|assigned_token"
signature  = HMAC-SHA256(shared_secret, message) → hex string

And append &hmac={signature} to the redirect URL. TRAC verifies the signature before accepting the callback.

Security properties

• Without HMAC a participant who knows their own token can self-confirm.
• With HMAC the return URL must be signed by someone who knows the secret — i.e. the remote system's backend, not the browser.
• Different hmac_secret_reference values for different remote systems ensure a compromise of one system cannot affect tasks on another.
• If hmac_secret_reference is absent, the original token-only flow is used (backward compatible).

4. Frontend Configuration

The frontend has a single settings file: frontend/src/settings/tud_settings.js. The most important setting is API_BASE_URL, which must point to the backend API:

const TUD_SETTINGS = {
    // URL of the backend API as seen from the user's browser.
    // If the backend is proxied at a sub-path, include that path here.
    API_BASE_URL: 'https://your.domain.example.com/tud_backend/api',

    // Short name of the study to load by default (must match 'name_short' in studies_config.json)
    DEFAULT_STUDY_NAME: 'default',

    // Whether to show navigation buttons for previous days
    SHOW_PREVIOUS_DAYS_BUTTONS: true
};

No build step is required. Once this file is configured, the entire frontend/src/ directory can be deployed as-is to any static file server.

5. Web Server and Reverse Proxy

In a typical production setup a single web server (or reverse proxy) handles all traffic:

• Static frontend files are served directly from the frontend/src/ directory.
• Requests to the backend API path (e.g., /tud_backend/) are forwarded to the running backend process.

Make sure the proxy passes the correct X-Forwarded-* headers so that the backend can construct correct URLs, and configure TUD_ROOTPATH and TUD_ALLOWED_ORIGINS accordingly.

6. Admin Interface

TRAC includes a small server-rendered admin interface implemented in the backend using FastAPI templates. It is separate from the participant frontend and is intended for researchers or administrators who need to inspect studies, manage participants, and export collected data.

Access is protected with HTTP Basic Auth using the credentials configured in the backend .env file via TUD_API_ADMIN_USERNAME and TUD_API_ADMIN_PASSWORD.

The main entry point is:

https://your.domain.example.com/<TUD_ROOTPATH>/admin

For example, if TUD_ROOTPATH=/tud_backend, the admin overview page will be available at:

https://your.domain.example.com/tud_backend/admin

Because the admin interface is part of the backend application, it must be exposed through the same reverse-proxy setup as the API. As with the rest of the backend, it should only be made available over HTTPS.

7. API and Automation

The backend exposes a REST API and also serves FastAPI's live interactive API documentation. A convenient entry point is:

https://your.domain.example.com/<TUD_ROOTPATH>/api/docs

This redirects to the automatically generated FastAPI docs for the running backend instance.

Some endpoints are especially useful for automation, monitoring, backup, and integration with other systems:

• GET /api/health checks that the backend is running and can reach the database. This is useful for uptime monitoring and health checks.
• POST /api/admin/studies/import-config creates one or more studies from a study configuration payload (including activities and study texts), which is useful for automated study setup and server-to-server provisioning workflows.
• PATCH /api/admin/studies/{study_name_short}/collection-window updates the study data-collection time window (data_collection_start/data_collection_end), which is useful for closing a study early, extending it, or pausing data collection by setting a date range that excludes today.
• GET /api/admin/export/studies-runtime-config exports the full runtime study configuration together with participant assignments and logged activities. This is useful for backups and server-to-server synchronization.
• GET /api/admin/export/{study_name_short}/activities exports all recorded activities for one study in CSV or JSON, which is useful for data pipelines and integration with external systems.
• POST /api/admin/studies/{study_name_short}/assign-participants assigns one or more participants to a study and can create participant records when needed, which is useful for automated invitation workflows.

The /api/admin/... endpoints are protected with the same admin authentication as the admin interface.

Security

Because TRAC collects research data from study participants over the internet, you must secure the deployment:

• Use HTTPS for all traffic. Never run the app over plain HTTP in production.
• Set strong, unique passwords for the database user and the admin interface.
• Restrict TUD_ALLOWED_ORIGINS to only the exact origin(s) from which the frontend is served.
• Protect the admin interface — it is served at <TUD_ROOTPATH>/admin/. Make sure the admin password is strong and that it is only transmitted over HTTPS.
• Follow general web-server hardening best practices (secure headers, rate limiting, firewall rules, etc.) appropriate for your server software and environment.

Developer Documentation

Development Without Docker

Make sure you have git, uv, postgresql and nginx. Python comes with every Linux distribution, so you should not need to install it. This will get you everything you need under Ubuntu 24 LTS:

sudo apt install nginx git postgresql
curl -LsSf https://astral.sh/uv/install.sh | sh  # get uv for your user

Clone repo and change into it:

git clone https://github.com/dfsp-spirit/trac
cd trac/

There is no need to do anything for the frontend, it is ready to run. So let's create an empty, new database for the app:

cp dev_tools/local_nginx/backend_settings/.env.dev-nginx backend/.env
./database/create_tud_db.sh backend/.env

Now let's install the backend dependencies first and run the unit tests:

cd backend/

# Create virtual environment and install dependencies
uv sync --dev

# Run backend unit tests to verify setup
uv run pytest

Great, now it is time to run everything:

cd ..     # back to repo root (`trac` directory)
./run_dev_nginx_both.bash

The web server is configured for hot reload, so you are good to edit away and instantly see the changes. You only need to restart the backend if you add a new endpoint or after database schema changes.

You can verify that all services are operational by running the integration tests in a new terminal:

# in the repo root (`trac` directory)
./test_backend_integration.sh

We recommend to also run the E2E tests, see next section.

You can now connect to http://localhost:3000 to access nginx. The default nginx page will show details on how to access the frontend, admin interface, and API. If you did not change the default configuration, the URLs are:

• frontend for default study: /report/index.html
• backend (requires admin credentials from your backend/.env file): /tud_backend/admin
• API: /tud_backend/api
• API docs: /tud_backend/docs

Running Tests Locally (Without Docker)

Now that you have all local test dependencies, you can run tests directly from your host machine:

• Unit tests do not require services to be running: ./test_backend_unit.sh
• Integration tests require the backend and database to be running: ./test_backend_integration.sh
• E2E tests require frontend and backend to be running: ./test_e2e.sh

If you want to run E2E tests, install Node.js and Playwright once:

cd frontend/
npm install
npx playwright install --with-deps chromium firefox webkit

Note: If you get errors on npm install, e.g. about unsupported engine, the most likely reason is an outdated npm installation. We highly recommend to install nvm, then run nvm install node to get the latest stable node/npm version.

Development With Docker Compose

If you prefer a container-based development setup, this repo also includes docker-compose.dev.yml. It starts three services:

• PostgreSQL database
• backend container with the local backend/ directory mounted for live code reload
• nginx container with the local frontend/src/ directory mounted and proxied under /report/

This setup mirrors the normal local nginx development layout:

• frontend: http://localhost:3000/report/
• backend via reverse proxy: http://localhost:3000/tud_backend/
• backend direct port: http://localhost:8000/
• admin interface: http://localhost:3000/tud_backend/admin
• API docs: http://localhost:3000/tud_backend/api/docs

Start the stack from the repo root:

docker compose -f docker-compose.dev.yml up --build

Stop it again with:

docker compose -f docker-compose.dev.yml down

If you also want to delete the Docker volumes, including the PostgreSQL data volume, use:

docker compose -f docker-compose.dev.yml down -v

The compose setup uses Docker-specific config overlays from dev_tools/docker/ and does not require you to overwrite your normal local development settings files manually.

Running Tests with Docker

Helper scripts are provided to run tests against the containerized stack:

# Run all backend tests (unit + integration) in Docker
./run_tests_docker.sh

# Or run specific test suites
./run_tests_docker_unit.sh          # backend unit tests only
./run_tests_docker_integration.sh   # backend integration tests only (PostgreSQL default)
./run_tests_docker_integration.sh postgres
./run_tests_docker_integration.sh mariadb
./run_tests_docker_integration.sh mssql

# Run E2E tests fully inside Docker (Playwright container)
./run_tests_docker.sh e2e
./run_tests_docker_e2e.sh

These scripts assume the docker-compose stack is already running.

For PostgreSQL (default):

docker compose -f docker-compose.dev.yml up -d --build

For MariaDB:

docker compose -f docker-compose.dev.yml -f docker-compose.dev.mariadb.yml up -d --build

For Microsoft SQL Server:

docker compose -f docker-compose.dev.yml -f docker-compose.dev.mssql.yml up -d --build

The E2E test command uses the e2e service in docker-compose.dev.yml, which installs frontend test dependencies inside the container and runs Playwright there. This is useful when you want to avoid installing Node.js/Playwright on the host.

How to make a Release

• record changes in CHANGES file
• bump version of backend in backend/src/o-timeusediary_backend/__init__.py
• bump version of frontend in frontend/src/js/constants.js
• create commit with the mentioned changes, with a commit message like 'Bump version to and log changes for v0.x.y'
• tag the commit with the new version_ git tag v0.x.y <hash>
• run git push --tags to publish
• manually run the GitHub Actions workflow TUD Backend Integration Tests (DBMS Matrix, Manual) and confirm postgres, mariadb, and mssql jobs pass
• in the backend/ dir, run uv build to create the wheel artefact
• log into Github account, draft/publish a new release based on the tag, copy change notes from CHANGES in there and attach the wheel artefact
• API docs are attached automatically on release publish via GitHub Actions as release assets (openapi.json, index.html, and a tar.gz bundle), no manual docs upload needed

Author, License and Citing

TRAC was written by Tim Schäfer as part of employment at the Department of Cognitive Neuropsychology, Max-Planck Institute for Empirical Aesthetics, Frankfurt am Main, Germany.

Copyright 2025-2026 Max-Planck Institute for Empirical Aesthetics.

The frontend is based on o-timeusediary (O-TUD) by Andrei Tamas Foldes et al..

Both O-TUD and TRAC are published under the very permissive MIT License.

There is no academic paper on TRAC yet, but this software has a DOI and is fully citeable. Please see the CITATION.cff file for full information on how to properly cite TRAC in academic work.