Synthetic Population Experiments

This workspace is organized to handle multiple experiments with different configurations, data, and analysis requirements. The LLMPopulationGenerator package is installed as an editable dependency (pip install -e), allowing direct imports while keeping your custom schemas, prompts, and data separate.

Directory Structure

Synthetic-Population-Experiments/
├── requirements.txt              # Python dependencies
├── .env                         # Environment variables (API keys, etc.)
├── .env.example                 # Example environment variable template
├── README.md                    # This file
├── setup_shared_files.py        # Validation script for package installation
│
├── shared/                      # Shared resources across experiments
│   ├── utils/                   # Utility functions and classes
│   │   ├── experiment_utils.py  # General experiment utilities
│   │   ├── fit_metrics.py       # Demographic fit metric calculations
│   │   ├── model_factory.py     # LLM model creation factory
│   │   ├── plot_config.py       # Shared plotting configuration
│   │   ├── srmse_calculator.py  # SRMSE calculation utilities
│   │   └── validation_rules.py  # Data validation rules
│   ├── prompts/                 # Custom prompt templates
│   └── schemas/                 # Custom JSON schemas
│
├── experiments/                 # Individual experiments
│   ├── 01_llm_benchmark/       # LLM benchmarking experiment
│   ├── 02_global/              # Global population generation
│   ├── 03_newcastle/           # Newcastle case study
│   ├── 04_dar_es_salaam/       # Dar es Salaam case study
│   ├── 05_prompt_sensitivity/  # Prompt sensitivity analysis
│   ├── 06_no_feedback/         # Generation without feedback
│   ├── 07_method_benchmark/    # Method benchmarking
│   └── 08_batch_generation/    # Batch generation experiment
│
├── data_repository/             # Canonical input data and data dictionary
├── checkpoints/                 # Saved experiment checkpoints
└── figures_for_publication/     # Publication-ready figures

Each experiment follows a common structure:

<experiment>/
├── config/             # Experiment-specific configuration
├── data/               # Experiment input/processed data
├── scripts/            # Experiment scripts
└── outputs/            # Generated results and reports

Setup Instructions

1. Install Dependencies

   cd Synthetic-Population-Experiments
   pip install -r requirements.txt

   Then clone and install the LLMPopulationGenerator package in editable mode:

   git clone git@github.com:MJones235/LLM-Population-Generator.git
   pip install -e LLM-Population-Generator

2. Validate Package Installation

   python setup_shared_files.py

   This verifies that the LLMPopulationGenerator package can be imported and is properly installed.

3. Add Your Custom Files

   • Copy or create prompts in shared/prompts/
   • Copy or create schemas in shared/schemas/

4. Environment Variables Create a .env file with your API credentials:

   AZURE_OPENAI_API_KEY=your_azure_key_here
   AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
   AZURE_OPENAI_API_VERSION=2024-08-01-preview
   AZURE_OPENAI_TEMPERATURE=0.7
   

Running the LLM Benchmark Experiment

1. Navigate to experiment directory

   cd experiments/01_llm_benchmark

2. Configure models to test Edit config/llm_models.yaml to add/remove models

3. Configure experiment parameters
   Edit config/experiment_config.yaml for generation settings

4. Run the benchmark

   python scripts/run_benchmark.py

Experiment Results

Results are saved to experiments/01_llm_benchmark/outputs/ with:

• Individual model results in timestamped folders
• Population data in JSON and CSV formats
• Cost analysis and failure statistics
• Comprehensive experiment summary

Adding New Experiments

To create a new experiment:

1. Create directory: experiments/09_new_experiment/
2. Copy structure from an existing experiment (e.g. 01_llm_benchmark/)
3. Update configurations for your specific needs
4. Implement experiment-specific scripts
5. Use shared utilities and data as needed

Analysis

Each experiment includes:

• Automated cost tracking
• Failure analysis and statistics
• Demographic distribution analysis
• Model comparison metrics
• Exportable reports and summaries