Add flexible API key management and local model support (Hugging Face, vLLM)

.gitignore

@@ -0,0 +1,37 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Virtual environments
+venv/
+ENV/
+env/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# OS
+.DS_Store
+Thumbs.db

IMPLEMENTATION_SUMMARY.md

@@ -0,0 +1,240 @@
+# Oracle Benchmark - Implementation Summary
+
+## Overview
+This implementation adds flexible API key management and local model support to the Oracle Benchmark, making it easier to test models without requiring all API keys and enabling evaluation of open-source small parameter models.
+
+## Changes Made
+
+### New Files
+
+1. **flexible_api_manager.py** (195 lines)
+   - On-demand API client initialization
+   - Only initializes clients when API keys are available
+   - Provides methods to check API key availability
+   - Singleton pattern for global access
+
+2. **local_model_support.py** (289 lines)
+   - HuggingFaceModel class for Hugging Face transformers
+   - VLLMModel class for vLLM inference
+   - LocalModelManager for managing local model instances
+   - Support for quantization (4-bit, 8-bit)
+
+3. **model_config.py** (233 lines)
+   - YAML-based model configuration system
+   - ModelConfig class for individual model settings
+   - ModelConfigManager for loading and managing configs
+   - Default configurations for 19 API models
+
+4. **model_config.yaml** (79 lines)
+   - Template for custom model configurations
+   - Examples for Hugging Face models
+   - Examples for vLLM models
+   - Comments explaining each field
+
+5. **test_improvements.py** (184 lines)
+   - Comprehensive test suite
+   - Tests for all new components
+   - Backward compatibility tests
+   - Integration tests
+
+6. **.gitignore** (38 lines)
+   - Python artifacts
+   - Virtual environments
+   - IDE files
+   - OS-specific files
+
+### Modified Files
+
+1. **eva_models.py**
+   - Updated imports to include new managers
+   - Modified `__init__` to accept local model parameters
+   - Added flexible API client initialization
+   - Added local model support in `normal_output` method
+   - Added message format handling for local models
+   - Maintained backward compatibility
+
+2. **auto_generation.py**
+   - Updated imports to include flexible API manager
+   - Modified Platform class to use flexible API manager
+   - Modified PolishModel class to use flexible API manager
+   - Modified TestSamplesGenerator to use flexible API manager
+   - Fixed default parameter for max_turns
+
+3. **main.py**
+   - Added imports for new managers
+   - Updated argument parser to support 'local' family
+   - Changed eva_model_name from choices to free-form string
+   - Added new CLI parameters:
+     - `--model_config`: Path to YAML config file
+     - `--local_model_path`: Path to local model
+     - `--vllm_base_url`: vLLM server URL
+     - `--check_api_keys`: Check available API keys
+   - Added API key checking functionality
+   - Updated map_model_name_to_api_name to support config manager
+   - Maintained backward compatibility
+
+4. **requirements.txt**
+   - Added PyYAML>=6.0 as core dependency
+   - Added comments for optional dependencies:
+     - transformers>=4.35.0
+     - torch>=2.0.0
+     - accelerate>=0.25.0
+     - bitsandbytes>=0.41.0
+     - vllm>=0.2.0
+
+5. **README.md**
+   - Added "New Features: Flexible Model Management" section
+   - Added examples for checking API keys
+   - Added examples for local model usage
+   - Added examples for vLLM usage
+   - Added custom configuration examples
+   - Updated news section
+   - Added installation instructions for optional dependencies
+
+## Key Features
+
+### 1. Flexible API Key Management
+- **Problem Solved**: Previously required all API keys to be configured
+- **Solution**: On-demand client initialization based on selected models
+- **Benefits**:
+  - Test models without configuring all API keys
+  - Graceful error handling for missing keys
+  - Easy verification of available keys with `--check_api_keys`
+
+### 2. Local Model Support
+- **Problem Solved**: No support for open-source small parameter models
+- **Solution**: Integration with Hugging Face transformers and vLLM
+- **Supported Models**:
+  - Llama (2, 3)
+  - Mistral
+  - Qwen
+  - Phi
+  - Gemma
+  - Yi
+  - Any Hugging Face compatible model
+- **Features**:
+  - Quantization support (4-bit, 8-bit)
+  - Device selection (CPU, CUDA, auto)
+  - vLLM integration for fast inference
+
+### 3. Configuration-Driven Model Management
+- **Problem Solved**: Hardcoded model configurations
+- **Solution**: YAML-based configuration system
+- **Benefits**:
+  - Easy to add custom models
+  - Centralized model configuration
+  - Supports model-specific parameters
+  - Auto-detection of available models
+
+### 4. Backward Compatibility
+- **Achievement**: 100% backward compatible
+- **Evidence**:
+  - All existing APIs unchanged
+  - Original command-line parameters work
+  - Test suite validates compatibility
+  - No breaking changes
+
+## Usage Examples
+
+### Check Available API Keys
+```bash
+python main.py --eva_model_family gpt --check_api_keys
+```
+
+### Test Local Hugging Face Model
+```bash
+python main.py --eva_model_family local --eva_model_name llama-2-7b \
+  --local_model_path meta-llama/Llama-2-7b-chat-hf --task code
+```
+
+### Test with vLLM
+```bash
+# Start vLLM server
+vllm serve Qwen/Qwen2-7B-Instruct --port 8000
+
+# Run evaluation
+python main.py --eva_model_family local --eva_model_name qwen2-7b \
+  --vllm_base_url http://localhost:8000/v1 --task puzzle
+```
+
+### Use Custom Config
+```bash
+python main.py --model_config my_models.yaml \
+  --eva_model_family local --eva_model_name my-model --task encryption
+```
+
+### Traditional Usage (Still Works)
+```bash
+python main.py --eva_model_family gpt --eva_model_name gpt-4o --task code
+```
+
+## Testing Results
+
+All tests pass successfully:
+- ✓ Flexible API key management
+- ✓ On-demand API client initialization
+- ✓ Model configuration system
+- ✓ Local model support infrastructure
+- ✓ Backward compatibility maintained
+- ✓ Component integration
+- ✓ Python syntax validation
+- ✓ Import validation
+- ✓ CLI argument parsing
+
+## File Statistics
+
+- Total new files: 6
+- Total modified files: 5
+- Total lines of new code: ~1,000
+- Total lines of documentation: ~130 (README)
+- Total lines of tests: ~190
+
+## Installation
+
+### Core Dependencies (Required)
+```bash
+pip install PyYAML>=6.0
+```
+
+### Local Model Support (Optional)
+```bash
+# For Hugging Face models
+pip install transformers torch accelerate
+
+# For quantization
+pip install bitsandbytes
+
+# For vLLM
+pip install vllm
+```
+
+## Benefits
+
+1. **Increased Flexibility**: Test models without configuring all API keys
+2. **Cost Reduction**: Use free local models instead of paid APIs
+3. **Research Enablement**: Easy evaluation of open-source models
+4. **Better Developer Experience**: Clear error messages and validation
+5. **Future Proof**: Easy to add new model types and configurations
+6. **Maintained Compatibility**: Zero breaking changes
+
+## Future Enhancements
+
+Possible future improvements:
+1. Support for more inference frameworks (TensorRT-LLM, etc.)
+2. Batch processing for local models
+3. Model performance benchmarking
+4. Automatic model recommendation based on task
+5. Cloud model deployment integration
+6. Model caching and optimization
+
+## Conclusion
+
+This implementation successfully addresses all requirements from the problem statement:
+- ✓ Flexible API key management
+- ✓ Open-source small parameter model support
+- ✓ Configuration-driven model management
+- ✓ Backward compatibility
+- ✓ Rich documentation and examples
+- ✓ Comprehensive testing
+
+The Oracle Benchmark is now more flexible, accessible, and ready for evaluating both API-based and local models.