Specialized AI Assistant for Bioinformatics with Advanced RAG System and Multi-Agent Workflows
BioAgent is an AI-powered bioinformatics analysis platform that combines a sophisticated RAG (Retrieval-Augmented Generation) system with intelligent per-message mode detection to provide expert bioinformatics assistance. The tool automatically generates code, executes analyses, and consults a knowledge base of 344+ bioinformatics protocols with automatic database population, real-time updates, and comprehensive testing (111 unit tests with 100% success rate).
- π€ Specialized AI Assistant: Expertise in genomics, transcriptomics, proteomics, and bioinformatics
- π Smart RAG System: 344+ indexed protocols with automatic database population and file change detection
- π§ Intelligent Mode Detection: Per-message complexity assessment automatically selects ask vs agent mode
- π Multi-Agent Workflows: CrewAI system for sophisticated and coordinated analyses
- π» Automatic Code Generation: Python, R, and Bash with automatic execution and debugging
- π Integrated Visualizations: Automatic generation of plots and reports
- π§ Modern Web Interface: Intuitive React interface with FastAPI backend and auto-reload
- π Session Management: Conversation history and analysis tracking with live transcripts
- π― Real-Time Status Updates: Precise tracking of analysis steps with streaming updates
- π Detailed Display: Interactive execution blocks, RAG metadata, and debugging information
- π₯ Live Code Execution Streaming: Real-time visibility into code execution, debugging steps, and error handling
- β‘ Auto-Reload Development: Automatic server restart on code changes for seamless development
- π§ͺ Comprehensive Testing: 111 unit tests with 100% success rate ensuring reliability and stability
- Python 3.12 or higher (recommended)
- Python 3.10+ minimum supported
- Git
- GPU Requirements: High-performance GPU with at least 30GB VRAM (RTX 4090, RTX 6000 Ada, A100, or equivalent)
- 32GB RAM minimum recommended for optimal performance
- Ollama with an LLM model (qwen3:32b recommended)
# Clone the project
git clone https://github.com/mickaelleclercq/bioAgent.git
cd bioAgent
# Automatic installation (recommended)
bash install.sh
# Launch the application
bash launch.sh# Create virtual environment
python3 -m venv .venv
source .venv/bin/activate
# Install dependencies
pip install -r requirements.txt
# Populate RAG database (automatic on first launch)
cd src/scripts
python3 populate_rag.py
cd ../..
# Launch application
bash launch.shOnce launched, the application is available at:
- Web Interface: http://localhost:8080 (JavaScript UI)
- REST API: http://localhost:8001
- API Documentation: http://localhost:8001/docs
- Health Check: http://localhost:8001/health
bioAgent/
βββ π src/ # Main source code
β βββ π bioagent/ # Core AI system
β β βββ π agents/ # 19 specialized AI agents
β β β βββ adaptive_orchestrator.py # Central coordinator
β β β βββ context_agent.py # Scientific context understanding
β β β βββ planning_agent.py # Workflow planning
β β β βββ code_generation_agent.py # Multi-language code generation
β β β βββ evaluation_agent.py # Result assessment
β β β βββ debugging_agent.py # Error resolution
β β β βββ data_profiler_agent.py # Data structure analysis
β β β βββ package_manager_agent.py # Dependency management
β β β βββ data_validator_agent.py # Data quality validation
β β β βββ error_recovery_agent.py # Automated error handling
β β β βββ r_code_validator_agent.py # R-specific validation
β β β βββ inline_code_agent.py # Quick code execution
β β β βββ session_report_agent.py # Analysis reporting
β β β βββ ai_*.py # Enhanced AI agents
β β βββ π orchestration/ # Multi-agent coordination
β β β βββ context.py # Analysis context
β β β βββ planning_handler.py # Workflow planning logic
β β β βββ execution_handler.py # Code execution management
β β β βββ evaluation_handler.py # Result evaluation
β β β βββ recovery_handler.py # Error recovery strategies
β β β βββ package_handler.py # Dependency resolution
β β βββ π prompts/ # Jinja2 prompt templates
β β βββ π tools/ # Analysis tools
β β βββ π utils/ # Utility functions
β β βββ π validation/ # Validation components
β β βββ rag_integration_complex.py # Knowledge retrieval system
β β βββ config.py # Centralized configuration
β β βββ session_manager.py # Session handling
β β βββ ai_request_classifier.py # Mode detection
β β βββ *.py # Additional modules
β βββ π ui/ # Web interface
β β βββ π api/ # FastAPI REST API
β β β βββ main.py # Application factory
β β β βββ π routes/ # API endpoints
β β β βββ π services/ # Business logic
β β βββ app.html # JavaScript frontend
β β βββ session_store.py # Session persistence
β β βββ index.html # Entry point
β βββ π scripts/ # Utility scripts
β βββ populate_rag.py # RAG database population
βββ π protocols/ # Knowledge base (344+ protocols)
β βββ *.sh # Shell scripts (Hi-C, RNA-seq, etc.)
β βββ *.py # Python analysis scripts
β βββ *.R # R statistical scripts
β βββ *.json # Structured knowledge bases
β βββ *.md # Documentation and protocols
β βββ *.yml # Environment configurations
β βββ *.ipynb # Jupyter notebooks
βββ π tests/ # Comprehensive test suite (111 tests)
β βββ π unit/ # Unit tests (7 files, 111 tests)
β βββ π integration/ # Integration tests
β βββ π performance/ # Performance benchmarks
β βββ conftest.py # Shared test fixtures
βββ π vector_store/ # ChromaDB vector database
βββ π data/ # User data and sessions
β βββ sessions.db # Session database
β βββ π chat-sessions/ # Conversation history
β βββ π user_data/ # Generated files and results
βββ π logs/ # Application logs
- Frontend: Modern JavaScript interface with real-time streaming capabilities
- Backend: FastAPI with modular API structure and service layers
- AI Framework: CrewAI with 19 specialized agents for multi-agent orchestration
- LLM Integration: Ollama with support for multiple models (qwen3:32b default)
- RAG System: ChromaDB vector database with 344+ indexed protocols
- Orchestration: Sophisticated handler system for workflow coordination
- Execution: Secure multi-language execution environment (Python, R, Bash)
- Sessions: SQLite-based persistence with comprehensive history tracking
- Configuration: Unified configuration system with environment variable support
- Testing: 111 unit tests with 100% success rate plus integration testing
BioAgent employs a sophisticated multi-agent system with 19 specialized AI agents coordinated by an adaptive orchestrator:
- AdaptiveAnalysisOrchestrator: Central coordinator that delegates tasks to specialized handlers
- Orchestration Handlers: Dedicated handlers for planning, execution, evaluation, recovery, and package management
- ContextAgent: Scientific context understanding and domain expertise
- PlanningAgent: Multi-step analysis workflow planning and optimization
- EvaluationAgent: Result assessment and quality validation
- CodeGenerationAgent: Multi-language code generation (Python, R, Bash)
- InlineCodeAgent: Quick code execution and testing
- RCodeValidatorAgent: R-specific syntax and logic validation
- DebuggingAgent: Automated error detection and code correction
- DataProfilerAgent: Comprehensive data structure analysis
- DataValidatorAgent: Data quality assessment and validation
- PackageManagerAgent: Intelligent dependency resolution and management
- ErrorRecoveryAgent: Sophisticated error handling and recovery strategies
- AI Error Recovery Agent: Enhanced AI-powered error analysis
- AI Data Validator Agent: Advanced data validation with machine learning
- AI Package Manager Agent: Smart package management with conflict resolution
- AI R Code Validator Agent: Enhanced R code validation and optimization
- SessionReportAgent: Comprehensive analysis reporting and documentation
- Session Manager: Conversation history and context preservation
class OrchestrationState(Enum):
INITIALIZING = "initializing"
ANALYZING_CONTEXT = "analyzing_context"
PLANNING = "planning"
PACKAGE_MANAGEMENT = "package_management"
GENERATING_CODE = "generating_code"
EXECUTING = "executing"
EVALUATING = "evaluating"
DEBUGGING = "debugging"
REVISING_PLAN = "revising_plan"
COMPLETED = "completed"
FAILED = "failed"The orchestrator uses specialized handlers for complex logic:
- PlanningHandler: Workflow planning and step optimization
- ExecutionHandler: Code execution with real-time monitoring
- EvaluationHandler: Result validation and success assessment
- RecoveryHandler: Error recovery and debugging coordination
- PackageHandler: Dependency management and environment setup
The RAG system is the core of BioAgent's expertise:
- 344+ protocols indexed and searchable with automatic updates
- Complete coverage: Genomics, RNA-seq, Hi-C, proteomics, single-cell analysis, nanopore sequencing
- Multiple formats: Shell scripts, Python, R, Markdown documentation, YAML configs, Jupyter notebooks
- Enriched metadata: Automatic context extraction from JSON files with multi-protocol support
- Smart Updates: Automatic database population when new files are added to protocols folder
- File Change Detection: Automatic re-indexing when protocols are modified
- Intelligent Mode Detection: Per-message complexity assessment determines ask vs agent workflow
- Mandatory RAG consultation: Every query automatically consults the unified protocol database
- Semantic search: Advanced vector similarity search with fallback keyword matching
- Multi-agent workflows: Coordinated analysis for complex requests with real-time streaming
- Contextual adaptation: Response customization based on detected analysis type and user context
- Transparent display: RAG metadata visible in interface with protocol count and reasoning
- Auto-population: Database automatically rebuilds when protocols folder changes
The RAG database updates automatically:
# Simply add your files to the protocols/ folder
cp your_protocol.py protocols/
# Launch the application - it will detect changes and auto-populate
bash launch.shAutomatic Features:
- Auto-detection:
launch.shautomatically detects if RAG database needs updating - File timestamps: Compares protocols folder with vector store modification times
- Smart rebuild: Only rebuilds when new files are added or existing files are modified
- Zero-downtime: RAG updates happen during application startup
- Manual rebuild: Run
python3 src/scripts/populate_rag.pyif needed
- Access http://localhost:8080 (JavaScript UI)
- Ask questions in natural language
- Follow real-time status updates:
- π Consulting knowledge base
- π Loading context
- π€ Complexity analysis
- π§ Code execution with live streaming
- β Finalization with results
- View results in interactive blocks
- Download generated files (plots, data)
The enhanced streaming system now provides:
- Real-time execution visibility: See Python/R code execution as it happens
- Debug step tracking: Watch automatic debugging attempts in real-time
- Error handling transparency: View bug detection and correction attempts
- Progress history: Complete chronological log of all analysis steps
- No silent periods: Continuous feedback throughout the entire process
# Dual model approach - separate reasoning and coding models for optimal performance
curl -X POST "http://localhost:8001/chat/stream" \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "message=Analyze RNA-seq data with DESeq2&session_id=my_session&reasoning_model=qwen3:32b&coding_model=qwen3-coder:30b"
# Same model for both tasks (simpler setup)
curl -X POST "http://localhost:8001/chat/stream" \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "message=Analyze RNA-seq data with DESeq2&session_id=my_session&reasoning_model=qwen3:32b&coding_model=qwen3:32b"Dual Model Benefits:
- π§ Reasoning Model: Optimized for analysis, planning, and interpretation (e.g.,
qwen3:32b,llama3.1:70b) - π» Coding Model: Specialized for code generation and debugging (e.g.,
qwen3-coder:30b,codellama:34b) - π Performance: Right model for each task type improves speed and accuracy
- π§ Flexibility: Use the same model for both tasks or specialize based on your Ollama setup
Recommended Model Combinations:
# High-performance setup (requires ~60GB VRAM)
reasoning_model=qwen3:32b&coding_model=qwen3-coder:30b
# Balanced setup (requires ~30GB VRAM)
reasoning_model=qwen3:14b&coding_model=codellama:13b
# Lightweight setup (requires ~15GB VRAM)
reasoning_model=qwen3:7b&coding_model=qwen3:7b
# Single model setup (minimal VRAM)
reasoning_model=qwen3:7b&coding_model=qwen3:7bAdditional Parameters:
thinking_enabled=true- Enable thinking mode (only supported by Qwen models, disabled by default)mode=agent- Force agent mode for complex analysis (default: auto-detection)mode=ask- Force simple Q&A mode
- "How to analyze WGS data?"
- "Create simulated gene expression data and generate a volcano plot"
- "Analyze my FastQ files and perform statistical analysis"
- "Perform differential analysis with DESeq2"
- "Generate a heatmap and PCA from RNA-seq data"
- "Analyze my Hi-C data and create contact maps"
The interface displays contextual status during processing:
- π "Consulting knowledge base..."
- β "Found X relevant protocols in knowledge base"
- π‘ "Analyzing protocols for guidance..."
- π "Loading conversation context..."
- π "Complex request detected - activating workflow system..."
- π§ "Executing Python code..." (with live updates)
- π "Bug detected - attempting automatic fix..."
- β "Bug fixed successfully after X attempts!"
- Execution details: Status, duration, language used
- Generated files: List with sizes, types, and direct download
- Complete logs: Standard output and errors
- Debug information: Automatic code correction attempts with real-time progress
- Automatic loading: Recovery of existing sessions
- Complete history: Preservation of RAG metadata and execution results
- Smooth navigation: Session switching with context preservation
BioAgent uses one simple configuration file: src/bioagent/config.py
To customize BioAgent:
- Edit the configuration sections in
src/bioagent/config.py - Or use environment variables (e.g.,
export BIOAGENT_API_PORT=8002)
Key configuration sections:
# API Configuration
config.api.host = "localhost"
config.api.port = 8001
config.api.cors_origins = ["http://localhost:8080"]
# LLM Configuration
config.llm.default_model = "qwen3:32b"
config.llm.max_tokens = 2048
config.llm.temperature = 0.7
# Debug Configuration
config.debug.enabled = False
config.debug.show_agent_prompts = True
# Security Configuration
config.security.max_execution_time = 300
config.security.allowed_extensions = [".py", ".r", ".sh", ".txt"]Environment variable support:
export BIOAGENT_API_PORT=8002
export BIOAGENT_LLM_MODEL="qwen3:8b"
export BIOAGENT_DEBUG=true
export BIOAGENT_MAX_EXECUTION_TIME=600# Import and use config
from bioagent.config import config
api_port = config.api.port
llm_model = config.llm.default_model
debug_mode = config.debug.enabledBioAgent uses a unified configuration system with multiple layers:
src/bioagent/config.py: Main configuration module with structured settingssrc/bioagent/mode_config.py: Mode-specific configurations for ask/agent workflowssrc/ui/api/config.py: API-specific configuration settingspyproject.toml: Python package configuration and dependenciesrequirements.txt: Detailed Python dependency specificationslaunch.sh: Launch script with runtime configuration detectioninstall.sh: Installation script with environment setup
# Example configuration access
from bioagent.config import config
# API settings
api_port = config.api.port
api_host = config.api.host
# LLM settings
default_model = config.llm.default_model
temperature = config.llm.temperature
# RAG settings
max_results = config.rag.max_results
enabled = config.rag.enabled
# Storage settings
db_path = config.storage.database_path
vector_store_path = config.storage.vector_store_pathBy default, BioAgent uses qwen3:32b. Change the config.llm.default_model setting or use environment variables.
BioAgent includes a comprehensive test suite with 111 unit tests, integration tests, and performance benchmarks, achieving 100% success rate in unit testing:
# Verify test structure and syntax
python3 tests/verify_tests.py# Run comprehensive test workflow
# Includes: server health check β unit tests β integration tests
python3 tests/run_comprehensive_tests.py# Run specific test categories
python3 -m pytest tests/unit/ -v # All unit tests (111 tests)
python3 -m pytest tests/integration/ -v # All integration tests
python3 -m pytest tests/performance/ -v # Performance benchmarks
# Run specific test files
python3 -m pytest tests/unit/test_api.py -v # API endpoints (17 tests)
python3 -m pytest tests/unit/test_modes.py -v # Mode detection (14 tests)
python3 -m pytest tests/unit/test_rag.py -v # RAG system (7 tests)
python3 -m pytest tests/unit/test_agents.py -v # Agent system (31 tests)
python3 -m pytest tests/unit/test_config.py -v # Configuration (11 tests)
# Run tests with detailed output
python3 -m pytest tests/ -v -s --tb=shorttests/
βββ conftest.py # Shared fixtures and mock components
βββ verify_tests.py # Test structure validation and syntax check
βββ run_comprehensive_tests.py # Complete test suite orchestrator
βββ run_clean_tests.py # Clean environment test runner
βββ run_dual_model_tests.py # Dual model configuration testing
βββ test_rag_efficiency.py # RAG system performance testing
βββ π unit/ (111 tests total) # Unit tests with 100% success rate
β βββ test_config.py # Configuration system (11 tests)
β βββ test_agents.py # Multi-agent system (31 tests)
β βββ test_api.py # API endpoints (17 tests)
β βββ test_modes.py # Mode detection (14 tests)
β βββ test_rag.py # RAG/Vector store (7 tests)
β βββ test_files_and_packages.py # File operations (16 tests)
β βββ test_main_functions.py # Core workflows (15 tests)
β βββ test_session_improvements.py # Session management tests
β βββ test_dual_model_integration.py # Dual model testing
β βββ test_fresh_restart.py # Clean restart validation
β βββ test_function_calls.py # Function execution testing
β βββ test_package_parsing.py # Package management testing
βββ π integration/ # End-to-end integration tests
β βββ test_api_complete.py # Complete API workflow testing
β βββ test_dual_model_api.py # Dual model API integration
β βββ test_orchestration.py # Multi-agent coordination testing
βββ π performance/ # Performance and benchmark tests
β βββ test_benchmarks.py # System performance metrics
βββ π development/ # Development and debugging tests
βββ π tools/ # Testing utilities and helpers
βββ π fixtures/ # Test data and mock files
Unit Tests (111 tests - 100% success rate):
- Agent System: 31 tests covering all 19 AI agents and orchestration
- API Functionality: 17 tests for REST endpoints and streaming
- Configuration: 11 tests for unified config system
- File Operations: 16 tests for secure file handling and package management
- Core Workflows: 15 tests for analysis pipelines and execution
- Mode Detection: 14 tests for intelligent ask/agent mode selection
- RAG System: 7 tests for vector database and knowledge retrieval
Integration Tests:
- Complete API Workflows: End-to-end request processing
- Multi-Agent Coordination: Complex analysis pipeline testing
- Dual Model Integration: Reasoning + coding model combinations
- Session Management: Persistence and history validation
Performance Tests:
- RAG Efficiency: Vector search and protocol retrieval benchmarks
- Execution Speed: Code generation and execution timing
- Memory Usage: Resource consumption monitoring
- Concurrent Sessions: Multi-user scenario testing
- Server must be running:
bash launch.shbefore integration tests - Virtual environment: Tests use
.venv/bin/python - Clean environment: Tests use mocks and fixtures for safety
- Comprehensive coverage: All major bioAgent components tested
- Proven reliability: 100% unit test success rate (111/111 tests passing)
- Safe execution: All tests use mocks - no real file operations
- Async support: Proper testing of streaming responses
- Detailed reporting: Success/failure analysis with timing
- Server health checks: Automatic API availability verification
- Performance tracking: Integration test timing and timeout handling
# AI & Machine Learning
crewai>=0.80.0 # Multi-agent orchestration framework
chromadb # Vector database for RAG system
tiktoken # Token counting for LLM optimization
# Web Framework & API
fastapi==0.116.1 # Modern REST API framework
uvicorn[standard]==0.35.0 # ASGI server with auto-reload
pydantic # Data validation and settings management
# Data Science & Analysis
pandas # Data manipulation and analysis
numpy # Numerical computing
matplotlib # Plotting and visualization
seaborn # Statistical data visualization
pillow # Image processing
# Utilities & Integration
jinja2 # Template engine for prompts
trafilatura # Web content extraction
duckduckgo-search # Web search capabilities
redis # Caching and task queuing
rq # Redis-based job queue
# Development & Testing
pytest # Testing framework
pytest-cov # Coverage reporting
black # Code formatting
isort # Import sorting
flake8 # Code linting- Backend Framework: FastAPI with modular route structure
- Frontend: Vanilla JavaScript with Server-Sent Events (SSE)
- AI Orchestration: CrewAI with custom agent implementations
- Vector Database: ChromaDB for semantic search and knowledge retrieval
- LLM Integration: Ollama for local model hosting and inference
- Session Management: SQLite database with file-based persistence
- Configuration: Python-based unified configuration system
- Testing: pytest with comprehensive mocking and fixtures
To contribute to the project:
- Fork the repository
- Create a branch for your feature
- Add your protocols to the
protocols/folder - Test your modifications
- Submit a pull request
Protocols can be added in different formats:
- Shell scripts (
.sh): Complete analysis pipelines - Python scripts (
.py): Specialized analyses - R scripts (
.R): Statistical analyses - Documentation (
.md): Guides and protocols - Configurations (
.json,.yaml): Parameters and metadata
This project is under MIT license. See the LICENSE file for details.
For any questions or issues:
- API Documentation: http://localhost:8001/docs (automatic OpenAPI docs)
- Application Logs: Check
logs/directory for detailed error information - Test Results: Run
python3 tests/verify_tests.pyfor system validation - Session Data: Review saved sessions in
data/chat-sessions/for analysis history - Configuration Issues: Verify settings in
src/bioagent/config.py
- Real-time Status: Web interface shows detailed progress and RAG consultation
- Execution Logs: Live streaming of code execution and debugging attempts
- Test Suite: 111 unit tests available for system verification
- Health Check: Visit http://localhost:8001/health for system status
Installation Problems:
# Verify Python version
python3 --version # Should be 3.10+
# Check virtual environment
source .venv/bin/activate
pip list | grep -E "(fastapi|crewai|chromadb)"
# Reinstall if needed
bash install.shServer Issues:
# Check if ports are available
lsof -i :8001 # API port
lsof -i :8080 # UI port
# Restart with clean logs
bash launch.shModel Issues:
# Verify Ollama is running
curl http://localhost:11434/api/tags
# Check model availability
ollama list | grep qwen3For contributors and developers:
# Install development dependencies
pip install -r requirements.txt
pip install -e .
# Run comprehensive tests
python3 tests/run_comprehensive_tests.py
# Start in development mode
export BIOAGENT_API_RELOAD=true
bash launch.sh- CPU: 8-core processor (Intel i7/AMD Ryzen 7 or better)
- RAM: 32GB DDR4/DDR5
- GPU: High-performance GPU with at least 30GB VRAM
- NVIDIA RTX 4090 (24GB) - minimum
- NVIDIA RTX 6000 Ada (48GB) - recommended
- NVIDIA A100 (40GB/80GB) - optimal
- NVIDIA H100 (80GB) - premium
- Storage: 500GB free space (SSD recommended)
- Network: Stable internet connection for model downloads
- CPU: 16-core processor or better
- RAM: 64GB or more
- GPU: 48GB+ VRAM for optimal performance
- Storage: 1TB+ NVMe SSD
- The 30GB VRAM requirement is for running large language models (e.g., qwen3:32b)
- Smaller models can run with less VRAM but may impact analysis quality
- Multiple GPUs can be used for distributed processing
- CPU-only mode is possible but significantly slower