flowchart TB
User[User Query] --> TA[Triaging Agent]
TA --> DPA[Data Processing Agent]
TA --> AA[Analysis Agent]
TA --> VA[Visualization Agent]
DPA --> DPT[Data Processing Tools]
AA --> DAT[Data Analysis Tools]
VA --> DVT[Data Visualization Tools]
DPT --> Results[Results]
DAT --> Results
DVT --> Results
Results --> User
flowchart LR
User[User] --> |1. Query| TA[Triaging Agent]
TA --> |2. Data Request| DPA[Data Processing Agent]
DPA --> |3. Clean Data| AA[Analysis Agent]
AA --> |4. Analyzed Data| VA[Visualization Agent]
VA --> |5. Visualizations| TA
TA --> |6. Final Response| User
flowchart TB
Input[Raw Data Input] --> Clean[Data Cleaning]
Clean --> Transform[Data Transformation]
Transform --> Aggregate[Data Aggregation]
Clean --> |Remove Duplicates| CD1[Duplicate Handler]
Clean --> |Handle Missing| CD2[Missing Value Handler]
Clean --> |Handle Outliers| CD3[Outlier Detector]
Transform --> |Scale Features| T1[Feature Scaler]
Transform --> |Encode Categories| T2[Category Encoder]
Transform --> |Process Dates| T3[DateTime Processor]
Aggregate --> |Group Data| A1[Group Operations]
Aggregate --> |Window Functions| A2[Time Windows]
Aggregate --> |Custom Aggregations| A3[Custom Functions]
flowchart LR
Input[Processed Data] --> Stats[Statistical Analysis]
Input --> Corr[Correlation Analysis]
Input --> Reg[Regression Analysis]
Stats --> |Basic Stats| S1[Mean/Median/Mode]
Stats --> |Distribution| S2[Standard Deviation]
Stats --> |Hypothesis| S3[T-tests/ANOVA]
Corr --> |Pearson| C1[Linear Correlation]
Corr --> |Spearman| C2[Rank Correlation]
Reg --> |Linear| R1[Simple/Multiple]
Reg --> |Logistic| R2[Binary/Multi-class]
flowchart TB
Input[Analysis Results] --> Charts[Chart Generation]
Charts --> Bar[Bar Charts]
Charts --> Line[Line Charts]
Charts --> Pie[Pie Charts]
Bar --> |Single| B1[Simple Bar]
Bar --> |Multiple| B2[Grouped Bar]
Bar --> |Stacked| B3[Stacked Bar]
Line --> |Single| L1[Simple Line]
Line --> |Multiple| L2[Multi-line]
Line --> |Area| L3[Area Chart]
Pie --> |Basic| P1[Simple Pie]
Pie --> |Donut| P2[Donut Chart]
Pie --> |Exploded| P3[Exploded Pie]
flowchart TB
Start[User Query] --> Valid{Valid Query?}
Valid --> |No| Error1[Query Error]
Valid --> |Yes| Process[Process Query]
Process --> DataCheck{Data Available?}
DataCheck --> |No| Error2[Data Error]
DataCheck --> |Yes| Execute[Execute Tools]
Execute --> Success{Execution Success?}
Success --> |No| Error3[Processing Error]
Success --> |Yes| Complete[Return Results]
Error1 --> Feedback[Error Feedback]
Error2 --> Feedback
Error3 --> Feedback
Feedback --> Start
Complete --> End[End]
- Introduction
- Project Overview
- System Architecture
- Getting Started
- Usage
- Configuration
- Project Structure
- Contributing
- License
This project implements a multi-agent system designed for comprehensive data analysis. It leverages the OpenAI API to create intelligent agents capable of processing, analyzing, and visualizing data. The system is built around a modular architecture, allowing for easy extension and customization.
The Multi-Agent Data Analysis System is designed to handle a variety of data-related tasks through the collaboration of specialized agents. Users can input queries, and the system will intelligently route these queries to the appropriate agents, execute the necessary tools, and return the results. The system supports data cleaning, transformation, aggregation, statistical analysis, correlation analysis, regression analysis, and the creation of various charts (bar, line, pie).
The system is composed of several specialized agents, each designed to handle specific tasks and implemented using OpenAI's GPT-4 model:
The Triaging Agent is the entry point for user queries, implemented in agents/triage_agent.py. It uses natural language processing to understand and categorize user requests.
Key Responsibilities:
- Assess user queries using semantic analysis
- Route queries to appropriate agents based on intent classification
- Interact with users for additional information using a structured dialogue system
- Maintain context across multiple interactions
- Handle error cases and provide meaningful feedback
Example Interaction:
User: "I need to analyze sales data trends"
Agent: "I'll help you analyze sales trends. Could you specify:
1. The time period you're interested in
2. Specific metrics you want to analyze
3. Any particular visualization preferences?"
The Data Processing Agent (agents/data_processing_agent.py) handles all data preparation tasks using pandas and numpy.
Key Responsibilities:
- Clean data by:
- Removing duplicates
- Handling missing values (imputation/deletion)
- Standardizing formats
- Detecting and handling outliers
- Transform data through:
- Feature scaling
- Encoding categorical variables
- Date/time processing
- Custom transformations
- Aggregate data using:
- Multiple grouping levels
- Various aggregation functions (sum, mean, median, custom)
- Time-based windows
The Analysis Agent performs statistical, correlation, and regression analysis on the data.
Key Responsibilities:
- Perform statistical analysis.
- Calculate correlation coefficients.
- Perform regression analysis.
The Visualization Agent creates various types of charts (bar, line, pie) from the data.
Key Responsibilities:
- Create bar charts.
- Create line charts.
- Create pie charts.
The system utilizes a set of tools to perform specific data operations:
clean_data(tools/data_processing.py):
- Input: DataFrame, cleaning parameters
- Output: Cleaned DataFrame
- Supports:
- Multiple imputation strategies
- Outlier detection algorithms
- Custom cleaning rules
transform_data:
- Supports multiple transformation types:
- Numerical: StandardScaler, MinMaxScaler, RobustScaler
- Categorical: OneHotEncoding, LabelEncoding
- Text: TF-IDF, Word Embeddings
- Configurable via JSON transformation rules
stat_analysis: Performs statistical analysis on the dataset.correlation_analysis: Calculates correlation coefficients between variables.regression_analysis: Performs regression analysis on the dataset.
create_bar_chart: Creates a bar chart from the provided data.create_line_chart: Creates a line chart from the provided data.create_pie_chart: Creates a pie chart from the provided data.
The utils/execution.py script handles the execution of tools and the overall workflow of the system. It includes functions to execute specific tools based on the agent's response and to manage the conversation flow between the user and the agents.
Key Functions:
execute_tool: Executes specific tools based on the agent's response.handle_user_message: Manages the conversation flow and interacts with the agents to process user queries.
- Python 3.8 or higher
- OpenAI API Key with GPT-4 access
- Required Python packages:
- pandas >= 1.5.0
- numpy >= 1.21.0
- matplotlib >= 3.5.0
- seaborn >= 0.11.0
- scikit-learn >= 1.0.0
- openai >= 1.0.0
- python-dotenv >= 0.19.0
-
Clone the repository:
git clone https://github.com/yourusername/multi-agent-data-analysis.git cd multi-agent-data-analysis -
Create and activate a virtual environment:
python -m venv venv # On Windows .\venv\Scripts\activate # On Unix or MacOS source venv/bin/activate
-
Install the required packages:
pip install -r requirements.txt
-
Set up your OpenAI API Key:
- Create a
.envfile in the root directory - Add your configuration:
OPENAI_API_KEY=your_api_key_here OPENAI_MODEL=gpt-4-0314 LOG_LEVEL=INFO MAX_TOKENS=2000 TEMPERATURE=0.7
- Create a
To run the system, execute the main.py script: