Memory Augmented Agents

A sophisticated chat agent designed with long-term memory capabilities. It leverages a Recursive Language Model (RLM) for intelligent retrieval and a Reflection Agent for continuous memory consolidation.

🏗️ System Architecture

This project implements a modular, memory-augmented architecture:

graph TD
    User((User)) <--> Main["main.py<br/>(Orchestrator)"]

    subgraph "Agent Layer"
        Main --injects--> Chat[ChatAgent]
        Chat --calls--> Router[RouterAgent]
        Chat --calls--> Retriever[MemoryRetriever]

        Scheduler[APScheduler] --triggers--> Reflection[ReflectionAgent]
    end

    subgraph "External Services"
        Gemini["Gemini API<br/>(LLM)"]
        Docker["Docker Container<br/>(RLM Sandbox)"]
    end

    subgraph "Data Persistence"
        Postgres[("PostgreSQL<br/>Chat History")]
        MDKeys["Markdown Files<br/>(Long-term Memory)"]
    end

    %% Flows
    Router --Classify Query--> Gemini
    Gemini --Return Decision--> Router

    Retriever --Generate Retrieval Code--> Gemini
    Retriever --Execute Code--> Docker
    Docker --Read--> MDKeys
    Docker --Return Context--> Retriever

    Chat --Read/Write--> Postgres
    Reflection --Read--> Postgres
    Reflection --Update--> MDKeys

    Chat --Generate Response--> Gemini

Loading

Core Components

1. main.py (Orchestrator):

   • Acts as the Composition Root.
   • Initializes all agents (RouterAgent, MemoryRetriever, ChatAgent).
   • Runs the main chat loop and the background APScheduler for reflection.

2. ChatAgent:

   • Role: Manages conversation flow and context.
   • Dependencies: Injected with RouterAgent and MemoryRetriever.
   • State: Maintains short-term context (default 20 messages) and persists full history to PostgreSQL.

3. RouterAgent:

   • Role: Analyzes user queries to determine if memory retrieval is necessary.
   • Logic: Uses LLM to classify intent (e.g., "What is Python?" -> No Memory, "What was my last project?" -> Needs Memory).

4. MemoryRetriever:

   • Role: Retrieves relevant information from long-term memory.
   • RLM Implementation: Uses a Docker sandbox to safely execute Python code generated by the LLM to search, filter, and summarize memories from Markdown files.
   • Simple Implementation: Fallback keyword search if RLM is disabled.

5. ReflectionAgent:

   • Role: Runs periodically in the background.
   • Process: Analyzes recent chat history, extracts new facts/preferences, and updates the Markdown memory files in memory_service/memory/.

---

💾 Data Persistence

The system uses a Hybrid Memory Architecture:

• Structured Database (PostgreSQL): Stores the raw, immutable Chat Logs. Everything said is recorded here.
• Unstructured Knowledge (Markdown Files): Stores Distilled Memories in memory_service/memory/. These are human-readable files organized by category (e.g., personal_info.md, goals.md).

---

🚀 Setup & Installation

Prerequisites

• Python 3.10+
• Docker Desktop: Must be installed and running (required for RLM secure execution).
• PostgreSQL: Database for chat history.

Installation

1. Clone the repository:

   git clone https://github.com/staru09/mem_agents.git
   cd mem_agents

2. Create a virtual environment:

   python -m venv .venv
   # Windows
   .venv\Scripts\activate
   # Mac/Linux
   source .venv/bin/activate

3. Install dependencies:

   pip install -r requirements.txt

4. Environment Configuration: Create a .env file in memory_service/ (or project root depending on your config):

   GOOGLE_API_KEY=your_gemini_api_key
   DATABASE_URL=postgresql://user:password@localhost:5432/memory_service
   # Optional: RLM configuration
   RLM_DOCKER_WORKSPACE_DIR=.rlm_workspace

5. Database Setup: Start your PostgreSQL instance. If using Docker Compose (included):

   docker-compose up -d

---

🏃 Usage

Running the Agent

To start the interactive chat session:

python memory_service/main.py

How it works

1. Chat: Type your message.
2. Route: The system decides if it needs to look up info about you.
3. Retrieve: If needed, it spins up a Docker container to search your memory files.
4. Respond: It answers using the retrieved context.
5. Reflect: In the background, it learns new facts about you and saves them for next time.

---

📂 Project Structure

memory_service/
├── agents/
│   ├── chat_agent.py       # Main interaction logic
│   ├── router_agent.py     # Intent classification
│   ├── memory_retriever.py # RLM-based retrieval
│   └── reflection_agent.py # Memory consolidation
├── database/
│   ├── connection.py       # DB connection & session
│   ├── models.py           # SQLAlchemy models
│   └── repository.py       # Data access layer
├── memory/                 # Markdown memory files (generated)
├── main.py                 # Entry point
└── ...