Cable RFP Automation (Agentic AI System)

An autonomous multi-agent system designed to discover, analyze, and bid on government cable tenders. The system uses a Hub-and-Spoke Agentic Architecture to automate the end-to-end RFP process, from discovery to final bid generation.

---

Pipeline Execution Demo

---

System Architecture

The system is built on a Main Agent architecture that coordinates specialized sub-agents.

graph TD
    %% Styling Definitions
    classDef mainAgent fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#01579b;
    classDef subAgent fill:#fff3e0,stroke:#e65100,stroke-width:2px,color:#e65100;
    classDef source fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#2e7d32;
    classDef output fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px,color:#7b1fa2;
    classDef user fill:#eeeeee,stroke:#616161,stroke-width:2px,color:#212121;

    %% Nodes
    User(["User"]) -- "Start Pipeline" --> Main{{"Main Agent (Coordinator)"}}:::mainAgent
    
    subgraph "Stage 1: Discovery"
        Crawlers["Official Sources Crawler"]:::subAgent
        NTPC["NTPC Portal"]:::source
        PG["POWERGRID PRANIT"]:::source
        SEB["State SEBs"]:::source
        
        Crawlers -- "Scrapes" --> NTPC
        Crawlers -- "Scrapes" --> PG
        Crawlers -- "Scrapes" --> SEB
        Crawlers -- "Returns Tenders" --> Main
    end
    
    subgraph "Stage 2: Processing"
        Main -- "1. Filter & Select" --> Sales["Sales Agent"]:::subAgent
        Sales -- "Selected RFP" --> Main
        
        Main -- "2. Tech Specs" --> Tech["Technical Agent"]:::subAgent
        Tech -- "Product Matches" --> Main
        
        Main -- "3. Costing" --> Price["Pricing Agent"]:::subAgent
        Price -- "Final Quote" --> Main
    end
    
    Main -- "4. Decision (BID/NO-BID)" --> Output["Final Decision & Bid Pack"]:::output

Loading

---

Project Structure

cable-rfp-automation/
├── agents/                 # Multi-agent system
│   ├── main_agent.py       # Hub-and-spoke orchestrator
│   ├── sales_agent.py      # RFP discovery & qualification
│   ├── technical_agent_enhanced.py  # SKU matching engine
│   └── pricing_agent_enhanced.py    # BOQ & quote generation
├── crawlers/               # Government portal scrapers
│   ├── official_sources_only.py     # Main aggregator
│   ├── ntpc_crawler.py     # NTPC eProcurement
│   ├── powergrid_crawler.py # POWERGRID PRANIT
│   ├── cppp_crawler.py     # Central Public Procurement Portal
│   └── state_seb_crawler.py # State Electricity Boards
├── utils/                  # Notification & utilities
│   ├── whatsapp_notifier.py
│   ├── telegram_notifier.py
│   └── email_notifier.py
├── config/                 # Configuration & policies
│   ├── settings.py         # Pydantic settings
│   └── client_policy.json  # Business rules
├── database/               # SQLAlchemy models
├── tests/                  # Test suite
├── run_pipeline_new.py     # Main entry point
├── run_monthly_scheduler.py # Automated scheduling
└── Dockerfile              # Container support

---

Agents & Capabilities

1. Main Agent (The Brain)

• Role: Orchestrator & Decision Maker
• Function: Coordinates data flow between agents, consolidates results, and calculates Win Probability.
• Logic:
  • Win Probability > 50% → BID
  • Win Probability < 50% → NO-BID

2. Sales Agent

• Role: Gatekeeper
• Function: Filters tenders based on:
  • Deadlines: Ensures tender is active (next 3-60 days).
  • Relevance: Checks for cable-specific keywords (XLPE, Power Cable, etc.).
• Output: Selects the most promising RFP for processing.

3. Technical Agent (Enhanced)

• Role: Engineer
• Function:
  • Parses complex technical specifications from tender documents.
  • Matches requirements against an OEM Product Catalog.
  • Handles Gap Analysis (e.g., matching 11kV requirements to 11kV products).
  • Generates "New SKU Requests" if no suitable product exists.

4. Pricing Agent

• Role: Commercial Lead
• Function:
  • Calculates Material Costs (Cable lengths * Unit rates).
  • Estimates Service Costs (Transport, Installation, Testing).
  • Adds Margins & Taxes (GST, Contingency).
  • Generates the final Bill of Quantities (BOQ).

---

Crawler System

The system includes robust crawlers for major Indian government e-procurement portals.

Source	Feature	URL	resilience
NTPC	eProcurement	eprocurentpc.nic.in	Proxy Supported
POWERGRID	PRANIT Portal	eprocure.powergrid.in	Proxy Supported
State SEBs	Maharashtra, etc.	mahadiscom.in	Auto-Retry

• Status Logging: Tracks successful/failed scrapes in output/crawler_status.json.
• Proxy Support: Configurable via .env for accessing geo-restricted portals.

---

Tech Stack

Category	Technologies
Language	Python 3.11+
Architecture	Multi-Agent (Hub-and-Spoke)
Web Scraping	Requests, BeautifulSoup4, aiohttp
Database	PostgreSQL, SQLAlchemy, Alembic
Notifications	Twilio (WhatsApp), Telegram Bot API, SMTP
Scheduling	APScheduler
Containerization	Docker
Configuration	Pydantic Settings, python-dotenv

---

How to Run

Prerequisites

• Python 3.11+
• Git

1. Clone the Repository

git clone https://github.com/somuai/cable-rfp-automation.git
cd cable-rfp-automation

2. Setup Environment

Rename the template file and configure your settings:

cp .env.example .env

Edit .env to add your keys (optional) and Proxy settings if needed.

3. Install Dependencies

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

4. Run the Pipeline

Execute the main pipeline script:

python run_pipeline_new.py

Outputs

Results are generated in the output/ directory:

• pipeline_results_new.json: Complete JSON dump of the process.
• crawler_status.json: Health check of all crawlers.
• new_sku_requests/: Markdown files for any custom manufacturing requests.

Sample Output

Sample Pipeline Output (Click to expand)

Discovery: 113 tenders from 7 sources (Seed URLs: 70, CPPP: 40, State SEBs: 3)

Sales Agent:
  Filtered (3-month deadline): 1
  Qualified (cable-related): 1
  Selected: NTPC-495832 (11kV Di-Electric Coating)

Technical Agent:
  Top Match: OEM-XLPE-11KV-3C-185 (71.43% match)
  3 OEM SKUs compared

Pricing Agent:
  Material Cost: ₹8,50,000
  Services Cost: ₹13,000  
  Final Quote: ₹12,20,990 (incl. GST)

Decision: BID (Win Probability: 55%)
Execution Time: 30.88 seconds

---

Decision Logic

The BID / NO-BID decision is based on a weighted probability algorithm:

1. Base Score: 50%
2. Product Match:
   • Match > 70%: +5% (Qualifies for BID)
   • Match < 70%: -5% (Rejects as NO-BID)
3. Price Competitiveness (if estimated value known):
   • Quote within range: +20%
4. Urgency:
   • Urgent (<30 days): +10%

---

Production Guardrails & Evaluation Loops

To ensure deterministic, audit-ready, and compliant outcomes for government procurement bids, the pipeline enforces guardrails across five operational stages:

1. Ingestion & Network Guardrails

• Circuit Breaker Gate (utils/source_gate.py, utils/reachability.py): Pre-checks host DNS and connectivity with TTL caching; automatically trips and falls back if government portals experience downtime or TLS handshake errors.
• Domain Relevance Gate (utils/cable_detector.py): Filters incoming raw text with keyword matching across voltage ratings, insulation, and conductor materials; discards noise with scores below the 0.30 threshold.
• Deduplication Gate (database/models.py): Enforces Tender-ID and payload uniqueness in PostgreSQL to avoid redundant compute and duplicate notifications.

2. Sales Qualification & Capacity Guardrails

• Submission Window Enclosure: Restricts active bidding candidates to a 90-day submission horizon (current_date <= deadline <= current_date + 90 days).
• Multi-Factor Scoring Policy (config/client_policy.json): Requires a composite qualification score >= 0.60 across Product Coverage (40%), Deadline (25%), Past Experience (20%), and Estimated Value (15%).
• Rotation Anti-Starvation Gate (output/last_selected.json): Tracks previously selected tenders across scheduled runs to prevent repetitive processing of the same RFP.
• Manufacturing Capability Ceiling: Enforces plant limits across supported voltage ratings (11 kV, 22 kV, 33 kV, 440V) and quarterly production length caps (1,500 km).

3. Technical Compliance & Engineering Escalation

• 7-Parameter Strict Spec Normalizer (utils/standards_normalizer.py): Normalizes voltage (with +-1 kV tolerance), insulation type (XLPE, PVC, EPR), conductor material, core counts, and standards (IS 7098, IEC 60502).
• Made-to-Order (MTO) Engineering Escalation: If the top catalogue match is below the 80% policy threshold, the system halts auto-quote and auto-generates a structured Made-to-Order New SKU Request (output/new_sku_requests/) with detailed parameter gap analysis for human engineering sign-off.

4. Commercial & Pricing Guardrails

• Raw Material Index Linking (agents/pricing_agent_enhanced.py): Links unit material costs to live copper and aluminum commodity multipliers.
• Profit Margin Boundary Enclosure: Hard clamps profit margins between a 4% safety floor and an 18% competitiveness ceiling.
• Strategic PSU Account Policies: Restricts margin discounting (1% to 2% delta) exclusively to designated public sector clients (POWERGRID, NTPC, Indian Railways).
• Statutory Compliance: Enforces mandatory routine/type testing fees and statutory 18% GST calculation.

5. Decision & Evaluation Loops

• Win Probability Boundary Clamping: Clamps final win probabilities strictly to [0.0%, 100.0%].
• Integrity Gate: Requires both quoted_value > 0 and Win Probability >= 50.0% for a BID recommendation; otherwise defaults to NO-BID.
• Automated Innovation Verification (verify_innovations.py): Built-in test harness verifying scheduler health, multi-source ingestion, spec scoring, and pricing calculation consistency.
• Continuous Integration Suite (tests/test_agents.py): 31 automated unit tests run via GitHub Actions on every push and pull request.

---

Notifications System

The system includes automated notifications via WhatsApp, Telegram, and Email.

1. WhatsApp Setup (Twilio)

1. Create Account: Twilio Console
2. Enable Sandbox: Messaging → Try it Out → WhatsApp
3. Configure:

   TWILIO_ACCOUNT_SID=your_sid
   TWILIO_AUTH_TOKEN=your_token
   TWILIO_WHATSAPP_FROM=whatsapp:+14155238886
   WHATSAPP_TO=whatsapp:+919876543210

2. Telegram Setup

1. Create Bot: Chat with @BotFather and send /newbot.
2. Get Chat ID: chat with your new bot or use @userinfobot.
3. Configure:

   TELEGRAM_BOT_TOKEN=123456:ABC-DEF1234ghIkl-zyx57W2v1u123ew11
   TELEGRAM_CHAT_ID=123456789

3. Email Setup (SMTP)

Configure any SMTP server (Gmail, Outlook, AWS SES). For Gmail, use App Passwords.

SMTP_SERVER=smtp.gmail.com
SMTP_PORT=587
SMTP_USERNAME=your_email@gmail.com
SMTP_PASSWORD=your_app_password
EMAIL_FROM=your_email@gmail.com
EMAIL_TO=recipient1@example.com,recipient2@example.com

Usage Options

Option 1: Run Pipeline with Specific Notifications

python run_pipeline_new.py --whatsapp --telegram --email

Option 2: Send Last Results (All Channels)

python run_monthly_scheduler.py --send-last

Option 3: Monthly Scheduled Automation The scheduler sends to ALL configured channels automatically.

python run_monthly_scheduler.py --day 1 --hour 9

Message Format

• WhatsApp/Telegram: Condensed text summary with key metrics.
• Email: Rich HTML report with tables, colors, and detailed breakdown.

---

Security

• .gitignore: Configured to exclude all sensitive data and outputs.
• Environment Variables: All secrets managed via .env.
• Crawler resilience: Handles SSL errors and proxies securely.

License

This project is proprietary software developed for Asian Paints subsidiary. All rights reserved.