full pipeline of llm

[yiftachashkenazi]

Complete Pipeline Flow with LLM Output Routing

flowchart TD
    A[Start Pipeline] --> B["1. Verify RAG Sources Indexed"]
    B --> C["2. Extract CFP Data from RAG Sources"]

    C --> C1["2.1 Get RAG Sources Data"]
    C1 --> C2["2.2 Format Sources for Prompt"]
    C2 --> C3["2.3 LLM Call - Extract CFP Data"]
    C3 --> C4["2.4 Cache Result"]
    C4 --> C5["2.5 Parse Organization & Date"]

    C5 --> D["3. Extract and Enrich Sections"]

    D --> D1["3.1 Handle Extract Sections"]
    D1 --> D11["3.1.1 Retrieve Organization Guidelines"]
    D11 --> D12["3.1.2 LLM Call - Extract Sections"]
    D12 --> D13["3.1.3 Filter Sections by Embeddings"]

    D1 --> D2["3.2 Handle Generate Grant Template"]
    D2 --> D21["3.2.1 Retrieve Organization Guidelines"]
    D21 --> D22["3.2.2 LLM Call - Generate Metadata"]
    D22 --> D23["3.2.3 Validate Section Dependencies"]

    D --> D3["3.3 Merge Sections with Metadata"]

    D3 --> E["4. Save to Database"]
    E --> E1["4.1 Update GrantTemplate Record"]
    E1 --> E2["4.2 Commit Transaction"]
    E2 --> F[Pipeline Complete]

    %% LLM Input Data
    RAG_DATA[("📄 RAG Sources Data<br/>• Full text content<br/>• Key chunks<br/>• Source types<br/>• Organization mapping")] -->|Input| C3

    SECTION_INPUT[("📝 Section Input<br/>• CFP content string<br/>• CFP subject<br/>• Organization guidelines<br/>• Exclude categories list")] -->|Input| D12

    META_INPUT[("⚙️ Metadata Input<br/>• Long-form sections<br/>• CFP content<br/>• CFP subject<br/>• Organization guidelines")] -->|Input| D22

    %% LLM Outputs and Their Destinations

    %% LLM #1 Output - Goes to multiple places
    C3 --> CFP_OUTPUT[("📋 CFP Structure<br/>• organization_id<br/>• cfp_subject<br/>• submission_date<br/>• content sections<br/>• titles & subtitles")]
    CFP_OUTPUT --> C5
    CFP_OUTPUT --> ORG_PARSE["3.1.1 Organization Detection"]
    CFP_OUTPUT --> DATE_PARSE["3.2.1 Date Parsing"]
    CFP_OUTPUT --> CONTENT_CONCAT["3.1.2 Content Concatenation"]

    %% LLM #2 Output - Goes to filtering and merging
    D12 --> SECTIONS_OUTPUT[("🏗️ Section Structure<br/>• Section hierarchy<br/>• IDs & titles<br/>• Parent-child relations<br/>• Research plan flags<br/>• Long-form markers")]
    SECTIONS_OUTPUT --> D13
    SECTIONS_OUTPUT --> FILTERED_SECTIONS["3.1.3 Filtered Sections"]
    FILTERED_SECTIONS --> D2
    FILTERED_SECTIONS --> LONGFORM_FILTER["3.2.1 Long-form Section Filter"]

    %% LLM #3 Output - Goes to merging process
    D22 --> META_OUTPUT[("📊 Section Metadata<br/>• Word counts<br/>• Keywords list<br/>• Topics list<br/>• Generation instructions<br/>• Search queries<br/>• Dependencies")]
    META_OUTPUT --> D23
    META_OUTPUT --> VALIDATED_META["3.2.3 Validated Metadata"]
    VALIDATED_META --> D3

    %% Final merging process
    FILTERED_SECTIONS --> MERGE_PROCESS["3.3 Section + Metadata Merge"]
    VALIDATED_META --> MERGE_PROCESS
    ORG_PARSE --> DB_ORG["4.1 Organization Field"]
    DATE_PARSE --> DB_DATE["4.1 Submission Date Field"]
    MERGE_PROCESS --> GRANT_SECTIONS["4.1 Grant Sections Array"]

    %% Final Database Structure
    DB_ORG --> E1
    DB_DATE --> E1
    GRANT_SECTIONS --> E1
    E1 --> FINAL_OUTPUT[("💾 Final Grant Template<br/>• granting_institution_id<br/>• submission_date<br/>• grant_sections[]<br/>  - GrantElement (title-only)<br/>  - GrantLongFormSection (full)")]

    %% LLM Call Details with output routing
    C3 -.-> LLM1["🤖 LLM Call #1<br/>Stage: 2.3<br/>Model: Gemini/Sonnet<br/>Prompt: extract_cfp_data_multi_source<br/><br/>📥 Input:<br/>• RAG sources (formatted)<br/>• Organization mapping<br/><br/>📤 Output goes to:<br/>• Organization parsing (C5)<br/>• Date parsing (C5)<br/>• Section input (D1)<br/>• Content concatenation (D12)"]
    
    LLM1 ===>|Output| CFP_OUTPUT

    D12 -.-> LLM2["🤖 LLM Call #2<br/>Stage: 3.1.2<br/>Model: Claude Sonnet<br/>Prompt: section_extraction<br/><br/>📥 Input:<br/>• CFP content (from LLM #1)<br/>• Organization guidelines<br/><br/>📤 Output goes to:<br/>• Embedding filter (D13)<br/>• Long-form filter (D2)<br/>• Section merger (D3)"]
    
    LLM2 ===>|Output| SECTIONS_OUTPUT

    D22 -.-> LLM3["🤖 LLM Call #3<br/>Stage: 3.2.2<br/>Model: Gemini<br/>Prompt: grant_template_generation<br/><br/>📥 Input:<br/>• Filtered sections (from LLM #2)<br/>• CFP content (from LLM #1)<br/><br/>📤 Output goes to:<br/>• Dependency validation (D23)<br/>• Section merger (D3)<br/>• Final grant sections array (E1)"]
    
    LLM3 ===>|Output| META_OUTPUT

    %% Data flow connections with labels
    CONTENT_CONCAT -.->|"content"| SECTION_INPUT
    LONGFORM_FILTER -.->|"long_form_sections"| META_INPUT

    %% Cross-LLM dependencies
    CFP_OUTPUT -.->|"feeds into"| LLM2
    SECTIONS_OUTPUT -.->|"feeds into"| LLM3

    %% Styling
    classDef llmNode fill:#e1f5fe,stroke:#0277bd,stroke-width:3px
    classDef mainPhase fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
    classDef subPhase fill:#f1f8e9,stroke:#388e3c,stroke-width:1px
    classDef dataNode fill:#fff3e0,stroke:#f57c00,stroke-width:2px
    classDef outputNode fill:#e8f5e8,stroke:#2e7d32,stroke-width:2px
    classDef processNode fill:#fce4ec,stroke:#c2185b,stroke-width:1px
    classDef finalNode fill:#e3f2fd,stroke:#1565c0,stroke-width:3px

    class LLM1,LLM2,LLM3 llmNode
    class A,B,C,D,E,F mainPhase
    class C1,C2,C3,C4,C5,D1,D2,D3,D11,D12,D13,D21,D22,D23,E1,E2 subPhase
    class RAG_DATA,SECTION_INPUT,META_INPUT dataNode
    class CFP_OUTPUT,SECTIONS_OUTPUT,META_OUTPUT outputNode
    class ORG_PARSE,DATE_PARSE,CONTENT_CONCAT,FILTERED_SECTIONS,LONGFORM_FILTER,VALIDATED_META,MERGE_PROCESS,DB_ORG,DB_DATE,GRANT_SECTIONS processNode
    class FINAL_OUTPUT finalNode

Loading

[yiftachashkenazi]

@Goldziher do you want to share this with other team members?

[yiftachashkenazi]

I am thinking to build a collector that see all test input and output and ai agent that can help us, to find our way in it. it might save us time and make our work better.