feat(knowledge): add OpenSearch as a first-class knowledge backend

[rhossi]

Summary

Adds OpenSearch as a built-in knowledge backend in AIQ 2.0, alongside the
existing LlamaIndex (ChromaDB) and Foundational RAG (Milvus) options.
Selectable through the standard backend: opensearch field in workflow YAML;
no fork, no custom base image.

OpenSearch is broadly available — self-hosted, and as a managed offering on
every major cloud (Amazon OpenSearch Service, Amazon OpenSearch Serverless,
others). Until now, customers running OpenSearch had to write their own
BaseRetriever/BaseIngestor adapter, rebuild base images to include it,
and handle their environment's authentication themselves — including inside
the Dask workers during ingestion. This PR removes that burden and
establishes a plugin pattern that other native backends can follow.

What is included

• OpenSearchRetriever + OpenSearchIngestor extending BaseRetriever /
  BaseIngestor, registered through the knowledge-layer factory, exposed as
  a NAT function (backend: opensearch in YAML).
• Three auth modes: none (development clusters), basic (self-hosted
  with username/password), sigv4 (Amazon OpenSearch Service via
  aws_service: es, Amazon OpenSearch Serverless via aws_service: aoss).
  Pluggable enough for additional auth schemes to slot in as new Literal
  values without adapter rewrites.
• Authentication inside the Dask worker process: the ingestion task
  constructs its own OpenSearch client inside each worker, so credentials
  resolve from the worker's environment — works with EKS Pod Identity,
  IRSA, SSO, env profiles, and instance profiles. No signer state is
  serialized across the cluster.
• Session isolation: AIQ collections map to per-session OpenSearch
  indexes (<prefix>-s_<uuid>) inside a single persistent OpenSearch
  cluster, with 24-hour TTL cleanup driven by _meta.updated_at. Lets
  multiple sessions share one OpenSearch resource without cross-session
  leakage.
• Helm chart override surface: example values at
  deploy/helm/examples/aws-opensearch-serverless-values.yaml shows the
  image, imagePullSecrets, and secretEnv shape for a managed-serverless
  EKS deployment. The chart's existing schema accepts the override directly —
  no chart fork required to point at a custom image or environment.
• Reference workflow YAML: configs/config_web_opensearch.yml
  (env-substitution driven; the same file works for self-hosted, managed,
  and serverless OpenSearch backends).
• Reference deployment guide:
  docs/source/deployment/aws-opensearch-serverless.md — end-to-end
  walkthrough for one concrete cloud (Amazon OpenSearch Serverless on EKS
  with Pod Identity), covering architecture, prerequisites, OpenSearch
  collection setup, IAM trust policy, data access policy, Pod Identity
  association, image pull secret, embedding endpoint, verification, and
  cleanup. The same pattern adapts to other cloud or self-hosted OpenSearch
  deployments.

Test plan

All paths validated against a real Amazon OpenSearch Serverless collection
(SigV4 via assumed role) prior to PR submission:

• Adapter unit tests (tests/knowledge_layer_tests/test_opensearch_adapter.py) — 19 passed.
• AOSS live test suite (tests/knowledge_layer_tests/test_opensearch_serverless_live.py) —
  2 passed in 160s. Covers SigV4 health, collection lifecycle, vector
  ingest, k-NN retrieval, filtered k-NN, and AOSS-aware delete batching.
• Reference YAML loads cleanly under nat serve against a live
  OpenSearch endpoint.
• Local-mode ingestion: POST /v1/collections/<name>/documents → embed
  → bulk index → status completed. Verified via direct doc-count
  query.
• Dask-mode ingestion: separate scheduler + worker; _bulk issued from
  the worker process (verified absent from backend log). Authentication
  resolved inside the worker.
• File deletion: search-then-bulk-delete pattern with eventual visibility
  wait. Doc count drops to 0 in the target index; other indexes
  unaffected.
• Real PDF ingestion: 30-page paper → 30 chunks via pypdf → multi-batch
  embedding → bulk index in 5.3s. Page-level citations preserved through
  retrieval.
• UI flow with session-bound collection: <prefix>-s_<uuid> index
  appears in OpenSearch alongside API-driven collections.
• Sphinx docs build under strict mode (SPHINXOPTS="-W --keep-going -n")
  — zero warnings.
• Reference YAML and helm example values both parse cleanly.

CONTRIBUTING.md compliance

• All 26 commits carry Signed-off-by: trailers per the DCO requirement
  (git rebase --signoff across the branch).
• Manual testing complete per §4 (no CI/CD in this repo today).

[greptile-apps]

Greptile Summary

This PR adds OpenSearch as a first-class knowledge backend alongside the existing LlamaIndex and Foundational RAG options, implementing OpenSearchRetriever and OpenSearchIngestor with three auth modes (none, basic, SigV4), Dask-based distributed ingestion, composite-pagination file listing, and a TTL cleanup mechanism for session-bound indexes.

• Core adapter (sources/knowledge_layer/src/opensearch/adapter.py): 1 553-line implementation covering index lifecycle, chunking, embedding, bulk indexing, AOSS-aware delete, and composite-aggregation-based file listing. All five issues flagged in the previous review round are resolved.
• Distributed worker (distributed.py): Dask worker entry point that reconstructs its own OpenSearchIngestor inside the worker so SigV4 credentials resolve from the worker environment.
• Config & registration (register.py, configs/config_web_opensearch.yml, deploy/helm/examples/): Extends KnowledgeRetrievalConfig with ~25 new OpenSearch fields backed by env vars, adds the backend to the factory, and ships a reference YAML and Helm values example.

Confidence Score: 5/5

Safe to merge. All five issues raised in the previous review round are resolved and the new code introduces no correctness regressions or data-safety concerns.

The adapter core paths are implemented correctly and backed by a 1100+ line unit test suite. Remaining findings are efficiency and consistency suggestions with no impact on correctness.

The _collection_info_from_index method in adapter.py and validate_backend_config in register.py are worth a second look before the feature sees heavy multi-tenant load.

Important Files Changed

Filename	Overview
sources/knowledge_layer/src/opensearch/adapter.py	Core 1553-line adapter. All previously-reviewed P1 issues are resolved. N+1 calls in list_collections and per-call OpenAI client allocation are the remaining observations.
sources/knowledge_layer/src/opensearch/distributed.py	Dask worker entry point. Correctly reconstructs its own ingestor so credentials resolve locally; temp files cleaned up in finally.
sources/knowledge_layer/src/register.py	Adds ~25 OpenSearch fields to the shared KnowledgeRetrievalConfig. Missing cross-backend warnings for embed_model/embed_base_url.
configs/config_web_opensearch.yml	Reference YAML using env-substitution for all sensitive values. Clean and portable.
tests/knowledge_layer_tests/test_opensearch_adapter.py	Comprehensive 1132-line unit test suite with fake OpenSearch clients covering all major code paths.

Sequence Diagram

sequenceDiagram
    participant Client as API Client
    participant Ingestor as OpenSearchIngestor
    participant Dask as Dask Worker
    participant OS as OpenSearch / AOSS

    Client->>Ingestor: submit_job(file_paths, collection)
    Ingestor->>Ingestor: _ensure_index (race-safe)
    alt Local mode
        Ingestor->>Ingestor: _start_local_ingestion (thread)
        Ingestor-->>Client: job_id (immediate)
        Ingestor->>OS: bulk index chunks
    else Dask mode
        Ingestor->>Dask: client.submit(run_opensearch_ingestion_task)
        Ingestor-->>Client: job_id (immediate)
        Ingestor->>Ingestor: _monitor_dask_ingestion (thread)
        Dask->>Dask: reconstruct OpenSearchIngestor
        Dask->>OS: bulk index chunks (SigV4 resolved in worker)
        Dask-->>Ingestor: result dict
        Ingestor->>Ingestor: _apply_dask_ingestion_result
    end

    Client->>Ingestor: retrieve(query, collection)
    Ingestor->>OS: knn_vector search + filter
    OS-->>Ingestor: hits
    Ingestor-->>Client: RetrievalResult (Chunks)

    loop TTL cleanup (every 1h)
        Ingestor->>OS: list_collections
        OS-->>Ingestor: collections + updated_at
        Ingestor->>OS: delete_collection (expired indexes)
    end

Loading

_{Reviews (8): Last reviewed commit: "fix(opensearch): stop indexing internal ..." | Re-trigger Greptile}

[AjayThorve]

Thank you @rhossi , we will review this soon