use @Once for Singletons

in genai_tk/utils/nemo_relay_setup.py, genai_tk/utils/ladybug/shared.py, genai_graph/kg/ingest/merge.py,

GenAI Toolkit (`genai-tk`)

A toolkit for building Gen AI and Agentic applications with LangChain, LangGraph, and 100+ LLM providers.

See also the great DeepWiki generated documentation.

Three Domains

The toolkit is organized around three complementary domains:

🧠 Core GenAI

Build intelligent applications with multi-provider LLM/embeddings support and state management.

Multi-provider LLM and embeddings factory (OpenAI, Groq, Anthropic, Ollama, local, …)
Vector stores: Chroma, PgVector, in-memory
LLM caching, prompt templates, structured output
See: cli core llm, cli info models, docs/core.md

🔤 NLP

Centralized spaCy and NLP functionality for PII detection, anonymization, BM25 preprocessing, and text classification.

PII detection with Presidio + spaCy (English, French, and more)
Reversible anonymization with Faker — shared by agent middleware and Prefect ETL flows
Configurable sensitivity scoring (regex + keywords + Presidio + heuristics)
BM25 lemmatization preprocessing, multi-language model management
See: docs/nlp.md, docs/middleware-pii-and-routing.md

🤖 Agents

Three agent frameworks (ReAct, Deep, DeerFlow) sharing YAML profiles, LLM factory, tools, and Docker sandbox — unified behind a shared harness layer (agents.harness) for CLI and UI.

ReAct — standard Thought → Action → Observation loop (LangChain)
Deep agent — multi-step planning + subagent delegation (LangChain / DeepAgents SDK)
DeerFlow — native web research, multi-agent orchestration (LangGraph / ByteDance)
Harness layer — one BaseHarness interface + one event model across LangChain and DeerFlow; cli agents run/cli agents list work across both
Skills system — SKILL.md domain-knowledge files loaded on demand; managed with cli skills
Docker sandbox — isolated execution, browser automation
MCP servers — protocol-standard tool integration
See: cli agents, docs/agents.md, AGENTS.md

⚙️ Workflows

Orchestrate multi-step AI pipelines with Prefect and a YAML DSL for composable, reusable workflows.

Workflow DSL — YAML-configured steps, dependencies, and sub-workflows (no Python needed)
Prefect server — explicit local server managed via cli prefect start/stop/status; auto-starts before workflow runs
Document pipelines — markdownize, OCR, PDF extraction, chunking
RAG pipeline — full retrieval pipeline with BM25 + dense hybrid search
Structured extraction — BAML-based extraction with type-safe output
See: cli workflow, cli prefect, docs/workflows.md, docs/prefect.md

📊 Monitoring & Observability

Built-in tracing for all LLM calls, agents, and workflows across multiple observability backends.

Trajectory store (ATOF / NeMo Relay) — local, structured, agent-readable record of every Deep Agents run (scope tree, tool args/results, skill loads, token usage); inspect with cli trajectory
Multi-backend support — LangSmith, LangFuse (cloud or self-hosted), OpenTelemetry, local JSONL
State management — .genai_tk file tracks which backends are active
Trace URL opening — cli monitoring open --trace fetches and opens the latest trace in your browser
JSONL logging — local file-based trace log (always on, no external service required)
Docker service control — just langfuse-server-start/stop for self-hosted LangFuse
See: cli monitoring, cli trajectory, docs/monitoring.md, docs/trajectory.md

What it gives you:

YAML-configured profiles — swap models, tools, MCP servers, and sandboxes without code changes
Rich CLI that mirrors every capability; easily extensible with one class + one YAML line

Installation

Start from scratch:

# Install uv (Python package manager)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Create a new project directory
mkdir my-genai-project && cd my-genai-project

# Initialize a Python project (creates pyproject.toml and a stub main.py)
uv init

# Add genai-tk to your project
uv add git+https://github.com/tclatos/genai-tk@main

# Interactive template picker (recommended)
# cli init scaffolds config/, skills/, justfile, sets hatchling as the build
# backend, removes the uv-init stub (main.py), and runs `uv sync` automatically.
uv run cli init

# Or choose a template directly:
uv run cli init -t agent-app  --name "My AI Project"   # tools, skills, agent profiles
uv run cli init -t rag-app    --name "My RAG App"       # document ingestion + retrieval
uv run cli init -t workflow-app                         # YAML-driven pipelines
uv run cli init -t minimal                              # config + justfile only

# (optional) also install the DeerFlow / DeepAgents harness
uv run cli init --extra harnessing

just run                                                # start the application

Add to existing project:

# Add to your project  
uv add git+https://github.com/tclatos/genai-tk@main

# With PostgreSQL / Playwright extras
uv add "genai-tk[extra] @ git+https://github.com/tclatos/genai-tk@main"

# Initialize config in the current directory
# (scaffolds config/, skills/, justfile, hatchling build backend, runs uv sync)
uv run cli init                      # interactive template picker
uv run cli init -t agent-app         # agent app with tools + skills
uv run cli init --extra harnessing   # also install the DeerFlow / DeepAgents harness

Development (clone & edit):

git clone https://github.com/tclatos/genai-tk.git && cd genai-tk
uv sync              # core + dev
uv sync --all-groups # + postgres, browser, evals

Using your clone in another project: add it as a local path dependency so changes are picked up immediately:
uv add --editable /path/to/genai-tk

Add your API key to .env in the project root (auto-loaded at startup):

OPENAI_API_KEY=sk-...
# ANTHROPIC_API_KEY=...  GROQ_API_KEY=...  etc.

Quick Start by Domain

🧠 Core GenAI

# Call any LLM
uv run cli core llm -i "tell me a joke" -m gpt-4o-mini@openai --stream

# List available models (1000+ supported)
uv run cli info models

# Show active config and API key status
uv run cli info config

Python:

from genai_tk.core.factories import get_llm

llm = get_llm()  # uses default from config
response = llm.invoke("Tell me a joke")
print(response.content)

🤖 Agents

# ReAct agent — interactive chat
uv run cli agents run --chat

# Single query with a specific profile (by KEY)
uv run cli agents run coding "Explain async generators"

# Show all agent profiles and frameworks
uv run cli agents list

Python:

from genai_tk.agents.langchain import LangchainAgent

agent = LangchainAgent("Research")
result = agent.run("What is the latest AI news?")
print(result)

⚙️ Workflows

# List available workflow profiles
uv run cli workflow list profiles

# Show the execution plan (dry-run)
uv run cli workflow run markdownize_docs --dry-run

# Execute a workflow
uv run cli workflow run markdownize_docs

YAML:

# config/workflows.yaml
workflows:
  my_pipeline:
    steps:
      - id: extract_pdf
        uses: genai_tk.workflow.prefect.flows.pdf_to_markdown_flow
        inputs:
          input_dir: "${paths.pdfs}"
          output_dir: "${paths.markdown}"

Then: uv run cli workflow run my_pipeline

📊 Monitoring & Observability

# Check monitoring status (config + enabled backends)
uv run cli monitoring status

# Enable monitoring (writes .genai_tk state file)
uv run cli monitoring start langfuse,local

# Start self-hosted LangFuse (Docker Compose)
just langfuse-server-start

# Open latest trace in browser
uv run cli monitoring open langfuse --trace

# View local trace log
uv run cli monitoring tail --n 50

Python:

from genai_tk.utils.tracing import setup_monitoring, get_monitoring_callbacks

setup_monitoring()  # Initialize all active backends
callbacks = get_monitoring_callbacks()

# Pass callbacks to LLM invocation
llm.invoke("Hello", config={"callbacks": callbacks})

More Examples

CLI Reference

After completing installation and running uv run cli init, explore the full CLI:

# Discover all available commands
uv run cli --help

# Inspect a specific model profile
uv run cli info llm-profile gpt-4o-mini

# Test with a fake model (no API key needed)
uv run cli core llm -i "tell me a joke" -m parrot_local@fake

# Try the generated example commands (after uv sync)
uv run cli example joke "software engineers"   # simple LLM call
uv run cli example agent "What is 2 + 2?"     # ReAct agent with tools

See docs/cli.md for the full command reference.

First steps — Web UI

Test agents interactively without code:

just webapp          # launches Streamlit on http://localhost:8501

One built-in agent demo page is included in an Agents section:

🤖 Agent — unified 2-panel trace + chat, artifacts, streaming; st.pills filter by kind (React, DeepAgent, DeerFlow) + profile picker across both harnesses

Downstream projects (like genai-blueprint) can embed this page in their own navigation using the genai_tk:// prefix — no copy-pasting, no wrappers:

# config/app_conf.yaml in your project
ui:
  pages_dir: myapp/webapp/pages
  navigation:
    agents:
      - genai_tk://demos/agent.py              # served from the installed package
    demos:
      - demos/my_custom_page.py               # your own page

See docs/webapp.md for configuration, cross-package navigation, custom pages, and running from a new project via cli init --name "My Project".

Monitoring and Observability

Track LLM calls, agent steps, and pipeline execution across LangSmith, LangFuse, OpenTelemetry, and local JSONL logs.

# Check monitoring status
uv run cli monitoring status

# View local trace log (most recent first)
uv run cli monitoring tail                    # last 20 entries
uv run cli monitoring tail --n 50 --json      # raw JSON for piping

# Start self-hosted LangFuse
uv run cli monitoring start langfuse
uv run cli monitoring open langfuse

Configuration — YAML aliases make it easy to switch between cloud and self-hosted:

monitoring:
  _langfuse_cloud: &langfuse_cloud
    host: https://cloud.langfuse.com
    public_key: ${oc.env:LANGFUSE_PUBLIC_KEY,""}
    secret_key: ${oc.env:LANGFUSE_SECRET_KEY,""}

  backends: [langfuse, local]      # Multiple backends active in parallel
  project: MyProject
  langfuse: *langfuse_cloud        # Change to *langfuse_local for docker-compose
  local_log:
    path: ${paths.data_root}/traces/llm_calls.jsonl
    include_prompts: true

Set API keys in ~/.env:

LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGSMITH_API_KEY=...

See docs/monitoring.md for configuration, self-hosted setup, and troubleshooting.

Python API Examples

See Quick Start by Domain above for domain-specific examples with explanations.

For complete Python examples:

Core GenAI: docs/core.md
Agents: docs/agents.md + AGENTS.md
Workflows: docs/workflows.md + docs/prefect.md

Agents Deep Dive

The toolkit ships three agent frameworks, all sharing the same YAML profile system, LLM factory, MCP servers, and Docker sandbox — unified behind a shared harness layer (see docs/agents.md).

Choosing an agent

Agent	Best for	Multi-turn	Planning	Web research	Code execution
ReAct	General tasks, tool use	✓	—	via tools	via tools
Deep agent	Complex multi-step planning	✓	✓	via tools	Docker sandbox
DeerFlow	Deep web research & reports	✓	✓	✓ native	Docker sandbox

ReAct agent (LangChain)

Standard Thought → Action → Observation loop. Good general-purpose default.

cli agents run --chat                                  # interactive session
cli agents run coding "Review this code"             # profile by KEY
cli agents list                                       # show profiles

from genai_tk.agents.langchain import LangchainAgent

agent = LangchainAgent("research")
result = agent.run("Summarise the 2024 GPT-4 technical report")

Profile in config/agents.yaml (unified agents: dict — harness: defaults to langchain when omitted):

agents:
  research:                       # profile KEY — used as: cli agents run research
    name: "Research"              # display name (shown in cli agents list)
    type: react
    llm: gpt_41mini@openai
    tools:
      - spec: web_search
    mcp_servers: []
    checkpointer:
      type: memory                # memory | postgres | sqlite

Deep agent (LangChain)

Extends ReAct with multi-step planning, subagent delegation, and optional Docker sandbox execution. Requires the deepagents extra package.

cli agents run research --chat

agents:
  research:                             # Profile KEY
    name: "Research"                    # Display name
    harness: langchain                  # defaults to langchain when omitted
    type: deep
    llm: gpt_41@openai
    enable_planning: true
    skills:
      directories:
        - ${paths.project}/skills       # SKILL.md files read on demand
    backend:
      type: aio_sandbox                 # optional Docker sandbox

DeerFlow (ByteDance / LangGraph)

DeerFlow is a multi-agent LangGraph system with native web search, planning, reporting, and sub-agents. genai-tk embeds it in-process — no separate server.

Setup (one-time):

cli init --extra harnessing   # installs deerflow-harness as a regular uv dependency

No DEER_FLOW_PATH env var is needed — deerflow-harness is a regular Python package.

Run:

cli agents run --chat                                   # interactive (default profile)
cli agents run research_assistant --trace "Explain quantum key distribution"
cli agents run research_assistant --mode ultra --chat   # full planning + sub-agents
cli agents list                                         # show profiles + modes

Modes:

Mode	Thinking	Planning	Sub-agents
`flash`	—	—	—
`thinking`	✓	—	—
`pro`	✓	✓	—
`ultra`	✓	✓	✓

Profile in config/agents.yaml (same unified agents: dict, harness: deerflow):

agents:
  "Research Assistant":         # profile key — used as: cli agents run "Research Assistant"
    harness: deerflow
    name: "Research Assistant"
    mode: pro
    llm: gpt_41@openai          # optional; falls back to server default
    mcp_servers: [tavily-mcp]
    skill_directories:
      - ${paths.project}/skills
    available_skills:           # filter which skills are exposed (omit = all)
      - public/deep-research
      - public/data-analysis
    sandbox: local              # local | docker

Skills configured under skill_directories are discovered automatically.

Skills — domain knowledge on demand

Skills are SKILL.md files that agents load when needed — not injected on every call. This keeps context lean and enables per-task specialisation.

skills/
├── custom/              # your project skills (committed)
│   └── my-domain/
│       └── SKILL.md
├── community/           # installed via cli skills add (gitignored)
└── bundled/             # copies of genai-tk bundled skills

Every cli init project gets a skills/ tree, a docs/SKILLS.md guide, and a getting-started example skill. Use the cli skills command group to manage them:

cli skills list                                           # all discovered skills
cli skills add getting-started                            # bundled skill
cli skills add --skillssh langchain-ai/langchain-skills   # from GitHub (skills.sh format)
cli skills add --git https://github.com/org/repo --path my-skill
cli skills create my-domain-skill                        # interactive scaffold
cli skills validate --all                                 # validate frontmatter + structure
cli skills info my-skill                                  # show full SKILL.md

Any agent that supports skill_directories (LangChain deep, all Deer-flow profiles) can discover and use skill files. In Docker sandbox mode the skill directories are automatically bind-mounted read-only at /mnt/skills/.

See docs/scaffolding.md for the complete skills guide and skills.sh for the community registry.

Sandbox

Agents can run code in an isolated Docker container via OpenSandbox:

# One-time warm-up (cuts container startup from ~28 s to ~5 s)
cli sandbox start
cli sandbox pull

# Use with any agent
cli agents run research "Write and run a Python script"            # LangChain deep profile (backend: aio_sandbox)
cli agents run research_assistant --sandbox docker --chat          # DeerFlow --sandbox override

The sandbox provides Chromium (VNC at localhost:8080/vnc), Python, Node.js, and a REST shell/file API. Skill directories are mount-inserted automatically.

See docs/sandbox_support.md for full setup instructions.

MCP Servers

Any agent profile can load tools from Model Context Protocol servers:

# config/mcp_servers.yaml
mcp_servers_config:
  tavily-mcp:
    command: npx
    args: ["-y", "tavily-mcp"]
    env:
      TAVILY_API_KEY: ${TAVILY_API_KEY}
  math_server:
    command: python
    args: ["-m", "genai_tk.mcp.math_server"]

cli agents run research_assistant --mcp math_server "…"
cli agents run research --mcp custom_server "…"

See docs/mcp-servers.md for the full reference.

Workflows

Orchestrate multi-step AI pipelines using a YAML-based DSL with Prefect execution.

Workflow DSL (YAML)

Define workflows as compositions of steps with dependencies, templates, and sub-workflows — no Python required:

# config/workflows.yaml
step_templates:
  markdownize_step:
    uses: genai_tk.workflow.markdownize.markdownize_flow
    inputs:
      root_dir: "${params.input_dir}"
      output_dir: "${params.output_dir}"

workflows:
  document_pipeline:
    steps:
      - id: convert
        ref: markdownize_step
        inputs:
          input_dir: "${paths.pdfs}"

workflow_profiles:
  process_docs:
    workflow: document_pipeline
    values:
      input_dir: ~/Documents/pdfs
      output_dir: ~/Documents/markdown

Run:

uv run cli workflow list profiles              # show available profiles
uv run cli workflow run process_docs --dry-run # show the plan
uv run cli workflow run process_docs           # execute

Prefect Flows

Ships with ready-to-use flows:

Flow	Purpose	CLI
markdownize	Documents → Markdown (MarkItDown, MessyExcel, EdgeParse, Mistral OCR, LightOn, AnyDoc, LLM)	`cli workflow run markdownize`
office2pdf	Office docs (PPT/Excel) → PDF	`cli workflow run office2pdf`
baml	Structured extraction	`cli baml run`
rag	RAG indexing + retrieval	`cli rag add-files`

All flows run in-process with an ephemeral Prefect client — no Prefect server needed.

See docs/markdownize.md, docs/workflows.md and docs/prefect.md for full reference.

LLM Selection

Models are referenced as model_id@provider — a short logical name plus the provider that serves it (openai, openrouter, groq, ollama, fake, …).

The toolkit ships with a built-in database sourced from models.dev covering 1 000+ models across all major providers. You only need llm.yaml entries for models that are not in that database or when you want to give a model a short alias:

# config/providers/llm.yaml
llm:
  exceptions:
    - model_id: gpt41mini          # short alias used in config and CLI
      providers:
        - openai: gpt-4.1-mini-2025-04-14  # maps to the actual API name
    - model_id: haiku
      providers:
        - openrouter: anthropic/claude-haiku-4-5
    - model_id: parrot_local       # built-in fake model, no API key needed
      providers:
        - fake: parrot

Override at runtime with -m / --llm:

cli core llm -i "Hello" -m gpt41mini@openai   # declared alias — explicit provider
cli core llm -i "Hello" -m fast_model         # named tag from genai_def.yaml
cli core llm -i "Hello" -m gpt-4o-mini        # raw model name — fuzzy-resolved from models.dev

See docs/llm-selection.md for the full reference (tags, cli info commands, models.dev database).

Configuration

The config system uses a hierarchy of YAML files loaded from config/ (auto-discovered by walking up the directory tree):

config/
├── app_conf.yaml           # entry point — profile selection, paths, env vars
├── profiles/
│   ├── local/
│   │   └── genai_def.yaml   # default LLM / embeddings / cache
│   └── pytest/
│       └── genai_def.yaml   # fake models for test runs
├── providers/
│   ├── llm.yaml            # LLM model declarations
│   └── embeddings.yaml     # Embeddings model declarations
├── agents.yaml             # unified agent profiles (agents: dict; harness: langchain|deerflow)
└── deerflow.yaml           # DeerFlow runtime settings (skills, general, default_profile)

Environment variables in .env override config values at any level.
Switch deployment environment with GENAITK_PROFILE=prod or in code with switch_profile("prod").
Activate a named in-session overlay (no reload) with global_config().use_context("training_local").

See docs/configuration.md for the full reference.

Capabilities by Domain

Core GenAI	CLI	Python	Docs
LLM / Embeddings	`cli core llm`	`get_llm()` / `get_embeddings()`	docs/core.md
Model selection	`cli info models`	`llm.yaml`	docs/llm-selection.md
Vector stores	—	`EmbeddingsStore`	docs/core.md
LLM caching	—	`LlmCache`	docs/core.md

Agents	CLI	Python	Docs
Unified harness	`cli agents run` / `cli agents list`	`create_harness()`	docs/agents.md
ReAct agent	`cli agents run <key>`	`LangchainAgent`	docs/agents.md
Deep agent	`cli agents run <deep>`	`LangchainAgent` (deep)	docs/agents.md
DeerFlow	`cli agents run <key>`	`EmbeddedDeerFlowClient`	docs/deer-flow.md
Skills	—	`skill_directories:` in config	AGENTS.md
Docker sandbox	`cli sandbox`	`SandboxBackend`	docs/sandbox_support.md
MCP servers	`cli mcpserver`	`McpClient`	docs/mcp-servers.md

Workflows	CLI	Python	Docs
Workflow DSL	`cli workflow`	`resolve_workflow_invocation()`	docs/workflows.md
Prefect flows	`cli tools *`	`run_flow_ephemeral()`	docs/prefect.md
RAG pipeline	`cli rag`	`RetrieverFactory`	docs/rag.md
BAML extraction	`cli baml`	`BamlStructuredProcessor`	docs/baml.md
Document loaders	—	`MarkdownLoader`	docs/workflows.md

Cross-cutting	CLI	Python	Docs
NLP / PII / spaCy	—	`genai_tk.extra.nlp`	docs/nlp.md
Configuration	`cli init`	`global_config()`	docs/configuration.md
Project scaffolding	`cli init --name`	`ProjectScaffolder`	docs/scaffolding.md
Copilot Agent support	`cli init`	—	docs/design/copilot-agent-support.md
CLI extension	—	`CliTopCommand`	docs/cli.md
Streamlit webapp	`just webapp`	`genai_tk.webapp`	docs/webapp.md
Testing	`cli test`	pytest	docs/TESTING_GUIDE.md
Browser automation	—	`browser_use` tools	docs/browser_control.md

Design and investigation notes: docs/design/.

Package Structure

genai_tk/
├── core/          # LLM factory, embeddings, vector stores, cache, MCP client
│   └── vector_backends/  # Chroma, InMemory, PgVector (+ Postgres connection mgmt)
├── agents/
│   ├── harness/   # Shared BaseHarness + event model across LangChain and DeerFlow
│   ├── langchain/ # Unified ReAct / Deep / Custom agents + middleware
│   └── deer_flow/ # DeerFlow embedded client + CLI
├── workflow/      # ETL orchestration: Prefect flows, RAG, loaders, retrievers
│   ├── prefect/   # run helpers + flows/ (markdownize, office2pdf, rag, baml)
│   ├── rag/       # chunkers, RAG CLI commands
│   ├── loaders/   # Markdown loader, Mistral OCR loader
│   └── retrievers/ # BM25, ZeroEntropy
├── extra/         # Non-pipeline tooling: document converters (genai_tk.extra.markdownize), agent graphs, NLP (spaCy/Presidio/classifiers), BAML, image analysis
├── tools/         # LangChain tool sets
├── utils/         # Config manager, Pydantic helpers, LangGraph utilities
└── main/          # CLI entry point + command modules

Development

just fmt    # ruff format + isort
just lint   # ruff lint
just test   # unit + integration (no API keys needed)
just check  # fmt + lint + test

# Run tests that need real models
uv run cli test full_integration

See AGENTS.md for code style, Pydantic conventions, and testing patterns.

License

MIT — see LICENSE.

Name		Name	Last commit message	Last commit date
.github		.github
.vscode		.vscode
browser_control		browser_control
config		config
deploy		deploy
docs		docs
examples		examples
genai_tk		genai_tk
notebooks		notebooks
scripts		scripts
skills		skills
tests		tests
vendor		vendor
.dockerignore		.dockerignore
.genai_tk		.genai_tk
.gitignore		.gitignore
.python-version		.python-version
AGENTS.md		AGENTS.md
CHANGELOG.md		CHANGELOG.md
IDEAS.md		IDEAS.md
None		None
README.md		README.md
browser_control_tests.md		browser_control_tests.md
fib.py		fib.py
fibonacci.py		fibonacci.py
justfile		justfile
pyproject.toml		pyproject.toml
tk.just		tk.just
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation

use @Once for Singletons

GenAI Toolkit (genai-tk)

Three Domains

🧠 Core GenAI

🔤 NLP

🤖 Agents

⚙️ Workflows

📊 Monitoring & Observability

Installation

Quick Start by Domain

🧠 Core GenAI

🤖 Agents

⚙️ Workflows

📊 Monitoring & Observability

More Examples

CLI Reference

First steps — Web UI

Monitoring and Observability

Python API Examples

Agents Deep Dive

Choosing an agent

ReAct agent (LangChain)

Deep agent (LangChain)

DeerFlow (ByteDance / LangGraph)

Skills — domain knowledge on demand

Sandbox

MCP Servers

Workflows

Workflow DSL (YAML)

Prefect Flows

LLM Selection

Configuration

Capabilities by Domain

Package Structure

Development

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

GenAI Toolkit (`genai-tk`)