AIDP

What This Project Is

AIDP is a system for automatically finding files in WEKA-backed storage and sending those files into an ingestion pipeline so they can be processed and used elsewhere.

Today, the main focus of this repository is the AIDP Watcher Control Plane. In simple terms, that means:

• people sign in through a browser
• they create a "watcher" for a WEKA filesystem
• the system prepares the storage connection and launches the watcher in Kubernetes
• the user can come back later to see whether that watcher is active, starting, inactive, or failed

This removes a lot of manual setup that previously lived in scripts, environment variables, and handwritten Kubernetes resources.

Who This Is For

This README is written for people who want to understand what the project does without needing to read Python, Kubernetes YAML, or operator code.

It is useful for:

• project sponsors
• product owners
• operations leads
• new users of the AIDP watcher interface
• anyone trying to understand the current shape of the platform

The Short Version

If you remember only one thing, remember this:

AIDP lets a user create and manage automated file-watching jobs from a web application instead of building them by hand.

Each watcher is tied to a WEKA filesystem and one or more folders to monitor. Once saved, Kubernetes and the operator do the backend work needed to run that watcher safely.

The Problem It Solves

Before this control plane work, setting up watcher-based ingestion required technical users to work directly with scripts, runtime configuration, and Kubernetes objects.

That approach created several problems:

• setup was hard for non-developers
• there was no simple browser-based workflow
• ownership was unclear in a shared environment
• it was difficult to see which watchers were healthy and which were not
• changes depended too heavily on manual technical steps

The control plane solves this by giving users a browser interface and making Kubernetes do the heavy lifting in the background.

What A User Can Do Today

Based on the current implementation and planning documents, the project now supports these core outcomes:

1. Sign in through Keycloak

Users access the AIDP GUI through a browser and authenticate through the existing Keycloak identity system.

The application distinguishes between:

• Administrators, who can see and manage all watchers
• Standard users, who can only see and manage the watchers they created

2. Create a watcher from the web UI

Instead of editing code or Kubernetes manifests, a user can fill in a form that captures:

• a display name
• the WEKA filesystem name
• one or more folders to watch

The GUI saves that request as a Kubernetes custom resource. That custom resource becomes the source of truth for what should exist.

3. Let the operator do the backend setup

Once a watcher is created, the operator takes over. It is responsible for:

• preparing the WEKA storage connection for that filesystem
• creating the persistent volume and persistent volume claim needed by the watcher
• launching a dedicated watcher runtime container
• updating watcher status as Kubernetes reconciliation progresses

This means the browser interface declares what the user wants, and the operator works to make reality match that request.

4. View watchers and their status

The GUI provides list and detail views so users can see:

• which watchers are visible to them
• who owns each watcher
• which filesystem a watcher is tied to
• whether it is active, starting, provisioning, inactive, failed, or unknown
• whether the health check is passing, pending, failing, or not applicable

5. Delete or purge watchers

The project supports lifecycle controls with different permission levels:

• users can delete watchers they own
• administrators can delete any watcher
• only administrators can purge a watcher completely

This protects shared environments from accidental destructive actions.

How The System Works In Plain English

Here is the end-to-end flow without the implementation details:

1. A user opens the AIDP web page and signs in.
2. The user creates a watcher for a WEKA filesystem and one or more folders.
3. The GUI records that request in Kubernetes.
4. The operator notices the new watcher request.
5. The operator prepares the shared WEKA storage objects for that filesystem.
6. The operator starts the watcher runtime workload.
7. The watcher begins monitoring the selected folders.
8. The GUI shows the watcher and its current state back to the user.

The Main Parts Of The Repository

This repository contains several related services. They are not all the same thing, but they support the wider AIDP platform.

aidp-gui/

This is the browser-based control plane.

It handles:

• sign-in
• session management
• role-aware access
• watcher create, list, detail, delete, and purge flows
• rendering the web pages users interact with

For most non-technical users, this is the face of the project.

aidp-gui-operator/

This is the Kubernetes operator that turns watcher requests into real runtime resources.

It handles:

• reconciliation of watcher custom resources
• shared WEKA filesystem mount resources
• persistent volume and persistent volume claim creation
• deployment of watcher runtime workloads
• runtime status reporting back into the watcher resource

For most users, this is invisible, but it is the engine that makes the GUI useful.

watcher/

This folder contains the watcher and ingestion support scripts that do the actual backend file-processing work.

In plain terms, these scripts are the workers that react to filesystem changes and push work into the ingestion pipeline.

k8s/

This folder contains Kubernetes deployment manifests for the GUI and the operator.

These files describe how the services should be deployed into a cluster.

Dockerfiles/

This folder defines how container images are built for the key services in the project.

rag-gateway/ and rag-bridge/

These are related services from the wider AIDP platform. They support retrieval and bridge-style API behavior and are part of the broader repository, but they are not the main user-facing part of the watcher control plane.

Key Concepts

Watcher

A watcher is a saved instruction that tells the system:

• which WEKA filesystem to use
• which folders to monitor
• who owns the watcher

GUI

The browser application where users sign in and manage watchers.

Operator

The background automation component that reads watcher requests and creates the needed Kubernetes resources.

Shared Namespace

All watchers live in one shared Kubernetes namespace. Because of that, the GUI applies visibility rules so normal users only see their own watchers.

WEKA Storage Provisioning

When a watcher is created, the system prepares storage resources tied to the selected WEKA filesystem so the watcher runtime can access the right data.

What "Status" Means

The project now includes user-visible runtime status. The labels are designed to answer a simple question: "Is my watcher working?"

Typical states include:

• Provisioning: the system is still preparing storage
• Starting: the watcher runtime is being launched
• Active: the watcher is running and appears healthy
• Failed: something went wrong and the watcher is not healthy
• Inactive: the watcher has been disabled
• Unknown: the system does not yet have enough information to confidently classify the watcher

The GUI also exposes a health result so the user can tell whether the watcher appears healthy, still pending, or clearly failing.

What Has Been Built So Far

The planning records in .planning/ show that the project has already completed the major build-out for:

• authentication and role-aware access
• watcher schema and browser-based creation
• operator reconciliation for runtime workloads
• WEKA filesystem-bound storage setup
• watcher list, detail, delete, and purge flows
• runtime status projection and automated test coverage
• Docker image build and publish workflow
• Kubernetes deployment manifests for the GUI and operator

In other words, this is no longer just an idea or early prototype. The repository contains a functioning control-plane design with implementation, tests, container packaging, and cluster manifests.

What Still Needs Attention

The planning history also shows one remaining caution:

• runtime status and hardening work is implemented, but the project notes that parts of Phase 5 were still awaiting live validation evidence

That means the platform is substantially built, but production confidence still depends on final end-to-end validation in a real environment.

What A New User Should Expect

If you are a first-time user of the project, the expected experience is:

1. Open the AIDP watcher web page.
2. Sign in with your organization credentials through Keycloak.
3. Create a watcher by naming it, selecting a WEKA filesystem, and entering the folders to watch.
4. Wait while the backend provisions the watcher runtime.
5. Return to the dashboard or watcher list to see whether it becomes active.
6. If you are an administrator, use the broader views and lifecycle controls to manage all watchers in the shared environment.

How To Read The .planning Folder

You asked specifically to use .planning, and it is the best place to understand the project story.

The most useful files are:

• .planning/PROJECT.md: the project purpose, goals, and constraints
• .planning/ROADMAP.md: the implementation phases and completion status
• .planning/codebase/ARCHITECTURE.md: the high-level system shape
• .planning/codebase/STRUCTURE.md: where major pieces live in the repo
• .planning/codebase/INTEGRATIONS.md: the external systems involved
• .planning/phases/: evidence of how each stage of the system was planned and delivered

For a non-technical reader, those files are best understood as the project diary behind the finished system.

In One Sentence

AIDP is becoming a browser-managed control plane for WEKA-backed file ingestion, giving users a simple way to create, run, and monitor filesystem watchers without doing the backend configuration by hand.