Documentation Update

.gitignore

@@ -165,4 +165,5 @@ config.json
 # Claude AI assistant working files
 CLAUDE.md
 
-bindings.rs
+bindings.rs
+site/

Doc/docs/cfcli/Agent-API-Overview.md

@@ -1,4 +1,4 @@
-# Agent API Overview
+# Agent API Commands Overview
 
 This page documents the CortexFlow Agent gRPC API for versions >= 0.1.3.
 

Doc/docs/contacts/contact.md

@@ -1,10 +1,3 @@
 - **Tettamanti Lorenzo**  [📧 lorenzotettamanti5@gmail.com](mailto:lorenzotettamanti5@gmail.com)  
 - **Lorenzo Bradanini**  [📧 lorenzolollobrada@gmail.com](mailto:lorenzolollobrada@gmail.com)  
-Right now we're looking for contributors and collaborators. If you have knowledge in DevOps/Kubernetes or Networks please write an email to lorenzotettamanti5@gmail.com  
-
-| **Role**                         | **Skills**                                           | **Tasks**                                                                                 | **Related Issues and Milestones** |
-|-----------------------------------|-----------------------------------------------------|------------------------------------------------------------------------------------------|----------------------------------|
-| **CortexBrain Core Developer**    | - Kubernetes  <br> - Networks  <br> - Rust         | - Work alongside us to build and optimize the core functionalities (Client, DNS, Proxy, Telemetry, etc.) | - [Rust](https://github.com/CortexFlow/CortexBrain/labels/rust) <br> - [Core](https://github.com/CortexFlow/CortexBrain/milestone/1) |
-| **CortexBrain Dashboard Developer** | - React  <br> - Frontend Development <br> - JavaScript/TypeScript | - Work alongside us to design and improve the dashboard | - [JavaScript](https://github.com/CortexFlow/CortexBrain/labels/javascript) |
-| **General Maintainers**           | - GitHub  <br> - Practical organization  <br> - Documentation | - Keep the repository organized and clean <br> - Write/Update documentation <br> - Spot typos in the repository | - [Documentation](https://github.com/CortexFlow/CortexBrain/labels/documentation) <br> - [Question](https://github.com/CortexFlow/CortexBrain/labels/question) |
-| **Code Reviewers/Testers**        | - Rust  <br> - JavaScript/TypeScript  <br> - Kubernetes <br> - Docker | - Review code and suggest changes/optimizations <br> - Write tests for CI/CD | - [Code refactoring](https://github.com/CortexFlow/CortexBrain/labels/code%20refactoring) |
+We're looking for contributors and collaborators. If you have knowledge in DevOps/Kubernetes, Monitoring, GPUs or Networking and Security please write an email to lorenzotettamanti5@gmail.com  

Doc/docs/developer-guide/agent-api.md

@@ -0,0 +1,80 @@
+# Agent API Architecture Overview
+
+The CortexBrain **Agent** is the user-space gRPC server that exposes the data collected by the eBPF kernel programs to clients such as [`cfcli`](../cfcli/overview.md). This page gives an overview of the service surface and the data flow behind it. The full message schema lives in [`core/api/protos/agent.proto`](https://github.com/CortexFlow/CortexBrain/blob/main/core/api/protos/agent.proto) in the repository.
+
+## What the Agent is
+
+The Agent is the `cortexflow_agent_api` crate (`core/api/` in the workspace). It is a [tonic](https://docs.rs/tonic/) gRPC server that:
+
+- Listens on `0.0.0.0:9090` (`core/api/src/main.rs`).
+- Registers the `agent.Agent` service together with a tonic **server reflection** service, so clients can enumerate the available RPCs at runtime - this is what `cfcli monitoring list` uses.
+- On startup, opens a set of **pinned BPF maps** from `/sys/fs/bpf/...` and spawns background reader tasks that drain the perf buffers into mpsc channels.
+- Serves the RPC methods (listed below) that drain those channels and return the data to the caller.
+
+The Agent is deployed as the `cortexflow-agent` Kubernetes service (ClusterIP, port 9090). The CLI reaches it via `kubectl port-forward svc/cortexflow-agent 9090:9090 -n cortexflow`.
+
+## Data flow
+
+The Agent is a **consumer and aggregator** of data produced by metrics service which load the eBPF programs and pin the maps
+
+### The BPF maps the Agent reads
+
+| Map (kernel name) | Pinned path | Producer | Userspace struct | Feeds RPC |
+|-------------------|-------------|----------|------------------|-----------|
+| `EventsMap` | `/sys/fs/bpf/maps/events_map` | Identity (`identity/src/map_handlers.rs`) | `PacketLog` | `ActiveConnections` |
+| `net_metrics` | `/sys/fs/bpf/trace_maps/net_metrics` | Metrics (`metrics/src/maps_handlers.rs`) | `NetworkMetrics` | `GetDroppedPacketsMetrics` |
+| `time_stamp_events` | `/sys/fs/bpf/trace_maps/time_stamp_events` | Metrics (`metrics/src/maps_handlers.rs`) | `TimeStampMetrics` | `GetLatencyMetrics` |
+| `Blocklist` | `/sys/fs/bpf/maps/blocklist_map` | Identity (seeded from the `cortexbrain-client-config` ConfigMap) | `[u8;4] ->[u8;4]` | `AddIpToBlocklist` / `CheckBlocklist` / `RmIpFromBlocklist` |
+
+!!! warning
+    The Agent hardcodes the paths `/sys/fs/bpf/maps/...` and `/sys/fs/bpf/trace_maps/...` (`core/api/src/api.rs`), while the Identity and Metrics services pin under `${PIN_MAP_PATH}` which defaults to `/sys/fs/bpf/cortexbrain-identity-service/...`. This implies a deployment-time mount/symlink convention that is not captured in code. If you change the pin paths, update both sides.
+
+## The `agent.Agent` service
+
+The gRPC service is defined in `core/api/protos/agent.proto` (package `agent`, service name `Agent`). It exposes unary RPC methods. All methods are discoverable via server reflection.
+
+| RPC method | Request | Response | Description |
+|------------|---------|----------|-------------|
+| `ActiveConnections` | `RequestActiveConnections { optional string pod_ip }` | `ActiveConnectionResponse { string status; repeated ConnectionEvent events }` | Returns the connection events drained from `events_map`. Each `ConnectionEvent` carries `event_id` (the PID), `src_ip_port`, and `dst_ip_port`. |
+| `AddIpToBlocklist` | `AddIpToBlocklistRequest { optional string ip }` | `BlocklistResponse { string status; map<string,string> events }` | Inserts an IPv4 into the `Blocklist` BPF map. If `ip` is omitted, just returns the current blocklist. |
+| `CheckBlocklist` | `google.protobuf.Empty` | `BlocklistResponse` | Reads the entire `Blocklist` BPF map and returns it as a `map<ip, ip>`. |
+| `RmIpFromBlocklist` | `RmIpFromBlocklistRequest { string ip }` | `RmIpFromBlocklistResponse { string status; map<string,string> events }` | Removes an IPv4 from the `Blocklist` BPF map and returns the remaining entries. |
+| `GetLatencyMetrics` | `google.protobuf.Empty` | `LatencyMetricsResponse { string status; repeated LatencyMetric metrics; uint32 total_count; double average_latency_us; double min_latency_us; double max_latency_us }` | Returns TCP connection latency metrics (`delta_us` per event, plus aggregate stats) drained from `time_stamp_events`. |
+| `GetDroppedPacketsMetrics` | `google.protobuf.Empty` | `DroppedPacketsResponse { string status; repeated DroppedPacketMetric metrics; uint32 total_drops }` | Returns socket-level drop/error metrics (only entries where `sk_drops > 0`) drained from `net_metrics`. |
+| `GetTrackedVeth` | `google.protobuf.Empty` | `VethResponse { string status; repeated string veth_names; int32 tot_monitored_veth }` | Returns tracked veth names from recent events. |
+| `GetTrackedVethFromHashMap` | `google.protobuf.Empty` | `VethHashMapResponse { string status; map<string,string> veths }` | Returns tracked veth entries from the pinned eBPF hash map. |
+
+!!! note "Full message schema"
+    The detailed field-by-field schema for every message (`ConnectionEvent`, `LatencyMetric`, `DroppedPacketMetric`, ...) lives in [`core/api/protos/agent.proto`](https://github.com/CortexFlow/CortexBrain/blob/main/core/api/protos/agent.proto). The [Integrated Metrics](metrics.md) page also describes the metric fields in tabular form. The [CLI Agent API page](../cfcli/Agent-API-Overview.md) has grpcurl examples.
+
+## How `cfcli` calls the Agent
+
+The CLI (`cli/`) is a tonic gRPC client. The mapping between `cfcli` commands and the Agent RPCs:
+
+| `cfcli` command | Agent RPC |
+|-----------------|-----------|
+| `cfcli monitoring list` | (server reflection - lists all `Agent` methods) |
+| `cfcli monitoring connections` | `ActiveConnections` |
+| `cfcli monitoring latencymetrics` | `GetLatencyMetrics` |
+| `cfcli monitoring droppedpackets` | `GetDroppedPacketsMetrics` |
+| `cfcli policy create-blocklist --flags <IP>` | `AddIpToBlocklist` |
+| `cfcli policy check-blocklist` | `CheckBlocklist` |
+| `cfcli policy remove-ip --flags <IP>` | `RmIpFromBlocklist` |
+
+## Relationship to the Identity service
+
+The **Identity** service (`core/src/components/identity/`) is the **producer** of the connection events and the blocklist. It:
+
+1. Loads the `conntracker` eBPF object.
+2. Extracts and pins `EventsMap` (->`events_map`), `Blocklist` (->`blocklist_map`), and other maps.
+3. Seeds the `Blocklist` map from the `cortexbrain-client-config` Kubernetes ConfigMap on startup.
+4. Attaches the TC classifier and kprobes that fill the perf buffers.
+
+The **Agent** never calls Identity directly - they communicate solely through the pinned BPF maps in `/sys/fs/bpf/`. The Agent is also a **mutator** of the `Blocklist` map that Identity owns: `AddIpToBlocklist` and `RmIpFromBlocklist` write to it at runtime, on top of the initial ConfigMap seeding.
+
+## Security notes
+
+The gRPC channel is currently **plaintext** HTTP/2:
+
+- The server binds to `0.0.0.0:9090` (`core/api/src/main.rs`, flagged with a `FIXME`).
+- The client connects to `http://127.0.0.1:9090` (`core/api/src/client.rs`, also flagged with a `FIXME`).

Doc/docs/developer-guide/architecture.md

@@ -0,0 +1,63 @@
+# Architecture Overview
+
+This page describes the CortexBrain architecture with a focus on how data flows from the Linux kernel, through eBPF programs and pinned BPF maps, to the user-space gRPC agent, and finally to the `cfcli` and the Grafana Dashboard.
+
+The conceptual diagram on the [home page](../index.md#architecture) gives the high-level picture; in this page you will find the full detailed pipeline.
+
+## Architecture Resume
+
+CortexBrain is built around a four-stage pipeline. Each stage has a clear responsibility and a well-defined boundary with the next.
+
+| Stage | Components | Role | Output |
+|-------|------------|------|--------|
+| **1. Kernel instrumentation** | `conntracker`, `metrics_tracer` (eBPF) | Observe at kernel level via TC classifiers and kprobes | PerfEventArray fills |
+| **2. Map pinning** | `identity`, `metrics` (user-space loaders) | Load + attach the eBPF programs, pin the maps to `/sys/fs/bpf/`, seed policy | Pinned maps in bpffs |
+| **3. Aggregation & serving** | `agent` (`core/api`) - gRPC on `:9090` and opentelemetry collector on `:4317` | Open pinned maps, drain perf buffers into mpsc channels, serve the RPC and aggregate the data for visualization | gRPC responses |
+| **4. Consumption** | `cfcli` (live), `prometheus` ,`dashboard` (grafana) | Forwards the data to the UI components |
+
+!!! tip "Reading order"
+    If you want the detail behind each stage, after this page read the [Agent API Overview](agent-api.md) for the gRPC surface and the [Integrated Metrics](metrics.md) page for the metric field schemas.
+
+## Full Pipeline 
+!!! warn
+    The **0.1.5** version adds OpenTelemetry instruments across socket/network, CPU, memory, scheduler, and SSL tracing (experimental) see the [Integrated Metrics](metrics.md#opentelemetry-metrics-incoming-metrics-patch) page for the full instrument list.
+
+![cf-architecture-v.0.1.5](../assets/cf-architecture-0.1.5.png)
+
+### Kernel hooks
+
+CortexBrain attaches eBPF programs to the following kernel hook points. The diagram below shows where these hooks sit in the Linux network stack.
+
+![Linux network stack and eBPF hooks](../assets/linux-net-stack.svg "Linux network stack with eBPF hook points")
+
+| Program | Attach type | Target function | BPF map written | Data produced |
+|---------|-------------|-----------------|------------------|---------------|
+| `identity_classifier` | TC classifier (ingress) | veth interfaces | `EventsMap` | `PacketLog` (per-packet metadata) |
+| `veth_creation_trace` | kprobe | `register_netdevice` | `veth_identity_map` | `VethLog` (event_type=1) |
+| `veth_deletion_trace` | kprobe | `unregister_netdevice_queue` | `veth_identity_map` | `VethLog` (event_type=2) |
+| `tcp_message_tracer` | kprobe | `tcp_v4_rcv`, `tcp_v4_connect` | `TcpPacketRegistry` | `TcpPacketRegistry` (TCP flow metadata) |
+| `metrics_tracer` | kprobe | `tcp_identify_packet_loss` | `net_metrics` | `NetworkMetrics` (socket stats + drops) |
+| `tcp_connect` | kprobe | `tcp_v4_connect`, `tcp_v6_connect` | `time_stamp_start` | start timestamp keyed by socket pointer |
+| `tcp_rcv_state_process` | kprobe | `tcp_rcv_state_process` | `time_stamp_events` | `TimeStampEvent` (latency `delta_us`) |
+
+**Source files**:
+- `core/src/components/conntracker/src/main.rs` and sub-modules (`tc.rs`, `veth_tracer.rs`, `tcp_analyzer.rs`, `data_structures.rs`)
+- `core/src/components/metrics_tracer/src/main.rs` and `data_structures.rs`
+
+### Deployment topology
+
+All CortexBrain components run in the `cortexflow` namespace. The core deployments use `hostPID: true`, `hostNetwork: true`, and `privileged: true` with the `BPF`, `SYS_ADMIN`, `NET_ADMIN`, `SYS_PTRACE`, and `SYS_RESOURCE` capabilities, and mount the host `/sys/fs/bpf` (bidirectional), `/proc`, and `/lib/modules`.
+
+| Pod (Deployment) | Container & binary | Image | Network exposure | BPF maps (producer / consumer) |
+|------------------|--------------------|-------|------------------|--------------------------------|
+| `cortexflow-agent` | `agent` -> `/usr/local/bin/agent-api` | `lorenzotettamanti/cortexflow-agent:latest` | Service `cortexflow-agent` ClusterIP TCP 9090 (grpc); reached by `cfcli` via `kubectl port-forward` | **Consumer only** - reads `events_map`, `net_metrics`, `time_stamp_events`, `blocklist_map` |
+| `cortexflow-identity` | initContainer `bpf-map-permissions` (mounts bpffs); `identity` -> `/usr/local/bin/cortexflow-identity-service`; sidecar `bpftool-control-manager` | `lorenzotettamanti/cortexflow-identity:latest` | None (no Service) | **Producer** - pins `events_map`, `veth_map`, `blocklist_map`, `tcp_packet_registry`; seeds `blocklist_map` from the `cortexbrain-client-config` ConfigMap |
+| `cortexflow-metrics` | `metrics` -> `/usr/local/bin/cortexflow-metrics`; sidecar `bpftool-control-manager` | `lorenzotettamanti/cortexflow-metrics:latest` | None (no Service) | **Producer** - pins `net_metrics`, `time_stamp_events` |
+
+**Manifests**: `core/src/testing/agent.yaml`, `core/src/testing/identity.yaml`, `core/src/testing/metrics.yaml`
+## Where to go next
+
+- **[Agent API Overview](../developer-guide/agent-api.md)** - the gRPC RPCs and the BPF maps behind them, in detail.
+- **[Integrated Metrics](../developer-guide/metrics.md)** - the field schemas for `ConnectionEvent`, `LatencyMetric`, `DroppedPacketMetric`, and the incoming OpenTelemetry instruments.
+- **[Development Workflow](../developer-guide/development-workflow.md)** - how to build the components locally and submit changes.
+- **[Development Goals & Discussions](../developer-guide/discussions.md)** - milestones, roadmap, and how to propose new features.

Doc/docs/developer-guide/development-workflow.md

@@ -0,0 +1,91 @@
+# Development Workflow
+
+This page groups all the best practices to help newcomers create their first pull request.
+
+## Branch conventions
+
+We review every type of PR that points to the **current development branch**.  
+The latest public version is contained in the **0.1.5** branch. The current development version is contained in the **0.1.6** branch.
+
+|Branch | Content|
+|-----|------|
+|**main**| current production version linked to the latest version |
+|**0.1.5**| latest version |
+|**0.1.6**| current development version (send your PR here)|
+
+
+## Commit message conventions
+
+- Reference the issue number: `#<n>: <description>` (e.g. _[#199]: updated documentation. Added development-workflow_)
+
+## Contribution flow
+
+Before opening a pull request, you **must** open an issue or a GitHub Discussion for any non-trivial change - pull requests without a related issue or discussion are not accepted (see `CONTRIBUTING.md`). This rule ensures that every update is pre-approved by the maintainers and is aligned with the latest project goals.
+
+- ### "Step 1 - Fork & clone"
+
+    ```bash
+    git clone https://github.com/<your-username>/CortexBrain.git
+    cd CortexBrain
+    git remote add upstream https://github.com/CortexFlow/CortexBrain.git
+    ```
+
+- ### "Step 2 - Open an issue or discussion"
+
+    New features and significant changes require a prior [GitHub Discussion](https://github.com/CortexFlow/CortexBrain/discussions). Bug fixes and small improvements can start directly from a [GitHub Issue](https://github.com/CortexFlow/CortexBrain/issues/new/choose). Reference the issue/discussion number in every commit and in the PR description.
+
+- ### "Step 3 - Create a dedicated branch"
+
+    ```bash
+    git checkout -b feature/<short-description>
+    ```
+
+    Pull requests without a dedicated branch are not accepted. Use a descriptive branch name prefixed by the type of change (`feature/`, `fix/`, `docs/`).
+
+- ### "Step 4 - Commit with a clear message"
+
+    Tag the related issue in every commit using the `#issue: message` convention:
+
+    ```bash
+    git commit -m "#78: added TCP connection tracing in conntracker"
+    ```
+
+    Keep commits focused. Avoid pull requests larger than ~1500 lines of code unless explicitly justified. (e.g. Grafana example dashboard update)
+
+- ### "Step 5 - Open a Pull Request"
+
+    Open the PR against `CortexFlow/CortexBrain:<current development branch>`. See the **Branch Conventions** to correctly link the PR to the development branch. The PR template asks for:
+
+    - A description of the change
+    - The type of change (bug fix, new feature, documentation, refactoring, other)
+    - A checklist (tested locally, docs updated, new tests added, builds successfully)
+    - Related issues tagged using `#`
+
+    The PR is auto-assigned to a maintainer (`@LorenzoTettamanti`) and an auto-reviewer assignment workflow runs.
+
+
+## Local build
+
+To build the core components locally, see the [Getting Started for developers](../getting-started/installation.md#getting-started-for-developers) section. The short version:
+
+```bash
+# From the repo root
+cargo +nightly build --release -p <component-name>
+```
+
+Each component also has its own build script under `core/` and `core/src/components/<name>/`.
+
+```bash
+# From the repo root CortexBrain/core
+cargo +nightly build --release -p <component-name>
+```
+
+
+## Containarization
+
+Every component can be containerized using a prebuilt scripts located in the `CortexBrain/core`.
+|Script name| content|
+|-----|------|
+|agent-api-build.sh| Build the cortexflow-agent container|
+|identity-build.sh| Build the identity service container| 
+|metrics-build.sh| Build the metrics service container|