diff --git a/.gitignore b/.gitignore index 302ff537..f9ca7d05 100644 --- a/.gitignore +++ b/.gitignore @@ -165,4 +165,5 @@ config.json # Claude AI assistant working files CLAUDE.md -bindings.rs \ No newline at end of file +bindings.rs +site/ \ No newline at end of file diff --git a/Doc/docs/assets/cf-architecture-0.1.5.png b/Doc/docs/assets/cf-architecture-0.1.5.png new file mode 100644 index 00000000..dff50f6c Binary files /dev/null and b/Doc/docs/assets/cf-architecture-0.1.5.png differ diff --git a/Doc/docs/assets/cf-new-architecture-readme.png b/Doc/docs/assets/cf-new-architecture-readme.png new file mode 100644 index 00000000..fa721896 Binary files /dev/null and b/Doc/docs/assets/cf-new-architecture-readme.png differ diff --git a/Doc/docs/assets/cortexflow-mcp.png b/Doc/docs/assets/cortexflow-mcp.png new file mode 100644 index 00000000..2f069a2e Binary files /dev/null and b/Doc/docs/assets/cortexflow-mcp.png differ diff --git a/Doc/docs/cfcli/Agent-API-Overview.md b/Doc/docs/cfcli/Agent-API-Overview.md index 22104a18..53be3a8b 100644 --- a/Doc/docs/cfcli/Agent-API-Overview.md +++ b/Doc/docs/cfcli/Agent-API-Overview.md @@ -1,4 +1,4 @@ -# Agent API Overview +# Agent API Commands Overview This page documents the CortexFlow Agent gRPC API for versions >= 0.1.3. diff --git a/Doc/docs/contacts/contact.md b/Doc/docs/contacts/contact.md index c4159540..a71dbdf7 100644 --- a/Doc/docs/contacts/contact.md +++ b/Doc/docs/contacts/contact.md @@ -1,10 +1,3 @@ - **Tettamanti Lorenzo** [📧 lorenzotettamanti5@gmail.com](mailto:lorenzotettamanti5@gmail.com) - **Lorenzo Bradanini** [📧 lorenzolollobrada@gmail.com](mailto:lorenzolollobrada@gmail.com) -Right now we're looking for contributors and collaborators. If you have knowledge in DevOps/Kubernetes or Networks please write an email to lorenzotettamanti5@gmail.com - -| **Role** | **Skills** | **Tasks** | **Related Issues and Milestones** | -|-----------------------------------|-----------------------------------------------------|------------------------------------------------------------------------------------------|----------------------------------| -| **CortexBrain Core Developer** | - Kubernetes
- Networks
- Rust | - Work alongside us to build and optimize the core functionalities (Client, DNS, Proxy, Telemetry, etc.) | - [Rust](https://github.com/CortexFlow/CortexBrain/labels/rust)
- [Core](https://github.com/CortexFlow/CortexBrain/milestone/1) | -| **CortexBrain Dashboard Developer** | - React
- Frontend Development
- JavaScript/TypeScript | - Work alongside us to design and improve the dashboard | - [JavaScript](https://github.com/CortexFlow/CortexBrain/labels/javascript) | -| **General Maintainers** | - GitHub
- Practical organization
- Documentation | - Keep the repository organized and clean
- Write/Update documentation
- Spot typos in the repository | - [Documentation](https://github.com/CortexFlow/CortexBrain/labels/documentation)
- [Question](https://github.com/CortexFlow/CortexBrain/labels/question) | -| **Code Reviewers/Testers** | - Rust
- JavaScript/TypeScript
- Kubernetes
- Docker | - Review code and suggest changes/optimizations
- Write tests for CI/CD | - [Code refactoring](https://github.com/CortexFlow/CortexBrain/labels/code%20refactoring) | +We're looking for contributors and collaborators. If you have knowledge in DevOps/Kubernetes, Monitoring, GPUs or Networking and Security please write an email to lorenzotettamanti5@gmail.com diff --git a/Doc/docs/developer-guide/agent-api.md b/Doc/docs/developer-guide/agent-api.md new file mode 100644 index 00000000..26481153 --- /dev/null +++ b/Doc/docs/developer-guide/agent-api.md @@ -0,0 +1,80 @@ +# Agent API Architecture Overview + +The CortexBrain **Agent** is the user-space gRPC server that exposes the data collected by the eBPF kernel programs to clients such as [`cfcli`](../cfcli/overview.md). This page gives an overview of the service surface and the data flow behind it. The full message schema lives in [`core/api/protos/agent.proto`](https://github.com/CortexFlow/CortexBrain/blob/main/core/api/protos/agent.proto) in the repository. + +## What the Agent is + +The Agent is the `cortexflow_agent_api` crate (`core/api/` in the workspace). It is a [tonic](https://docs.rs/tonic/) gRPC server that: + +- Listens on `0.0.0.0:9090` (`core/api/src/main.rs`). +- Registers the `agent.Agent` service together with a tonic **server reflection** service, so clients can enumerate the available RPCs at runtime - this is what `cfcli monitoring list` uses. +- On startup, opens a set of **pinned BPF maps** from `/sys/fs/bpf/...` and spawns background reader tasks that drain the perf buffers into mpsc channels. +- Serves the RPC methods (listed below) that drain those channels and return the data to the caller. + +The Agent is deployed as the `cortexflow-agent` Kubernetes service (ClusterIP, port 9090). The CLI reaches it via `kubectl port-forward svc/cortexflow-agent 9090:9090 -n cortexflow`. + +## Data flow + +The Agent is a **consumer and aggregator** of data produced by metrics service which load the eBPF programs and pin the maps + +### The BPF maps the Agent reads + +| Map (kernel name) | Pinned path | Producer | Userspace struct | Feeds RPC | +|-------------------|-------------|----------|------------------|-----------| +| `EventsMap` | `/sys/fs/bpf/maps/events_map` | Identity (`identity/src/map_handlers.rs`) | `PacketLog` | `ActiveConnections` | +| `net_metrics` | `/sys/fs/bpf/trace_maps/net_metrics` | Metrics (`metrics/src/maps_handlers.rs`) | `NetworkMetrics` | `GetDroppedPacketsMetrics` | +| `time_stamp_events` | `/sys/fs/bpf/trace_maps/time_stamp_events` | Metrics (`metrics/src/maps_handlers.rs`) | `TimeStampMetrics` | `GetLatencyMetrics` | +| `Blocklist` | `/sys/fs/bpf/maps/blocklist_map` | Identity (seeded from the `cortexbrain-client-config` ConfigMap) | `[u8;4] ->[u8;4]` | `AddIpToBlocklist` / `CheckBlocklist` / `RmIpFromBlocklist` | + +!!! warning + The Agent hardcodes the paths `/sys/fs/bpf/maps/...` and `/sys/fs/bpf/trace_maps/...` (`core/api/src/api.rs`), while the Identity and Metrics services pin under `${PIN_MAP_PATH}` which defaults to `/sys/fs/bpf/cortexbrain-identity-service/...`. This implies a deployment-time mount/symlink convention that is not captured in code. If you change the pin paths, update both sides. + +## The `agent.Agent` service + +The gRPC service is defined in `core/api/protos/agent.proto` (package `agent`, service name `Agent`). It exposes unary RPC methods. All methods are discoverable via server reflection. + +| RPC method | Request | Response | Description | +|------------|---------|----------|-------------| +| `ActiveConnections` | `RequestActiveConnections { optional string pod_ip }` | `ActiveConnectionResponse { string status; repeated ConnectionEvent events }` | Returns the connection events drained from `events_map`. Each `ConnectionEvent` carries `event_id` (the PID), `src_ip_port`, and `dst_ip_port`. | +| `AddIpToBlocklist` | `AddIpToBlocklistRequest { optional string ip }` | `BlocklistResponse { string status; map events }` | Inserts an IPv4 into the `Blocklist` BPF map. If `ip` is omitted, just returns the current blocklist. | +| `CheckBlocklist` | `google.protobuf.Empty` | `BlocklistResponse` | Reads the entire `Blocklist` BPF map and returns it as a `map`. | +| `RmIpFromBlocklist` | `RmIpFromBlocklistRequest { string ip }` | `RmIpFromBlocklistResponse { string status; map events }` | Removes an IPv4 from the `Blocklist` BPF map and returns the remaining entries. | +| `GetLatencyMetrics` | `google.protobuf.Empty` | `LatencyMetricsResponse { string status; repeated LatencyMetric metrics; uint32 total_count; double average_latency_us; double min_latency_us; double max_latency_us }` | Returns TCP connection latency metrics (`delta_us` per event, plus aggregate stats) drained from `time_stamp_events`. | +| `GetDroppedPacketsMetrics` | `google.protobuf.Empty` | `DroppedPacketsResponse { string status; repeated DroppedPacketMetric metrics; uint32 total_drops }` | Returns socket-level drop/error metrics (only entries where `sk_drops > 0`) drained from `net_metrics`. | +| `GetTrackedVeth` | `google.protobuf.Empty` | `VethResponse { string status; repeated string veth_names; int32 tot_monitored_veth }` | Returns tracked veth names from recent events. | +| `GetTrackedVethFromHashMap` | `google.protobuf.Empty` | `VethHashMapResponse { string status; map veths }` | Returns tracked veth entries from the pinned eBPF hash map. | + +!!! note "Full message schema" + The detailed field-by-field schema for every message (`ConnectionEvent`, `LatencyMetric`, `DroppedPacketMetric`, ...) lives in [`core/api/protos/agent.proto`](https://github.com/CortexFlow/CortexBrain/blob/main/core/api/protos/agent.proto). The [Integrated Metrics](metrics.md) page also describes the metric fields in tabular form. The [CLI Agent API page](../cfcli/Agent-API-Overview.md) has grpcurl examples. + +## How `cfcli` calls the Agent + +The CLI (`cli/`) is a tonic gRPC client. The mapping between `cfcli` commands and the Agent RPCs: + +| `cfcli` command | Agent RPC | +|-----------------|-----------| +| `cfcli monitoring list` | (server reflection - lists all `Agent` methods) | +| `cfcli monitoring connections` | `ActiveConnections` | +| `cfcli monitoring latencymetrics` | `GetLatencyMetrics` | +| `cfcli monitoring droppedpackets` | `GetDroppedPacketsMetrics` | +| `cfcli policy create-blocklist --flags ` | `AddIpToBlocklist` | +| `cfcli policy check-blocklist` | `CheckBlocklist` | +| `cfcli policy remove-ip --flags ` | `RmIpFromBlocklist` | + +## Relationship to the Identity service + +The **Identity** service (`core/src/components/identity/`) is the **producer** of the connection events and the blocklist. It: + +1. Loads the `conntracker` eBPF object. +2. Extracts and pins `EventsMap` (->`events_map`), `Blocklist` (->`blocklist_map`), and other maps. +3. Seeds the `Blocklist` map from the `cortexbrain-client-config` Kubernetes ConfigMap on startup. +4. Attaches the TC classifier and kprobes that fill the perf buffers. + +The **Agent** never calls Identity directly - they communicate solely through the pinned BPF maps in `/sys/fs/bpf/`. The Agent is also a **mutator** of the `Blocklist` map that Identity owns: `AddIpToBlocklist` and `RmIpFromBlocklist` write to it at runtime, on top of the initial ConfigMap seeding. + +## Security notes + +The gRPC channel is currently **plaintext** HTTP/2: + +- The server binds to `0.0.0.0:9090` (`core/api/src/main.rs`, flagged with a `FIXME`). +- The client connects to `http://127.0.0.1:9090` (`core/api/src/client.rs`, also flagged with a `FIXME`). diff --git a/Doc/docs/developer-guide/architecture.md b/Doc/docs/developer-guide/architecture.md new file mode 100644 index 00000000..ad7029a4 --- /dev/null +++ b/Doc/docs/developer-guide/architecture.md @@ -0,0 +1,63 @@ +# Architecture Overview + +This page describes the CortexBrain architecture with a focus on how data flows from the Linux kernel, through eBPF programs and pinned BPF maps, to the user-space gRPC agent, and finally to the `cfcli` and the Grafana Dashboard. + +The conceptual diagram on the [home page](../index.md#architecture) gives the high-level picture; in this page you will find the full detailed pipeline. + +## Architecture Resume + +CortexBrain is built around a four-stage pipeline. Each stage has a clear responsibility and a well-defined boundary with the next. + +| Stage | Components | Role | Output | +|-------|------------|------|--------| +| **1. Kernel instrumentation** | `conntracker`, `metrics_tracer` (eBPF) | Observe at kernel level via TC classifiers and kprobes | PerfEventArray fills | +| **2. Map pinning** | `identity`, `metrics` (user-space loaders) | Load + attach the eBPF programs, pin the maps to `/sys/fs/bpf/`, seed policy | Pinned maps in bpffs | +| **3. Aggregation & serving** | `agent` (`core/api`) - gRPC on `:9090` and opentelemetry collector on `:4317` | Open pinned maps, drain perf buffers into mpsc channels, serve the RPC and aggregate the data for visualization | gRPC responses | +| **4. Consumption** | `cfcli` (live), `prometheus` ,`dashboard` (grafana) | Forwards the data to the UI components | + +!!! tip "Reading order" + If you want the detail behind each stage, after this page read the [Agent API Overview](agent-api.md) for the gRPC surface and the [Integrated Metrics](metrics.md) page for the metric field schemas. + +## Full Pipeline +!!! warn + The **0.1.5** version adds OpenTelemetry instruments across socket/network, CPU, memory, scheduler, and SSL tracing (experimental) see the [Integrated Metrics](metrics.md#opentelemetry-metrics-incoming-metrics-patch) page for the full instrument list. + +![cf-architecture-v.0.1.5](../assets/cf-architecture-0.1.5.png) + +### Kernel hooks + +CortexBrain attaches eBPF programs to the following kernel hook points. The diagram below shows where these hooks sit in the Linux network stack. + +![Linux network stack and eBPF hooks](../assets/linux-net-stack.svg "Linux network stack with eBPF hook points") + +| Program | Attach type | Target function | BPF map written | Data produced | +|---------|-------------|-----------------|------------------|---------------| +| `identity_classifier` | TC classifier (ingress) | veth interfaces | `EventsMap` | `PacketLog` (per-packet metadata) | +| `veth_creation_trace` | kprobe | `register_netdevice` | `veth_identity_map` | `VethLog` (event_type=1) | +| `veth_deletion_trace` | kprobe | `unregister_netdevice_queue` | `veth_identity_map` | `VethLog` (event_type=2) | +| `tcp_message_tracer` | kprobe | `tcp_v4_rcv`, `tcp_v4_connect` | `TcpPacketRegistry` | `TcpPacketRegistry` (TCP flow metadata) | +| `metrics_tracer` | kprobe | `tcp_identify_packet_loss` | `net_metrics` | `NetworkMetrics` (socket stats + drops) | +| `tcp_connect` | kprobe | `tcp_v4_connect`, `tcp_v6_connect` | `time_stamp_start` | start timestamp keyed by socket pointer | +| `tcp_rcv_state_process` | kprobe | `tcp_rcv_state_process` | `time_stamp_events` | `TimeStampEvent` (latency `delta_us`) | + +**Source files**: +- `core/src/components/conntracker/src/main.rs` and sub-modules (`tc.rs`, `veth_tracer.rs`, `tcp_analyzer.rs`, `data_structures.rs`) +- `core/src/components/metrics_tracer/src/main.rs` and `data_structures.rs` + +### Deployment topology + +All CortexBrain components run in the `cortexflow` namespace. The core deployments use `hostPID: true`, `hostNetwork: true`, and `privileged: true` with the `BPF`, `SYS_ADMIN`, `NET_ADMIN`, `SYS_PTRACE`, and `SYS_RESOURCE` capabilities, and mount the host `/sys/fs/bpf` (bidirectional), `/proc`, and `/lib/modules`. + +| Pod (Deployment) | Container & binary | Image | Network exposure | BPF maps (producer / consumer) | +|------------------|--------------------|-------|------------------|--------------------------------| +| `cortexflow-agent` | `agent` -> `/usr/local/bin/agent-api` | `lorenzotettamanti/cortexflow-agent:latest` | Service `cortexflow-agent` ClusterIP TCP 9090 (grpc); reached by `cfcli` via `kubectl port-forward` | **Consumer only** - reads `events_map`, `net_metrics`, `time_stamp_events`, `blocklist_map` | +| `cortexflow-identity` | initContainer `bpf-map-permissions` (mounts bpffs); `identity` -> `/usr/local/bin/cortexflow-identity-service`; sidecar `bpftool-control-manager` | `lorenzotettamanti/cortexflow-identity:latest` | None (no Service) | **Producer** - pins `events_map`, `veth_map`, `blocklist_map`, `tcp_packet_registry`; seeds `blocklist_map` from the `cortexbrain-client-config` ConfigMap | +| `cortexflow-metrics` | `metrics` -> `/usr/local/bin/cortexflow-metrics`; sidecar `bpftool-control-manager` | `lorenzotettamanti/cortexflow-metrics:latest` | None (no Service) | **Producer** - pins `net_metrics`, `time_stamp_events` | + +**Manifests**: `core/src/testing/agent.yaml`, `core/src/testing/identity.yaml`, `core/src/testing/metrics.yaml` +## Where to go next + +- **[Agent API Overview](../developer-guide/agent-api.md)** - the gRPC RPCs and the BPF maps behind them, in detail. +- **[Integrated Metrics](../developer-guide/metrics.md)** - the field schemas for `ConnectionEvent`, `LatencyMetric`, `DroppedPacketMetric`, and the incoming OpenTelemetry instruments. +- **[Development Workflow](../developer-guide/development-workflow.md)** - how to build the components locally and submit changes. +- **[Development Goals & Discussions](../developer-guide/discussions.md)** - milestones, roadmap, and how to propose new features. \ No newline at end of file diff --git a/Doc/docs/developer-guide/development-workflow.md b/Doc/docs/developer-guide/development-workflow.md new file mode 100644 index 00000000..920bebcb --- /dev/null +++ b/Doc/docs/developer-guide/development-workflow.md @@ -0,0 +1,91 @@ +# Development Workflow + +This page groups all the best practices to help newcomers create their first pull request. + +## Branch conventions + +We review every type of PR that points to the **current development branch**. +The latest public version is contained in the **0.1.5** branch. The current development version is contained in the **0.1.6** branch. + +|Branch | Content| +|-----|------| +|**main**| current production version linked to the latest version | +|**0.1.5**| latest version | +|**0.1.6**| current development version (send your PR here)| + + +## Commit message conventions + +- Reference the issue number: `#: ` (e.g. _[#199]: updated documentation. Added development-workflow_) + +## Contribution flow + +Before opening a pull request, you **must** open an issue or a GitHub Discussion for any non-trivial change - pull requests without a related issue or discussion are not accepted (see `CONTRIBUTING.md`). This rule ensures that every update is pre-approved by the maintainers and is aligned with the latest project goals. + +- ### "Step 1 - Fork & clone" + + ```bash + git clone https://github.com//CortexBrain.git + cd CortexBrain + git remote add upstream https://github.com/CortexFlow/CortexBrain.git + ``` + +- ### "Step 2 - Open an issue or discussion" + + New features and significant changes require a prior [GitHub Discussion](https://github.com/CortexFlow/CortexBrain/discussions). Bug fixes and small improvements can start directly from a [GitHub Issue](https://github.com/CortexFlow/CortexBrain/issues/new/choose). Reference the issue/discussion number in every commit and in the PR description. + +- ### "Step 3 - Create a dedicated branch" + + ```bash + git checkout -b feature/ + ``` + + Pull requests without a dedicated branch are not accepted. Use a descriptive branch name prefixed by the type of change (`feature/`, `fix/`, `docs/`). + +- ### "Step 4 - Commit with a clear message" + + Tag the related issue in every commit using the `#issue: message` convention: + + ```bash + git commit -m "#78: added TCP connection tracing in conntracker" + ``` + + Keep commits focused. Avoid pull requests larger than ~1500 lines of code unless explicitly justified. (e.g. Grafana example dashboard update) + +- ### "Step 5 - Open a Pull Request" + + Open the PR against `CortexFlow/CortexBrain:`. See the **Branch Conventions** to correctly link the PR to the development branch. The PR template asks for: + + - A description of the change + - The type of change (bug fix, new feature, documentation, refactoring, other) + - A checklist (tested locally, docs updated, new tests added, builds successfully) + - Related issues tagged using `#` + + The PR is auto-assigned to a maintainer (`@LorenzoTettamanti`) and an auto-reviewer assignment workflow runs. + + +## Local build + +To build the core components locally, see the [Getting Started for developers](../getting-started/installation.md#getting-started-for-developers) section. The short version: + +```bash +# From the repo root +cargo +nightly build --release -p +``` + +Each component also has its own build script under `core/` and `core/src/components//`. + +```bash +# From the repo root CortexBrain/core +cargo +nightly build --release -p +``` + + +## Containarization + +Every component can be containerized using a prebuilt scripts located in the `CortexBrain/core`. +|Script name| content| +|-----|------| +|agent-api-build.sh| Build the cortexflow-agent container| +|identity-build.sh| Build the identity service container| +|metrics-build.sh| Build the metrics service container| \ No newline at end of file diff --git a/Doc/docs/developer-guide/discussions.md b/Doc/docs/developer-guide/discussions.md new file mode 100644 index 00000000..9f63ce41 --- /dev/null +++ b/Doc/docs/developer-guide/discussions.md @@ -0,0 +1,64 @@ +# Development Goals & Discussions + +This page collects the project milestones, the roadmap, and the GitHub references that shape CortexBrain development. Use it to find where to contribute and how to propose new work. + +## Current milestones + +The development stage of every component is tracked in the [home page](../index.md#current-development-stage). The current milestones are: + +| Component | Stage | Branch | Milestone | +|-----------|-------|--------|-----------| +| Dashboard | Under development | `feature/frontend` | CortexBrain v 0.1.0 Launch | +| Identity Service | Under development | `feature/ebpf-core` | CortexBrain core v 0.1.0 | +| Agent | Under development | `feature/core` | CortexBrain core v 0.1.0 | +| CLI | Under development | `feature/cli` | CortexBrain CLI v 0.1.0 | + +The Kubernetes-native "Core" milestone is tracked as [GitHub milestone #1](https://github.com/CortexFlow/CortexBrain/milestone/1) + +## Roadmap + +The broader roadmap is described in the [`README.md`](https://github.com/CortexFlow/CortexBrain/blob/main/README.md#current-development-focus) "Current Development Focus" section. The active and experimental tracks are: + +| Track | Status | +|-------|--------| + +## Proposing new features + +New features and significant changes **must** be discussed before a pull request is opened. The flow is: + +1. Open a [GitHub Discussion](https://github.com/CortexFlow/CortexBrain/discussions) describing the feature and how it aligns with the roadmap above. +2. Once there is rough agreement, open a [GitHub Issue](https://github.com/CortexFlow/CortexBrain/issues/new/choose) to track the work. +3. Reference the discussion/issue in your commits and PR (see the [Development Workflow](../developer-guide/development-workflow.md) page for the conventions). + +Pull requests without a related issue or discussion are not accepted. + +## GitHub references + +### Discussions & issues + +- [GitHub Discussions](https://github.com/CortexFlow/CortexBrain/discussions) - for feature proposals and design questions. +- [GitHub Issues](https://github.com/CortexFlow/CortexBrain/issues) - for bug reports and tracked work. +- [Core milestone (#1)](https://github.com/CortexFlow/CortexBrain/milestone/1) - the "Core" milestone. +- [Conntracker issue (#78)](https://github.com/CortexFlow/CortexBrain/issues/78) and [Identity issue (#92)](https://github.com/CortexFlow/CortexBrain/issues/92) - the two main core-component tracking issues. + +### Labels + +Issues and PRs are labeled to help routing and discovery. The canonical set (referenced from `README.md` and `contacts/contact.md`): + +| Label | Use | +|-------|-----| +| [`rust`](https://github.com/CortexFlow/CortexBrain/labels/rust) | Rust / core development | +| [`ebpf`](https://github.com/CortexFlow/CortexBrain/labels/ebpf) | eBPF-specific work | +| [`documentation`](https://github.com/CortexFlow/CortexBrain/labels/documentation) | Docs work | +| [`question`](https://github.com/CortexFlow/CortexBrain/labels/question) | Questions and clarifications | +| [`code refactoring`](https://github.com/CortexFlow/CortexBrain/labels/code%20refactoring) | Refactoring tasks | +| `cortexflow cli` | CLI-specific issues (also routes auto-assignment in CI) | + +### Roles we are looking for +The project is in his early stages of developement so users might encounters in bugs and unexpected behaviours. We are actively looking for motivated contributors that wants to share knowledge and create a community around the project and its main themes. + +If you have knowledge in DevOps, Kubernetes, GPUs, Docker or networking, email `lorenzotettamanti5@gmail.com` - see the [Contacts](../contacts/contact.md) page for details. + +## External resources + +Because the project sits at the intersection of several deep topics, the [Developer Guide index](index.md#external-resources) lists the best starting points for eBPF, the Aya Rust framework, gRPC/tonic, and Kubernetes networking. \ No newline at end of file diff --git a/Doc/docs/developer-guide/glossary.md b/Doc/docs/developer-guide/glossary.md new file mode 100644 index 00000000..f776b7ca --- /dev/null +++ b/Doc/docs/developer-guide/glossary.md @@ -0,0 +1,81 @@ +# Glossary + +This page defines the terms and acronyms used across the documentation. It is grouped in categories to simplify fast consulting +## eBPF & kernel + +| Term | Definition | +|------|------------| +| **eBPF** | Extended Berkeley Packet Filter. A virtual machine inside the Linux kernel that runs sandboxed programs in response to events (network packets, system calls, kprobes). CortexBrain uses eBPF to observe the cluster at kernel level without modifying the kernel source. | +| **BPF verifier** | The kernel component that statically analyzes a BPF program before execution, ensuring it terminates safely, does not access out-of-bounds memory, and meets security constraints. | +| **JIT** | Just-In-Time compiler. Translates verified BPF bytecode into native CPU instructions for maximum performance. | +| **kprobe** | Kernel probe. An eBPF attach point on a kernel function entry. CortexBrain uses kprobes on `tcp_v4_connect`, `tcp_rcv_state_process`, `register_netdevice`, `tcp_identify_packet_loss`, etc. | +| **uprobe** | User-space probe. An eBPF attach point on a userspace binary symbol. | +|**uretprobe**| an alternative to breakpoint instructions for triggering return uprobe consumers. Calls to `uretprobe()` are only made from the user-space trampoline provided by the kernel| +| **TC** | Traffic Control. The Linux kernel packet scheduler. CortexBrain attaches the `identity_classifier` eBPF program as a TC classifier on veth interfaces ingress. | +|**Trampolines**| In the context of Linux this refers to locations in memory containing addresses of logic to jump to. `Trampolines` are also referred to as _indirect jump vectors_. It is a mechanism that has a number of use cases such as interrupt service routines or I/O routines. In these classic use cases the hardware hard-codes memory locations to which execution will jump when certain events such as interrupt happen. A trampoline typically jumps immediately to some other function where the actual handler lives, hence the term trampoline. [reference](https://docs.ebpf.io/linux/concepts/trampolines/)| +| **XDP** | eXpress Data Path. An eBPF attach point on the network driver, before the skb is allocated. Mentioned in the architecture diagram as a future hook. | +| **PerfEventArray** | A BPF map type that pushes events from kernel to user space via a perf ring buffer. CortexBrain uses it to stream `PacketLog`, `NetworkMetrics`, `TimeStampEvent` to the userspace loaders and consumers. | +| **BPF map** | A kernel data structure shared between eBPF programs and userspace. | +| **bpffs** | The BPF filesystem, mounted at `/sys/fs/bpf`. Pinned BPF maps live here as files. | +| **Pinning** | Persisting a BPF map to bpffs so it survives the loading process and can be opened by another process. The Identity and Metrics services pin maps; the agent opens them. | +| **Aya** | The Rust eBPF framework CortexBrain uses to load, attach, and read BPF maps. See [aya-rs.dev](https://aya-rs.dev/). | + +## CortexBrain components + +| Term | Definition | +|------|------------| +| **conntracker** | The eBPF kernel crate (`core/src/components/conntracker/`) that produces connection events via a TC classifier and TCP kprobes. | +| **identity** | The userspace service (`core/src/components/identity/`) that loads the conntracker eBPF object, pins `events_map` and `blocklist_map`, seeds the blocklist from the `cortexbrain-client-config` ConfigMap (Kubernetes Only), and attaches the TC classifier to veths. | +| **metrics_tracer** | The eBPF kernel crate (`core/src/components/metrics_tracer/`) that produces socket-level metrics and TCP latency via kprobes. | +| **metrics** | The userspace service (`core/src/components/metrics/`) that loads the metrics_tracer eBPF object | +| **agent** | The gRPC server crate (`core/api/`, binary `agent-api`) that opens the pinned maps, drains the perf buffers into mpsc channels, and serves the `agent.Agent` RPCs on `:9090`. | +| **cfcli** | The Rust CLI client (`cli/`, binary `cfcli`) that calls the agent gRPC and manages the install/uninstall lifecycle via `kubectl`. | +| **MCP server** | The Model Context Protocol server (`mcp/`, binary `mcp`) that exposes tools querying Prometheus, so AI assistants (opencode, Claude) can read CortexBrain metrics. See [MCP Server](mcp-server.md). | + +## Kubernetes + +| Term | Definition | +|------|------------| +| **ClusterIP** | A Kubernetes Service type reachable only from inside the cluster. The `cortexflow-agent` Service is ClusterIP on port 9090. | +| **port-forward** | `kubectl port-forward` opens a tunnel from the local host to a Kubernetes Service | +| **hostPath** | A volume type that mounts a path from the host node. CortexBrain mounts `/sys/fs/bpf`, `/proc`, `/lib/modules` this way. | +| **hostPID / hostNetwork** | Pod options that share the host PID and network namespaces. CortexBrain pods use both for eBPF visibility. | +| **privileged** | A security context flag that gives a container nearly all host capabilities. CortexBrain pods require it for BPF map and kprobe access. | +| **ConfigMap** | A Kubernetes object holding non-secret key-value data. `cortexbrain-client-config` stores the blocklist. | +| **ServiceAccount** | A Kubernetes identity for pods. CortexBrain deployments currently use the `default` ServiceAccount. | +| **RBAC** | Role-Based Access Control. The `configmap-reader` Roles grant the deployments read access to ConfigMaps. | + +## Networking + +| Term | Definition | +|------|------------| +| **veth** | Virtual Ethernet pair. The per-pod network interface created by Container Networking Interfaces (CNI) | +| **TC classifier** | A Traffic Control program that classifies packets. | +| **ingress / egress** | Incoming / outgoing traffic directions. The TC classifier is attached on ingress. | +| **TCP handshake** | The SYN -> SYN-ACK -> ACK exchange that establishes a TCP connection. The `tcp_connect` and `tcp_rcv_state_process` kprobes measure the time spent in `TCP_SYN_SENT` as the connection latency. | +| **`struct sock`** | The kernel structure representing a socket. The `metrics_tracer` kprobe reads its fields (`sk_drops`, `sk_err`, `sk_backlog_len`, ...) at hardcoded offsets. | + +## Observability + +| Term | Definition | +|------|------------| +| **OpenTelemetry (OTel)** | The CNCF standard for observability data. | +| **OTLP** | OpenTelemetry Protocol. The gRPC/HTTP transport for OTel data. The exporter is at `localhost:4317` (gRPC) or `localhost:4318` (HTTP) by default. | +| **Prometheus** | A metrics database that scrapes and stores time-series. The MCP server queries it at `localhost:9090`. | +| **Counter** | An OTel instrument that only increases (e.g. `events_total`). | +| **Gauge** | An OTel instrument that can go up or down. | +| **Histogram** | An OTel instrument that records a distribution (e.g. `latency_us`). | +| **mpsc channel** | Multi-producer, single-consumer channel from the Rust `tokio` library. The agent uses mpsc channels to move events from the reader tasks to the gRPC handlers. | + +## Model Context Protocol (MCP) + +| Term | Definition | +|------|------------| +| **Model Context Protocol** | The Anthropic standard for exposing tools, resources, and prompts to LLM-based assistants. | +| **stdio transport** | The MCP transport that uses the process stdin/stdout for JSON-RPC. The CortexBrain MCP server uses this. | +| **tool** | An MCP capability that lets the assistant call a function. The CortexBrain MCP server exposes tools querying Prometheus. | +| **resource** | An MCP capability that lets the assistant read data. CortexBrain does not currently expose resources. | +| **prompt** | An MCP capability that lets the assistant use a templated prompt. CortexBrain does not currently expose prompts. | +| **ServerInfo** | The MCP `initialize` response containing the server name, version, and capabilities. | +| **ToolRouter** | The `rmcp` component that dispatches `tools/call` requests to the matching handler method. | +| **rmcp** | The Rust MCP SDK crate used by the CortexBrain MCP server. | \ No newline at end of file diff --git a/Doc/docs/developer-guide/index.md b/Doc/docs/developer-guide/index.md new file mode 100644 index 00000000..b070994f --- /dev/null +++ b/Doc/docs/developer-guide/index.md @@ -0,0 +1,43 @@ +# Developer Guide + +!!! note + CortexBrain is currently in active development. Resources on eBPF and the Aya Rust framework are sparse across the internet, and existing references such as [ebpf.io](https://ebpf.io) and [aya-rs.dev](https://aya-rs.dev) only cover the top of the iceberg. This Developer Guide aims to flatten the learning curve so that newcomers can easily learn, contribute and become maintainers of the project. + +## Why this section exists +The project has a very steep curve because of the combination of eBPF and Rust at the same time. +This section consolidates the architecture, the Agent API surface, the integrated metrics, and the contribution workflow into a single, detailed reference so that you don't have to guess what the codebase does to get started. + +## What you'll find here + +| Page | What it covers | +|------|----------------| +| [Glossary](glossary.md) | Definitions for eBPF, kernel, Kubernetes, networking, observability, GPU, and MCP terms used across the docs | +| [Architecture Overview](architecture.md) | Covers kernel instrumentation, maps pinning, aggregation and consumption | +| [Development Workflow](../developer-guide/development-workflow.md) | Covers the complete developer workflow: Fork, development branch and PR flow, CI pipeline, commit and branch conventions | +| [Agent API Overview](agent-api.md) | The Cortexflow agent service. Covers the RPC methods, the BPF maps the agent reads, and the data flow from eBPF kernel programs to `cfcli` | +| [Integrated Metrics](metrics.md) | Covers the live metrics and the OpenTelemetry metric format from `metrics-patch` | +| [MCP Server](mcp-server.md) | The MCP server that exposes CortexBrain metrics to AI assistants via Prometheus queries | +| [GPU Metrics (Upcoming)](../gpu-metrics/gpu-metrics.md) | The complete CUDA observability roadmap | +| [Troubleshooting](troubleshooting.md) | Common issues (BPF fs not mounted, pinned map path mismatch, port-forward, permissions) and their fixes | +| [Development Goals & Discussions](discussions.md) | Milestones, roadmap, GitHub Discussions, labels, and how to propose new features | + +## Prerequisites + +Make sure you have the development environment set up. The guided setup (Rust nightly, eBPF toolchain, Minikube/Kind, Calico CNI, Docker setup) is documented in the [Getting Started for developers](../getting-started/installation.md#getting-started-for-developers) section of the installation page. + +The core build requirements (kernel `>= 5.15`, `bpftool`, `bcc`, `clang`, `llvm`, `libbpf-dev`, `rustc >= 1.85.0` nightly) are also listed there. + +## External resources + +CortexBrain sits at the intersection of Rust and eBPF no single resource covers everything at the same time, in the same repository. The following are the best starting points beyond this guide: + +- **Linux Kernel docs** - [Linux Kernel](https://elixir.bootlin.com/linux/v7.2-rc6/source/kernel) the complete linux kernel reference divided by version +- **eBPF** - [ebpf.io](https://ebpf.io/what-is-ebpf/) (concept overview), the [Cilium eBPF documentation](https://docs.cilium.io/) (practical kernel hook reference), and the [BPF and XDP reference guide](https://docs.kernel.org/networking/filter.html) in the Linux kernel docs. +- **Aya (Rust eBPF)** - [aya-rs.dev](https://aya-rs.dev/) (the framework CortexBrain uses to load and pin BPF maps), the [Aya book](https://aya-rs.dev/book/), and the [Aya examples](https://github.com/aya-rs/aya/tree/main/examples). +- **gRPC / tonic** - the [tonic](https://docs.rs/tonic/) Rust gRPC stack documentation, which powers the `agent.Agent` service. +- **Kubernetes networking** - the [Kubernetes CNI documentation](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-network/network-plugins/) and the [Calico docs](https://docs.tigera.io/calico/latest/about), since CortexBrain attaches eBPF programs to veth interfaces created by Calico. +- **Docker docs** - [Docker docs](https://docs.docker.com/) + +## Contributing + +CortexBrain is open source under Apache 2.0. We actively look for contributors and collaborators. If you have knowledge in DevOps,GPUs, Kubernetes,Docker or networking. \ No newline at end of file diff --git a/Doc/docs/developer-guide/mcp-server.md b/Doc/docs/developer-guide/mcp-server.md new file mode 100644 index 00000000..913aede4 --- /dev/null +++ b/Doc/docs/developer-guide/mcp-server.md @@ -0,0 +1,84 @@ +# MCP Server (Experimental) + +CortexBrain ships a **Model Context Protocol (MCP) server** that exposes its eBPF-derived metrics to AI assistants (opencode, Claude Desktop, etc.) so they can query the cluster state in natural language. The server is a thin stdio JSON-RPC process that translates tool calls into PromQL queries against a local Prometheus. + +## What is the Model Context Protocol? + +The [Model Context Protocol](https://modelcontextprotocol.io/) is an open standard (originated by Anthropic) for exposing **tools**, **resources**, and **prompts** to LLM-based assistants over a standard JSON-RPC transport. An MCP host (opencode, Claude Desktop, Cursor) spawns an MCP server as a local child process and calls its tools when the model needs to interact with an external system. CortexBrain uses MCP to let an assistant query Prometheus metrics collected from eBPF. + +## Architecture +![cortexflow-mcp](../assets/cortexflow-mcp.png) + + +The MCP server is a **local standalone binary**, not a Kubernetes workload and not a sidecar. The operator builds it and configures an MCP host to spawn it; the host owns the process lifecycle. + +## The tools +Metrics are exposed as tools that your AI agent can call. It's also possible to use it in **agentic-workflows**. All the tools share the same input schema: + +```json +{ + "type": "object", + "properties": { + "container_name": { "type": "string" }, + "timeframe": { "type": "string" } + }, + "required": ["container_name", "timeframe"] +} +``` + +- `container_name` - a string matched against the Prometheus `container_name` label (e.g. `"grafana"`). Note that **container_name** refers directly to the container image used, this is a semantic inefficiency that will be fixed in future +- `timeframe` - a PromQL range-vector duration (e.g. `"10m"`). + +Each tool runs a `sum by(container_name) (rate(...[timeframe]))` query and returns the Prometheus JSON response as a pretty-printed string. + +!!! note + CortexFlow MCP is in its early stages and doesn't have full access to the full [metrics](./metrics.md). If you are willing to make any tests please refer to [contact](../contacts/contact.md) page + + +| Tool name | Description | PromQL metric | OTel instrument family | +|-----------|-------------|----------------|------------------------| +| `get_cpu_bytes` | CPU bytes allocation per event | `cortexbrain_cpu_bytes_alloc` | gauge | +| `get_memory_allocated_bytes` | Bytes requested via `mmap` syscalls | `cortexbrain_enter_mem_alloc` | gauge | +| `get_events` | Total eBPF events processed across all perf buffers | `cortexbrain_events_total` | counter | +| `get_l4_events` | Total socket state events processed | `cortexbrain_socket_events_total` | counter | +| `get_ssl_write_events` | Total bytes requested by `ssl_write` | `cortexbrain_ssl_write_bytes` | gauge | +| `get_ssl_read_events` | Total bytes requested by `ssl_read` | `cortexbrain_ssl_read_bytes` | gauge | + +The `cortexbrain_` prefix comes from the OTLP collector's Prometheus exporter `namespace: cortexbrain` setting (see `Examples/run-with-docker/otel-collector-config.yaml`). The base instrument names (`cpu_bytes_alloc`, `events_total`, ...) are defined in `core/common/src/semantic.rs` on the `metrics-patch` branch. + +## Configuration in opencode + +The MCP server is configured in `~/.config/opencode/opencode.jsonc` (or the project-local `.opencode/opencode.jsonc`): + +```jsonc +{ + "mcp": { + "cortexflow-mcp": { + "type": "local", + "command": ["/home//CortexBrain/mcp/target/release/mcp"], + "enabled": true + } + } +} +``` + +- `"type": "local"` + the `command` array tells opencode to spawn the binary as a child process and speak MCP over its stdin/stdout. +- `"enabled": true` auto-starts the server when opencode launches. +- The `"environment"` block (commented out in the example) can inject env vars into the child process - currently unused because the Prometheus URL is hardcoded. + +When the assistant needs a metric, opencode sends a `tools/call` request with the tool name and arguments; the server runs the PromQL and returns the JSON result as `TextContent`. + +## Building the server + +From the repository root (the `mcp/` crate must be checked out - it is on `origin/0.1.5` and `origin/metrics-patch`): + +```bash +cargo build --release -p mcp +``` + +## See also + +- [Architecture Overview](../developer-guide/architecture.md) - the four-stage pipeline and where Prometheus sits. +- [Integrated Metrics](../developer-guide/metrics.md) - the OpenTelemetry instruments that feed Prometheus. +- [Agent API Overview](../developer-guide/agent-api.md) - the gRPC service the MCP server currently does *not* call (planned for the roadmap). +- [Troubleshooting](../developer-guide/troubleshooting.md) - "MCP server cannot reach Prometheus". \ No newline at end of file diff --git a/Doc/docs/developer-guide/metrics.md b/Doc/docs/developer-guide/metrics.md new file mode 100644 index 00000000..cb89bee3 --- /dev/null +++ b/Doc/docs/developer-guide/metrics.md @@ -0,0 +1,152 @@ +# Integrated Metrics + +CortexBrain's metrics are documented below. Since version **0.1.5** we've introduced the OpenTelemetry procol (OTLP) to process and transport the metrics accross the full pipeline. Below we've documented all the featured metrics + +See the [Agent API Overview](agent-api.md) for how the gRPC methods map to the BPF maps behind these metrics. + +## CLI-Featured Metrics + +These metrics are collected by eBPF perf buffers and surfaced through the cortexflow agent. There is no Prometheus endpoint on the current branch - the data is pulled via `cfcli` or any tonic gRPC client. + +### Connection events - `ActiveConnections` + +Connection events are produced by the `conntracker` TC classifier and read from the `events_map` perf buffer. The agent converts each `PacketLog` into a `ConnectionEvent` proto. + +| Field | Type | Description | +|-------|------|-------------| +| `event_id` | `string` | The PID of the process that triggered the event | +| `src_ip_port` | `string` | Source IP and port, formatted as `ip:port` | +| `dst_ip_port` | `string` | Destination IP and port, formatted as `ip:port` | + +**Source**: `core/api/protos/agent.proto` (`ConnectionEvent`), `core/src/components/conntracker/src/data_structures.rs` (`PacketLog`), `core/api/src/api.rs` (handler). + +**CLI**: `cfcli monitoring connections` + +### Latency metrics - `GetLatencyMetrics` + +Latency metrics measure the time spent in `TCP_SYN_SENT` for IPv4/IPv6 connections. The `metrics_tracer` eBPF program kprobes `tcp_v4_connect`/`tcp_v6_connect` (start timestamp) and `tcp_rcv_state_process` (delta computation), then emits `TimeStampEvent`s to the `time_stamp_events` perf buffer. + +Each `LatencyMetric`: + +| Field | Type | Description | +|-------|------|-------------| +| `delta_us` | `uint64` | Latency in microseconds (SYN → established) | +| `timestamp_us` | `uint64` | Event timestamp, µs since boot (`bpf_ktime_get_ns`) | +| `tgid` | `uint32` | Thread group ID (process) | +| `process_name` | `string` | Process name (`comm`, 16 bytes) | +| `local_port` | `uint32` | Local port | +| `remote_port` | `uint32` | Remote port (big-endian) | +| `address_family` | `uint32` | `AF_INET=2` / `AF_INET6=10` | +| `src_address_v4` / `dst_address_v4` | `string` | IPv4 source / destination | +| `src_address_v6` / `dst_address_v6` | `string` | IPv6 source / destination | + +The response (`LatencyMetricsResponse`) also includes aggregates: + +| Field | Type | Description | +|-------|------|-------------| +| `total_count` | `uint32` | Number of metrics returned | +| `average_latency_us` | `double` | Average `delta_us` | +| `min_latency_us` | `double` | Minimum `delta_us` | +| `max_latency_us` | `double` | Maximum `delta_us` | + +**Source**: `core/api/protos/agent.proto` (`LatencyMetric`, `LatencyMetricsResponse`), `core/src/components/metrics_tracer/src/main.rs` (kprobes), `core/api/src/api.rs` (handler). + +**CLI**: `cfcli monitoring latencymetrics` + +### Dropped packet metrics - `GetDroppedPacketsMetrics` + +Dropped packet metrics come from the `metrics_tracer` kprobe that reads `struct sock` fields at hardcoded kernel offsets. The agent only forwards entries where `sk_drops > 0`. Each `DroppedPacketMetric`: + +| Field | Type | Kernel offset | Description | +|-------|------|---------------|-------------| +| `tgid` | `uint32` | - | Thread group ID | +| `process_name` | `string` | - | Process name (`comm`) | +| `timestamp_us` | `uint64` | - | Event timestamp (µs) | +| `sk_drops` | `int32` | 136 | Socket drops | +| `sk_err` | `int32` | 284 | Socket errors | +| `sk_err_soft` | `int32` | 600 | Soft errors | +| `sk_backlog_len` | `int32` | 196 | Backlog length (congestion indicator) | +| `sk_wmem_queued` | `int32` | 376 | Write memory queued | +| `sk_rcvbuf` | `int32` | 244 | Receive buffer size | +| `sk_ack_backlog` | `uint32` | 604 | ACK backlog | + +The response (`DroppedPacketsResponse`) also includes `total_drops` (`uint32`), the sum of `sk_drops` across all returned metrics. + +**Source**: `core/api/protos/agent.proto` (`DroppedPacketMetric`, `DroppedPacketsResponse`), `core/src/components/metrics_tracer/src/main.rs` (kprobe + offsets), `core/api/src/api.rs` (handler). + +**CLI**: `cfcli monitoring droppedpackets` + +The **0.1.5** version adds a full OpenTelemetry pipeline: + +- **Exporter**: OTLP, via gRPC (`http://localhost:4317`) or HTTP (`http://localhost:4318`), controlled by the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable. +- **Reader**: `PeriodicReader` with a 5-second export interval. +- **Meter name**: `cortexbrain-metrics`. + +The instrument definitions live in `core/common/src/otel_metrics.rs` and `core/common/src/semantic.rs` (on the `metrics-patch` branch). They are grouped below by category. + +## Prometheus-Featured metrics + +These metrics are collected by eBPF perf buffers and forwarded to the userspace using the OTLP protocol. These metrics are collected and aggregated by the OpenTelemetry agent, the collector expose the metrics through the API and the Prometheus scraper crawls and make them available in the Grafana Dashboard. The `CortexBrain Dashboard` use PromQL to create the visualizations. + +### Common attributes (labels) + +All OTel instruments share the following attribute set: + +| Attribute | Description | +|-----------|-------------| +| `tgid` | Thread group ID | +| `command` | Process name | +| `container.name` | Container name | +| `container.id` | Container ID (optional) | +| `k8s.pod.name` | Pod name (optional) | +| `k8s.namespace.name` | Namespace name (optional) | + +### Socket / network metrics + +| Instrument name | Type | Description | +|-----------------|------|-------------| +| `events_total` | `Counter` | Total eBPF events processed across all perf buffers | +| `socket_events_total` | `Counter` | Total socket state events processed | +| `sk_drops` | `Gauge` | Socket drop count per event | +| `sk_err` | `Gauge` | Socket error count per event | +| `latency_us` | `Histogram` | Distribution of latency values from timestamp events | + +### CPU metrics + +| Instrument name | Type | Description | +|-----------------|------|-------------| +| `bytes_alloc_events_total` | `Counter` | Total `bytes_alloc` events in the CPU | +| `cpu_bytes_alloc` | `Gauge` | CPU bytes allocation per event | +| `cpu_idle_state` | `Gauge` | Current CPU idle C-state per `cpu_id`, updated on state change | + +### Memory metrics + +| Instrument name | Type | Description | +|-----------------|------|-------------| +| `mem_alloc_events_total` | `Counter` | Total memory allocation (`mmap`) events | +| `enter_mem_alloc` | `Gauge` | Bytes requested via `mmap` syscalls | + +### Scheduler metrics + +| Instrument name | Type | Description | +|-----------------|------|-------------| +| `sched_stat_wait` | `Gauge` | Scheduler wait time (ns) from `sched_stat_wait` | +| `sched_stat_wait_distribution` | `Histogram` | Distribution of scheduler wait times (ns) | +| `sched_stat_runtime` | `Gauge` | Scheduler runtime (ns) from `sched_stat_runtime` | +| `sched_stat_runtime_distribution` | `Histogram` | Distribution of scheduler runtimes (ns) | + +### SSL metrics + +| Instrument name | Type | Description | +|-----------------|------|-------------| +| `ssl_read_bytes` | `Gauge` | Total bytes requested by `SSL_read` | +| `ssl_write_bytes` | `Gauge` | Total bytes requested by `SSL_write` | + +### Metrics source code + +The **0.1.5** version update the following modules (`core/src/components/metrics_tracer/src/`): + +- `cpu.rs` - CPU frequency and bytes-alloc events +- `memory.rs` - `mmap` syscall tracing +- `network.rs` - packet-loss and timestamp events (extended from current) +- `ssl.rs` - `SSL_read` / `SSL_write` tracing via `ssl_ctx_map` diff --git a/Doc/docs/developer-guide/troubleshooting.md b/Doc/docs/developer-guide/troubleshooting.md new file mode 100644 index 00000000..6943f942 --- /dev/null +++ b/Doc/docs/developer-guide/troubleshooting.md @@ -0,0 +1,142 @@ +# Troubleshooting + +This page resumes the most common issues you may encounter while running CortexBrain. For the initial setup, see the [Installation guide](../getting-started/installation.md). + +## Quick diagnostic table + +| Symptom | Likely cause | Fix | +|---------|--------------|-----| +| `cfcli status` reports the `cortexflow` namespace missing | Components not installed | `cfcli install cortexflow` | +| `cfcli monitoring *` returns `connection refused` | No port-forward to the agent | `kubectl port-forward svc/cortexflow-agent 9090:9090 -n cortexflow` | +| Agent pod `CrashLoopBackOff`, log: `No such file or directory` | Pinned BPF map path mismatch | See [Pinned map path mismatch](#pinned-map-path-mismatch) | +| Agent log: `permission denied` opening `/sys/fs/bpf/...` | bpffs not mounted or wrong permissions | See [BPF filesystem not mounted](#bpf-filesystem-not-mounted-minikube) | +| `cfcli monitoring list` returns an error | Agent not running or reflection not registered | Check `cfcli status` and agent logs | +| MCP server tools return empty / error | Prometheus not reachable at `localhost:9090` | Port-forward Prometheus or run the docker-compose stack | + +## BPF filesystem not mounted (Minikube) + +Minikube does not mount the BPF filesystem automatically. CortexBrain uses BPF maps with the pinning operation ([What's Pinning?](https://docs.ebpf.io/linux/concepts/pinning/)), so `/sys/fs/bpf` must be mounted and writable. + +```bash +minikube ssh +mount -t bpf bpffs /sys/fs/bpf +ls -ld /sys/fs/bpf +``` + +Expected output: + +``` +drwx-----T 3 root root 0 Sep 4 16:34 /sys/fs/bpf +``` + +If the permissions are wrong, fix them: + +```bash +chmod 700 /sys/fs/bpf +``` + +The `identity` Deployment includes an initContainer `bpf-map-permissions` that runs `mount -t bpf bpf /sys/fs/bpf` on startup, so in a Kubernetes deployment this is handled automatically as long as the pod is privileged and mounts the host `/sys/fs/bpf` as a bidirectional volume. + +## Pinned map path mismatch + +!!! warning + The Agent hardcodes the pinned map paths `/sys/fs/bpf/maps/events_map`, `/sys/fs/bpf/maps/blocklist_map`, `/sys/fs/bpf/trace_maps/net_metrics`, and `/sys/fs/bpf/trace_maps/time_stamp_events` (see `core/api/src/api.rs`). The Identity and Metrics services pin under `${PIN_MAP_PATH}`, which the Dockerfiles set to `/sys/fs/bpf/maps` and `/sys/fs/bpf/trace_maps` respectively. The deployment must reconcile these paths. + +**Symptoms**: the `cortexflow-agent` pod crashes on startup with `No such file or directory` when calling `MapData::from_pin(...)`. + +**Diagnosis**: + +1. Check which maps are actually pinned on the node: + ```bash + kubectl exec -it cortexflow-identity- -n cortexflow -- ls -R /sys/fs/bpf/ + ``` +2. Confirm the Identity and Metrics env vars: + ```bash + kubectl exec -it cortexflow-identity- -n cortexflow -- env | grep PIN_MAP_PATH + kubectl exec -it cortexflow-metrics- -n cortexflow -- env | grep PIN_MAP_PATH + ``` +3. Confirm the agent expects maps at `/sys/fs/bpf/maps/...` and `/sys/fs/bpf/trace_maps/...`. + +**Fix**: ensure the `PIN_MAP_PATH` of Identity is `/sys/fs/bpf/maps` and of Metrics is `/sys/fs/bpf/trace_maps` (these are the defaults in their Dockerfiles). If you changed them, update the agent's hardcoded paths in `core/api/src/api.rs` accordingly, or create symlinks under `/sys/fs/bpf/`. + +## Port-forward to the agent + +The `cortexflow-agent` Service is `ClusterIP` only (port 9090, gRPC). The `cfcli` client connects to `127.0.0.1:9090`, so you must open a port-forward from your host: + +```bash +kubectl port-forward svc/cortexflow-agent 9090:9090 -n cortexflow +``` + +Keep this running in a separate terminal while you use `cfcli monitoring *` commands. If the forward is missing, every `cfcli monitoring` command returns `connection refused`. + +## Permissions on `/sys/fs/bpf` + +The agent needs read-write access to `/sys/fs/bpf`. The deployment manifests set `privileged: true` with the `BPF`, `SYS_ADMIN`, `NET_ADMIN`, `SYS_PTRACE`, and `SYS_RESOURCE` capabilities and mount `/sys/fs/bpf` as a bidirectional hostPath volume. If you run outside Kubernetes (or with a reduced security context), verify: + +```bash +ls -ld /sys/fs/bpf +# must be drwx for root +``` + +## Debugging with bpftool + +The `cortexflow-identity` and `cortexflow-metrics` Deployments ship a `bpftool-control-manager` sidecar (`danielpacak/bpftool-runner:latest`) so you can inspect the BPF state from inside the cluster. + +List all pinned maps: + +```bash +kubectl exec -it cortexflow-identity- -c bpftool-control-manager -n cortexflow -- bpftool map show +``` + +Inspect a specific pinned map: + +```bash +kubectl exec -it cortexflow-identity- -c bpftool-control-manager -n cortexflow -- bpftool map show pinned /sys/fs/bpf/maps/blocklist_map +``` + +List loaded BPF programs: + +```bash +kubectl exec -it cortexflow-identity- -c bpftool-control-manager -n cortexflow -- bpftool prog show +``` + +## Reading pod logs + +CortexBrain pods run in the `cortexflow` namespace. Use `cfcli logs` or `kubectl logs` directly: + +```bash +cfcli logs --component agent --namespace cortexflow +kubectl logs -n cortexflow -l app=cortexflow-agent +kubectl logs -n cortexflow -l app=cortexflow-identity +kubectl logs -n cortexflow -l app=cortexflow-metrics +``` + +The agent logs startup messages on stderr (`Starting cortexflow-mcp`, `cortexflow-mcp running`) and gRPC traffic on the configured logger. + +## Namespace not found + +If `cfcli status` reports that the `cortexflow` namespace does not exist, the components have not been installed: + +```bash +cfcli install cortexflow +``` + +This creates the namespace, applies the RBAC, deploys the agent, identity, and metrics components, and seeds the `cortexbrain-client-config` ConfigMap. See the [CLI overview](../cfcli/overview.md) for the full command list. + +## MCP server cannot reach Prometheus + +The MCP server (see [MCP Server](mcp-server.md)) queries Prometheus at the hardcoded URL `http://localhost:9090/api/v1/query`. If the MCP tools return empty results or connection errors: + +1. Ensure Prometheus is running and reachable on `localhost:9090` (e.g. via the `Examples/run-with-docker/docker-compose.yaml` stack, or `kubectl port-forward svc/prometheus 9090:9090`). +2. Verify the OTLP collector is exporting metrics into Prometheus with the `cortexbrain_` namespace prefix (configured in `otel-collector-config.yaml`). +3. Test the Prometheus query directly: + ```bash + curl 'http://localhost:9090/api/v1/query?query=up' + ``` + +## See also + +- [Architecture Overview](../developer-guide/architecture.md) - the four-stage pipeline and the BPF map paths. +- [Agent API Overview](../developer-guide/agent-api.md) - the gRPC service and the BPF maps behind each RPC. +- [MCP Server](../developer-guide/mcp-server.md) - the Prometheus-backed MCP server. +- [Quickstart](../getting-started/quickstart.md) - the 5-minute end-to-end flow. \ No newline at end of file diff --git a/Doc/docs/getting-started/installation.md b/Doc/docs/getting-started/installation.md index e0ca4af0..2f3a7cfe 100644 --- a/Doc/docs/getting-started/installation.md +++ b/Doc/docs/getting-started/installation.md @@ -85,16 +85,3 @@ For RedHat/CentOS/Fedora based distributions you can copy and paste the followin ```bash cfcli install ``` - -### **Dashboard Development** - -1. Install [Node.js](https://nodejs.org/en/download) -2. Open the dashboard folder and install the required packages - ```bash - cd dashboard - npm install - ``` -3. Run the local development server - ```bash - npm start - ``` diff --git a/Doc/docs/cfcli/quick-start-guide.md b/Doc/docs/getting-started/quickstart.md similarity index 99% rename from Doc/docs/cfcli/quick-start-guide.md rename to Doc/docs/getting-started/quickstart.md index 61860328..c7e3499f 100644 --- a/Doc/docs/cfcli/quick-start-guide.md +++ b/Doc/docs/getting-started/quickstart.md @@ -1,5 +1,5 @@ -## CortexFlow Agent: Quick Start Guide +## Quickstart !!! warning Up to now the only supported cluster environment is Kubernetes diff --git a/Doc/docs/gpu-metrics/gpu-metrics.md b/Doc/docs/gpu-metrics/gpu-metrics.md new file mode 100644 index 00000000..b7c7f17a --- /dev/null +++ b/Doc/docs/gpu-metrics/gpu-metrics.md @@ -0,0 +1,9 @@ +# GPU Metrics (Upcoming) + +We're actively working to add GPU metrics support in the next updates. Nowadays GPU plays a crucial role in Artificial Intelligence (AI) and Machine Learning (ML) workflows. We believe that the current profiling systems does not integrate seamlessly in the current production systems that are constantly demanding resources. Optimization is quick becoming an essential feature in order to avoid resources and budget wasting and so we're constantly evolving to provide a better optimized platform. + +## See also + +- [Integrated Metrics](../developer-guide/metrics.md) - the live eBPF/gRPC metrics and the incoming OpenTelemetry suite +- [Architecture Overview](../developer-guide/architecture.md) - the four-stage pipeline (kernel → maps → agent → consumption) +- [Development Goals & Discussions](../developer-guide/discussions.md) - the "GPU observability" roadmap track \ No newline at end of file diff --git a/Doc/docs/index.md b/Doc/docs/index.md index f94b41ed..865b4ebf 100644 --- a/Doc/docs/index.md +++ b/Doc/docs/index.md @@ -2,17 +2,37 @@ CortexBrain is currently in active development so that occasional bugs may occur. Your contributions and feedback are foundamental for refining and enhancing the project! 🚀 **CortexBrain** is an ambitious open-source project designed to build an intelligent, lightweight, and highly efficient monitoring platform for distributed cloud and hybrid (cloud–edge) workflows. + +--- + +## v0.1.5 Changelog +- Added memory, cpu and SSL (experimental) metrics +- Introduced OpenTelemetry protocol to aggregate the exported metrics +- Added metadata injection +- Added a startup example +- Improved core (refered as common) library structure +- Introduced MCP server (experimental) +- minor bug fixes + +--- + ## **Current Development Stage** You can see the development stage of every component here: -| **Component** | **Stage** | **Latest Commit** | **Referring Branch** |**Related Milestone** +| **Component** | **Latest Version** | **Latest Commit** | **Referring Branch** |**Related Milestone** |---------------------|--------------------------|-------------------|-----------------------|-----------------------| -| **Dashboard** | Under development | [![GitHub last commit](https://img.shields.io/github/last-commit/CortexFlow/CortexBrain?style=flat-square&logo=github&color=success)](https://github.com/CortexFlow/CortexBrain/commits/feature/frontend) | `feature/frontend` | CortexBrain v 0.1.0 Launch | -| **Identity Service** | Under development | [![GitHub last commit](https://img.shields.io/github/last-commit/CortexFlow/CortexBrain?style=flat-square&logo=github&color=success)](https://github.com/CortexFlow/CortexBrain/commits/feature/ebpf-core) | `feature/ebpf-core` | CortexBrain core v 0.1.0 | -| **Agent** | Under development | [![GitHub last commit](https://img.shields.io/github/last-commit/CortexFlow/CortexBrain?style=flat-square&logo=github&color=success)](https://github.com/CortexFlow/CortexBrain/commits/core) | `feature/core` | CortexBrain core v 0.1.0 | -| **CLI** | Under development |[![GitHub last commit](https://img.shields.io/github/last-commit/CortexFlow/CortexBrain?style=flat-square&logo=github&color=success)](https://github.com/CortexFlow/CortexBrain/commits/feature/cli) | `feature/cli` | CortexBrain CLI v .0.1 | +| **Identity Service** | 0.1.1 | [![GitHub last commit](https://img.shields.io/github/last-commit/CortexFlow/CortexBrain?style=flat-square&logo=github&color=success)](https://github.com/CortexFlow/CortexBrain/commits/0.1.5) | `0.1.5` | CortexBrain core v 0.1.0 | +| **Agent** | 0.1.1 | [![GitHub last commit](https://img.shields.io/github/last-commit/CortexFlow/CortexBrain?style=flat-square&logo=github&color=success)](https://github.com/CortexFlow/CortexBrain/commits/0.1.5) | `0.1.5` | CortexBrain core v 0.1.0 | +| **CLI** | 0.1.5 |[![GitHub last commit](https://img.shields.io/github/last-commit/CortexFlow/CortexBrain?style=flat-square&logo=github&color=success)](https://github.com/CortexFlow/CortexBrain/commits/0.1.5) | `0.1.5` | CortexBrain CLI v .0.1 | + +--- +## Shortcuts +- [MCP server](./developer-guide/mcp-server.md) +- [Architecture overview](./developer-guide/architecture.md) +- [Glossary & Resources](./developer-guide/glossary.md) +--- # **An Introduction to Service Mesh** A **service mesh** is a specialized infrastructure layer embedded within a software application that manages communication between services. It handles critical functions such as traffic routing, security, observability, and resiliency, while shielding individual services from these complexities. @@ -34,7 +54,10 @@ eBPF is a powerful technology that allows for high-performance networking and se # **Architecture** The CortexFlow architecture is designed to ensure a robust, scalable, and fault-tolerant system that can operate seamlessly without interruptions. It is composed of several key components that work together to provide a continuous and reliable infrastructure. These components are orchestrated within a Kubernetes cluster, ensuring efficient resource management, high availability, and scalability. Below is a GIF that visually represents the architecture and illustrates how the components interact within the cluster. -![Architecture](./cf_architecture.svg "Cortexflow architecture") +![Architecture](./assets/cf-new-architecture-readme.png "Cortexflow architecture") + +!!! note + Since **v0.1.5** the metrics service can be deployed standalone using Docker like in this [example](https://github.com/CortexFlow/CortexBrain/tree/0.1.5/Examples/run-with-docker) ## What's eBPF? Extended Berkeley Packet Filter (eBPF) presents a transformative approach to building service meshes by eliminating the need for the traditional sidecar model, which often introduces significant complexity and overhead in microservices architecture. eBPF allows for the implementation of service mesh functionalities directly in the kernel, resulting in a more efficient and streamlined data plane. This native integration minimizes the number of proxies required, reduces additional network connections, and simplifies redirection logic for network traffic, thereby enhancing performance. @@ -57,15 +80,3 @@ The execution flow of a BPF program can be resumed in four key steps: - Execution: The compiled code is executed by the kernel This rigorous process ensures safety and maximum performance by molding itself with an event-driven approach, and so can be used as a foundation of an efficient monitoring system (Metrics and Observability in the graph above) and a fast networking plugin that we refer to in the illustration above as Identity Service. - - -# **CortexBrain components** -Cortexflow core components, also referred to as CortexBrain components, are composed of a list of services and utilities that empower users to efficiently observe networking events and resource usage. It also includes a command line interface (CLI) referred to as cfcli (cortexflow-cli) and a dashboard. Every component is carefully and detaily documented below. - -## **CLI** -The command line interface, also known as CLI, is an essential part of the CortexFlow User Experience. It allows users and developers to interact with all the core components without directly managing the manifests' YAML files. The CLI stores the relevant information, such as the underlying cluster environment (e.g., Kubernetes, Docker Swarm, etc), to support multiple environments without changing the user experience. Right now, the CLI only supports **Kubernetes** as an orchestrator. - - The CLI is available to install with the cargo package manager; we have carefully documented the installation in this [page](./cfcli/overview.md). - -!!! warning - Right now, the identity service, metrics, and dashboard are under development until 2026. We will release the first documentation snippet soon \ No newline at end of file diff --git a/Doc/docs/stylesheets/style.css b/Doc/docs/stylesheets/style.css index 8369b035..6571157d 100644 --- a/Doc/docs/stylesheets/style.css +++ b/Doc/docs/stylesheets/style.css @@ -1,6 +1,12 @@ html { width: 100%; } +[dir="ltr"] .md-header__title{ + margin-top: 30px; +} +.md-ellipsis{ + line-height: 0.9rem; +} .md-content { width: 800px; margin: auto auto; diff --git a/Doc/docs/test-report/reports/March2025/March2025.md b/Doc/docs/test-report/reports/March2025/March2025.md deleted file mode 100644 index d8471fd2..00000000 --- a/Doc/docs/test-report/reports/March2025/March2025.md +++ /dev/null @@ -1,1326 +0,0 @@ ---- -title: 🚀 March 2025 Test Report -date: 2025-03-29 -categories: - - Test - - Test report -tags: - - performance - - sidecar-proxy - - metrics export - - proxy communication ---- - -## Introduction: - -This report summarizes the test results for CortexBrain during March 2025. -Several tests were conducted to evaluate the performance of CortexBrain **proxy-sidecar** and CortexBrain **proxy-injector** in various scenarios. -The results are presented in the following sections: - -- **Proxy-Injector Tests**: Contains a list of tests performed to evaluate the injection functionalities. -- **Sidecar-Proxy Tests**: Contains a list of tests performed to assess the proxy functionalities, including communication, service discovery, asynchronous communication, and metrics export. - -## 🌟 CortexBrain Proxy-Injector tests - -Below there's the complete list of tests - -### Test : Injection - -The tests confirm that the injector component is fully functional.The container [patch](https://github.com/CortexFlow/CortexBrain/blob/4d9c1092ccc083d7c6830d0891c96175d9b2dd77/core/src/components/proxy-injector/src/vars.rs#L4) is correctly injected into all pods with the appropriate label (`proxy-injection: enabled`). Tests were conducted using a test pod named test-proxy and the bash script [test-sidecar-proxy.sh](https://github.com/CortexFlow/CortexBrain/blob/main/Scripts/test-sidecar-proxy.sh). - -```bash -lorenzo@Lorenzo:~/CortexBrain/Scripts$ ./test-sidecar-proxy.sh -Testing Sidecar proxy injection -Checking pods -NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES -cortexflow-proxy-9c94f5d6c-2w5t8 1/1 Running 16 (58s ago) 5d1h 10.244.0.179 minikube -proxy-injector 1/1 Running 14 (2m14s ago) 5d1h 10.244.0.181 minikube -test-proxy 2/2 Running 6 (2m14s ago) 3d12h 10.244.0.184 minikube -test-proxy2 2/2 Running 6 (2m14s ago) 3d12h 10.244.0.182 minikube - -Checking if the sidecar proxy is present -"proxy-sidecar" -"proxy-injector" -"nginx" -"proxy-sidecar" -"nginx" -"proxy-sidecar" -``` - -## 🌟 CortexBrain Proxy-Sidecar tests - -Available Functionalities - -- **UDP Traffic**: Supports communication over port `5053`. -- **TCP Traffic**: Supports communication over port `5054`. -- **Metrics Export**: Provides real-time metrics for monitoring and analysis. - -Results: - -| **Service** | **Test** | **Result (✅/❌)** | **Additional Notes** | -|----------------------------------|--------------------------------------------------------------------------|----------------------------------------------------------------------------|-------------------------------------------------------------------------------------| -| **Service discovery** | Pod IP resolution
| ✅| restricted to the cortexflow namespace only| -| **Messaging** | Support for JSON messages | ✅|/| -| **Communication** | - UDP (5053) port
- TCP (5054) port
| ✅|/| -| **Metrics** | Metrics are successfully collected | ✅|/| - - -!!! warning - At the time this report is written, all the other functionalities, such as caching, mTLS, and policies, are in the development stage. We will gradually introduce eBPF into the project from April 2025. - - -### Test 1: Checking proxy open ports -The test confirm that the proxy component is functional. Tests were conducted using a test pod -called test-proxy and the bash script [test-sidecar-proxy.sh](https://github.com/CortexFlow/CortexBrain/blob/main/Scripts/test-sidecar-proxy.sh) - -
- Click here to open the test details - -```bash -Checking open ports in test-proxy -[{"containerPort":5053,"protocol":"UDP"},{"containerPort":5054,"protocol":"TCP"}] [{"containerPort":5054,"protocol":"TCP"},{"containerPort":5053,"protocol":"UDP"}] -[{"containerPort":5053,"protocol":"UDP"},{"containerPort":5054,"protocol":"TCP"}] [{"containerPort":5054,"protocol":"TCP"},{"containerPort":5053,"protocol":"UDP"}] - -Checking network connections in test-proxy pod -Active Internet connections (only servers) -Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name -tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN - -tcp 0 0 0.0.0.0:5054 0.0.0.0:* LISTEN 1/cortexflow-proxy -tcp 0 0 0.0.0.0:9090 0.0.0.0:* LISTEN 1/cortexflow-proxy -tcp6 0 0 :::80 :::* LISTEN - -udp 0 0 0.0.0.0:5053 0.0.0.0:* 1/cortexflow-proxy - -Checking network connections in test-proxy2 pod -Active Internet connections (only servers) -Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name -tcp 0 0 0.0.0.0:9090 0.0.0.0:* LISTEN 1/cortexflow-proxy -tcp 0 0 0.0.0.0:5054 0.0.0.0:* LISTEN 1/cortexflow-proxy -tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN - -tcp6 0 0 :::80 :::* LISTEN - -udp 0 0 0.0.0.0:5053 0.0.0.0:* 1/cortexflow-proxy -``` - -
- -### Test 2: UDP communication - -The test confirm that the UDP communication is functional. Tests were conducted using a test pod called test-proxy as sender and test-proxy2 as receiver -
- Click here to open the test details - - -```bash -TEST 2: Sending a message from test-proxy to test-proxy2 (UDP) -Starting UDP listener on port 5053... -Test: Incoming Message ⏳ -{"status":"received"}{"status":"received"}✅ Test completed -``` - -
- -Using tcpdump, we can capture UDP packets processed on port 5053. The following is the detailed output: - -
- Click here to open the test details - - -```bash -lorenzo@Lorenzo:~$ kubectl exec -it test-proxy2 -c proxy-sidecar -n cortexflow -- tcpdump -i any port 5053 -nn -X -tcpdump: data link type LINUX_SLL2 -tcpdump: verbose output suppressed, use -v[v]... for full protocol decode -listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes -10:59:25.095047 eth0 In IP 10.244.0.184.51969 > 10.244.0.182.5053: UDP, length 125 - 0x0000: 4500 0099 38ba 4000 4011 ea44 0af4 00b8 E...8.@.@..D.... - 0x0010: 0af4 00b6 cb01 13bd 0085 17ec 7b22 7365 ............{"se - 0x0020: 7276 6963 6522 3a22 7465 7374 2d70 726f rvice":"test-pro - 0x0030: 7879 322e 636f 7274 6578 666c 6f77 222c xy2.cortexflow", - 0x0040: 2264 6972 6563 7469 6f6e 223a 2249 6e63 "direction":"Inc - 0x0050: 6f6d 696e 6722 2c22 7061 796c 6f61 6422 oming","payload" - 0x0060: 3a22 6579 4a74 5a58 4e7a 5957 646c 496a :"eyJtZXNzYWdlIj - 0x0070: 6f67 496b 686c 6247 7876 4947 5a79 6232 ogIkhlbGxvIGZyb2 - 0x0080: 3067 6348 4a76 6548 6b74 6332 6c6b 5a57 0gcHJveHktc2lkZW - 0x0090: 4e68 6369 4a39 227d 0a NhciJ9"}. -10:59:25.148297 eth0 Out IP 10.244.0.182.5053 > 10.244.0.184.51969: UDP, length 21 - 0x0000: 4500 0031 177e 4000 4011 0be9 0af4 00b6 E..1.~@.@....... - 0x0010: 0af4 00b8 13bd cb01 001d 1784 7b22 7374 ............{"st - 0x0020: 6174 7573 223a 2272 6563 6569 7665 6422 atus":"received" - 0x0030: 7d } -10:59:25.148423 eth0 Out IP 10.244.0.182.5053 > 10.244.0.184.51969: UDP, length 21 - 0x0000: 4500 0031 177f 4000 4011 0be8 0af4 00b6 E..1..@.@....... - 0x0010: 0af4 00b8 13bd cb01 001d 1784 7b22 7374 ............{"st - 0x0020: 6174 7573 223a 2272 6563 6569 7665 6422 atus":"received" - 0x0030: 7d } -``` - -
- - -### Test 3: TCP communication - -The test confirm that the TCP communication is functional. Tests were conducted using a test pod called test-proxy as sender and test-proxy2 as receiver - -```bash -TEST 2: Checking if test-proxy can communicate with test-proxy2 (TCP) -Test: Incoming Message ⏳ -{"status":"received"}Message ReceivedMessage Received✅ Test completed -``` - -Using tcpdump, we can capture TCP packets processed on port 5054. The following is the detailed output: - -
- Click here to open the test details - -```bash -lorenzo@Lorenzo:~$ kubectl exec -it test-proxy2 -c proxy-sidecar -n cortexflow -- tcpdump -i any port 5054 -nn -X -tcpdump: data link type LINUX_SLL2 -tcpdump: verbose output suppressed, use -v[v]... for full protocol decode -listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes -10:59:19.215746 eth0 In IP 10.244.0.184.45108 > 10.244.0.182.5054: Flags [S], seq 2592051025, win 64240, options [mss 1460,sackOK,TS val 3611482804 ecr 0,nop,wscale 7], length 0 - 0x0000: 4500 003c 5c4b 4000 4006 c71b 0af4 00b8 E..<\K@.@....... - 0x0010: 0af4 00b6 b034 13be 9a7f 8f51 0000 0000 .....4.....Q.... - 0x0020: a002 faf0 1784 0000 0204 05b4 0402 080a ................ - 0x0030: d742 dab4 0000 0000 0103 0307 .B.......... -10:59:19.215876 eth0 Out IP 10.244.0.182.5054 > 10.244.0.184.45108: Flags [S.], seq 907372687, ack 2592051026, win 65160, options [mss 1460,sackOK,TS val 2389105439 ecr 3611482804,nop,wscale 7], length 0 - 0x0000: 4500 003c 0000 4000 4006 2367 0af4 00b6 E..<..@.@.#g.... - 0x0010: 0af4 00b8 13be b034 3615 688f 9a7f 8f52 .......46.h....R - 0x0020: a012 fe88 1784 0000 0204 05b4 0402 080a ................ - 0x0030: 8e66 db1f d742 dab4 0103 0307 .f...B...... -10:59:19.215915 eth0 In IP 10.244.0.184.45108 > 10.244.0.182.5054: Flags [.], ack 1, win 502, options [nop,nop,TS val 3611482804 ecr 2389105439], length 0 - 0x0000: 4500 0034 5c4c 4000 4006 c722 0af4 00b8 E..4\L@.@..".... - 0x0010: 0af4 00b6 b034 13be 9a7f 8f52 3615 6890 .....4.....R6.h. - 0x0020: 8010 01f6 177c 0000 0101 080a d742 dab4 .....|.......B.. - 0x0030: 8e66 db1f .f.. -10:59:19.216138 eth0 In IP 10.244.0.184.45108 > 10.244.0.182.5054: Flags [F.], seq 1, ack 1, win 502, options [nop,nop,TS val 3611482804 ecr 2389105439], length 0 - 0x0000: 4500 0034 5c4d 4000 4006 c721 0af4 00b8 E..4\M@.@..!.... - 0x0010: 0af4 00b6 b034 13be 9a7f 8f52 3615 6890 .....4.....R6.h. - 0x0020: 8011 01f6 177c 0000 0101 080a d742 dab4 .....|.......B.. - 0x0030: 8e66 db1f .f.. -10:59:19.218764 eth0 Out IP 10.244.0.182.5054 > 10.244.0.184.45108: Flags [.], ack 2, win 510, options [nop,nop,TS val 2389105442 ecr 3611482804], length 0 - 0x0000: 4500 0034 6d57 4000 4006 b617 0af4 00b6 E..4mW@.@....... - 0x0010: 0af4 00b8 13be b034 3615 6890 9a7f 8f53 .......46.h....S - 0x0020: 8010 01fe 177c 0000 0101 080a 8e66 db22 .....|.......f." - 0x0030: d742 dab4 .B.. -10:59:19.239614 eth0 Out IP 10.244.0.182.5054 > 10.244.0.184.45108: Flags [P.], seq 1:17, ack 2, win 510, options [nop,nop,TS val 2389105463 ecr 3611482804], length 16 - 0x0000: 4500 0044 6d58 4000 4006 b606 0af4 00b6 E..DmX@.@....... - 0x0010: 0af4 00b8 13be b034 3615 6890 9a7f 8f53 .......46.h....S - 0x0020: 8018 01fe 178c 0000 0101 080a 8e66 db37 .............f.7 - 0x0030: d742 dab4 4d65 7373 6167 6520 5265 6365 .B..Message.Rece - 0x0040: 6976 6564 ived -10:59:19.239732 eth0 In IP 10.244.0.184.45108 > 10.244.0.182.5054: Flags [R], seq 2592051027, win 0, length 0 - 0x0000: 4500 0028 0000 4000 4006 237b 0af4 00b8 E..(..@.@.#{.... - 0x0010: 0af4 00b6 b034 13be 9a7f 8f53 0000 0000 .....4.....S.... - 0x0020: 5004 0000 aac5 0000 P....... -10:59:19.669707 eth0 In IP 10.244.0.184.45120 > 10.244.0.182.5054: Flags [S], seq 235715412, win 64240, options [mss 1460,sackOK,TS val 3611483258 ecr 0,nop,wscale 7], length 0 - 0x0000: 4500 003c df9e 4000 4006 43c8 0af4 00b8 E..<..@.@.C..... - 0x0010: 0af4 00b6 b040 13be 0e0c bb54 0000 0000 .....@.....T.... - 0x0020: a002 faf0 1784 0000 0204 05b4 0402 080a ................ - 0x0030: d742 dc7a 0000 0000 0103 0307 .B.z........ -10:59:19.669739 eth0 Out IP 10.244.0.182.5054 > 10.244.0.184.45120: Flags [S.], seq 2600685464, ack 235715413, win 65160, options [mss 1460,sackOK,TS val 2389105893 ecr 3611483258,nop,wscale 7], length 0 - 0x0000: 4500 003c 0000 4000 4006 2367 0af4 00b6 E..<..@.@.#g.... - 0x0010: 0af4 00b8 13be b040 9b03 4f98 0e0c bb55 .......@..O....U - 0x0020: a012 fe88 1784 0000 0204 05b4 0402 080a ................ - 0x0030: 8e66 dce5 d742 dc7a 0103 0307 .f...B.z.... -10:59:19.669776 eth0 In IP 10.244.0.184.45120 > 10.244.0.182.5054: Flags [.], ack 1, win 502, options [nop,nop,TS val 3611483258 ecr 2389105893], length 0 - 0x0000: 4500 0034 df9f 4000 4006 43cf 0af4 00b8 E..4..@.@.C..... - 0x0010: 0af4 00b6 b040 13be 0e0c bb55 9b03 4f99 .....@.....U..O. - 0x0020: 8010 01f6 177c 0000 0101 080a d742 dc7a .....|.......B.z - 0x0030: 8e66 dce5 .f.. -10:59:19.669894 eth0 In IP 10.244.0.184.45120 > 10.244.0.182.5054: Flags [P.], seq 1:126, ack 1, win 502, options [nop,nop,TS val 3611483258 ecr 2389105893], length 125 - 0x0000: 4500 00b1 dfa0 4000 4006 4351 0af4 00b8 E.....@.@.CQ.... - 0x0010: 0af4 00b6 b040 13be 0e0c bb55 9b03 4f99 .....@.....U..O. - 0x0020: 8018 01f6 17f9 0000 0101 080a d742 dc7a .............B.z - 0x0030: 8e66 dce5 7b22 7365 7276 6963 6522 3a22 .f..{"service":" - 0x0040: 7465 7374 2d70 726f 7879 322e 636f 7274 test-proxy2.cort - 0x0050: 6578 666c 6f77 222c 2264 6972 6563 7469 exflow","directi - 0x0060: 6f6e 223a 2249 6e63 6f6d 696e 6722 2c22 on":"Incoming"," - 0x0070: 7061 796c 6f61 6422 3a22 6579 4a77 5958 payload":"eyJwYX - 0x0080: 6c73 6232 466b 496a 6f67 496b 686c 6247 lsb2FkIjogIkhlbG - 0x0090: 7876 4947 5a79 6232 3067 6348 4a76 6548 xvIGZyb20gcHJveH - 0x00a0: 6b74 6332 6c6b 5a57 4e68 6369 4a39 227d ktc2lkZWNhciJ9"} - 0x00b0: 0a . -10:59:19.669904 eth0 Out IP 10.244.0.182.5054 > 10.244.0.184.45120: Flags [.], ack 126, win 509, options [nop,nop,TS val 2389105893 ecr 3611483258], length 0 - 0x0000: 4500 0034 8131 4000 4006 a23d 0af4 00b6 E..4.1@.@..=.... - 0x0010: 0af4 00b8 13be b040 9b03 4f99 0e0c bbd2 .......@..O..... - 0x0020: 8010 01fd 177c 0000 0101 080a 8e66 dce5 .....|.......f.. - 0x0030: d742 dc7a .B.z -10:59:19.733476 lo In IP 10.244.0.182.49858 > 10.244.0.182.5054: Flags [S], seq 2495395789, win 65495, options [mss 65495,sackOK,TS val 1960120607 ecr 0,nop,wscale 7], length 0 - 0x0000: 4500 003c 800a 4000 4006 a35e 0af4 00b6 E..<..@.@..^.... - 0x0010: 0af4 00b6 c2c2 13be 94bc b7cd 0000 0000 ................ - 0x0020: a002 ffd7 1782 0000 0204 ffd7 0402 080a ................ - 0x0030: 74d5 111f 0000 0000 0103 0307 t........... -10:59:19.733496 lo In IP 10.244.0.182.5054 > 10.244.0.182.49858: Flags [S.], seq 4063665784, ack 2495395790, win 65483, options [mss 65495,sackOK,TS val 1960120607 ecr 1960120607,nop,wscale 7], length 0 - 0x0000: 4500 003c 0000 4000 4006 2369 0af4 00b6 E..<..@.@.#i.... - 0x0010: 0af4 00b6 13be c2c2 f236 9e78 94bc b7ce .........6.x.... - 0x0020: a012 ffcb 1782 0000 0204 ffd7 0402 080a ................ - 0x0030: 74d5 111f 74d5 111f 0103 0307 t...t....... -10:59:19.733514 lo In IP 10.244.0.182.49858 > 10.244.0.182.5054: Flags [.], ack 1, win 512, options [nop,nop,TS val 1960120607 ecr 1960120607], length 0 - 0x0000: 4500 0034 800b 4000 4006 a365 0af4 00b6 E..4..@.@..e.... - 0x0010: 0af4 00b6 c2c2 13be 94bc b7ce f236 9e79 .............6.y - 0x0020: 8010 0200 177a 0000 0101 080a 74d5 111f .....z......t... - 0x0030: 74d5 111f t... -10:59:19.734675 lo In IP 10.244.0.182.49858 > 10.244.0.182.5054: Flags [P.], seq 1:139, ack 1, win 512, options [nop,nop,TS val 1960120609 ecr 1960120607], length 138 - 0x0000: 4500 00be 800c 4000 4006 a2da 0af4 00b6 E.....@.@....... - 0x0010: 0af4 00b6 c2c2 13be 94bc b7ce f236 9e79 .............6.y - 0x0020: 8018 0200 1804 0000 0101 080a 74d5 1121 ............t..! - 0x0030: 74d5 111f 7b22 7061 796c 6f61 6422 3a22 t...{"payload":" - 0x0040: 6579 4a74 5a58 4e7a 5957 646c 496a 6f69 eyJtZXNzYWdlIjoi - 0x0050: 6531 7769 6347 4635 6247 3968 5a46 7769 e1wicGF5bG9hZFwi - 0x0060: 4f69 4263 496b 686c 6247 7876 4947 5a79 OiBcIkhlbGxvIGZy - 0x0070: 6232 3067 6348 4a76 6548 6b74 6332 6c6b b20gcHJveHktc2lk - 0x0080: 5a57 4e68 636c 7769 6653 4a39 222c 2273 ZWNhclwifSJ9","s - 0x0090: 6572 7669 6365 223a 2274 6573 742d 7072 ervice":"test-pr - 0x00a0: 6f78 7932 222c 2264 6972 6563 7469 6f6e oxy2","direction - 0x00b0: 223a 224f 7574 636f 6d69 6e67 227d ":"Outcoming"} -10:59:19.734740 lo In IP 10.244.0.182.5054 > 10.244.0.182.49858: Flags [.], ack 139, win 511, options [nop,nop,TS val 1960120609 ecr 1960120609], length 0 - 0x0000: 4500 0034 ca9c 4000 4006 58d4 0af4 00b6 E..4..@.@.X..... - 0x0010: 0af4 00b6 13be c2c2 f236 9e79 94bc b858 .........6.y...X - 0x0020: 8010 01ff 177a 0000 0101 080a 74d5 1121 .....z......t..! - 0x0030: 74d5 1121 t..! -10:59:19.734972 lo In IP 10.244.0.182.5054 > 10.244.0.182.49858: Flags [P.], seq 1:22, ack 139, win 512, options [nop,nop,TS val 1960120609 ecr 1960120609], length 21 - 0x0000: 4500 0049 ca9d 4000 4006 58be 0af4 00b6 E..I..@.@.X..... - 0x0010: 0af4 00b6 13be c2c2 f236 9e79 94bc b858 .........6.y...X - 0x0020: 8018 0200 178f 0000 0101 080a 74d5 1121 ............t..! - 0x0030: 74d5 1121 7b22 7374 6174 7573 223a 2272 t..!{"status":"r - 0x0040: 6563 6569 7665 6422 7d eceived"} -10:59:19.735041 lo In IP 10.244.0.182.49858 > 10.244.0.182.5054: Flags [.], ack 22, win 512, options [nop,nop,TS val 1960120609 ecr 1960120609], length 0 - 0x0000: 4500 0034 800d 4000 4006 a363 0af4 00b6 E..4..@.@..c.... - 0x0010: 0af4 00b6 c2c2 13be 94bc b858 f236 9e8e ...........X.6.. - 0x0020: 8010 0200 177a 0000 0101 080a 74d5 1121 .....z......t..! - 0x0030: 74d5 1121 t..! -10:59:19.735069 lo In IP 10.244.0.182.5054 > 10.244.0.182.49858: Flags [P.], seq 22:38, ack 139, win 512, options [nop,nop,TS val 1960120609 ecr 1960120609], length 16 - 0x0000: 4500 0044 ca9e 4000 4006 58c2 0af4 00b6 E..D..@.@.X..... - 0x0010: 0af4 00b6 13be c2c2 f236 9e8e 94bc b858 .........6.....X - 0x0020: 8018 0200 178a 0000 0101 080a 74d5 1121 ............t..! - 0x0030: 74d5 1121 4d65 7373 6167 6520 5265 6365 t..!Message.Rece - 0x0040: 6976 6564 ived -10:59:19.735085 lo In IP 10.244.0.182.49858 > 10.244.0.182.5054: Flags [.], ack 38, win 512, options [nop,nop,TS val 1960120609 ecr 1960120609], length 0 - 0x0000: 4500 0034 800e 4000 4006 a362 0af4 00b6 E..4..@.@..b.... - 0x0010: 0af4 00b6 c2c2 13be 94bc b858 f236 9e9e ...........X.6.. - 0x0020: 8010 0200 177a 0000 0101 080a 74d5 1121 .....z......t..! - 0x0030: 74d5 1121 t..! -10:59:19.735126 lo In IP 10.244.0.182.5054 > 10.244.0.182.49858: Flags [F.], seq 38, ack 139, win 512, options [nop,nop,TS val 1960120609 ecr 1960120609], length 0 - 0x0000: 4500 0034 ca9f 4000 4006 58d1 0af4 00b6 E..4..@.@.X..... - 0x0010: 0af4 00b6 13be c2c2 f236 9e9e 94bc b858 .........6.....X - 0x0020: 8011 0200 177a 0000 0101 080a 74d5 1121 .....z......t..! - 0x0030: 74d5 1121 t..! -10:59:19.735193 lo In IP 10.244.0.182.49858 > 10.244.0.182.5054: Flags [F.], seq 139, ack 39, win 512, options [nop,nop,TS val 1960120609 ecr 1960120609], length 0 - 0x0000: 4500 0034 800f 4000 4006 a361 0af4 00b6 E..4..@.@..a.... - 0x0010: 0af4 00b6 c2c2 13be 94bc b858 f236 9e9f ...........X.6.. - 0x0020: 8011 0200 177a 0000 0101 080a 74d5 1121 .....z......t..! - 0x0030: 74d5 1121 t..! -10:59:19.735229 lo In IP 10.244.0.182.5054 > 10.244.0.182.49858: Flags [.], ack 140, win 512, options [nop,nop,TS val 1960120609 ecr 1960120609], length 0 - 0x0000: 4500 0034 caa0 4000 4006 58d0 0af4 00b6 E..4..@.@.X..... - 0x0010: 0af4 00b6 13be c2c2 f236 9e9f 94bc b859 .........6.....Y - 0x0020: 8010 0200 177a 0000 0101 080a 74d5 1121 .....z......t..! - 0x0030: 74d5 1121 t..! -10:59:19.735285 eth0 Out IP 10.244.0.182.5054 > 10.244.0.184.45120: Flags [P.], seq 1:38, ack 126, win 509, options [nop,nop,TS val 2389105958 ecr 3611483258], length 37 - 0x0000: 4500 0059 8132 4000 4006 a217 0af4 00b6 E..Y.2@.@....... - 0x0010: 0af4 00b8 13be b040 9b03 4f99 0e0c bbd2 .......@..O..... - 0x0020: 8018 01fd 17a1 0000 0101 080a 8e66 dd26 .............f.& - 0x0030: d742 dc7a 7b22 7374 6174 7573 223a 2272 .B.z{"status":"r - 0x0040: 6563 6569 7665 6422 7d4d 6573 7361 6765 eceived"}Message - 0x0050: 2052 6563 6569 7665 64 .Received -10:59:19.735394 eth0 In IP 10.244.0.184.45120 > 10.244.0.182.5054: Flags [.], ack 38, win 502, options [nop,nop,TS val 3611483323 ecr 2389105958], length 0 - 0x0000: 4500 0034 dfa1 4000 4006 43cd 0af4 00b8 E..4..@.@.C..... - 0x0010: 0af4 00b6 b040 13be 0e0c bbd2 9b03 4fbe .....@........O. - 0x0020: 8010 01f6 177c 0000 0101 080a d742 dcbb .....|.......B.. - 0x0030: 8e66 dd26 .f.& -10:59:19.735423 eth0 Out IP 10.244.0.182.5054 > 10.244.0.184.45120: Flags [P.], seq 38:54, ack 126, win 509, options [nop,nop,TS val 2389105958 ecr 3611483323], length 16 - 0x0000: 4500 0044 8133 4000 4006 a22b 0af4 00b6 E..D.3@.@..+.... - 0x0010: 0af4 00b8 13be b040 9b03 4fbe 0e0c bbd2 .......@..O..... - 0x0020: 8018 01fd 178c 0000 0101 080a 8e66 dd26 .............f.& - 0x0030: d742 dcbb 4d65 7373 6167 6520 5265 6365 .B..Message.Rece - 0x0040: 6976 6564 ived -10:59:19.735468 eth0 In IP 10.244.0.184.45120 > 10.244.0.182.5054: Flags [.], ack 54, win 502, options [nop,nop,TS val 3611483323 ecr 2389105958], length 0 - 0x0000: 4500 0034 dfa2 4000 4006 43cc 0af4 00b8 E..4..@.@.C..... - 0x0010: 0af4 00b6 b040 13be 0e0c bbd2 9b03 4fce .....@........O. - 0x0020: 8010 01f6 177c 0000 0101 080a d742 dcbb .....|.......B.. - 0x0030: 8e66 dd26 .f.& -10:59:19.735505 eth0 Out IP 10.244.0.182.5054 > 10.244.0.184.45120: Flags [F.], seq 54, ack 126, win 509, options [nop,nop,TS val 2389105958 ecr 3611483323], length 0 - 0x0000: 4500 0034 8134 4000 4006 a23a 0af4 00b6 E..4.4@.@..:.... - 0x0010: 0af4 00b8 13be b040 9b03 4fce 0e0c bbd2 .......@..O..... - 0x0020: 8011 01fd 177c 0000 0101 080a 8e66 dd26 .....|.......f.& - 0x0030: d742 dcbb .B.. -10:59:19.735669 eth0 In IP 10.244.0.184.45120 > 10.244.0.182.5054: Flags [F.], seq 126, ack 55, win 502, options [nop,nop,TS val 3611483324 ecr 2389105958], length 0 - 0x0000: 4500 0034 dfa3 4000 4006 43cb 0af4 00b8 E..4..@.@.C..... - 0x0010: 0af4 00b6 b040 13be 0e0c bbd2 9b03 4fcf .....@........O. - 0x0020: 8011 01f6 177c 0000 0101 080a d742 dcbc .....|.......B.. - 0x0030: 8e66 dd26 .f.& -10:59:19.735685 eth0 Out IP 10.244.0.182.5054 > 10.244.0.184.45120: Flags [.], ack 127, win 509, options [nop,nop,TS val 2389105959 ecr 3611483324], length 0 - 0x0000: 4500 0034 8135 4000 4006 a239 0af4 00b6 E..4.5@.@..9.... - 0x0010: 0af4 00b8 13be b040 9b03 4fcf 0e0c bbd3 .......@..O..... - 0x0020: 8010 01fd 177c 0000 0101 080a 8e66 dd27 .....|.......f.' - 0x0030: d742 dcbc .B.. -``` - -
- -### Test 4: Checking proxy metrics export - -The test indicates that the metrics export is functional. Tests were conducted using the bash script [test-proxy-endpoints.sh](https://github.com/CortexFlow/CortexBrain/blob/main/Scripts/test-proxy-endpoints.sh) - -
- Click here to open the test details - - -```bash -🔨 Testing /metrics endpoint -Defaulted container "nginx" out of: nginx, proxy-sidecar, init-iptables (init) -✅ Server is working - Checking /metrics endpoint -Defaulted container "nginx" out of: nginx, proxy-sidecar, init-iptables (init) - % Total % Received % Xferd Average Speed Time Time Time Current - Dload Upload Total Spent Left Speed - 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 127.0.0.1:9090... -* Connected to localhost (127.0.0.1) port 9090 (#0) -> GET /metrics HTTP/1.1 -> Host: localhost:9090 -> User-Agent: curl/7.88.1 -> Accept: */* -> -< HTTP/1.1 200 OK -< content-type: text/plain; charset=utf-8 -< content-length: 2092 -< date: Fri, 28 Mar 2025 11:00:29 GMT -< -{# HELP dns_response_time DNS_response_time -# TYPE dns_response_time histogram -dns_response_time_bucket{server="service_discovery",le="0.005"} 0 -dns_response_time_bucket{server="service_discovery",le="0.01"} 0 -dns_response_time_bucket{server="service_discovery",le="0.025"} 1 -dns_response_time_bucket{server="service_discovery",le="0.05"} 1 -dns_response_time_bucket{server="service_discovery",le="0.1"} 1 -dns_response_time_bucket{server="service_discovery",le="0.25"} 1 -dns_response_time_bucket{server="service_discovery",le="0.5"} 1 -dns_response_time_bucket{server="service_discovery",le="1"} 1 -dns_response_time_bucket{server="service_discovery",le="2.5"} 1 -dns_response_time_bucket{server="service_discovery",le="5"} 1 -dns_response_time_bucket{server="service_discovery",le="10"} 1 -dns_response_time_bucket{server="service_discovery",le="+Inf"} 1 -dns_response_time_sum{server="service_discovery"} 0.015117346 -dns_response_time_count{server="service_discovery"} 1 -dns_response_time_bucket{server="service_discovery_udp",le="0.005"} 0 -dns_response_time_bucket{server="service_discovery_udp",le="0.01"} 0 -dns_response_time_bucket{server="service_discovery_udp",le="0.025"} 1 -dns_response_time_bucket{server="service_discovery_udp",le="0.05"} 1 -dns_response_time_bucket{server="service_discovery_udp",le="0.1"} 1 -dns_response_time_bucket{server="service_discovery_udp",le="0.25"} 1 -dns_response_time_bucket{server="service_discovery_udp",le="0.5"} 1 -dns_response_time_bucket{server="service_discovery_udp",le="1"} 1 -dns_response_time_bucket{server="service_discovery_udp",le="2.5"} 1 -dns_response_time_bucket{server="service_discovery_udp",le="5"} 1 -dns_response_time_bucket{server="service_discovery_udp",le="10"} 1 -dns_response_time_bucket{server="service_discovery_udp",le="+Inf"} 1 -dns_response_time_sum{server="service_discovery_udp"} 0.011202774 -dns_response_time_count{server="service_discovery_udp"} 1 -# HELP total_dns_requests Total_DNS_Requests -# TYPE total_dns_requests counter -total_dns_requests{client_ip="10.244.0.182:5054"} 1 -total_dns_requests{client_ip="10.244.0.184:51969"} 1 - [2092 bytes data] -100 2092 100 2092 0 0 1338k 0 --:--:-- --:--:-- --:--:-- 2042k -* Connection #0 to host localhost left intact -``` - -
- -We can produce the results giving them a better visualization: -![Report1](report_1.png "Test Report 1") - -Since there's only one measure, no advanced statistical analysis can be used. Below a real case will be investigated - -### Test 5: Asynchronus messaging - -In this test, we evaluate the latency differences between the TCP and UDP protocols. -We use the following scripts: - -- [`test-sidecar-advanced-tcp.sh`](https://github.com/CortexFlow/CortexBrain/blob/feature/frontend/Scripts/test-sidecar-advanced-tcp.sh) -- [`test-sidecar-advanced-udp.sh`](https://github.com/CortexFlow/CortexBrain/blob/feature/frontend/Scripts/test-sidecar-advanced-udp.sh) - -The test setup includes four testing pods: `test-proxy`, `test-proxy2`, `test-proxy3`, and `test-proxy4`. Every pod has two containers with `NGINX` and `cortexflow-proxy` - -**Note:** -NGINX is not mandatory, you can use any other application as long as it is containerized. [How to containerize and application?](https://docs.docker.com/get-started/workshop/02_our_app/) - - -### Test Structure - -The communication follows this sequence: - -1. `test-proxy` sends a message to `test-proxy2`. -2. `test-proxy2` responds with a message to `test-proxy`. -3. `test-proxy3` sends a message to `test-proxy2`. -4. `test-proxy4` sends a message to `test-proxy2`. - -These actions are looped for 5 minutes, with multiple asynchronous requests sent at random intervals. - -The communication flow is illustrated in the following diagram: - -![test-architecture](test-architecture.png) - -After 5 minutes, the test concludes, and results can be analyzed using the **metrics endpoint** of `test-proxy` (`/metrics`). - -
-Click here to open the raw results - -```bash - -🔨 Testing /metrics endpoint -Defaulted container "nginx" out of: nginx, proxy-sidecar, init-iptables (init) -✅ Server is working - Checking /metrics endpoint -Defaulted container "nginx" out of: nginx, proxy-sidecar, init-iptables (init) - % Total % Received % Xferd Average Speed Time Time Time Current - Dload Upload Total Spent Left Speed - 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 127.0.0.1:9090... -* Connected to localhost (127.0.0.1) port 9090 (#0) -> GET /metrics HTTP/1.1 -> Host: localhost:9090 -> User-Agent: curl/7.88.1 -> Accept: */* -> -< HTTP/1.1 200 OK -< content-type: text/plain; charset=utf-8 -< content-length: 46860 -< date: Sat, 29 Mar 2025 14:15:18 GMT - -{ [46860 bytes data] -100 46860 100 46860 0 0 10.2M 0 --:--:-- --:--:-- --:--:-- 11.1M -* Connection #0 to host localhost left intact -# HELP dns_response_time DNS_response_time -# TYPE dns_response_time histogram -dns_response_time_bucket{server="service_discovery_tcp",le="0.005"} 900 -dns_response_time_bucket{server="service_discovery_tcp",le="0.01"} 900 -dns_response_time_bucket{server="service_discovery_tcp",le="0.025"} 900 -dns_response_time_bucket{server="service_discovery_tcp",le="0.05"} 900 -dns_response_time_bucket{server="service_discovery_tcp",le="0.1"} 900 -dns_response_time_bucket{server="service_discovery_tcp",le="0.25"} 900 -dns_response_time_bucket{server="service_discovery_tcp",le="0.5"} 900 -dns_response_time_bucket{server="service_discovery_tcp",le="1"} 900 -dns_response_time_bucket{server="service_discovery_tcp",le="2.5"} 900 -dns_response_time_bucket{server="service_discovery_tcp",le="5"} 900 -dns_response_time_bucket{server="service_discovery_tcp",le="10"} 900 -dns_response_time_bucket{server="service_discovery_tcp",le="+Inf"} 900 -dns_response_time_sum{server="service_discovery_tcp"} 0.00011784500000000017 -dns_response_time_count{server="service_discovery_tcp"} 900 -dns_response_time_bucket{server="service_discovery_udp",le="0.005"} 767 -dns_response_time_bucket{server="service_discovery_udp",le="0.01"} 774 -dns_response_time_bucket{server="service_discovery_udp",le="0.025"} 781 -dns_response_time_bucket{server="service_discovery_udp",le="0.05"} 785 -dns_response_time_bucket{server="service_discovery_udp",le="0.1"} 786 -dns_response_time_bucket{server="service_discovery_udp",le="0.25"} 788 -dns_response_time_bucket{server="service_discovery_udp",le="0.5"} 788 -dns_response_time_bucket{server="service_discovery_udp",le="1"} 788 -dns_response_time_bucket{server="service_discovery_udp",le="2.5"} 788 -dns_response_time_bucket{server="service_discovery_udp",le="5"} 788 -dns_response_time_bucket{server="service_discovery_udp",le="10"} 788 -dns_response_time_bucket{server="service_discovery_udp",le="+Inf"} 788 -dns_response_time_sum{server="service_discovery_udp"} 1.2394329659999987 -dns_response_time_count{server="service_discovery_udp"} 788 -# HELP total_dns_requests Total_DNS_Requests -# TYPE total_dns_requests counter -total_dns_requests{client_ip="10.244.0.246:5054_tcp"} 1800 -total_dns_requests{client_ip="10.244.0.247:32846_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:32883_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:32984_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:33020_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:33085_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:33096_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:33186_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:33462_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:33488_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:33647_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:33663_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:33678_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:33738_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:34012_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:34193_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:34524_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:34687_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:34771_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:34785_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:34841_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:34894_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:35046_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:35102_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:35421_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:35437_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:35466_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:35525_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:35563_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:35573_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:35712_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:35720_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:35795_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:35970_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:36116_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:36270_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:36350_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:36425_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:36432_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:36448_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:36449_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:36714_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:36718_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:36961_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:37339_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:37342_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:37415_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:37724_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:37740_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:37853_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:37872_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:37913_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:37927_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:38126_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:38281_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:38444_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:38752_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:38833_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:38884_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:38917_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:38955_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:38960_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:39083_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:39142_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:39192_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:39227_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:39228_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:39262_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:39330_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:39464_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:39509_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:39662_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:39843_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:39868_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:39917_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:39985_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:40013_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:40039_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:40118_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:40149_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:40177_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:40194_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:40238_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:40263_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:40265_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:40383_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:40413_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:40430_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:40480_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:40659_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:41052_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:41139_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:41142_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:41176_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:41183_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:41249_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:41287_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:41499_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:41504_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:41598_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:41620_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:41632_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:41687_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:41720_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:41819_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:41926_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:42044_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:42087_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:42232_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:42251_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:42637_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:42685_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:42833_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:43041_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:43054_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:43445_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:43587_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:43729_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:43750_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:43866_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:43926_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:44129_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:44155_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:44409_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:44454_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:44480_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:44505_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:44658_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:44832_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:44848_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:44853_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:44886_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:45048_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:45230_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:45268_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:45456_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:45570_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:45700_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:45757_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:45774_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:45840_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:45920_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:45993_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:46131_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:46190_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:46226_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:46502_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:47028_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:47228_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:47324_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:47333_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:47334_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:47415_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:47666_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:47685_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:47691_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:47737_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:47761_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:47777_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:47784_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:47788_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:48014_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:48199_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:48261_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:48345_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:48361_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:48752_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:48803_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:48965_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:48982_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:49027_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:49038_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:49047_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:49171_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:49302_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:49414_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:49437_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:49562_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:49998_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:50827_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:50984_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:51010_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:51011_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:51104_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:51133_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:51152_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:51192_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:51193_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:51196_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:51341_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:51440_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:51607_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:52050_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:52067_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:52202_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:52221_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:52287_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:52367_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:52582_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:52793_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:52839_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:53043_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:53238_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:53293_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:54067_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:54169_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:54216_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:54409_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:54661_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:54993_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:55050_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:55111_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:55437_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:55524_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:55689_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:55833_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:55860_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:55960_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:55966_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:56209_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:56236_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:56314_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:56402_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:56491_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:56612_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:56839_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:57049_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:57094_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:57452_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:57524_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:57542_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:57675_udp"} 2 -total_dns_requests{client_ip="10.244.0.247:57808_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:57820_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:57906_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:57939_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:58035_udp"} 2 -total_dns_requests{client_ip="10.244.0.247:58154_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:58462_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:58600_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:58734_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:58758_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:58829_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:58903_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:59149_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:59727_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:59736_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:59848_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:59882_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:59974_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:60230_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:60250_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:60335_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:60353_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:60416_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:60614_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:60619_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:60722_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:60803_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:60853_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:60872_udp"} 1 -total_dns_requests{client_ip="10.244.0.247:60955_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:33031_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:33364_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:33385_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:33453_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:33483_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:33533_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:33631_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:33852_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:33876_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:33953_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:34105_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:34331_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:34399_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:34462_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:34541_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:34571_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:34772_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:34817_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:34871_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:34933_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:34976_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:35336_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:35417_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:35612_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:35667_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:35708_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:35714_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:35763_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:35901_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:35956_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:36020_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:36054_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:36124_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:36189_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:36347_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:36359_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:36435_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:36533_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:36658_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:36801_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:36832_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:36836_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:36963_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:37267_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:37431_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:37557_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:37579_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:37786_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:37796_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:37843_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:38207_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:38236_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:38267_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:38414_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:38607_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:38648_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:38656_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:38728_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:38897_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:38936_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:38967_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:39095_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:39109_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:39134_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:39148_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:39203_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:39462_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:39521_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:39920_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:40041_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:40112_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:40205_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:40225_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:40289_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:40344_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:40371_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:40581_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:40600_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:40773_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:40887_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:41128_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:41184_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:41190_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:41212_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:41285_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:41595_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:41600_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:41656_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:41673_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:41735_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:41749_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:41755_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:42024_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:42516_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:42706_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:42788_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:42803_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:42936_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:43021_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:43023_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:43307_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:43320_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:43345_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:43445_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:43607_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:43704_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:44029_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:44087_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:44094_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:44134_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:44597_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:44948_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:45013_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:45069_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:45092_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:45179_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:45458_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:45467_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:45678_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:45680_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:45693_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:45773_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:45835_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:45849_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:45960_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:46014_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:46369_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:46431_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:46568_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:46797_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:46970_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:46989_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:47092_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:47234_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:47568_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:47582_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:47611_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:47655_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:47661_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:47741_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:47922_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:47950_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:47966_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:48006_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:48059_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:48569_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:48728_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:48736_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:48799_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:48901_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:49049_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:49199_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:49265_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:49382_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:49574_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:49708_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:50102_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:50286_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:50461_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:50492_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:50833_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:50928_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:51004_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:51073_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:51151_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:51243_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:51340_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:51352_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:51376_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:51729_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:51893_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:51959_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:52296_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:52307_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:52366_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:52480_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:52573_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:52696_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:52732_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:52745_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:52911_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:52972_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:53042_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:53061_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:53137_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:53153_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:53565_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:53781_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:54010_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:54110_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:54141_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:54296_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:54345_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:54406_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:54414_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:54535_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:54537_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:54564_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:54744_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:54865_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:54980_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:55013_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:55047_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:55129_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:55282_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:55302_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:55512_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:55513_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:55525_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:55640_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:55672_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:55792_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:56118_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:56249_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:56613_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:56700_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:56707_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:56751_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:56802_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:56881_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:57090_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:57178_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:57450_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:57558_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:58020_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:58141_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:58145_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:58191_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:58313_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:58332_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:58423_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:58440_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:58496_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:58539_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:58775_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:59035_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:59056_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:59153_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:59154_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:59190_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:59231_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:59265_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:59290_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:59332_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:59416_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:59530_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:59553_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:59609_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:59713_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:59754_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:59800_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:59917_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:60082_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:60194_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:60235_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:60283_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:60406_udp"} 2 -total_dns_requests{client_ip="10.244.0.248:60409_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:60477_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:60623_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:60745_udp"} 1 -total_dns_requests{client_ip="10.244.0.248:60819_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:32864_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:32924_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:33104_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:33126_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:33179_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:33314_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:33579_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:33929_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:34027_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:34080_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:34152_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:34378_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:34784_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:34848_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:34985_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:35097_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:35169_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:35202_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:35385_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:35417_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:35521_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:35609_udp"} 2 -total_dns_requests{client_ip="10.244.0.249:35648_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:35657_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:35695_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:35823_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:36016_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:36727_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:36757_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:36786_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:36891_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:36913_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:36940_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:36944_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:37098_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:37103_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:37281_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:37304_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:37449_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:37465_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:37488_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:37518_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:37560_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:37730_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:37871_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:37886_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:37911_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:37913_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:38109_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:38130_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:38193_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:38277_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:38348_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:38352_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:38414_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:38596_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:38906_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:38962_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:39016_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:39123_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:39277_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:39359_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:39397_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:39631_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:39719_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:39742_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:39800_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:39887_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:40062_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:40137_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:40170_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:40267_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:40409_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:40430_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:40497_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:40516_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:40540_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:40696_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:40746_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:40777_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:40923_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:40993_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:41094_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:41313_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:41319_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:41425_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:41473_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:41542_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:41686_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:41699_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:41701_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:41853_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:41945_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:42046_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:42066_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:42216_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:42270_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:42356_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:42373_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:42429_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:42537_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:42545_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:42553_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:42622_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:42685_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:42973_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:43057_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:43347_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:43397_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:43447_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:43487_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:43648_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:43664_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:43712_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:43799_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:43939_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:44326_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:44456_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:44548_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:44553_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:44583_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:44925_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:44926_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:44974_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:45033_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:45077_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:45188_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:45373_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:45422_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:45511_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:45654_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:45985_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:45993_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:46261_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:46326_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:46432_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:46531_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:46586_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:46686_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:46841_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:46985_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:46999_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:47092_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:47425_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:47497_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:47597_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:48112_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:48154_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:48269_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:48736_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:48849_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:49009_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:49438_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:49457_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:49498_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:49536_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:49735_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:49754_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:49925_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:50084_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:50088_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:50150_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:50212_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:50583_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:50755_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:50984_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:51033_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:51035_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:51088_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:51228_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:51355_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:51392_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:51619_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:51687_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:51934_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:51975_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:52089_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:52261_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:52384_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:52401_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:52411_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:52417_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:52461_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:52509_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:52677_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:52968_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:53146_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:53169_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:53292_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:53735_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:53955_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54005_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54015_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54115_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54153_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54298_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54357_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54459_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54462_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54491_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54575_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54588_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54704_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54712_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54744_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54770_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54786_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54866_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54892_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54899_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:54927_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:55008_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:55059_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:55060_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:55247_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:55264_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:55297_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:55299_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:55459_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:55567_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:55837_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:55849_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:56069_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:56111_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:56241_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:56390_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:56504_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:56777_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:56789_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:56801_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:57216_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:57557_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:57815_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:58081_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:58596_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:58621_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:58640_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:58751_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:58863_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:58891_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:59056_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:59107_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:59240_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:59403_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:59416_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:59435_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:59572_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:59618_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:59746_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:59760_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:59763_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:59766_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:59797_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:59978_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:60138_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:60176_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:60360_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:60673_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:60689_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:60927_udp"} 1 -total_dns_requests{client_ip="10.244.0.249:60933_udp"} 1 -``` -
- - -The results of our tests can be resumed with this table: - -| Protocol | Total dns requests | Mean response time (ms) | -| ------- | ------- |-------| -| TCP | 788 | 0.00013 | -| UDP | 900 | 1.57288 | - -Without too much effort we can extract more details from the raw data using a boxplot to better visualize and compare the results that we obtained from the test: -![dns-response-comparison](dns_response_analysis_seaborn.png) - -## Summary and conclusions - -In this report, we analyze the proxy-injector and proxy functionalities following their latest introduction in March 2025. -Specifically, we evaluate TCP and UDP communication between services in a distributed Kubernetes environment. The results confirm that the proxy is fully functional and capable of processing TCP and UDP JSON messages. -Future updates will focus on expanding its capabilities, with a particular emphasis on transitioning to a **proxyless architecture**. -Once implemented, we will conduct a new performance comparison against the current test results. \ No newline at end of file diff --git a/Doc/docs/test-report/reports/March2025/dns_response_analysis_seaborn.png b/Doc/docs/test-report/reports/March2025/dns_response_analysis_seaborn.png deleted file mode 100644 index ac291af9..00000000 Binary files a/Doc/docs/test-report/reports/March2025/dns_response_analysis_seaborn.png and /dev/null differ diff --git a/Doc/docs/test-report/reports/March2025/report_1.png b/Doc/docs/test-report/reports/March2025/report_1.png deleted file mode 100644 index 7aaaedc5..00000000 Binary files a/Doc/docs/test-report/reports/March2025/report_1.png and /dev/null differ diff --git a/Doc/docs/test-report/reports/March2025/test-architecture.png b/Doc/docs/test-report/reports/March2025/test-architecture.png deleted file mode 100644 index c157f8ec..00000000 Binary files a/Doc/docs/test-report/reports/March2025/test-architecture.png and /dev/null differ diff --git a/Doc/docs/test-report/test-report.md b/Doc/docs/test-report/test-report.md index 9a2595a4..b3df3d27 100644 --- a/Doc/docs/test-report/test-report.md +++ b/Doc/docs/test-report/test-report.md @@ -5,7 +5,5 @@ type: blog --- Testing is the foundation of development. A well-tested architecture ensures smooth and reliable performance over time. -Below, you’ll find a list of the latest test reports, each with a detailed explanation of the tests performed. You are welcome to replicate these tests, suggest improvements, or contribute new test cases by submitting a Pull Request. If you encounter any bugs, please report them via GitHub Issues. -For security vulnerabilities, do not open a public issue on GitHub. Instead, please contact directly. -## Latest reports -- [March 2025](reports/March2025/March2025.md) +Below, you’ll find a list of the latest test reports, each with a detailed explanation of the tests performed. You are welcome to replicate these tests, suggest improvements, or contribute new test cases by submitting a Pull Request. +If you encounter any bugs, please report them via GitHub Issues. For security vulnerabilities, do not open a public issue on GitHub. Instead, please contact directly. diff --git a/Doc/docs/user-guide/getting-started.md b/Doc/docs/user-guide/getting-started.md deleted file mode 100644 index e69de29b..00000000 diff --git a/Doc/mkdocs.yml b/Doc/mkdocs.yml index 00cbd621..2c992d15 100644 --- a/Doc/mkdocs.yml +++ b/Doc/mkdocs.yml @@ -96,15 +96,29 @@ plugins: nav: - Home: index.md - Getting Started: getting-started/installation.md + - Quickstart: getting-started/quickstart.md + - Developer Guide: + - developer-guide/index.md + - developer-guide/glossary.md + - developer-guide/architecture.md + - developer-guide/development-workflow.md + - developer-guide/agent-api.md + - developer-guide/metrics.md + - developer-guide/mcp-server.md + - developer-guide/troubleshooting.md + - developer-guide/discussions.md + - GPU (Coming Soon): + - gpu-metrics/gpu-metrics.md - CLI: - cfcli/overview.md - - cfcli/quick-start-guide.md - cfcli/Agent-API-Overview.md + - MCP Server: developer-guide/mcp-server.md + - Test Report: - test-report/test-report.md - Reports: - - test-report/reports/March2025/March2025.md + - test-report/ - Contact: contacts/contact.md diff --git a/README.md b/README.md index 3194ce94..8d66e916 100644 --- a/README.md +++ b/README.md @@ -2,11 +2,11 @@ **Enabling developers to effectively monitor and analyze distributed applications for rapid and efficient problem solving** -_Written in Rust and eBPF for minimum overhead and maximum performance_ +_Written in Rust and eBPF for low-overhead and high-performance_ +
+

+ Supported enviroments   + + Kubernetes + + + Docker + +

+
+ ## 📬Contacts - **Tettamanti Lorenzo** [📧 lorenzotettamanti5@gmail.com](mailto:lorenzotettamanti5@gmail.com) @@ -27,26 +39,37 @@ _Written in Rust and eBPF for minimum overhead and maximum performance_ **CortexBrain** is an ambitious open-source project designed to build an intelligent, lightweight, and highly efficient monitoring platform for distributed cloud and hybrid (cloud–edge) workflows. By leveraging the power of eBPF, CortexBrain can successfully manage **networking** and **observability** in a distributed cluster, limiting resource waste and improving overall performance. -Comprehensive information about CortexBrain’s core architecture, installation, and practical applications is available in the [Official Documentation](https://docs.cortexflow.org/) and on the [CortexFlow blog](https://blog.cortexflow.org/). +Comprehensive information about CortexBrain’s core architecture, installation, and practical applications is available in the [Official Documentation](http://docs.cortexflow.org/getting-started/quickstart/) and on the [CortexFlow blog](https://blog.cortexflow.org/). ## ⚡ Why CortexBrain ? -- **🔎 Deeper Insights**: CortexBrain integrates eBPF with the KubeAPI to produce deeper kernel-level insights of your system +- **🔎 Deeper Insights**: CortexBrain integrates eBPF to produce deeper kernel-level insights of your system without modifying your application code -- **🚁 No sidecar overhead:** Sidecarless architecture that eliminates memory waste and processing overhead +- **🚁 No sidecar overhead:** Sidecarless architecture that eliminates additional CPU and memory overhead associated with sidecar proxies -- **🔒 Safety:** The linux **BPF Verifier** ensures that all the programs are safe to run.A **JIT** compiler converts bytecode into native CPU instructions for optimal execution efficiency. CortexBrain can trace network traffic such as **ingress** (incoming) and **egress** (outgoing) TCP/UDP connections and apply policies directly at **kernel level** by attaching the programs in different hooks such as TC (traffic control) and XDP hooks. All the intercepted events are successfully propagated in the **user space** thanks to BPF maps. +- **🔒 Safety:** The linux **BPF Verifier** ensures that all the programs are safe to run.A **JIT** compiler converts bytecode into native CPU instructions for optimal execution efficiency. CortexBrain can trace network traffic such as **ingress** (incoming) TCP/UDP connections and apply policies directly at **kernel level** by attaching the programs in different hooks such as TC (traffic control) and XDP hooks. All the intercepted events are successfully propagated in the **user space** thanks to BPF maps and dedicated data structures. ## **🧑🏻‍🔬 Current Development Focus** Our current development efforts are dedicated to the following key features: -- **🌐 Open Telemetry Integration:** Integrating the open telemetry stack to our metrics to produce industry standard metrics formats -- **📈 Metrics enhancement:** Expanding the current [metrics](https://docs.cortexflow.org/cfcli/overview/#monitoring-and-status-commands) landscape to provide deeper system understanding -- **📊 DashBoard Integration:** Delivering beautiful _user centered_ data visualization from the collected metrics -- **📡 [Experimental] Cloud-Edge Multi-Cluster Integration:** Enabling orchestration between hybrid cloud and edge environments - **🧪[Experimental] GPU Observability**: Introducing GPU tracing and monitoring capabilities to efficiently support AI/ML applications -![Architecture](Doc/docs/cf_architecture.svg "Cortexflow architecture") +- **🤖 Extending AI integrations** - Extending the current MCP server to seamlessly integrate with coding agent platforms and enable AI-assisted system analysis + +- **🚁 Simplify the pipeline** - Simplify the monitoring pipeline to reduce the overall overhead and reduce points of failures + +- **📡 [Experimental] Cloud-Edge Multi-Cluster Integration:** Extending observability accross hybrid cloud and edge environments + +![Architecture](Doc/docs/assets/cf-new-architecture-readme.png "Cortexflow architecture") + +# Documentation +## Table of Contents +- [Architecture](http://docs.cortexflow.org/developer-guide/architecture/#kernel-hooks): Latest version architecture overview +- [Quickstart](http://docs.cortexflow.org/getting-started/quickstart/): Quickstart guide +- [Developer Guide](http://docs.cortexflow.org/developer-guide/): Full developer guide +- [Common Issues](http://docs.cortexflow.org/getting-started/quickstart/#common-issues-while-using-ebpf-in-a-local-setup): Common documented issues encountered while programming with the eBPF framework +- [MCP server](http://docs.cortexflow.org/developer-guide/mcp-server/): Architecture, building and configuration with opencode +- [CLI](http://docs.cortexflow.org/cfcli/overview/): Full CLI documentation covering setup and commands ## 🤖 Getting Started @@ -75,23 +98,23 @@ cfcli service list ## 💪🏻 Contributing Do you think the project is missing something? Contributing is the best way to show your skills and leave your mark on a project. -If you know DevOps/Kubernetes, networking, security, or you enjoy maintaining a repository, please write an email to lorenzotettamanti5@gmail.com +If you know DevOps/Kubernetes, networking, security, or you just enjoy maintaining a repository, please write an email to lorenzotettamanti5@gmail.com | **Role** | **Skills** | **Tasks** | **Related Issues and Milestones** | | ------------------------- | ------------------------------------------------------------------------- | --------------- |--------| -| **CortexBrain Core Developer** | - Kubernetes
- Networks
- Rust programming language | - Work alongside us to build and optimize the core functionalities (Client,DNS,Proxy,Telemetry,etc..)
| - [Rust](https://github.com/CortexFlow/CortexBrain/labels/rust)
- [Core](https://github.com/CortexFlow/CortexBrain/milestone/1)
- [eBPF](https://github.com/CortexFlow/CortexBrain/labels/ebpf) -| **CortexBrain Dashboard Developer** | - React
- Frontend Development
- Javascript/TypeScript | - Work alongside us to design and improve the dashboard
| [Javascript](https://github.com/CortexFlow/CortexBrain/labels/javascript) +| **CortexBrain Core Developer** | - Kubernetes
- Networks
- Rust programming language | - Help us to build and optimize the core functionalities (Client,DNS,Proxy,Telemetry,etc..)
| - [Rust](https://github.com/CortexFlow/CortexBrain/labels/rust)
- [Core](https://github.com/CortexFlow/CortexBrain/milestone/1)
- [eBPF](https://github.com/CortexFlow/CortexBrain/labels/ebpf) | **General Mantainers** | - Github
- Practical organition
- Documentation | - Keep the repository organized and clean
- Write/Update documentation
- Spot typos in the repository | - [Documentation](https://github.com/CortexFlow/CortexBrain/labels/documentation)
- [question](https://github.com/CortexFlow/CortexBrain/labels/question) -| **Code Reviewers/Testers** | - Rust
- Javascript/TypeScript
- Kubernetes
- Docker | - Review code and suggest changes/optimizations
- Write tests for CI/CD | [Code refactoring](https://github.com/CortexFlow/CortexBrain/labels/code%20refactoring) +| **Code Reviewers/Testers** | - Rust
- Kubernetes
- Docker | - Review code and suggest changes/optimizations
- Write tests for CI/CD | [Code refactoring](https://github.com/CortexFlow/CortexBrain/labels/code%20refactoring) ## 🤖 How to Contribute? We welcome contributions from the community! To contribute to the project, please follow these steps: - Fork the repository. -- Check out [Contributing Best Practices](https://github.com/CortexFlow/CortexBrain/blob/main/CONTRIBUTING.md) +- Check out [Developer guide](http://docs.cortexflow.org/developer-guide/) - Create a new branch for your feature (`git checkout -b feature/feature-name`). - Submit a Pull Request with a detailed explanation of your changes. + ## 🙋**Proposing New Features** If you would like to contribute a new feature, we ask you to open a discussion before submitting a PR. This is to ensure that all new features align with the project's goals and to avoid overlapping work or conflicting views. @@ -100,6 +123,12 @@ Please initiate a discussion in the [GitHub Discussions](https://github.com/Cort By collaborating in this manner, we can maintain clarity and consistency, ensuring that all contributors are working towards the same objectives. Thank you for your understanding and contributions! +## AI Policy +We accept Pull Requests containing AI-generated code, subject to certain conditions. First of all, make sure the new functionalities are well documented; secondly, prove that you can understand and explain the idea behind the code you have submitted. We don't discourage the use of AI to help make new updates; we just want to make sure that there was a clear human thinking process behind the code you submit. + +### Towars AI generated content +The codebase is entirely written by humans, and the AI is used to flatten the knowledge gap required to fully understand the Linux kernel codebase- things like: where is this tracepoint located? Or what is the offset for the tgid field in this kernel structure?. We used AI-assisted development to speed up the building of the backbones of the documentation, but all the sections were carefully reviewed by humans. The illustration of the [architecture](./Doc/docs/assets/cf-new-architecture-readme.png) was enhanced with AI, this is the [version](./Doc/docs/assets/cf-architecture-0.1.5.png) we submitted to the AI agent + ## 🐐 Top contributors [![Top contributors](https://images.repography.com/54717595/CortexFlow/CortexBrain/top-contributors/bRL3WTk3lP0LlkiA2QM-GAH_NLqgBwcXYg8aH_s_9Fg/_YHQeQ-ptyH2aRy6rfxNfiMSSDWLoxKWQgKovd2sKJM_table.svg)](https://github.com/CortexFlow/CortexBrain/graphs/contributors)