A deep dive into the design and implementation of GhostScope's eBPF-based runtime tracing system.
The normative guarantees and failure semantics are defined in Design Guarantees and Trust Model. This document explains the mechanisms that enforce that contract.
GhostScope does not obtain source semantics from any single component. A rendered value is the end of an evidence chain:
requested target scope
|
v
runtime PID / namespace / module mapping
|
v
module-relative probe PC + matching DWARF
|
v
PC-specific scope, type, and location plan
|
v
bounded eBPF read + attributed event
|
v
validated protocol record or explicit failure state
| Stage | Invariants | Enforcement and failure behavior |
|---|---|---|
| Target selection | SCOPE-1, IDENT-1 |
Linux x86_64 build guards and target ELF validation establish the supported platform; -p/-t mode semantics, PID filters, and namespace-aware discovery define the allowed runtime scope. |
| Module and debug information | IDENT-1, SEM-1 |
Runtime mappings, module cookies, load offsets, and available debug-file identity checks bind semantics to a module; unverifiable or loose matches are surfaced as weakened evidence. |
| Semantic planning | SEM-1, FAIL-1 |
The DWARF engine resolves scope, type, location, and relocation at the probe PC; unsupported plans produce diagnostics instead of guessed values. |
| eBPF execution | SAFE-1, COST-1 |
The compiler emits bounded observational programs, and the kernel verifier rejects unsafe programs. |
| Event transport | LOSS-1 |
RingBuf or PerfEventArray carries events; output-helper failures increment per-trace loss counters. |
| Protocol and rendering | IDENT-1, FAIL-1 |
Trace/PID/TID metadata and structured unavailable, expression-error, and backtrace states remain visible to consumers. |
Static display limits and runtime read statuses travel separately. The DWARF
layer resolves a ValueReadPlanResolution containing an optional capture plan,
path-qualified static notes, and any rejected root adapter report. Notes survive
even when no semantic capture is selected; ordinary DWARF reads retain their
runtime address checks. The compiler adds limits discovered during bounded
capture lowering and finalizes notes for both memory-backed and register-backed
arguments, binding them to expression/type indices in TraceContext. Setup messages deduplicate
resolved expressions, types, and reasons even when internal indices differ.
CLI, TUI creation results, and trace snapshots consume the same structured
notes. Snapshots retain them for info trace, independently of logs and loader
actor ownership. Runtime read failures retain VariableStatus and individual
nested-child statuses; the formatter never parses logs to infer failures.
Diagnostic metadata adds no eBPF memory reads and does not change the binary
status layout. Optional sequence-width metadata identifies proven element-limit
truncation; legacy metadata retains a general capture-limit explanation.
At semantic depth boundaries, bounded type-only lookahead avoids diagnosing plain field-only structs as failed adapters. It does not traverse runtime pointer graphs. Static enum-path notes describe possible branches, not observed read failures. User-facing explanations and next steps are in value diagnostics.
The CLI embeds that same Markdown file for --value-diagnostics-help, using the same
early-exit path as --script-help. CI compares both outputs with their source
documents. Skills route to the installed binary's reference instead of
maintaining a separate copy of the guide.
┌──────────────────────────────────────────────────────────┐
│ Terminal UI (TUI) │
│ ┌──────────────────────────────────┐ │
│ │ TEA Architecture │ │
│ │ (Model-Update-View Pattern) │ │
│ └────────────┬─────────────────────┘ │
│ │ Action Events │
└──────────────────────┼───────────────────────────────────┘
│
┌──────────▼──────────┐
│ Event Registry │ Channel-based Communication
│ (mpsc channels) │
└──────────┬──────────┘
│
┌──────────────────────▼────────────────────────────────────┐
│ Runtime Coordinator │
│ (Tokio-based async orchestration) │
│ │
│ ┌─────────────┐ ┌────────────┐ ┌─────────────┐ │
│ │ GhostSession│ │ DWARF │ │ Trace │ │
│ │ (State) │ │ Analyzer │ │ Manager │ │
│ └─────────────┘ └────────────┘ └─────────────┘ │
│ │
│ Event Loop: tokio::select! { │
│ - Wait for eBPF events (from all loaders) │
│ - Handle runtime commands (from TUI) │
│ - Send status updates │
│ } │
└───────────┬────────────────────────────┬──────────────────┘
│ │
┌────────▼─────────┐ ┌────────▼──────────┐
│ Script Compiler │ │ eBPF Loaders │
│ (Multi-stage) │ │ (Per-Trace Pool) │
└──────────────────┘ └───────────────────┘
│ │
└────────────┬───────────────┘
│
┌──────▼──────┐
│ Target │
│ Process │
│ (uprobes) │
└─────────────┘
GhostScope uses Cargo workspace for modular design:
| Crate | Purpose |
|---|---|
| ghostscope | Main binary and runtime coordinator - orchestrates all components via async event loop |
| ghostscope-compiler | Script compilation pipeline - transforms user scripts into verified eBPF bytecode via LLVM |
| ghostscope-dwarf | PC-context DWARF semantic engine - resolves source locations, visible variables, type layouts, address mappings, and compiler read plans |
| ghostscope-loader | eBPF program lifecycle manager - handles uprobe attachment and RingBuf/PerfEventArray event transport via Aya |
| ghostscope-ui | Terminal user interface - implements interactive TUI with TEA (The Elm Architecture) pattern |
| ghostscope-protocol | Communication protocol - defines message format for eBPF-userspace data exchange |
| ghostscope-platform | Platform abstraction - encapsulates architecture-specific code (calling conventions, ABIs) |
| ghostscope-process | Runtime process introspection and offsets — single source of truth for module cookies and ASLR section offsets in both -p and -t modes; provides PID/module enumeration and cached offsets for loaders/compilers |
Role: Async orchestrator that multiplexes eBPF events and UI commands.
Key responsibilities:
- Polls eBPF ring buffers for trace events (non-blocking)
- Receives commands from UI (script execution, trace enable/disable)
- Forwards events to UI for display
- Manages trace lifecycle
Role: Central state container for the entire tracing session.
Manages:
- DWARF analyzer (debug info for all loaded modules)
- Trace manager (pool of active traces)
- Target process information (PID, binary path)
- Configuration state
Key feature: Progressive loading with callbacks for UI progress updates.
Role: High-performance multi-module debug information system and PC-context semantic planner.
Core Optimizations:
-
Parallel Module Loading
- Asynchronous parallel loading of all process modules (main executable + dynamic libraries)
- Progress callbacks for real-time UI feedback during initialization
- Efficient discovery via
/proc/PID/mapsparsing
-
Cross-Module Symbol Resolution
- Unified namespace across all loaded modules
- Function lookup spanning main binary and shared libraries
- Source line to address mapping with inline function support
- Type resolution across module boundaries
-
Memory-Efficient Caching
- Multi-level cache for frequently accessed symbols
- Lazy evaluation of debug information (parsed on-demand)
- Minimizes memory footprint for large binaries with extensive debug info
-
Address Translation
- Automatic ASLR/PIE address handling
- Virtual address to file offset conversion
- Runtime address mapping for process-specific traces
-
PC-Context Read Planning
- Resolves locals, parameters, globals, and inline scopes at a specific probe PC
- Produces typed read plans for the compiler instead of exposing raw DWARF locations
- Preserves semantic distinctions such as optimized-out values, rebased absolute addresses, and value-backed aggregates
- Reports compile-time diagnostics when a variable is visible but cannot be safely lowered
TODO: Still slow, need to research how GDB optimizes DWARF parsing performance.
Multi-stage pipeline with type safety at each level:
┌──────────────────────────────────────────────────────────┐
│ Stage 1: Script Parsing │
│ │
│ User Script (*.gs) │
│ ↓ │
│ Pest Parser (PEG grammar) │
│ ↓ │
│ Abstract Syntax Tree (AST) │
└──────────────────────────────────────────────────────────┘
↓
┌──────────────────────────────────────────────────────────┐
│ Stage 2: LLVM IR Generation │
│ │
│ AST + PC Context + DWARF Read Plans │
│ ↓ │
│ Plan Lowering (variables, types, availability) │
│ ↓ │
│ LLVM IR (type-safe intermediate representation) │
└──────────────────────────────────────────────────────────┘
↓
┌──────────────────────────────────────────────────────────┐
│ Stage 3: eBPF Backend │
│ │
│ LLVM IR │
│ ↓ │
│ LLVM BPF Backend (optimizations + codegen) │
│ ↓ │
│ eBPF Bytecode (verifier-friendly) │
└──────────────────────────────────────────────────────────┘
The diagram below is from Crafting Interpreters, with the red path highlighting GhostScope's compilation flow. Of course, Pest and LLVM do the heavy lifting for us.
Compilation pipeline diagram (red path shows GhostScope's flow)
Role: Manages lifecycle of multiple independent trace points.
Architecture:
- Each trace has its own eBPF program and ring buffer
- Traces can be independently enabled/disabled
- Resource isolation: one trace's failure doesn't affect others
- Concurrent execution: all uprobes run in parallel in kernel space
Pattern: The Elm Architecture (Model-Update-View)
┌──────────────┐
│ Model │ AppState (immutable UI state snapshots)
└──────┬───────┘
│
┌──────▼───────┐
│ Update │ Event handlers (keypress → Action → State mutation)
└──────┬───────┘
│
┌──────▼───────┐
│ View │ Rendering (State → Terminal output)
└──────────────┘
Benefits:
- Testable: Pure functions for state updates
- Predictable: Same input always produces same output
- Debuggable: Can replay event sequences
- Maintainable: Clear data flow
Communication: Channels to runtime (send commands, receive trace events).
Core mechanism: A kernel event transport selected at startup. GhostScope prefers RingBuf on supported kernels and falls back to PerfEventArray. RingBuf is one shared multi-producer buffer across CPUs; PerfEventArray uses independent per-CPU buffers.
┌───────────────────────────────────────────────────────┐
│ Kernel Space │
│ │
│ ┌────────────┐ │
│ │ eBPF │ Trace event occurs │
│ │ Program │ ↓ │
│ │ (uprobe) │ Collect data (registers, memory) │
│ └─────┬──────┘ ↓ │
│ │ Serialize to protocol format │
│ │ ↓ │
│ │ bpf_ringbuf_output() / │
│ │ bpf_perf_event_output() │
│ │ ↓ │
│ └────────►┌─────────────────────┐ │
│ │ RingBuf (shared) │ │
│ │ or Perf buffers │ │
│ │ (per CPU) │ │
│ │ │ │
│ │ [Event1][Event2]...│ │
│ └──────────┬──────────┘ │
└─────────────────────────────┼─────────────────────────┘
│ Memory-mapped
↓
┌─────────────────────────────┼─────────────────────────┐
│ User Space │ │
│ │ │
│ ┌──────────────────────────▼──────────┐ │
│ │ Trace Manager │ │
│ │ (polls selected transport) │ │
│ └──────────────────────┬───────────────┘ │
│ │ │
│ Read events (non-blocking) │
│ ↓ │
│ ┌──────────────────────────────────────┐ │
│ │ Streaming Parser │ │
│ │ (handles variable-length messages) │ │
│ └──────────────────────┬───────────────┘ │
│ │ │
│ Parsed trace events │
│ ↓ │
│ ┌──────────────────────────────────────┐ │
│ │ Runtime Coordinator │ │
│ │ (forwards to UI) │ │
│ └──────────────────────────────────────┘ │
└───────────────────────────────────────────────────────┘
-
Event Generation (Kernel):
- Uprobe fires when target instruction executes
- eBPF program collects data (registers, stack, memory via DWARF locations)
- Serializes data according to protocol format
- Calls the selected RingBuf or PerfEventArray output helper
- Increments the trace's loss counter if the output helper fails
-
Event Polling (User Space):
- Trace manager polls the selected transport (via Aya framework)
- Non-blocking: Returns immediately if no events
- RingBuf uses a shared memory-mapped buffer; PerfEventArray drains per-CPU buffers
-
Event Parsing:
- Streaming parser handles variable-length messages
- Parser applies transport-specific framing and tracks partial RingBuf reads
- Reconstructs complete events
-
Event Delivery:
- Parsed events sent to runtime coordinator
- Coordinator forwards to UI via channel
- UI updates display in real-time
-
Loss Reporting:
- Runtime periodically reads each trace's eBPF output-failure counter
- CLI and TUI report interval and cumulative loss totals
- A nonzero counter marks the observation interval as incomplete
GhostScope uses an instruction-based protocol for flexible trace event representation:
┌─────────────────────────────────────────────────────┐
│ TraceEventHeader (16 bytes) │
│ - magic: u32 (0x43484C53 "CHLS") │
│ - reserved: u32 │
│ - generation: u64 (captured in eBPF) │
├─────────────────────────────────────────────────────┤
│ TraceEventMessage (24 bytes) │
│ - trace_id: u64 │
│ - timestamp: u64 │
│ - pid: u32 │
│ - tid: u32 │
├─────────────────────────────────────────────────────┤
│ Instruction Sequence (variable length) │
│ │
│ ┌──────────────────────────────────────┐ │
│ │ InstructionHeader (4 bytes) │ │
│ │ - inst_type: u8 │ │
│ │ - data_length: u16 │ │
│ │ - reserved: u8 │ │
│ ├──────────────────────────────────────┤ │
│ │ InstructionData (variable length) │ │
│ │ - Depends on instruction type │ │
│ └──────────────────────────────────────┘ │
│ │
│ ... (more instructions) ... │
│ │
│ ┌──────────────────────────────────────┐ │
│ │ EndInstruction (final marker) │ │
│ │ - total_instructions: u16 │ │
│ │ - execution_status: u8 │ │
│ └──────────────────────────────────────┘ │
└─────────────────────────────────────────────────────┘
Instruction Types:
| Type | Code | Purpose |
|---|---|---|
| PrintStringIndex | 0x01 | Print static string (indexed) |
| PrintVariableIndex | 0x02 | Print simple variable with type info |
| PrintComplexVariable | 0x03 | Print struct/array with access path |
| PrintComplexFormat | 0x05 | Formatted print with complex variables |
| Backtrace | 0x10 | Stack backtrace with frame addresses |
| EndInstruction | 0xFF | Marks end of instruction sequence |
Backtrace is a compact frame stream, not pre-rendered text. The compiler asks
ghostscope-dwarf for compact DWARF CFI rows, loads those rows into a BPF array
map, and the uprobe program records module cookies plus module-normalized PCs.
Userspace then resolves raw IPs through the process module map and asks
ghostscope-dwarf for function, source line, and inline-chain information.
bt always means DWARF unwinding; the script language intentionally does not
expose helper/fp/backend selection.
Variable Status Tracking:
Each variable instruction includes a status field (u8) indicating data acquisition result:
| Status | Value | Meaning |
|---|---|---|
| Ok | 0 | Variable read successfully |
| NullDeref | 1 | Attempted to dereference null pointer |
| ReadError | 2 | Memory read failed (invalid address) |
| AccessError | 3 | Memory access denied (permissions) |
| Truncated | 4 | Data truncated (exceeded size limit) |
This per-variable error reporting allows:
- Partial success: Print successfully read variables even if some fail
- Precise diagnostics: Identify exact failure point in complex expressions
- Safe operation: eBPF program continues execution despite individual read failures