Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -296,11 +296,14 @@ jobs:
- name: Build debug
run: cargo build --all-features

- name: Verify embedded script help matches docs
- name: Verify embedded help matches docs
run: |
tmpfile="$(mktemp)"
trap 'rm -f "$tmpfile"' EXIT
target/debug/ghostscope --script-help > "$tmpfile"
diff -u docs/scripting.md "$tmpfile"
target/debug/ghostscope --value-diagnostics-help > "$tmpfile"
diff -u docs/value-diagnostics.md "$tmpfile"

- name: Build release
run: cargo build --release --all-features
6 changes: 5 additions & 1 deletion README-zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,10 @@ GhostScope 把编译后的二进制重新变成“可观测系统”。在 TUI
- [**使用限制**](docs/zh/limitations.md)
已知的限制和约束

- [**值诊断**](docs/zh/value-diagnostics.md)
理解值不可用、内存读取失败、展示降级和采集限制;
使用 `ghostscope --value-diagnostics-help` 离线查看

</td>
<td width="33%" valign="top">

Expand All @@ -237,7 +241,7 @@ GhostScope 把编译后的二进制重新变成“可观测系统”。在 TUI
输入模式所有可用命令

- [**脚本语言**](docs/zh/scripting.md)
编写强大的追踪脚本
编写追踪脚本;使用 `ghostscope --script-help` 离线查看

</td>
<td width="33%" valign="top">
Expand Down
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,10 @@ See [Limitations](docs/limitations.md) for unsupported and explicitly degraded s
- [**Limitations**](docs/limitations.md)
Known limitations and constraints

- [**Value Diagnostics**](docs/value-diagnostics.md)
Understand unavailable values, read failures, display fallbacks, and capture limits;
read offline with `ghostscope --value-diagnostics-help`

</td>
<td width="33%" valign="top">

Expand All @@ -242,7 +246,7 @@ See [Limitations](docs/limitations.md) for unsupported and explicitly degraded s
All available commands for Input Mode

- [**Script Language**](docs/scripting.md)
Write powerful trace scripts
Write trace scripts; read offline with `ghostscope --script-help`

</td>
<td width="33%" valign="top">
Expand Down
30 changes: 30 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,36 @@ validated protocol record or explicit failure state
| Event transport | `LOSS-1` | RingBuf or PerfEventArray carries events; output-helper failures increment per-trace loss counters. |
| Protocol and rendering | `IDENT-1`, `FAIL-1` | Trace/PID/TID metadata and structured unavailable, expression-error, and backtrace states remain visible to consumers. |

## Value Display Diagnostics

Static display limits and runtime read statuses travel separately. The DWARF
layer resolves a `ValueReadPlanResolution` containing an optional capture plan,
path-qualified static notes, and any rejected root adapter report. Notes survive
even when no semantic capture is selected; ordinary DWARF reads retain their
runtime address checks. The compiler adds limits discovered during bounded
capture lowering and finalizes notes for both memory-backed and register-backed
arguments, binding them to expression/type indices in `TraceContext`. Setup messages deduplicate
resolved expressions, types, and reasons even when internal indices differ.

CLI, TUI creation results, and trace snapshots consume the same structured
notes. Snapshots retain them for `info trace`, independently of logs and loader
actor ownership. Runtime read failures retain `VariableStatus` and individual
nested-child statuses; the formatter never parses logs to infer failures.
Diagnostic metadata adds no eBPF memory reads and does not change the binary
status layout. Optional sequence-width metadata identifies proven element-limit
truncation; legacy metadata retains a general capture-limit explanation.

At semantic depth boundaries, bounded type-only lookahead avoids diagnosing
plain field-only structs as failed adapters. It does not traverse runtime
pointer graphs. Static enum-path notes describe possible branches, not observed
read failures. User-facing explanations and next steps are in
[value diagnostics](value-diagnostics.md).

The CLI embeds that same Markdown file for `--value-diagnostics-help`, using the same
early-exit path as `--script-help`. CI compares both outputs with their source
documents. Skills route to the installed binary's reference instead of
maintaining a separate copy of the guide.

## System Overview

```
Expand Down
4 changes: 4 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,9 @@ ghostscope --script-file trace.gs
# Print the embedded script language reference and exit
ghostscope --script-help

# Explain value failures, display fallbacks, and capture limits offline
ghostscope --value-diagnostics-help

# Choose script-mode event stdout rendering
ghostscope --script-output pretty # default: formatted stdout
ghostscope --script-output plain # payload-only stdout
Expand Down Expand Up @@ -290,6 +293,7 @@ index is reported in CLI/TUI startup status before falling back.
| `--script <SCRIPT>` | `-s` | Inline script to execute | None |
| `--script-file <PATH>` | | Script file to execute | None |
| `--script-help` | | Print the embedded script language reference and exit | Off |
| `--value-diagnostics-help` | | Print the embedded value diagnostics guide and exit | Off |
| `--script-output <MODE>` | | Script event stdout mode: pretty, plain | pretty |
| `--backtrace-depth <N>` | | Max DWARF-unwound frames captured by each `bt`/`backtrace` instruction (`1..=128`) | 128 |
| `--no-backtrace-runtime-modules` | | Disable compact-CFI loading for newly mapped `bt`/`backtrace` modules; events still render with available symbols, module offsets, or raw addresses | Off |
Expand Down
2 changes: 2 additions & 0 deletions docs/input-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -307,6 +307,8 @@ Displays list of available info subcommands.

### info trace - View Trace Status

Individual trace details also retain static value-display limits, affected paths, reasons, and documentation links. See [value diagnostics](value-diagnostics.md).

**Syntax:**
```
info trace [id]
Expand Down
2 changes: 2 additions & 0 deletions docs/limitations.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Limitations

For value failure and fallback messages and next steps, see [value diagnostics](value-diagnostics.md).

This document describes conditions outside the supported operating envelope or
cases that degrade explicitly. The guarantees that remain mandatory inside the
supported envelope are defined in the
Expand Down
11 changes: 7 additions & 4 deletions docs/scripting.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,8 @@ Practical guidance:

### Rust Value Presentation

For unavailable values, read failures, display fallbacks, and capture limits, see [value diagnostics](value-diagnostics.md), also available offline with `ghostscope --value-diagnostics-help`. CLI prints static display notes by default; TUI retains them in `info trace <id>`.

Rust value support does not add Rust-specific syntax to the DSL. After the DSL
resolves an expression, the DWARF layer may select a bounded semantic read plan
for its root type. If the concrete type identity or layout does not validate,
Expand Down Expand Up @@ -220,13 +222,14 @@ trace do_stuff {
explicit memory-read behavior.
- `ebpf.mem_dump_cap` bounds each indirect argument and defaults to 256 bytes.
A semantic value whose logical length exceeds the captured prefix is marked
`<truncated>`.
`<truncated: byte limit>`, `<truncated: element limit>`, or the more general
`<truncated: capture limit>`, depending on the available evidence.

For example, with `mem_dump_cap = 3`, an `&str` containing
`"hello from rust"` is rendered as:

```text
"hel" <truncated>
"hel" <truncated: byte limit>
```

The value can also be read explicitly when its target DWARF exposes known
Expand Down Expand Up @@ -376,7 +379,7 @@ DWARF variables include locals, parameters, and globals from the traced program.
| Enum | `enum E` | Enum (via base int) | Printed as `Type::Variant`; arithmetic/compare uses base integer |
| Bitfield | `int flags:3` | Bitfield → integer view | Extracted integer; mixes with script ints/bools |
| Typedef/Qualified | `typedef`, `const`, `volatile` | Typedef/QualifiedType | Treated as underlying type |
| Optimized‑out | variable optimized away | OptimizedOut | Read fails; renders `<OPTIMIZED_OUT>`; operations follow failure semantics |
| Optimized‑out | variable optimized away | OptimizedOut | No recoverable value at this location; renders `<unavailable: optimized out>`; operations follow failure semantics |
| Unknown | unsupported/unknown | Unknown | Renders `<UNKNOWN_TYPE_N_BYTES>` |

#### Supported Complex Access
Expand Down Expand Up @@ -532,7 +535,7 @@ Notes:
- Reads are type-sensitive. For `{:x}`, the DWARF/script type controls the captured value size and how the value is materialized. For `{:x.N}`/`{:s.N}`, pointers use the pointer value as the read address, arrays/aggregates use their base address, and addressable scalar DWARF variables use their storage address. A pure script integer is not addressable and will be rejected unless you explicitly cast it to a pointer type.
- Kernel performs bounded reads for memory-dump forms; user space renders hex/ASCII. For `{:s}` ASCII, rendering stops at first NUL; non‑printables show as `\xNN`.
- Per‑argument read cap is controlled by `ebpf.mem_dump_cap` (default 256 bytes). Requests beyond cap are truncated; if event payload is exceeded, output may also truncate with `…`.
- On read failure (e.g., null deref, offsets unavailable, permission), extended specifiers print `<MISSING_ARG>`.
- Extended specifiers preserve runtime read-failure messages (such as null dereference or unavailable offsets). `<MISSING_ARG>` means a formatting argument is missing. See [value diagnostics](value-diagnostics.md).

Example:

Expand Down
175 changes: 175 additions & 0 deletions docs/value-diagnostics.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
# Understanding value diagnostics

GhostScope distinguishes an unavailable value, a failed memory read, a display
fallback, and a partial capture. A display note does not mean the trace failed
to attach. Successfully captured fields remain available.

| Output | Meaning | Next step |
| --- | --- | --- |
| `<unavailable: optimized out>` | No recoverable value at this probe location | [Choose another location](#optimized-out) |
| `<unreadable: memory read failed; errno=…; address=…>` | This read failed; possible causes include a page that cannot be faulted in or an invalid address | [Check memory reads; consider sleepable uprobes](#memory-read-failed) |
| `<internal fields: layout unsupported>` | The type's contents could not be interpreted; physical fields are shown | [Check supported layouts](#layout-unsupported) |
| `<internal fields: read plan unsupported>` | Debug information could not describe the required reads | [Inspect the debug information](#read-plan-unsupported) |
| `<truncated: byte limit>` | Only a prefix of the bytes was captured | [Check the byte budget](#byte-limit) |
| `<truncated: element limit>` | The nested sequence reached the configured width | [Check the element limit](#element-limit) |
| `<truncated: capture limit>` | Capture was incomplete; the status alone does not identify which limit | [Check capture limits](#capture-limit) |
| `<not expanded: capture budget>` | Nested capture did not fit; the root display is retained | [Check the nested budget](#capture-budget) |
| `<nested display limits: depth limit; see trace details>` | Some nested types have static display limits | Read the trace's display notes |

## Finding the reason

Read this guide offline from the installed binary:

```bash
ghostscope --value-diagnostics-help
```

Like `--script-help`, this prints the embedded English reference to stdout and
exits without loading configuration, inspecting a target, or requiring eBPF
privileges. The content ships with the binary, so it is also available to LLM
tools without a source checkout or network access.

CLI mode prints static display notes to stderr once per expression/type/reason
in each compiled trace, even with `--no-log` and `--no-status`. TUI mode shows
them in trace creation results and retains them in `info trace <id>`. This also
applies to traces loaded from a saved file. Dry runs report the same static
notes; runtime reads cannot be checked until a probe runs.

A note names the expression and affected path, the type, a stable reason, the
technical detail, and a link to this page. `[]` denotes collection elements;
`::Some.__0` denotes a possible enum payload (field names follow DWARF).
These are **capture-plan limits**:
they do not claim that a collection is nonempty or that a variant is active.
Runtime read errors belong to the actual captured value or child slot.

Normal structs do not need a special adapter. Their ordinary field display is
not itself a failure. Debug logs and `dwarf-tool rust-adapter` remain useful for
deeper investigation, but are not required to see a display fallback.

## optimized-out

`<unavailable: optimized out>` means the selected location has no recoverable
value for that variable. Direct printing can emit this placeholder; using the
variable in arithmetic or taking its address fails during script compilation.

Try a source location where the value is still live. If rebuilding the target
is possible, retain debug information and consider reducing optimization.
Increasing capture limits or enabling sleepable uprobes cannot recover a value
that the compiler removed.

## memory-read-failed

`<unreadable: memory read failed>` means a target-memory read failed on this
event. **The errno alone, including `-EFAULT`, cannot distinguish a page that
cannot be faulted in from an invalid address.** Other possibilities include
inaccessible memory, a value observed during mutation, mismatched debug
information, or incorrect address calculation by GhostScope. This is a runtime
read failure; a static depth-limit note does not explain its cause. Other
successfully captured fields remain useful; the event is not a process-wide
atomic snapshot.

First confirm that the binary and separate debug information match, and that
the probe location and object's lifetime are appropriate. A target pointer may
itself be null, dangling, or changing concurrently even with matching debug
information. **If GhostScope computes an invalid address for an object known
to be valid and live, with matching debug information, that is a GhostScope
bug.** Report the expression, source/probe location, exact output (including
errno and address), binary/debug-file identity, GhostScope version, and a
minimal reproducer when possible. A failed read alone does not establish which
component is responsible.

If the memory is valid but may not be resident, retry the existing command
with **`--sleepable-uprobe`**, or use:

```toml
[ebpf]
sleepable_uprobe = true
```

Sleepable uprobes allow supported reads to fault in pages. They require Linux
5.18+ and RingBuf output. Fixed-length reads use a fault-capable helper;
NUL-terminated reads that use `bpf_probe_read_user_str()` still cannot fault in
pages. This option cannot fix an invalid address, missing DWARF, or an
unsupported type layout.

Enabling it can add probe latency, especially when a read actually faults or
blocks on I/O. The impact may be small for a particular workload, but is not
guaranteed to be small; measure the target's latency and probe frequency.
See [Sleepable Uprobe](configuration.md#sleepable-uprobe) for kernel checks,
output compatibility, and backtrace behavior.

Null-pointer dereference, address-computation failure, and unavailable process
offsets have their own messages. Do not assume that enabling sleepable uprobes
resolves those conditions.

## layout-unsupported

The type matched a known adapter, but its target DWARF layout did not satisfy
the adapter's checks. GhostScope retains the ordinary field representation.
These are implementation fields, not the semantic contents of a string or
collection. Increasing a capture budget will not make this layout supported.

Check [Rust value rendering](scripting.md#rust-value-presentation). For a report,
include the type, rejection detail, target rustc version, matching debug
information, and a minimal reproducer. If ordinary DWARF capture also fails,
the compilation error includes the adapter's rejection report.

## read-plan-unsupported

The required dependent type, pointer target, projection, width, or alignment
could not be resolved from the target debug information. The note includes the
available technical reason. Check that the target's matching, full debug
information is available; report a reproducible unsupported case. Raising
memory limits or enabling sleepable uprobes does not supply missing DWARF.

## byte-limit

A string or byte-string capture returned only a prefix. Check
`[ebpf].mem_dump_cap` (default 256 bytes per indirect argument). Nested values
share this budget with their root, metadata, and sibling values, so a child
can receive fewer bytes than that setting. An omitted payload instead reports
the broader `capture limit`, because it could also reflect the event budget.

## element-limit

The captured nested sequence reached `value_adapters.max_sequence_elements`.
Only the displayed elements were captured. Print a specific element or increase
that setting if more elements are needed; the shared byte budget still applies.

## capture-limit

The result is partial. The available runtime status does not always distinguish
byte, element, sparse-bucket, tree-node, and total-event limits. GhostScope does
not invent a more specific cause when the capture metadata cannot prove it.

Check `[ebpf].mem_dump_cap`, `[ebpf].max_trace_event_size`, and
`[value_adapters].max_sequence_elements` (default 4). The last setting limits
nested sequence elements or hash buckets; sparse buckets may contain fewer
actual entries. See [Value Adapter limits](configuration.md#value-adapter-limits).
Increase only the relevant limit: larger captures can increase eBPF program
size, event size, and probe overhead.

## depth-limit

The semantic planner stopped expanding nested contents at
`[value_adapters].max_nesting_depth` (default 4). The note identifies the path
beyond the limit. A plain struct or enum payload also consumes a semantic edge.
This is not a memory-read error and does not imply that the root value is
missing. Inspect a shallower expression or increase the limit deliberately.
The separate native DWARF formatting limit reports `<MAX_DEPTH_EXCEEDED>`.

## recursive-type

The planner encountered a repeated DWARF type on the current expansion path.
It keeps the existing representation rather than following the type
indefinitely. This concerns the type graph; it does not prove that runtime
objects form a cycle. Raising the nesting limit does not disable this guard.
Inspect a specific field or explicitly dereference a known pointer instead.

## capture-budget

The compiled nested capture could not fit within the shared byte budget.
The root display or other captured fields remain, and the note identifies the
affected path when available. Print the field separately, reduce requested
nesting or collection width, or increase `ebpf.mem_dump_cap` deliberately.
This limitation is determined before the probe runs.
Loading
Loading