Trace. Analyze. Optimize.
ETWView is a high-performance Python pipeline that automates WPA and RAMMap analysis: it parses exports from multiple runs across different scenarios, enriches process data with svchost service resolution, aggregates per-process metrics, generates ranked performance comparison charts for side-by-side evaluation, and produces memory breakdown charts from RAMMap snapshots.
- Python 3.9+
- Windows Performance Toolkit (WPT) (Windows ADK) for ETL capture and
wpaexporter.exeexport. - RAMMap (Download) to capture memory snapshots.
- kaleido (optional, for
export_images: true— Plotly static image export; install withpip install -e ".[export]")
pip install -e . # base install
pip install -e ".[export]" # + kaleido for PNG export- Capture — run the appropriate capture script as Administrator (see Capture Scripts below). This produces
.etlfiles. - Configure — edit
cfg.yamlwith your trace paths, scenarios, and metrics. See Configuration. - Discover — inspect available metrics before finalizing the config:
python etwview.py --discover
- Run — process everything and generate charts:
Charts are written to
python etwview.py
<output_dir>/<trace_type>/. A consolidated dataset is written to<output_dir>/_raw_all.csv.
The scripts in trace_scripts/ are examples provided for reference. They demonstrate how to automate WPR/WPA captures for different trace types. You can adapt them for other WPA/WPR metrics by changing the -start profiles, -onoffscenario arguments, and output paths.
Run each as Administrator from trace_scripts/.
Records a battery/power trace using WPR's Power profile. Runs 3 × 30 min iterations across 3 reboots (scheduled task). ETLs saved to %USERPROFILE%\Desktop\Battery\Scenario_<timestamp>\.
Records a boot trace using WPR's Boot scenario. Runs 3 iterations automatically (-onoffscenario Boot -numiterations 3). ETLs saved to %USERPROFILE%\Desktop\Boot\Scenario_<timestamp>\.
Records a General trace (CPU, DiskIO, Power profiles) with WPR and captures a RAMMap memory snapshot per run. Each run: 5 min stabilize + 5 min WPR recording = ~10 min per run. Runs 3 iterations across reboots (scheduled task). Produces both .etl and .rmp files. ETLs/RMPs saved to %USERPROFILE%\Desktop\General\Scenario_<timestamp>\.
When wpa_exporter.enabled: true, ETWView processes each (trace_type, scenario, run_name) combination and extracts the numeric portion of the run name, such as Run1 → 1. It then searches for ETLs in this order:
- Preferred layout:
<trace_path>/<scenario>/<run_name>/*.etl - Fallback layout:
<trace_path>/<scenario>/*.etl, filtered by run number in the filename
A root-level fallback to <trace_path>/*.etl is not attempted for scenario matching — the scenario is not encoded in ETL filenames at that level, making the result ambiguous.
The run-number filter matches filenames by numeric token boundaries (_1_, -1-, _1.) so Run1 maps to ETLs containing run 1 without matching 10 or 21.
Naming guidance:
Preferred:
<trace_path>/
└── <scenario>/
└── Run1/
└── Boot-Run1.etl
Fallback:
<trace_path>/
└── <scenario>/
├── Boot-Run1.etl
├── Boot-Run2.etl
└── Boot-Run3.etl
<base_dir>/
├── <trace_type>/
│ └── <scenario>/
│ ├── svchost/ # svchost resolver CSVs (if enabled; per-scenario)
│ └── <run_name>/
│ └── *.csv # WPA exports
Trace types = trace_metrics keys. Scenarios = scenarios keys or auto-discovered folders. Run names = run_names list or auto-discovered from filenames.
| Key | Required | Description |
|---|---|---|---|
| trace_metrics | yes | Maps trace type (e.g. Boot, General) to list of metric keys. Mapping keys define active trace types. |
| scenarios | no | Maps scenario folder name → display label in charts. Mapping keys define active scenarios; if empty, scenarios are auto-discovered from exported folders. |
| scenario_labels | no | Fallback dict when scenarios is absent. Maps scenario key → display label. |
| trace_path | yes | Input folder per source trace type. Used by ETL discovery (when exporter is enabled). |
| baseline_scenario | no | Scenario key used as baseline for deltas when a block's hybrid_metrics includes delta. Defaults to None (required when delta is used). |
| base_dir | no | Root for output data. Auto-computed as common parent of trace_path + /export when not set. |
| run_names | no | Explicit list of run folder names (e.g. [Run1, Run2, Run3]). Auto-discovered from ETL/CSV filenames when empty (e.g. filenames containing Run1, Boot_1). |
| hybrid_tiebreaker | no | Sort mode for hybrid selection: value_first (default), rank_first, or baseline_rank. See Hybrid Metrics. |
| wpa_exporter | no | ETL export settings (forcepath, workers). Runs automatically if forcepath points to a valid wpaexporter.exe. See subsection below. |
| rammap | no | RAMMap export settings (enabled, path). Exports .rmp → CSV when enabled. See subsection below. |
| svchost | no | Service-name enrichment (enabled). See Svchost Service Enrichment. |
| output_dir | no | Subfolder of base_dir for charts/report (default: _plots). |
| export_images | no | Set true to write PNG files alongside .editor.json sidecars. |
Global settings global_top_n (default: 25), ignore_processes, global_transforms, and entity_column_sets are stored in cfg.yaml, not in style.yaml. See Configuration.
ignore_processes: a list of process names (case-insensitive) to filter out at parse time. Rows whose process column matches an entry are dropped before aggregation and charting. Default: [idle, registry, "?", cmd.exe, reg.exe].
wpa_exporter:
forcepath: "C:\\Path\\To\\wpaexporter.exe" # optional — skipped if not found
workers: 4
optimize_profile: true| Key | Default | Description |
|---|---|---|
forcepath |
(auto-detect) | Override path to wpaexporter.exe. If not set or invalid, ETL export is skipped with a warning. |
workers |
CPU count | Parallel export threads. |
optimize_profile |
true |
When enabled, the WPA profile is optimized to export only columns required by active metrics — reduces CSV output size and export time. Set false to export all profile columns unchanged. |
ETL export runs automatically when forcepath points to a valid wpaexporter.exe. It looks up wpaProfiles/<TraceType>.wpaProfile for each active trace type and writes exported CSVs into the strict <base_dir>/<trace_type>/<scenario>/<run_name>/ layout. base_dir is auto-derived from the common parent of trace_path + /export when not explicitly set.
Existing CSVs in the output folder are skipped on subsequent runs (incremental caching). When svchost.enabled: true, each ETL is exported twice — once with the main profile and once with SvchostEng.wpaProfile to a svchost/ subfolder.
rammap:
enabled: true
path: "C:\\Path\\To\\General" # optional| Key | Default | Description |
|---|---|---|---|
| enabled | false | When true, RAMMap .rmp files are parsed and exported to CSV automatically during the pipeline run. |
| path | (auto-detect) | Root directory with scenario subfolders (e.g. General/WindowsDefault/*.rmp). If omitted, scans trace_path folders and local <trace_type>/ directories. |
| hidden_lists | (from RamMap_Hidden) | PFN list names to exclude from the system memory bar chart. Set via the RamMap_Hidden meta key under trace_metrics (see below). |
RamMap_Hidden is a special trace_metrics key (suffix _Hidden) that is not validated against WPA metrics. Its value is a list of PFN list names to exclude from RAMMap system memory charts. The _Hidden suffix on any trace_metrics key suppresses its entries from CLI output but still processes them.
trace_metrics:
RamMap_Hidden: [ModifiedNoWrite, Transition, Zeroed]RAMMap exports produce three CSVs per .rmp file in a rammap/ subfolder alongside the source file:
| CSV | Content |
|---|---|
*_rammap_usecounts.csv |
Memory by Use Type × List (Process Private, Mapped File, Page Table, etc., broken down by Active / Standby / Modified / Free) |
*_rammap_systemlist.csv |
System memory totals per PFN list (Active, Standby, Modified, Free, TOTAL) |
*_rammap_processes.csv |
Per-process memory with PFN page-list breakdown (Active / Standby / Modified / Free pages and KB, plus Private working set) |
Use python etwview.py --rammap to export RAMMap CSVs and generate RAMMap charts without re-running the full WPA pipeline.
| Flag | Behavior |
|---|---|---|
| (no flags) | Load config, optionally export ETLs, parse CSVs, write _raw_all.csv, render charts. After completion, prints a timing summary (WPA parse, chart generation, total). |
| --discover | Scan exports and print usable WPA metric keys + schema suggestions |
| --validate | Reparse source CSVs, compare against existing _raw_all.csv, check data quality |
| --report | (Re)build report.html from existing *.editor.json sidecars under the output directory without re-running the pipeline. Useful after manually editing chart sidecars or when the report was skipped during the main run (e.g. no editor files existed yet). |
| --rammap | Export RAMMap .rmp files to CSV and generate RAMMap charts (also runs automatically when rammap.enabled: true in config). |
| File | Description |
|---|---|---|
| <output_dir>/<trace_type>/<metric>.png | Chart image (written only when export_images: true) |
| <output_dir>/<trace_type>/<metric>.editor.json | Chart sidecar for chart_editor.html (always written) |
| <output_dir>/_raw_all.csv | Consolidated parsed dataset |
| <output_dir>/report.html | Report page |
| ./_issues.csv | Parse warnings and errors (written to CWD). Columns: severity, kind, message, scenario, trace_type, run, block, csv_path. |
| ./_validate.log | Validation report from --validate (written to CWD) |
Hybrid selection ranks processes using a multi-metric sort. Each block in cfg.yaml defines its metrics via hybrid_metrics — a list of metric names from two categories:
Value metrics — how values are aggregated across runs. Sort order is always descending (higher is better):
| Metric | Description |
|---|---|
peak_value |
Maximum value across runs within a scenario. Picks the highest single run. |
mean_value |
Arithmetic mean across runs within a scenario. |
median_value |
Median across runs within a scenario. Robust against outliers. |
Selection metrics — tiebreaker dimensions. Sort direction varies per metric:
| Metric | Direction | Description |
|---|---|---|
appear_count |
Descending | Number of scenarios the process appears in (more = better). |
process |
Ascending | Alphabetical process name (stable tiebreaker only). |
delta |
Descending | scenario_value - baseline_value, max across scenarios. Requires baseline_scenario. |
Example: hybrid_metrics: [peak_value, mean_value, appear_count] sorts by peak value first, then mean value, then by how many scenarios the process appears in.
When delta is used, baseline_scenario must be set. The delta is computed as scenario_value - baseline_value for each process-scenario pair; the max delta across scenarios is the sort key.
After selection, peakification reorders processes by their peak per-scenario value — highest on the left, lowest on the right (Plotly autorange="reversed"). This is presentation-only; selection and exported columns are unchanged.
Peakification can be disabled per block with peakification: false to preserve the hybrid selection order. The x-axis reversal can also be controlled independently per block with autorange_reversed: false.
Hybrid tiebreaker (hybrid_tiebreaker in cfg.yaml) controls how the value and selection metrics combine when sorting hybrid results:
| Mode | Behavior | Metric Priority | Cross-run default |
|---|---|---|---|
value_first (default) |
Sorts by value metric first, then by selection metrics. | (mean_value, peak_value, appear_count, process) |
mean_value |
rank_first |
Converts all value metrics to ranks (1 = best), then sorts by selection metrics. | (appear_count, median_value, process) |
median_value |
baseline_rank |
Like rank_first, but ranks are computed relative to the baseline scenario. Requires baseline_scenario. |
(delta, median_value, peak_value, appear_count, process) |
median_value |
Cross-run default is the reducer used to collapse multiple runs into one value per scenario (what charts plot), unless a block sets hybrid_metrics explicitly — e.g. hybrid_metrics: mean_value on boot_phases or cpu_util.
With value_first, cross-run chart values use mean across runs; peak_value in the priority tuple is a selection tiebreaker only (after mean).
Before cross-run hybrid ranking, rows within the same (scenario, run, entity) group are aggregated. When multiple CSV rows map to the same entity within a single run — e.g. a process with multiple threads, each appearing as a separate row — values are summed to produce one value per entity per run. This ensures each entity contributes a single value to the cross-run median/mean/peak computation.
Set chart_type in a cfg.yaml block definition to use a non-ranked layout:
| Type | Description |
|---|---|---|
| ranked (default) | Ranked scatter plot with connecting lines, one point per scenario per entity. |
| bar_summary | Grouped bar per entity per scenario, or one bar per scenario. |
| balls_only | Ranked scatter with large markers, no connecting lines. |
| (no entity) | Metrics without an entity column fall back to aggregate comparison — a scatter plot with one dot per scenario showing the aggregated value. |
wpaProfiles/ contains WPA export profiles used by wpaexporter.exe:
| Profile | Purpose |
|---|---|
Battery.wpaProfile |
Power, only for battery metrics |
Boot.wpaProfile |
CPU, disk, boot-phase metrics from Boot ETLs |
General.wpaProfile |
CPU, disk, UI delay, process metrics from General ETLs |
SvchostEng.wpaProfile |
Svchost PID-to-service mapping (see below) |
ETWView auto-looks up <TraceType>.wpaProfile for each trace type. Missing profiles produce a warning and the exporter skips that type. Add new trace types by creating a matching .wpaProfile and adding a block entry in cfg.yaml.
When svchost.enabled: true and wpa_exporter.enabled: true, ETWView runs an extra export pass using SvchostEng.wpaProfile to resolve svchost.exe (PID) entries into named services (e.g. svchost (SysMain)).
| Sub-key | Default | Description |
|---|---|---|
enabled |
false |
Enable svchost service resolution |
profile_path |
wpaProfiles/SvchostEng.wpaProfile |
Override path to the svchost WPA export profile |
Resolution happens at parse time: the PID is mapped to its hosted service set. Two instances with identical services across scenarios are treated as one logical process for ranking. Each scenario keeps its own metric values end-to-end.
If SvchostEng.wpaProfile is missing, svchost enrichment is silently disabled.
In the ranked chart, each svchost service group has a dot indicating its presence in each scenario:
| Dot | Meaning |
|---|---|
● (filled) |
The exact same svchost service group exists in this scenario. |
○ (hollow) |
One or more services from this svchost exist in this scenario, but are grouped differently inside another svchost. |
| (blank) | None of the services from this svchost were found in this scenario. |
| File | Purpose |
|---|---|---|
| cfg_default.yaml | Annotated template with all configurable keys and their defaults — rename to cfg.yaml and customize |
| style.yaml | Chart styling defaults (fonts, colors, sizes, layouts) |
All schema files are loaded at runtime with strict validation. Invalid structures raise explicit errors.
Each block in cfg.yaml supports:
| Field | Description |
|---|---|---|
| csv_filename_aliases | Normalized CSV filename stem (from --discover) |
| metrics | Mapping of metric key → column header (aliases), optional transform / title / entity (per-metric override of block-level entity) |
| chart_type | Layout: ranked (default), bar_summary, or balls_only |
| entity | Entity column set key from entity_column_sets, or an exact CSV column header (e.g. Disk). Defines what identifies a row (e.g. process name). |
| hybrid_metrics | Comma-separated string or list of sort metrics for process ranking |
| top_n | Override global_top_n for this block's metrics. Applies to all metrics in the block. |
| subtitle_text | Custom chart subtitle (overrides auto-generated default) |
| autorange_reversed | Set false to disable x-axis reversal (default: true). When true, highest value is on the left. |
| peakification | Set false to skip reordering by per-scenario peak value (default: true). Uses hybrid selection order instead. |
| merge_entities | Set true to merge per-entity values into a single bar_summary bar per scenario (e.g. aggregate CPU utilization across all cores). |
| legend_y / bottom / left / right | Per-block margin overrides for chart layout (defaults from chart_layouts in style.yaml). |
Transforms convert raw CSV column values into the display unit. Each metric can define a per-metric transform in its block entry. Additionally, global_transforms in cfg.yaml defines a list of transforms that are auto-applied to any metric whose CSV column header matches a known unit pattern — unless that metric explicitly sets transform: false to opt out.
| Transform | Effect |
|---|---|---|
| (omit) | Fallback: if a global_transforms pattern matches the column header, it is applied; otherwise values are stored as-is. |
| false | Disable any automatic transform for this metric (neither global nor pattern-based). |
| us_to_ms | ÷1000 (microseconds → milliseconds) |
| auto | Detect unit from column header: µs/μs → ÷1000, s → ×1000, else no-op |
| bytes_to_mib | ÷1,048,576 (bytes → mebibytes) |
WPA-exported CSVs contain many columns — metadata (Timestamp, Process, Thread ID), labels, and numeric values. The parser automatically discovers which columns hold metric data using three thresholds in cfg.yaml:
| Key | Default | What it does |
|---|---|---|
min_numeric_frac |
0.35 |
Drops columns where fewer than 35% of values can be coerced to numbers. Filters out text-only or sparse columns. |
min_usable_rows |
1 |
Minimum number of usable rows required. Applied both to value columns (numeric non-null count) and entity columns (non-empty label count). Acts as a floor against zero-row columns. |
identifier_uniqueness_threshold |
0.95 |
Excludes columns that look like row identifiers — Timestamp, Sample #, Thread ID, etc. A column is excluded if >95% of its numeric values are unique and all values are integers. The all-integer check prevents real metrics with mostly-unique values from being incorrectly filtered out. |
Discovery runs in _discover_value_columns() (src/wpa_parser.py): for each column that isn't the entity column or in the exclude set, it checks _looks_like_identifier(), then _column_usability(). Columns that pass both are sorted by numeric fraction (highest first) and assigned to block metrics in schema order.
Define reusable entity column groups in cfg.yaml:
entity_column_sets:
list: [Process, "New Process", Module, Function]
processor: [Processor]Reference them in a block via entity: list.
- Create or locate the exported CSV and note the exact filename stem and column headers
- Add a block entry in
cfg.yamlunderblocks:with:csv_filename_aliases— the exact filename stemmetrics— metric key → column header (aliases) + optionaltransform/title- Optionally
chart_type,entity,hybrid_metrics,top_n,subtitle_text,autorange_reversed,peakification
- Add the metric key to
trace_metricsincfg.yaml - Run
--discoverto verify the metric is found
Plot appearance is controlled by style.yaml, not cfg.yaml. Use chart_editor.html to visually tweak and export YAML snippets. Key groups:
- Image:
image_format,export_scale - Fonts:
title_font_family,subtitle_font_family,axis_label_font_family,tick_label_font_family,legend_font_family,body_font_family,unit_font_family,ui_font_family - Sizes:
title_size,subtitle_size,axis_label_size,tick_label_size,legend_font_size,body_font_size,unit_font_size - Colors:
title_color,subtitle_color,ui_text_color,grid_color,chart_bg,marker_border_color,legend_border_color,unit_font_color,axis_line_color - Layout:
chart_width,chart_height,top_margin,chart_layouts— per-type margins underchart_layouts(keys:ranked,balls_only,bar_summary, each withleft,right,bottom,legend_y) - Axes:
line_width,marker_size,marker_border_width,grid_line_width,axis_line_width,tick_length,tick_angle,zero_line - Labels:
process_name_max_len,show_value_in_legend,labels_bold - Legends:
legend_border_width,legend_y_offset - Toggles:
show_subtitle - Svchost:
svchost_wrap_per_line(service names per line in rank labels; default 3);svchost_two_columns(split the svchost table into two columns in report/chart editor) - Sequences:
color_sequence,marker_sequence(defaultnull— whennull, all scenarios use"circle"marker; set to a list of Plotly marker names to assign per-scenario markers on line charts)
Parse warnings and errors are written to _issues.csv with these columns: severity (error / warning / info), kind, message, scenario, trace_type, run, block, csv_path.
Common issue kinds:
| Kind | Severity | When |
|---|---|---|
csv_read_error |
error | Failed to read a CSV file |
unknown_block |
warning | CSV stem did not match any csv_filename_aliases |
rows_filtered |
info | Empty entity labels, invalid/negative values |
empty_after_cleaning |
warning | No rows remaining after entity/value cleaning |
missing_columns |
warning / error | Usable numeric columns not found |
Open chart_editor.html in a browser. Drag-drop an .editor.json file to tweak chart styling (fonts, colors, margins, marker sizes). Export a YAML fragment to merge into style.yaml.
- The svchost table always appears in the top-left corner and must be manually repositioned using the chart_editor, which is a usability problem.
The --validate flag re-parses all source CSVs from scratch and compares the result against the existing _raw_all.csv. It reports:
- Row mismatches — values that differ between the cached and fresh parse, with a likely cause analysis
- Parse issues — files with bad lines or missing columns
Output is written to _validate.log in the current directory.