elastic-eventgen

An event simulator that produces ECS-style NDJSON for Elastic demos and learning. Inspired by Splunk Eventgen, but designed from the ground up for modern Elastic Stack workflows: ECS, ES|QL, observability, and detection engineering.

The goal is not to produce random fake logs. It is to generate ECS-style data with light behavioral patterns that are useful for ES|QL practice, dashboards, basic SOC labs, and detection-engineering practice. The output is also designed to be a useful substrate for AI Assistant and Attack Discovery demos in Elastic, though the repo itself does not integrate with those products directly — it just produces data they can consume. Customer workshops and PoCs are an intended use case as the project grows.

What it does today

The project ships two independent generators in the same repo. Each one is a self-contained vertical slice: its own factory, its own CLI, its own scenario YAML, its own frozen sample, its own tests.

linux.auth — Linux auth.log activity for a small company (security / SOC-leaning):

• 20 employees across 6 departments
• 40 internal IPs and 15 VPN IPs
• 20 workstation hosts and 8 server hosts
• 5 attacker IPs, each mapped to a different country (hardcoded demo geo)
• brute-force attack sessions with shared session IDs
• an optional successful login at the end of some attack sessions
• simulates realistic password-typo retries — short bursts of failed logins from a legitimate workstation that end in success. Same surface shape as brute-force-with-success, opposite ground truth: the realistic false-positive that detection rules must learn to distinguish. Keeps the overall login failure rate around 10–11%, like a healthy environment with everyday mistakes rather than an artificially clean dataset.
• background sudo activity (~10% of events) so the auth log doesn't look like an unrealistic clean room of SSH logins only. Gives analysts noise to filter through during pivots and a privilege-escalation surface to practice on.
• per-event severity (1 / 3 / 5 / 8)
• ECS-style source.geo.* for attacker IPs only
• related.ip / related.user / related.hosts for cross-field pivoting

network.flow — service-to-service traffic for a small Japanese e-commerce backend (observability / SRE-leaning):

• 4 microservices (frontend, checkout-api, users-api, orders-api) with a real service dependency graph
• 8 hosts (2 per service), each pinned to one Tokyo availability zone
• baseline ~1.5% failure rate (healthy environment)
• a latency_spike scenario — windowed periods where one destination service's event.duration is elevated (100–1000 ms instead of the 1–100 ms baseline)
• an error_spike scenario — windowed periods where one destination service's failure rate is elevated (~30%), surfaced via the eventgen.scenario marker

Both generators share only a dataset-agnostic bulk-conversion script and a validation script that checks every event against the current rules. There is no shared factory, no plugin registry, no base class. This is deliberate: two readable parallel tracks instead of one abstraction-heavy one.

See README_NETWORK_FLOW.md for the network.flow walkthrough; the rest of this README focuses on linux.auth.

The output uses flat-keyed ECS-style field names (event.action, user.name, etc.) rather than nested objects. Elastic's ingest pipeline accepts both. We call this "ECS-style" because we have not yet validated against the official ECS schema.

Architecture

The two generators sit side by side. Each track is a self-contained vertical slice with the same shape — scenario YAML, CLI, factory, output.

linux.auth                              network.flow
────────────                            ─────────────
scenarios/linux_auth_failed_login.yaml  scenarios/network_flow_baseline.yaml
        │                                       │
        ▼                                       ▼
generator/main.py                       generator/network_flow_main.py
        │                                       │
        ▼                                       ▼
generator/event_factory.py              generator/network_flow_factory.py
        │                                       │
        ▼                                       ▼
output/generated_linux_auth.ndjson      output/generated_network_flow.ndjson

Within each track, scenario configuration, generation loop, and event-building logic each live in their own file. Output writing currently lives inline in each track's *_main.py and may be extracted later. The two tracks share only the dataset-agnostic scripts/to_bulk.py and scripts/validate_output.py.

Supported datasets

Both are first-class and tested independently. Each has its own generator, scenario YAML, frozen sample, index template, and regression hash sentinel.

• linux.auth — Linux auth.log activity: SSH logins (normal employees plus brute-force attackers with optional successes), realistic password-typo retries from legitimate users, and background sudo invocations. Walkthrough is in this file (continue reading).
• network.flow — service-to-service traffic for a 4-microservice Japanese e-commerce backend, with latency_spike and error_spike scenario windows that analysts detect from the event shape. Dedicated walkthrough: README_NETWORK_FLOW.md.

Inspect without installing

If you only want to see what the events look like, browse samples/example_linux_auth.ndjson — a frozen 500-event sample committed to this repo. No uv, no Python, no Elasticsearch needed. See samples/README.md for how it was generated and when it gets refreshed.

How to run

Generate events (the recommended workflow uses uv because it handles PyYAML and the Python toolchain in one step):

uv run python generator/main.py

If you don't have uv installed, a plain pip workflow works too:

python -m venv .venv
source .venv/bin/activate          # on Windows: .venv\Scripts\activate
pip install -r requirements.txt
python generator/main.py
pytest                             # run the test suite

requirements.txt is a minimal mirror of the runtime + test dependencies declared in pyproject.toml (PyYAML and pytest). For all other workflows, pyproject.toml is the source of truth.

The generator accepts four optional CLI flags:

Flag	Default	Purpose
--scenario PATH	scenarios/linux_auth_failed_login.yaml	scenario YAML file to load
--output PATH	output/generated_linux_auth.ndjson	where to write NDJSON output
--count N	YAML event_count, else 1000	number of events; overrides YAML
--seed N	unseeded (non-deterministic)	integer seed for reproducible runs

Precedence: CLI args > YAML values > built-in defaults. Run uv run python generator/main.py --help for full usage.

Examples:

uv run python generator/main.py --count 5000
uv run python generator/main.py --output output/quick_test.ndjson --count 100
uv run python generator/main.py --seed 42 --count 100   # reproducible

Validate the output (pure stdlib, no uv needed):

python scripts/validate_output.py

A typical workflow chains the two:

uv run python generator/main.py && python scripts/validate_output.py

Tests

The test suite documents the project's current behavior so future changes (realism, new datasets, refactors) can land without silently regressing existing contracts.

uv sync --group dev         # install dev dependencies (pytest)
uv run pytest               # run all tests
uv run pytest -v            # verbose

The suite covers event shape, attack-session invariants, related.* contracts, severity mapping, --seed determinism, the to_bulk.py / validate_output.py scripts, and one byte-identical regression sentinel that catches any drift in serialized output.

Loading into Elasticsearch (Linux SSH auth)

These instructions are documented but not yet verified end-to-end against a running cluster — please verify in your environment.

1. Install the index template

Required first. Without the template, source.ip and similar fields get auto-classified as keyword and CIDR queries silently fail.

ES_URL=http://localhost:9200
curl -s -H "Content-Type: application/json" \
     -XPUT "$ES_URL/_index_template/eventgen-linux-auth" \
     --data-binary @examples/index-template.json

2. Generate events

uv run python generator/main.py

3. Bulk import via curl

INDEX_NAME=eventgen-linux-auth-test
OUTPUT_FILE=output/generated_linux_auth.ndjson
python scripts/to_bulk.py $OUTPUT_FILE $INDEX_NAME > output/bulk.ndjson
curl -s -H "Content-Type: application/x-ndjson" \
     -XPOST "$ES_URL/_bulk" --data-binary @output/bulk.ndjson | head -c 200

Check the response for "errors": false. Mapping errors here usually mean the template was not installed before the import.

4. Alternative: Kibana File Upload

Stack Management → File Upload → choose output/generated_linux_auth.ndjson. Set the index name to match eventgen-linux-auth-* before clicking Import — otherwise Kibana auto-detects mappings and silently misclassifies the IP fields.

Verifying the data

In Kibana → Discover, select the eventgen-linux-auth-* data view and confirm:

• Time field is @timestamp (auto-detected)
• source.ip shows the ip type icon, not t (keyword)
• event.severity sorts numerically (1 → 3 → 5 → 8)
• source.geo.country_name shows values like China, Russia, Brazil
• Filtering by event.outcome:success reduces the count

Then run these ES|QL queries to confirm field types:

// CIDR query — confirms source.ip is `ip` type
FROM eventgen-linux-auth-*
| WHERE CIDR_MATCH(source.ip, "10.10.0.0/16")
| KEEP source.ip, user.name, event.outcome
| LIMIT 5

// Multi-value IP — confirms related.ip works across source + destination
FROM eventgen-linux-auth-*
| WHERE MV_COUNT(related.ip) > 1
| STATS count = COUNT(*)

// Full-text — confirms message is `match_only_text`
FROM eventgen-linux-auth-*
| WHERE MATCH(message, "Failed password")
| STATS count = COUNT(*) BY user.name
| SORT count DESC
| LIMIT 5

About the "Unknown data source" warning. The Kibana ES|QL editor may show a warning like Unknown data source "eventgen-linux-auth-test" even when the query executes successfully. This is editor-level static validation against registered Kibana data views, not a query failure. If results appear, the query is working.

For an editor-independent verification, run ES|QL directly against the _query API:

curl -s -H "Content-Type: application/json" -XPOST "$ES_URL/_query" -d '{
  "query": "FROM eventgen-linux-auth-* | STATS count = COUNT(*) BY source.ip | SORT count DESC | LIMIT 5"
}'

Exploring the data with ES|QL

The queries below are analyst-flavored: they ask investigation questions about the data rather than verifying field types. They assume a reasonably sized dataset (a few thousand events). Smaller samples may return fewer rows — or no rows — depending on the seed and attack_probability.

Which brute-force sessions actually succeeded?

FROM eventgen-linux-auth-*
| WHERE eventgen.scenario == "brute_force_ssh" AND event.outcome == "success"
| KEEP @timestamp, eventgen.session_id, source.ip, source.geo.country_name, user.name, host.name
| SORT @timestamp DESC

Every row is the closing successful login of a brute-force session — the highest-severity events in the dataset. source.ip and source.geo.country_name are preserved so you can pivot on the attacker.

Did anyone run sudo against sensitive files?

FROM eventgen-linux-auth-*
| WHERE event.action == "sudo" AND MATCH(message, "shadow")
| KEEP @timestamp, user.name, host.name, message
| SORT @timestamp DESC
| LIMIT 10

Surfaces sudo commands touching /etc/shadow. Swap "shadow" for "tcpdump" or "passwd" to look for other sensitive activity.

Multiple failures with a success — brute-force or typo cluster?

FROM eventgen-linux-auth-*
| WHERE event.action == "ssh_login"
| STATS attempts  = COUNT(*),
        failures  = COUNT(*) WHERE event.outcome == "failure",
        successes = COUNT(*) WHERE event.outcome == "success"
  BY user.name, source.ip
| WHERE failures >= 2 AND successes >= 1
| SORT failures DESC
| LIMIT 10

This is the dataset's training point. The result set deliberately mixes two very different things: legitimate password-typo bursts (employee logging in from their assigned workstation IP) and real brute-force sessions that ended in success (attacker logging in from an external IP). The two patterns are identical in shape — same field, same sequence — so an analyst has to distinguish them using source.ip, source.geo.*, and user.name. The pattern alone is not enough; that is the whole point.

Login failure rate per host

FROM eventgen-linux-auth-*
| WHERE event.action == "ssh_login"
| STATS total    = COUNT(*),
        failures = COUNT(*) WHERE event.outcome == "failure"
  BY host.name
| WHERE total >= 20
| SORT failures DESC
| LIMIT 10

Across the whole dataset, the typical login failure rate hovers around 10–11% — a healthy environment with everyday mistakes rather than an artificially clean room. Hosts that drift far above that band are worth a closer look.

Where are the attacks coming from?

FROM eventgen-linux-auth-*
| WHERE eventgen.scenario == "brute_force_ssh"
| STATS attacks         = COUNT(*),
        unique_sessions = COUNT_DISTINCT(eventgen.session_id)
  BY source.geo.country_name
| SORT attacks DESC

Demonstrates that source.geo.* is populated only for attacker IPs. Internal and VPN sources have no entries in the demo geo map, so they do not appear here.

Why these design choices

Nested mappings vs flat-keyed wire format. The NDJSON ships flat keys ("event.action": "ssh_login"), but the index template uses nested objects (event: { properties: { action: ... } }). Elasticsearch auto-expands dotted JSON keys, so both forms work — but flat-keyed mapping definitions ("event.action": { ... }) can silently create a literal event.action field instead of the nested path. Always nest the mapping.

Why ip field types. source.ip, destination.ip, and related.ip are mapped as ECS ip, not keyword. This unlocks CIDR queries, range comparisons, and ingest-time IP validation. keyword mapping would silently break those.

Why _id = event.id. The bulk script derives _id from the UUID4 in event.id, making re-imports idempotent. Re-running the same import overwrites documents instead of creating duplicates.

Why match_only_text for message. ECS 8.0+ recommends this for free-text fields. It supports MATCH() queries but skips positional indexing, making storage noticeably cheaper than text.

Configuration

Edit scenarios/linux_auth_failed_login.yaml:

event_count: 5000
attack_enabled: true
attack_probability: 0.003
attack_session_enabled: true
attack_success_after_session: true
attack_success_probability: 0.3
attack_burst_enabled: true

Key	Meaning
event_count	total events to generate
attack_enabled	whether attacker events are mixed in
attack_probability	per-event chance of starting a new attack session
attack_session_enabled	tie failures into sessions sharing IP / host / user
attack_success_after_session	end some sessions with one successful login
attack_success_probability	fraction of sessions that end in success (0–1)
attack_burst_enabled	brute-force events cluster outside business hours

The YAML controls attack-scenario tuning. The other realism dials — how often a legitimate user mistypes a password, how long a typo retry burst runs, how much background sudo noise mixes into the stream — live as named constants near the top of generator/event_factory.py. They're kept there so each value is searchable, testable, and locked against accidental drift. To dial them, edit the constants directly and re-run uv run pytest; the test suite will tell you if the change shifts the seeded output.

Example event shape

The values below are illustrative — actual output will use different timestamps, ports, and randomized choices.

{
  "@timestamp": "2026-05-15T12:32:11+00:00",
  "event.kind": "event",
  "event.category": ["authentication"],
  "event.type": ["start"],
  "event.dataset": "linux.auth",
  "eventgen.scenario": "brute_force_ssh",
  "eventgen.session_id": "attack-session-000004",
  "event.action": "ssh_login",
  "event.outcome": "success",
  "event.severity": 8,
  "user.name": "admin",
  "organization.department": null,
  "source.ip": "198.51.100.23",
  "source.geo.country_name": "China",
  "source.geo.city_name": "Beijing",
  "host.name": "jumpbox-01",
  "service.type": "system",
  "log.syslog.facility.name": "auth",
  "message": "Accepted password for admin from 198.51.100.23 port 51234 ssh2",
  "related.ip": ["198.51.100.23"],
  "related.user": ["admin"],
  "related.hosts": ["jumpbox-01"]
}

A background sudo event from the same dataset looks like this. Note that sudo events keep the everyday severity (1 for success, 3 for failure) — they're realistic auth.log clutter, not threats in themselves:

{
  "@timestamp": "2026-05-15T14:22:33+00:00",
  "event.kind": "event",
  "event.category": ["authentication"],
  "event.type": ["start"],
  "event.dataset": "linux.auth",
  "eventgen.scenario": "sudo_command",
  "event.action": "sudo",
  "event.outcome": "success",
  "event.severity": 1,
  "user.name": "tanaka",
  "organization.department": "sales",
  "source.ip": "10.10.1.25",
  "host.name": "jp-sales-ws-003",
  "service.type": "system",
  "log.syslog.facility.name": "auth",
  "message": "sudo:  tanaka : TTY=pts/0 ; PWD=/home/tanaka ; USER=root ; COMMAND=/usr/bin/apt update",
  "related.ip": ["10.10.1.25", "10.110.2.3"],
  "related.user": ["tanaka"],
  "related.hosts": ["jp-sales-ws-003"]
}

Password-typo retry events use the same shape as a normal SSH login event (no special marker field) — the retry pattern is visible only in the sequence of events, not in any one event on its own. That's deliberate: it forces analysts to reason about behavior over time rather than rely on a tag.

Important note

This is demo / simulation data, not production telemetry. The geographic attributions, attacker IPs, and behavioral patterns are designed to look plausible in a workshop or PoC context but should never be treated as real threat intelligence.

Roadmap (intentionally small)

The v1 scope is in place: two independent generators (linux.auth and network.flow), each with its own scenario YAML, frozen sample, index template, byte-identical regression sentinel, and ES|QL examples.

Planned next, in rough order of priority:

• a Japanese translation of README_NETWORK_FLOW.md (English-only today)
• additional datasets (Windows Sysmon, AWS CloudTrail, firewall, VPN, Kubernetes audit / container logs)
• formal ECS schema validation (load + query examples are documented; schema validation tooling is still future work)
• replay mode (Eventgen-style replay of a recorded NDJSON file with adjustable speed)
• sample log mutation (take a real log line and produce realistic near-variants for training)
• burst / anomaly simulation as a first-class scenario type (the current latency_spike / error_spike windows are the prototype)
• future detection-engineering scenarios (multi-stage incidents, correlated activity across datasets)

Each item is a separate future task. The current code is deliberately beginner-readable and prefers small, testable improvements over over-engineering.