Standing up a self-managed Elastic Stack on a local VM, ingesting real Linux authentication logs, and writing EQL sequence queries to correlate multi-step authentication behavior.
Methodology: Detect - Analyze - Correlate - Harden - Validate
This is a personal, self-directed lab built on my own hardware. It is not professional SOC experience. Where something was best-effort or hit a known limitation, it is labeled as such below. Four real failures happened during this build and all four are documented in TROUBLESHOOTING.md, because the debugging was the most useful part.
I already had detection work in Microsoft Sentinel (Kusto) and Splunk (SPL). What I did not have was hands-on time with Elastic, and specifically with EQL sequences, which express multi-step attack behavior in a way neither Kusto nor SPL does natively.
The goal was fluency: being able to read Elastic queries, follow an analyst conversation, and speak accurately about the stack. It was scoped deliberately small, a weekend ramp, not a full detection engineering lab.
| Component | Detail |
|---|---|
| Host | Windows 11 laptop, VMware Workstation |
| VM | Ubuntu 25.10, 12 GB RAM, 2 vCPU |
| Elasticsearch | 8.19.19, self-managed, basic license |
| Kibana | 8.19.19 |
| Elastic Agent | 8.19.19, standalone (not Fleet-managed) |
| Data source | /var/log/auth.log via the System integration |
Everything ran locally at zero cost. No cloud spend, no paid-tier trial. The self-managed basic license covers EQL and the Security app.
Elasticsearch is a Java application, and the JVM claims roughly half the host's RAM for its heap by default. On a 12 GB VM already running other services, that is a problem before a single document is indexed. So the first action was a memory decision, not an install.
free -h
systemctl is-active splunk ollama
sudo systemctl stop ollamaI read the available column rather than free, since available reflects what a new process could actually claim including reclaimable cache. With 9.6 GB available I capped the Elasticsearch heap explicitly rather than resizing the VM:
echo -e "-Xms2g\n-Xmx2g" | sudo tee /etc/elasticsearch/jvm.options.d/heap.options-Xms is the starting heap, -Xmx the ceiling. Setting both to 2g means it starts bounded and stays bounded. The running service settled at 2.4 GB total, heap plus JVM overhead.
Install and verification steps are in SETUP.md.
Modern Elastic ships with TLS and authentication on by default. The enrollment token handshake between Elasticsearch and Kibana is a real first-hour stumbling block, and I worked through it rather than disabling security to save time, since that friction is representative of a production environment.
I used the System integration with Elastic Agent in standalone mode rather than Fleet. Fleet gives central management across many agents; for a single VM it adds a Fleet Server to stand up and resource for no benefit.
The critical detail is ECS. ECS (Elastic Common Schema) is Elastic's standardized field-naming layer, roughly what CIM is to Splunk. EQL depends on it absolutely: without @timestamp and event.category on a document, every EQL query returns zero hits and looks exactly like a broken query. The System integration maps auth events into ECS automatically, which is why it was the right choice over raw Filebeat.
So before writing any EQL, I verified the mapping directly rather than trusting the UI:
curl -k -u elastic 'https://localhost:9200/_cat/indices/logs-*?v'Then confirmed the actual field values on a real SSH event:
"event": {
"action": "ssh_login",
"category": ["authentication"],
"outcome": "failure"
},
"source": { "ip": "127.0.0.1" },
"user": { "name": "sean" }Correctly mapped, with event.category, event.outcome, source.ip, and user.name all populated. That is the shape EQL sequences need.
Kibana KQL and Kusto KQL are unrelated languages that share an acronym. My background is Kusto (Microsoft Sentinel). At an Elastic shop, "KQL" means Kibana Query Language: simple field-colon-value filters, no piping, no aggregation. Confusing the two is an easy and visible mistake.
| Kusto KQL | Kibana KQL | |
|---|---|---|
| Example | SecurityEvent | where EventID == 4625 |
event.category : "authentication" |
| Aggregation | Yes | No |
| Runs in | Sentinel Logs blade | Kibana Discover |
Not in Discover. EQL runs in exactly three places: the Security app's Timelines, Correlation tab; a detection rule of type eql; or the _eql/search API. Pasting EQL into Discover produces what looks like a syntax error but is a wrong-surface error.
A Kusto or SPL brute-force rule counts: "10 or more failures in an hour from one source." That is a volume threshold.
An EQL sequence describes order: "this event, then that event, involving the same entity, within a window."
sequence by source.ip with maxspan=1h
[authentication where event.outcome == "failure"]
[authentication where event.outcome == "success"]
| Clause | Function |
|---|---|
sequence by source.ip |
The join key. Groups events by the same actor. |
with maxspan=1h |
Bounds the window. Must come immediately after sequence by, not at the end of the query. |
[authentication where ...] |
Each bracketed block is one ordered stage. |
Result: one sequence matched on 127.0.0.1, a failed SSH login at 22:44:42 UTC followed by a successful one at 22:44:48 UTC.
This is higher signal than a count threshold. It does not flag noisy failures, it flags failures that were followed by an actual successful login.
To express the same logic in Kusto or SPL you would self-join the table against itself on a time-ordered condition, or run two queries and correlate in a second pass. EQL does it in one readable statement. That is the capability worth knowing.
All queries are in queries/eql-queries.md.
Exercise B was framed as a hunt rather than a detection, which means it gets a hypothesis, a query, and a verdict.
Hypothesis: an attacker pacing password guesses beneath a volume threshold (say 10 failures per hour) is invisible to that rule, but would still show as sustained repeated failures against the same account over a longer window with no intervening success.
sequence by source.ip with maxspan=4h
[authentication where event.outcome == "failure"]
[authentication where event.outcome == "failure"]
[authentication where event.outcome == "failure"]
Result: 3 sequences matched.
Verdict: ruled out. Every match was my own test traffic. Two of the three stitched failures from two separate test sessions into a single match, because EQL only knew "three failures from this IP within four hours," which was true but not an attack.
This is the useful finding. A bare "N failures in a row" sequence is a weak hunt query. It fires on any repeated but harmless activity, including mistyped passwords. EQL sequences encode order, not rate or density. A serious low-and-slow hunt needs a rate-based signal, failures per distinct time bucket, not just a chain of events. I would rather document that limitation honestly than present a query that looks like it works.
| Criterion | Status |
|---|---|
| Elasticsearch and Kibana running, start/stop unaided | Confirmed |
| Real auth data ingesting, ECS mapping verified | Confirmed via _cat/indices and document inspection |
| Kibana KQL search by IP and username in Discover | Confirmed |
| EQL sequence written and run in Elastic Security | Confirmed, both via API and the Correlation tab UI |
| Kibana KQL vs Kusto KQL distinction | Documented above |
| Elastic vocabulary usable in conversation | See below |
| Elastic | Splunk | Sentinel |
|---|---|---|
| Index | Index | Table |
| Document | Event | Row |
| Index pattern / data view | Index or sourcetype scope | Table scope |
| Elastic Agent | Universal Forwarder | Log Analytics agent / AMA |
| Discover | Search | Logs blade |
| ECS | CIM | Normalized table schema |
| Detection rule | Correlation search | Analytics rule |
| Elastic Security | Enterprise Security | Sentinel workspace |
- EQL sequences. Ordered multi-stage correlation as a first-class language feature.
- The mapping model. Splunk's schema-on-read forgives bad field extraction. Elastic's EQL punishes bad ECS mapping silently, with zero results and no error.
- The KQL collision. No equivalent trap in Splunk or Sentinel.
- Setup friction. Security-by-default is heavier out of the box.
| Path | Contents |
|---|---|
README.md |
This writeup |
SETUP.md |
Full install and configuration commands with reasoning |
TROUBLESHOOTING.md |
Four real failures hit during this build and how each was diagnosed |
queries/eql-queries.md |
Every EQL, Kibana KQL, and Query DSL query used |
screenshots/ |
Numbered proof-point screenshots |
MIT. See LICENSE.
#ERSec