automatic troubleshooting endpoint context docs

[joeypoon]

Change Summary

Adds Automatic Troubleshooting context docs useful for troubleshooting common endpoint issues.

[ferullo]

There's a lot to review here so I'm going to submit a review for each file. The first one is done. It's reasonable/expected to sit on my recommendations until an Endpoint developer reviews too (so they can see my comments and contradict me, etc).

[ferullo]

Because this driver runs at the kernel level, bugs or incompatibilities can cause system-wide crashes (BSODs) rather than isolated process failures.

Does an LLM need to be told that?

[ferullo]

I don't expect customers to do this. Perhaps text that describes how to collect a memory dump and a note that says to share it with us?

[ferullo]

he crash occurs in the file system filter driver's post-FsControl handler (bkPostFsControl) when processing offloaded write completions. The faulting call stack typically shows elastic_endpoint_driver!bkPostFsControl followed by FLTMGR!FltGetStreamContext.

I doubt this is useful info for a Kibana user.

[ferullo]

I doubt this paragraph is useful info for a Kibana user.

[ferullo]

Agent doesn't support downgrade (or am I mistaken). This should say "Upgrade to a version with the fix" I think.

[ferullo]

Suggested change

- **Trellix Access Control**: Trellix's kernel driver can intercept the Windows Base Filtering Engine (BFE) service, causing Defend's WFP (Windows Filtering Platform) driver initialization to hang or take an extremely long time. When another thread within Defend tasks the kernel driver before initialization completes, the system crashes. The call stack typically shows `elastic_endpoint_driver!HandleIrpDeviceControl` calling `bkRegisterCallbacks` with `KeAcquireGuardedMutex` on an uninitialized mutex. This interaction was introduced by a WFP driver refactor in 8.16.0. Fixed in 8.17.6, 8.18.1, and 9.0.1. Workaround: disable Trellix Access Protection or add a Trellix exclusion for the BFE service (`svchost.exe`).

- **Trellix Access Control**: Trellix's kernel driver can intercept the Windows Base Filtering Engine (BFE) service, causing Defend's WFP (Windows Filtering Platform) driver initialization to hang or take an extremely long time. This interaction was introduced by an Elastic Defend refactor in 8.16.0. Fixed in 8.17.6, 8.18.1, and 9.0.1. Workaround: disable Trellix Access Protection or add a Trellix exclusion for the BFE service (`svchost.exe`).

Question for Defend developers: is "add a Trellix exclusion for the BFE service (svchost.exe)." an acceptable recommendation?

[bjmcnic]

I don't think that's a good recommendation to give a customer after the fixed versions were released. At this point the recommendation should be to upgrade.

[ferullo]

IDK, is this actionable by users?

[ferullo]

Something is missing here, we added support for Windows Server 2012 R2 back in 8.16.0

[ferullo]

This paragraph doesn't seem useful to a user.

[ferullo]

Suggested change

	1) Collect the full kernel memory dump (`C:\Windows\MEMORY.DMP` or minidumps from `C:\Windows\Minidump\`). Run WinDbg `!analyze -v` to identify the exact bugcheck code, faulting module, and call stack. Without the dump, root cause cannot be determined.
	1) Collect the full kernel memory dump (`C:\Windows\MEMORY.DMP` or minidumps from `C:\Windows\Minidump\`). Share the dump with Elastic.

[ferullo]

These two situations are completely different. Should a single MD doc cover them both or is it better to break this doc up? I'm holding off on reading this file until this is answered.

I have no personal preference, I'm just bringing this up in case it helps with context windows.

[nicholasberlin]

Suggested change

Elastic Defend on Linux uses eBPF and fanotify to monitor process, file, network, and DNS activity in real time. Each event is enriched, hashed, evaluated against behavioral rules, and forwarded to the configured output. The most common drivers of high CPU on Linux are monitoring scripts that spawn many short-lived child processes (each generating process events that trigger behavioral rules), output server disconnections causing retry storms, the Events plugin hashing large binaries during policy application with an empty cache, and memory scanning of large processes.

Elastic Defend on Linux uses eBPF or tracefs to monitor process, file, network, and DNS activity in real time. Each event is enriched, hashed, evaluated against behavioral rules, and forwarded to the configured output. The most common drivers of high CPU on Linux are monitoring scripts that spawn many short-lived child processes (each generating process events that trigger behavioral rules), output server disconnections causing retry storms, the Events plugin hashing large binaries during policy application with an empty cache, and memory scanning of large processes.

[nicholasberlin]

8.13.4 is really old, do we need to call this out?

[nicholasberlin]

5-30 minutes? is that supposed to be seconds?

[nicholasberlin]

I suppose not, because 30 seconds wouldn't be a problem.

[ferullo]

We should add another doc for when Alert documents do not appear in logs-endpoint.alerts-*

• A shipping error from Endpoint
• A configuration error in Kafka or Logstash
• Actually Endpoint isn't actively blocking activity, it's caused by some sort of interaction with another AV or process -- targeted Trusted Apps is the most common solution

[ferullo]

This is not true. We plan to ship per-policy exceptions in 9.4 and that'll be opt-in.

[ferullo]

Suggested change

In architectures where Fleet-managed agents send data to a "data cluster" and analysts query via CCS from a separate "analyst cluster", Endpoint Alert Exceptions must be created on the data cluster (the one running Fleet Server). Exceptions created on the analyst cluster are not distributed to endpoints because CCS does not support Fleet Actions.

In some architectures Fleet-managed agents are managed via an "analyst cluster" but write bulk data to a "data cluster".. In these architectures, analysts query data via CCS from the data cluster, Endpoint Alert Exceptions must be created on the analyst cluster (the one running Fleet Server). Exceptions created on the data cluster are not distributed to endpoints because CCS does not support Fleet Actions.

[ferullo]

Suggested change

	If exceptions exist only on the analyst cluster, alert documents will continue to be generated by endpoints and indexed on the data cluster. The analyst cluster will then alert on those documents via CCS.
	If exceptions exist only on the data cluster, alert documents will continue to be generated by endpoints and indexed on the data cluster. The analyst cluster will then alert on those documents via CCS.

[ferullo]

This whole section makes no sense. It's not possible for the SIEM rule to create an alert if there is no underlying Endpoint alert in logs-endpoint.alerts-*.

Maybe we mean to say alerts created by most SIEM rules need SIEM exceptions not Endpoint exceptions?

[ferullo]

Suggested change

To suppress on pre-9.2: Trusted Applications do not suppress behavioral detections on these older versions. Use an **Endpoint Alert Exception** targeting `rule.id` and the relevant process fields.

[ferullo]

Suggested change

	- A third-party driver or application causes memory corruption that makes non-executable memory pages appear executable, leading to unexpected scanning of heap regions.
	- Faulty RAM causes bit flips in page table entries, changing page protection bits and exposing non-code memory to the scanner.

[ferullo]

This was a very edge case, I think we should remove it. It won't generalize well.

Suggested change

	### Alert field data corrupted by product bug
	In rare cases, the endpoint may populate alert fields with corrupted data (e.g. truncated `dll.path` or `dll.name` values due to kernel-level memory corruption from a third-party driver). When the corrupted field values happen to match a behavioral detection rule's criteria — for example, a DLL path truncated to appear as if it has an unusual extension — the rule fires on data that does not reflect the actual system state.
	These are product bugs, not fixable via exceptions in the general case. However, a targeted Endpoint Alert Exception can suppress the specific false-positive pattern as a workaround. For example, suppressing a rule when the DLL's code signature status indicates it was never actually loaded (`dll.code_signature.status: "errorCode_endpoint: Initital state, no attempt to load signature was made"`).
	If you encounter alerts where field values appear truncated or nonsensical, collect agent diagnostics and the full alert document for Elastic Support. Check for third-party kernel drivers that may be causing memory corruption — the Windows Driver Verifier can help identify misbehaving drivers on non-production systems.

[ferullo]

Suggested change

This is a known product limitation. If the goal is to prevent specific trusted-signed binaries from running, the blocklist is not the correct mechanism. Consider using operating system-level application control policies (e.g. Windows AppLocker or WDAC) instead.

[ferullo]

Suggested change

	1) Determine the detection engine that generated the alert by checking `event.code`: `malicious_file` (malware engine), `behavior` (behavioral protection), `memory_signature` (memory scanning). This determines which artifact type to use for suppression.
	1) Determine the detection engine that generated the alert by checking `event.code`: `malicious_file` (malware protection), `behavior` (malicious behavior protection), `memory_signature` (memory threat protection), `ransomware` (ransomware protection). This determines which artifact type to use for suppression.

[ferullo]

Suggested change

A typical pattern is hourly CPU spikes lasting 5–10 minutes, aligning with cron schedules (e.g. xx:31–xx:41 every hour). In one case, a script at `/var/cache/system-monitoring/helper/compare-inventory.sh` that used `curl` to collect data triggered the behavioral rule "Suspicious Download and Redirect by Web Server" 86,340 times in a single diagnostics window, driving endpoint service CPU above 200% (out of 800% on 8 cores).

A typical pattern is hourly CPU spikes lasting 5–10 minutes, aligning with cron schedules (e.g. xx:31–xx:41 every hour).

[ferullo]

@nicholasberlin Trusted Application Descendants or Event Filter Descendants will help here, right?

[ferullo]

Suggested change

	- Review `linux.advanced.events` settings to disable unnecessary event types (e.g. `linux.advanced.events.dns` if DNS monitoring is not needed).
	- Review `linux.advanced.events` settings to disable unnecessary event types (e.g. `linux.advanced.events.dns` if malicious behavior rules based on DNS monitoring is not needed).

[ferullo]

Suggested change

	- Use **Event Filters** to reduce event volume from known-noisy directories without creating a monitoring blind spot.
	- Use **Event Filters** to reduce event volume from known-noisy directories without creating a active protection blind spot.

[ferullo]

Suggested change

During policy application, Elastic Defend hashes all running processes and their executables. When the file cache (`/opt/Elastic/Endpoint/state/cache.db`) is empty — first install, cache deleted, or after upgrade — the endpoint must hash every binary from scratch. Large binaries (e.g. Oracle at `/data/app/oracle/product/*/bin/oracle`) can cause the Events plugin to hang in `ConfigurationCallback` while performing SHA1 hashing, driving CPU to 100% for extended periods.

During policy application, Elastic Defend hashes all running processes and their executables. When the file cache (`/opt/Elastic/Endpoint/state/cache.db`) is empty — first install, cache deleted, or after upgrade — the endpoint must hash every binary from scratch. Large binaries (e.g. Oracle at `/data/app/oracle/product/*/bin/oracle`) can drive CPU to 100% for extended periods.

[tomsonpl]

Looks alright, left a few questions just for clarity sake :)

[tomsonpl]

I see that only 3 files contain the link field, just checking if this is intentional?

[joeypoon]

yeah, this is an optional field

[tomsonpl]

no OS field here?

[ferullo]

Suggested change

Elastic Defend performs real-time file hashing, digital signature verification, memory scanning, behavioral rule evaluation, and event enrichment. Each protection layer adds processing overhead, and the cumulative effect depends on the workload profile of the host. The most common drivers of high CPU on Windows are third-party security product conflicts creating mutual scanning loops, high-volume security events (logon/logoff) overwhelming the rules engine, file-intensive operations triggering repeated hashing of large binaries, and VDI/Citrix environments where the file metadata cache is empty on every session.

Elastic Defend performs real-time file hashing, digital signature verification, memory scanning, behavioral rule evaluation, and event enrichment. Each protection layer adds processing overhead, and the cumulative effect depends on the workload profile of the host. The most common drivers of high CPU on Windows are third-party security product conflicts creating mutual scanning loops, high-volume security events (logon/logoff) overwhelming the Malicious Behavior engine, file-intensive operations triggering repeated hashing of large binaries, and VDI/Citrix environments where the file metadata cache is empty on every session.

[ferullo]

pre 8.0.1 is so old it is no longer supported.

Suggested change

The endpoint maintains a file metadata cache to avoid re-hashing known files. On versions prior to 8.0.1, the cache size (`FILE_OBJECT_CACHE_SIZE`) was limited to 500 entries, leading to frequent cache evictions and redundant hashing. Upgrading to 8.0.1+ significantly improves caching behavior.

[ferullo]

Suggested change

For immediate relief on older versions, set `windows.advanced.kernel.asyncimageload` and `windows.advanced.kernel.syncimageload` to `false` in advanced policy settings. This reduces CPU by approximately 85% for DLL load processing at the cost of reduced visibility into library load events.

[intxgo]

Flushing some comments/suggestions

[intxgo]

Suggested change

A Trusted Application tells Elastic Defend to stop monitoring a process entirely — it creates an intentional blind spot. This is fundamentally different from an Endpoint Alert Exception, which only suppresses alerts while continuing to monitor. Many "trusted app not working" issues stem from using the wrong artifact type, misconfiguring the entry's condition fields, or running a version affected by known event-processing bugs.

A Trusted Application tells Elastic Defend to stop monitoring a process entirely — it creates an intentional blind spot - implicitly meaning it only works on executables. This is fundamentally different from an Endpoint Alert Exception, which only suppresses alerts while continuing to monitor. Many "trusted app not working" issues stem from using the wrong artifact type, misconfiguring the entry's condition fields, or running a version affected by known event-processing bugs.

[ferullo]

• implicitly meaning it only works on executables

This is true for standard trusted apps but Advanced Trusted Apps lets users trust non-process activity. The original version seems more correct to me.

[intxgo]

@ferullo I have to verify that, but I seriously doubt it, last time I walked the code, TA was evaluated only on execution callbacks, so anything like file access/modification/creation won't work here, but that's what people assume this can do

[intxgo]

Suggested change

	## Common issues
	### Wrong condition field or value
	## Common issues
	### Logical misuse
	Trusted application makes an application process trusted, allowing it to do whatever it does, modify files, start other applications, etc. When you're creating a Trusted Application entry you're adding a filter telling {{elastic-defend}} to fully trust a process matching the filter and all its actions. However it can't be used the other way around, to whitelist a particular action for some process(es), for example to trust any app to modify a particular csv file whilst doing full monitoring for other actions mabe by the process(es).
	### Wrong condition field or value

or ### Conceptual misues ?

I can see from SDHs that people really misuse TA, most likely because we don't have a commonly used feature "Suppress protections (trust) everything matching the path: xxx".

The problem is that they hardly ever cleanup unsuccessful TA entries, either they don't know or don't care, but the more entries the more CPU is spend on every process start evaluating the TA list.

We've seen TA artifacts containing paths like C:\somethin\*.sys, C:\*\*.txt

[ferullo]

Replace {{elastic-defend}} with Elastic Defend.

we don't have a commonly used feature "Suppress protections (trust) everything matching the path: xxx"

It's not obvious but the solution are Endpoint Alert Exceptions for that (either based on file.path MATCHES c:\whatever\* or process.executable MATCHES c:\whatever\*

[intxgo]

agree, we need to walk-by-hand the users here, so explain them TA is not what they think is the right tool -> point them towards exceptions... and here is another rabbit hole. I've recently tried to ask public LLM how to make an exception in Elastic Defend for a particular alert, it got it all wrong! LLMs (likewise me, looking at the public documents) are really confused about how to set Endpoint exceptions. The whole documentation is written for BEATS, loosely adding in Elastic Defend to the picture as "Endpoint exception" but IMO this is really confusing when you're having actions/processes blocked on your machine, going to Kibana to "suppress them". The verbiage in the UI and docs could be much more improved. I know I'm biased as Endpoint developer, but the UI/doc kind of hide/obscure Endpoint alerts and exception lists mechanics.

[intxgo]

Suggested change

	## Symptom
	Elastic Agent reports the Endpoint component as `FAILED` or `DEGRADED` with the message "Failed: endpoint service missed 3 check-ins". The Endpoint may appear unhealthy in Fleet, and endpoint events, alerts, and metadata stop being ingested. In some cases thousands of endpoints go unhealthy simultaneously.
	## Symptom
	The Agent appears `ORPHANED` in Fleet, and Endpoint policy changes do not apply nor response actions can execute. In some cases thousands of endpoints go unhealthy simultaneously.
	## Summary
	Elastic Endpoint has lost communication with Elastic Agent, but is still protecting the system according to last known policy and sends all events and alerts to the stack.
	## Common issues
	### Elastic Agent not running
	Elastic Agent crashed, has been explicitly disabled or removed.
	### Communication between Agent and Endpoint services not working
	Elastic Agent reports the Endpoint component as `FAILED` or `DEGRADED` with the message "Failed: endpoint service missed 3 check-ins". For troubleshooting see next symptom below.
	## Symptom
	Elastic Agent appears `UNHEALTHY` in fleet and Elastic Defend integration indicates errors. Elastic Agent reports the Endpoint component as `FAILED` or `DEGRADED` with the message "Failed: endpoint service missed 3 check-ins". Endpoint events, alerts, and metadata stop being ingested. In some cases thousands of endpoints go unhealthy simultaneously.

[intxgo]

Similar to windows case, please also explain Elastic Agent crash case.

[intxgo]

Suggested change

A third-party application malfunctions, crashes, or causes system instability when Elastic Defend (or the Endgame Sensor) is installed. Common manifestations include: application crashes (e.g. browsers closing unexpectedly), network policies or VPN functionality breaking, CPU spikes rendering the system unresponsive, or virtual machines hanging during policy changes. The issue resolves when the endpoint security product is uninstalled or specific protection features are disabled.

A third-party application malfunctions, crashes, or causes system instability when Elastic Defend is installed. Common manifestations include: application crashes (e.g. browsers closing unexpectedly), network policies or VPN functionality breaking, CPU spikes rendering the system unresponsive, or virtual machines hanging during policy changes. The issue resolves when the endpoint security product is uninstalled or specific protection features are disabled.

why that discontinued product appeared here? Furthermore it kind of suggested that Endpoint and Sensor are recommended to run together.

[intxgo]

Suggested change

Elastic Defend operates at multiple system levels — kernel-mode drivers for file system and network filtering, user-mode DLL injection for exploit protection (Endgame Sensor), and eBPF probes on Linux for host isolation and event collection. These deep integrations can conflict with other software that operates at similarly privileged levels: other security products with kernel drivers, eBPF-based networking tools, VPN clients that intercept DNS or network traffic, and applications that generate extreme volumes of system activity.

Elastic Defend operates at multiple system levels — kernel-mode drivers for file system and network filtering, user-mode DLL injection for exploit protection, and eBPF probes on Linux for host isolation and event collection. These deep integrations can conflict with other software that operates at similarly privileged levels: other security products with kernel drivers, eBPF-based networking tools, VPN clients that intercept DNS or network traffic, and applications that generate extreme volumes of system activity.

[intxgo]

Suggested change

	To confirm, check `elastic-endpoint top` for the third-party product's processes. If they dominate `overall.week_ms` in the metrics, add them as **Trusted Applications** in Elastic Defend. Also add Elastic Defend's paths to the third-party product's exclusion list.
	Add all 3rd party security applications as **Trusted Applications** in Elastic Defend to break feedback loops. Also add Elastic Defend's paths to the third-party product's exclusion list.
	To confirm the problem on Endpoint side check `elastic-endpoint top` for the third-party product's processes. If they dominate `overall.week_ms` in the metrics.

[intxgo]

Suggested change

A Windows system running Elastic Defend experiences a Blue Screen of Death (BSOD) or kernel crash. The memory dump analysis references `elastic_endpoint_driver.sys` or `elastic-endpoint-driver.sys`. Common bugcheck codes include `IRQL_NOT_LESS_OR_EQUAL`, `KERNEL_MODE_HEAP_CORRUPTION` (0x13A), `SYSTEM_SERVICE_EXCEPTION` (0x3B), or `PAGE_FAULT_IN_NONPAGED_AREA`. The crash may occur shortly after an agent upgrade, during heavy I/O workloads, or after the system has been running for some time with many network connections. In severe cases the system enters a boot loop.

A Windows system running Elastic Defend experiences a Blue Screen of Death (BSOD) or kernel crash. The memory dump analysis references `elastic_endpoint_driver.sys` or `elastic-endpoint-driver.sys`. The crash may occur shortly after an agent upgrade, 3rd party security product installation or it's configuration change, during heavy I/O workloads, or after the system has been running for some time with many network connections. In severe cases the system enters a boot loop.

I don't think we're commonly causing crashes, so this sentence makes absolutely no sense to me.

[ferullo]

Suggested change

Elastic Defend operates at multiple system levels — kernel-mode drivers for file system and network filtering, user-mode DLL injection for exploit protection (Endgame Sensor), and eBPF probes on Linux for host isolation and event collection. These deep integrations can conflict with other software that operates at similarly privileged levels: other security products with kernel drivers, eBPF-based networking tools, VPN clients that intercept DNS or network traffic, and applications that generate extreme volumes of system activity.

Elastic Defend operates at multiple system levels — kernel-mode drivers for file system and network filtering on Windows, system extension and network content filtering on macOS, and eBPF probes on Linux. These deep integrations can conflict with other software that operates at similarly privileged levels: other security products with kernel drivers, eBPF-based networking tools, VPN clients that intercept DNS or network traffic, and applications that generate extreme volumes of system activity.

[ferullo]

Suggested change

	### Browser crashes from exploit protection DLL after Windows updates (Endgame Sensor)
	On systems running the Endgame Sensor, Chrome, Edge, and potentially other Chromium-based browsers crash within seconds of opening. The crash dump shows a heap corruption or access violation in `esensordbi.dll` (the Endgame exploit protection injection DLL), specifically in `HookManager_InstallExternalHook`. This issue is triggered by specific Windows security updates (e.g. KB5074109 on Windows 11 25H2) that change the set of DLLs loaded into browser processes.
	The root cause is a bug in the Endgame Sensor's hook manager linked list. When the maximum number of hooks (4) for a particular function in a module is reached, the error handling path frees a `HOOK_ENTRY` that is still part of the global hook list. The next traversal of the list encounters corrupted `flink`/`blink` pointers and crashes. Windows updates that add new DLLs to browser processes increase the number of hooks installed, making it more likely to hit the limit and trigger the bug.
	**Fixed in**: Endgame Sensor 3.65.3.
	**Workaround** (for older sensor versions): In the Endgame policy under Threats > Exploit, uncheck Chrome and Edge to disable exploit protection injection into those browsers. Additionally, under Settings > Event Collection > Windows Event Collection, disable beta event sources. This prevents `esensordbi.dll` from being injected into the affected processes.
	Disabling exploit protection for browsers reduces defense against browser exploitation. Keep browsers and the OS fully patched to minimize the window of exposure.

[ferullo]

mac.advanced.capture.env.dns does not exist. I don't see any advanced option that can do this, @ricardo-estc is there one?

[ferullo]

Suggested change

On Windows Server systems running Distributed File System Replication (DFSR), Elastic Defend's file system filter driver can interfere with replication operations. Symptoms include replication backlogs, unexpected staging folder growth, or DFSR service errors in the event log. The issue is exacerbated when rollback self-healing (`windows.advanced.artifacts.global.rollback.self_healing.enabled`) is active, as Elastic Defend may detect and attempt to roll back legitimate DFSR staging operations that involve suspicious file patterns.

On Windows Server systems running Distributed File System Replication (DFSR), Elastic Defend's file system filter driver can interfere with replication operations. Symptoms include replication backlogs, unexpected staging folder growth, or DFSR service errors in the event log. The issue is exacerbated when rollback self-healing (`windows.advanced.alerts.rollback.self_healing.enabled`) is active, as Elastic Defend may detect and attempt to roll back legitimate DFSR staging operations that involve suspicious file patterns.

[ferullo]

Suggested change

	4) For browser crashes on Windows, collect crash dumps from `%HOMEPATH%\AppData\Local\Google\Chrome\User Data\Crashpad\reports` or `%LOCALAPPDATA%\CrashDumps` and check for `esensordbi.dll` in the faulting module. Verify the Endgame Sensor version and check if a fix is available.
	4) For browser crashes on Windows, collect crash dumps from `%HOMEPATH%\AppData\Local\Google\Chrome\User Data\Crashpad\reports` or `%LOCALAPPDATA%\CrashDumps`.

[ferullo]

This log message is wrong. @intxgo do you know what log is correct here?

[intxgo]

I'll park it for tomorrow, thanks

[intxgo]

Indeed, that message is for output only.

Endpoint -> Agent connection issue can be breaking in multiple places, I think we can point users at the 2 major logs

- ### "No valid comms client available" in Endpoint logs
+ ### "Unable to retrieve connection info from Agent" or "Agent GRPC connection failed to establish within deadline" in Endpoint logs

[ferullo]

Suggested change

	This message in the Endpoint state logs (`endpoint-*.log`) indicates that the Endpoint process started but cannot establish a communication channel with Elastic Agent. Common causes:
	This message in the Endpoint state logs (`logs-elastic_agent.endpoint_security-*`) indicates that the Endpoint process started but cannot establish a communication channel with Elastic Agent. Common causes:

[ferullo]

Suggested change

	- The Endpoint process is aborting shortly after startup due to another issue (look for `Aborting due to signal` immediately after the comms error).
	- The Endpoint process is aborting shortly after startup due to another issue (look for `Aborting due to signal` immediately after the comms error).
	- Another system process is listening on these ports.

[ferullo]

or the Endpoint communication port can be changed via advanced policy settings.

Is that true?

[intxgo]

old Endpoints hardcoded 6788, newer endpoints hardcode named pipe name

6789 was always given by Agent as bootstrap message, if it can be configured on Agent, then Endpoint will use what's given in bootstrap

[ferullo]

Suggested change

7) Check `metrics-endpoint.policy-*` for FAILED policy responses indicating the endpoint tried to start but could not apply its policy

[ferullo]

Suggested change

The bug is tracked in elastic/security#1973 and was fixed in Endpoint 8.13.3. Check the Endpoint logs for crash stack traces mentioning `CompletionPort.cpp` or `BigChief.cpp` during the `Connecting to minifilter` phase.

[ferullo]

Drop filename/line number since that could easily change.

Suggested change

	KafkaClient.cpp:561 KafkaClient failed to deliver record with unrecoverable error: Broker: Message size too large [10] | non-retriable
	KafkaClient failed to deliver record with unrecoverable error: Broker: Message size too large [10] | non-retriable

[ferullo]

Suggested change

	AgentContext.cpp:565 Endpoint is setting status to DEGRADED, reason: Unable to connect to output server
	Endpoint is setting status to DEGRADED, reason: Unable to connect to output server

[ferullo]

Suggested change

	LogstashClient.cpp:662 SSL handshake with Logstash server at [host]:[port] encountered an error
	SSL handshake with Logstash server at [host]:[port] encountered an error

[ferullo]

Suggested change

	BulkQueueConsumer.cpp:192 Logstash connection is down
	Logstash connection is down

[ferullo]

@brian-mckinney is this or was it ever true?

[ferullo]

Suggested change

	ServiceCommsFunctions.cpp:234 Canceling output backoff due to 'test output' command
	Canceling output backoff due to 'test output' command

[ferullo]

• implicitly meaning it only works on executables

This is true for standard trusted apps but Advanced Trusted Apps lets users trust non-process activity. The original version seems more correct to me.

[ferullo]

Replace {{elastic-defend}} with Elastic Defend.

we don't have a commonly used feature "Suppress protections (trust) everything matching the path: xxx"

It's not obvious but the solution are Endpoint Alert Exceptions for that (either based on file.path MATCHES c:\whatever\* or process.executable MATCHES c:\whatever\*

[ferullo]

If using "Global" assignment, confirm the affected endpoint's policy is not explicitly excluded.

Is that a feature I'm not aware of?

[intxgo]

A gneral question, is it going to be stack versioned? Is the current PR for the lastest 9.3 stack, or what? I agree with other comments here that it doesn't make sense to train AI on past already fixed bug symptoms, also quoting "Endgame" or "sensor" really makes no sense here, it's unrelated product, even not supported anymore.