Zeek integration - #11
Open
Dhar01 wants to merge 10 commits into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds a Zeek-based sensing path to hexwall so it can (a) correlate live connections with observed TLS SNI from ssl.log and (b) ingest Zeek DNS-bypass notices from notice.log, then evaluate those signals against Pi-hole policy (allow/deny/gravity) and the existing trust store.
Changes:
- Introduces
internal/zeektailers forssl.log(SNI correlation cache) andnotice.log(DNS-bypass events). - Expands Pi-hole integration to include gravity/policy lookups and adds domain/SNI matching + new store tables for per-IP domain sets and observations.
- Refactors the monitor into a “decision ladder”, adds domain-based deghost checks, and replaces somo-based enforcement with a stopgap
internal/enforcerprocess terminator.
Reviewed changes
Copilot reviewed 22 out of 23 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| zeek-integration-guide.md | Design/ops guide for Zeek DNS-bypass notice ingestion. |
| SNI_DETECTION_DESIGN.md | Design notes for SNI mismatch detection and correlation strategy. |
| SETUP.md | Local setup guide for Pi-hole + tool testing. |
| OVERVIEW.md | High-level product overview and threat-model explanation. |
| main.go | Adds flags, logging setup, Zeek watchers, pruning, and updated monitor wiring. |
| internal/zeek/zeek.go | Implements ssl.log tailer + notice.log watcher + parsers. |
| internal/zeek/zeek_test.go | Unit tests for Zeek log parsing/tailing behavior. |
| internal/store/store.go | Adds schema/tables, migrations, pruning, and new domain/SNI/IP observation APIs. |
| internal/store/store_test.go | Extensive tests for new store schema and behaviors. |
| internal/somo/somo.go | Removes PID kill helper (enforcement moved elsewhere). |
| internal/pihole/policy_test.go | Adds tests for gravity/policy lookup and counts. |
| internal/pihole/pihole.go | Adds gravity.db support, policy lookups, and domain matching helper. |
| internal/pihole/domain_match_test.go | Tests for DomainMatches. |
| internal/pihole/cache.go | Stores full IP→domains sets (not single domain per IP) and writes new store table. |
| internal/monitor/monitor.go | New decision ladder, Zeek event handler, domain policy gating, and enforcement path. |
| internal/enforcer/enforcer.go | Adds stopgap process-based enforcement and PID parsing. |
| internal/enforcer/enforcer_test.go | Tests for enforcer guardrails and PID parsing. |
| internal/deghost/deghost.go | Adds domain reputation endpoint + block predicate. |
| internal/deghost/deghost_test.go | Tests for domain reputation client + predicate. |
| CLAUDE.md | Adds repo guidance doc (currently out of sync with new flow/enforcement). |
| CHAT_SUMMARY_2026-04-06.md | Historical implementation summary (now partially out of date). |
| .golangci.yml | Adds golangci-lint configuration. |
| .gitignore | Adds local DB/inventory ignores. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+143
to
+153
| event := Event{Timestamp: time.Now()} | ||
| if ts, ok := payload["ts"].(string); ok { | ||
| if parsed, err := time.Parse(time.RFC3339Nano, ts); err == nil { | ||
| event.Timestamp = parsed | ||
| } | ||
| } | ||
| if ts, ok := payload["timestamp"].(string); ok { | ||
| if parsed, err := time.Parse(time.RFC3339Nano, ts); err == nil { | ||
| event.Timestamp = parsed | ||
| } | ||
| } |
Comment on lines
+85
to
+90
| selectedMode := normalizeMode(mode) | ||
| blocked, err := checker.IsBlockedByPolicy(event.SNI) | ||
| if err != nil { | ||
| slog.Warn("zeek policy lookup failed", "sni", event.SNI, "err", err) | ||
| return | ||
| } |
Comment on lines
+411
to
+426
| if err := hexwallStore.LogKill(ipStr, conn.PID, conn.Program); err != nil { | ||
| slog.Error("failed to log kill", "address", conn.RAddress, "err", err) | ||
| } | ||
|
|
||
| // Connection-level termination via `ss -K` (requires CONFIG_INET_DIAG_DESTROY) | ||
| // is deferred work. KillProcess terminates the owning process, which is coarser. | ||
| pidInt, err := enforcer.ParsePID(conn.PID) | ||
| if err != nil { | ||
| slog.Warn("invalid PID, skipping kill", "pid", conn.PID, "address", conn.RAddress, "err", err) | ||
| return | ||
| } | ||
| if err := enforcer.KillProcess(pidInt, conn.Program, conn.RAddress); err != nil { | ||
| slog.Warn("failed to kill process", "address", conn.RAddress, "pid", conn.PID, "err", err) | ||
| } else { | ||
| slog.Info("killed connection", "address", conn.RAddress) | ||
| } |
Comment on lines
+29
to
+34
| **Flow:** | ||
| 1. `main.go` — entry point; runs `runCheck()` immediately, then on a 10s ticker | ||
| 2. `runCheck()` calls `somo.GetEstablishedConnections()` to get live TCP/UDP connections | ||
| 3. Each connection's remote IP is passed to `isVulnerable()` (currently a no-op stub) | ||
| 4. Vulnerable connections are killed via `somo.KillConnection(pid)` | ||
|
|
| - `ShouldKill(report) == false` -> log as unrecognized but clean; allow. | ||
| - `ShouldKill(report) == true`: | ||
| - if mode is `watch`: log "would kill" only. | ||
| - if mode is `enforce`: `store.LogKill(...)` then `somo.KillConnection(pid)`. |
Comment on lines
+206
to
+229
| scanner := bufio.NewScanner(file) | ||
| for scanner.Scan() { | ||
| line := strings.TrimSpace(scanner.Text()) | ||
| if line == "" { | ||
| continue | ||
| } | ||
|
|
||
| event, err := ParseNoticeLine(line) | ||
| if err != nil { | ||
| continue | ||
| } | ||
|
|
||
| select { | ||
| case events <- event: | ||
| default: | ||
| } | ||
| } | ||
| if err := scanner.Err(); err != nil { | ||
| slog.Debug("zeek: notice log scan failed", "path", logPath, "err", err) | ||
| } | ||
| } | ||
| _ = file.Close() | ||
| offset = fi.Size() | ||
| } |
Adds a second trust signal alongside Pi-hole's IP history. Zeek reads the TLS SNI off the wire, giving the domain a connection actually claims rather than just the IP it dialed — closing the gap where a shared CDN edge IP is trusted for one domain and abused for another. - internal/zeek: new package. Tails ssl.log into a short-lived per-connection SNI cache keyed on local port, and watches notice.log for DNS-bypass alerts. Holds partial trailing lines across polls so records flushed mid-write are not silently dropped. - internal/pihole: IsBlockedByPolicy checks Pi-hole's blocklist directly, independent of query history. - internal/deghost: CheckDomain + ShouldBlockDomain for domain-shaped reputation lookups. Unlike the IP endpoint, 400/403 carry a real body and are decoded rather than treated as "safe". - internal/store: domain_checks cache (6h), plus zeek_alerts and sni_observations audit tables. - internal/monitor: SNI-bypass connections escalate through blocklist, then cache, then the domain API — never the IP reputation check, which reads clean on shared CDN addresses regardless of what is fronted through them. - main.go: --zeek-log, --zeek-notice-log, --enable-zeek flags; flag parsing extracted into parseFlags/runConfig. Zeek support defaults to off. Watch mode remains the default.
…thod for clearing the database
Dhar01
force-pushed
the
zeek_integration
branch
from
August 3, 2026 12:23
9f415f8 to
86ee303
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 16 out of 17 changed files in this pull request and generated no new comments.
Suppressed comments (1)
internal/enforcer/enforcer.go:90
KillProcesstreats any error fromsyscall.Kill(pid, 0)as meaning the process is gone, butkill(pid, 0)can also fail withEPERMwhen the process still exists (e.g., insufficient permissions). That would incorrectly report success and skip the SIGKILL escalation. OnlyESRCHshould be treated as “process no longer exists”; other errors should fall through to the escalation path (or be handled explicitly).
// Check whether the process still exists.
if err := syscall.Kill(pid, 0); err != nil {
// Process is gone — SIGTERM worked.
slog.Info("process terminated by SIGTERM", "pid", pid, "program", program)
return nil
}
Dhar01
force-pushed
the
zeek_integration
branch
from
August 4, 2026 07:50
d264331 to
6fe9c88
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 16 out of 17 changed files in this pull request and generated no new comments.
Suppressed comments (6)
internal/store/store.go:617
- RecentZeekAlert queries by (src_ip, sni) using exact equality. Without normalizing inputs to match LogZeekAlert's stored canonical form, lookups can miss equivalent values that differ only in case/whitespace.
cutoff := time.Now().Add(-time.Hour).Unix()
internal/monitor/monitor.go:413
- applyBlockedOutcome logs a kill to killed_connections before validating the PID or attempting the termination. With the new enforcer guardrails (protected programs, invalid PID, own PID, etc.), this can record "killed" rows even when nothing was actually killed, reducing audit accuracy. Log the kill only after KillProcess succeeds (or rename/store intent separately).
if err := hexwallStore.LogKill(ipStr, conn.PID, conn.Program); err != nil {
slog.Error("failed to log kill", "address", conn.RAddress, "err", err)
}
internal/monitor/monitor.go:83
- HandleZeekEvent persists Zeek notices to zeek_alerts, but it only checks event.SNI. The zeek_alerts schema requires src_ip/dst_ip/dst_port/sni NOT NULL, so events missing any of these fields will trigger repeated insert errors and noisy logs. Normalize SNI before policy lookup/persist and skip events missing required fields.
if event.SNI == "" {
return
}
internal/store/store.go:383
- UpsertAllowedIPDomain lowercases/trims the domain but still allows an empty string, which can create meaningless rows and waste space (and complicate later matching/debugging). Reject empty domains explicitly.
This issue also appears on line 617 of the same file.
func (s *Store) UpsertAllowedIPDomain(ip, domain string) error {
domain = strings.ToLower(strings.TrimSpace(domain))
now := time.Now().Unix()
internal/zeek/zeek.go:206
- WatchNoticeLog uses bufio.Scanner with the default 64K token limit; a long JSON notice line will cause scanner.Err() (typically ErrTooLong) and the event will be skipped. Increase the scanner buffer to avoid silently dropping legitimate Zeek notices.
scanner := bufio.NewScanner(file)
internal/store/store.go:603
- LogZeekAlert stores SNI and IP fields without normalization/validation. Because RecentZeekAlert matches on exact sni/src_ip and the table columns are NOT NULL, mixed-case or whitespace in SNI can prevent dedupe/lookups and missing fields will fail at INSERT time. Normalize/validate inputs before writing.
func (s *Store) LogZeekAlert(srcIP, dstIP, dstPort, sni string, blocked bool, confidence, actionTaken string) error {
blockedInt := 0
if blocked {
blockedInt = 1
}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
name: Pull request
about: Submit a pull request for hexwall
title: Integration of zeek to catch network connections
labels: 'new feature'
assignees:
LoknathDescription
The previous version of
hexwallwas designed to fix the gappi-holeintroduced, but it still fails for alternate resolvers, connecting via CNAME alias. A shared CDN edge address legitimately serves thousands of unrelated domains, an attacker can ride an already trusted IP by setting the TLS SNI to their hostname.This branch closes that gap by reading the SNI directly off the wire via
Zeek, by checking domains against pi-hole's real policy lists rather than inferring from history.Type of change
How has this been tested?
List the commands you ran and any manual verification steps.
If you skipped any checks, explain why.
go test ./...golangci-lint run ./...Test Configuration:
Linux pop-os 7.0.11-76070011-genericgo1.26.1 linux/amd64somoversion:1.3.1pihole-FTL.db,gravity.db.watchorenforce): bothChecklist