diff --git a/README_EN.md b/README_EN.md index c94c13b..f328aa9 100644 --- a/README_EN.md +++ b/README_EN.md @@ -4,381 +4,265 @@ [中文文档](./readme.md) | **English** -**KSubdomain is a stateless subdomain enumeration tool that delivers unprecedented scanning speed with extremely low memory consumption.** Say goodbye to traditional tool bottlenecks and experience lightning-fast DNS queries with a reliable state table retransmission mechanism ensuring result completeness. KSubdomain supports Windows, Linux, and macOS, making it ideal for large-scale DNS asset discovery. - -![](image.gif) +**KSubdomain is a stateless subdomain enumeration tool delivering unprecedented scanning speed with extremely low memory consumption.** By directly operating network adapters via raw sockets — bypassing the OS kernel stack — paired with a reliable state table retransmission mechanism, it ensures result completeness. Supports Windows, Linux, and macOS. ## 🚀 Core Advantages -* **Lightning-Fast Speed:** Utilizing stateless scanning technology, it directly operates network adapters for raw socket packet sending, bypassing the system kernel's network protocol stack to achieve astonishing packet rates. Use the `test` command to probe your local network adapter's maximum sending speed. -* **Extremely Low Resource Consumption:** Innovative memory management mechanisms, including object pools and global memory pools, significantly reduce memory allocation and GC pressure, maintaining low memory footprint even when processing massive domain lists. -* **Stateless Design:** Similar to Masscan's stateless scanning, it doesn't maintain a state table from the system, building a lightweight state table instead, fundamentally solving traditional scanning tools' memory bottlenecks and performance limitations, as well as stateless scanning packet loss issues. -* **Reliable Retransmission:** Built-in intelligent retransmission mechanism effectively handles network jitter and packet loss, ensuring result accuracy and completeness. -* **Cross-Platform Support:** Perfect compatibility with Windows, Linux, and macOS. -* **Easy to Use:** Simple command-line interface, providing verify and enum modes, with built-in common dictionaries. - -## ⚡ Performance Highlights - -KSubdomain far exceeds similar tools in speed and efficiency. Here's a comparison test using a 100k dictionary in a 4-core CPU, 5M bandwidth network environment: - -| Tool | Mode | Method | Command | Time | Success | Notes | -| ------------ | ------ | ------------ | -------------------------------------------------------------------------- | -------------- | ------- | ------------------------- | -| **KSubdomain** | Verify | pcap network | `time ./ksubdomain v -b 5m -f d2.txt -o k.txt -r dns.txt --retry 3 --np` | **~30 sec** | 1397 | `--np` disables real-time printing | -| massdns | Verify | pcap/socket | `time ./massdns -r dns.txt -t A -w m.txt d2.txt --root -o L` | ~3 min 29 sec | 1396 | | -| dnsx | Verify | socket | `time ./dnsx -a -o d.txt -r dns.txt -l d2.txt -retry 3 -t 5000` | ~5 min 26 sec | 1396 | `-t 5000` sets 5000 concurrent | - -**Conclusion:** KSubdomain is **7x faster** than massdns and **10x faster** than dnsx! - -## 🛠️ Technical Innovations (v2.0) - -KSubdomain 2.0 introduces multiple underlying optimizations to further squeeze performance potential: - -1. **State Table Optimization:** - * **Sharded Locks:** Replaces global locks, significantly reducing lock contention and improving concurrent write efficiency. - * **Efficient Hashing:** Optimizes key-value storage, evenly distributing domains, and enhancing lookup speed. -2. **Packet Sending Optimization:** - * **Object Pools:** Reuses DNS packet structures, reducing memory allocation and GC overhead. - * **Template Caching:** Reuses Ethernet/IP/UDP layer data for the same DNS servers, reducing redundant construction overhead. - * **Parallel Sending:** Multi-goroutine parallel packet sending, fully utilizing multi-core CPU performance. - * **Batch Processing:** Batch sends domain requests, reducing system calls and context switching. -3. **Receiving Optimization:** - * **Object Pools:** Reuses parsers and buffers, reducing memory consumption. - * **Parallel Processing Pipeline:** Receive → Parse → Process three-stage parallelism, improving processing pipeline efficiency. - * **Buffer Optimization:** Increases internal Channel buffer size, avoiding processing blockage. - * **Efficient Filtering:** Optimizes BPF filter rules and packet processing logic, quickly discarding invalid packets. -4. **Memory Management Optimization:** - * **Global Memory Pool:** Introduces `sync.Pool` to manage common data structures, reducing memory allocation and fragmentation. - * **Structure Reuse:** Reuses DNS query structures and serialization buffers. -5. **Architecture and Concurrency Optimization:** - * **Dynamic Concurrency:** Automatically adjusts goroutine count based on CPU cores. - * **Efficient Random Numbers:** Uses more performant random number generators. - * **Adaptive Rate:** Dynamically adjusts packet sending rate based on network conditions and system load. - * **Batch Loading:** Batch loads and processes domains, reducing per-domain processing overhead. +- **Lightning-Fast:** Stateless scanning operating directly on network adapters — **7× faster** than massdns, **10× faster** than dnsx +- **Low Resource Usage:** Object pools + global memory pools keep memory footprint minimal even on massive domain lists +- **Dynamic Timeout:** RTT sliding window (TCP RFC 6298 EWMA, α=0.125) auto-adjusts timeouts — no manual tuning needed +- **Multi-NIC Parallel Sending:** Repeat `--interface` for multiple adapters; goroutines share a single `domainChan` for natural load balancing +- **Streaming SDK:** `EnumStream`/`VerifyStream` callback APIs for real-time result processing +- **Cross-Platform:** Windows, Linux, macOS -## 📦 Installation +## ⚡ Performance -### Quick Install +4-core CPU, 5M bandwidth, 100k wordlist: -```bash -# One-line install (Linux/macOS) -curl -sSL https://raw.githubusercontent.com/boy-hack/ksubdomain/main/install.sh | bash - -# Or use wget -wget -qO- https://raw.githubusercontent.com/boy-hack/ksubdomain/main/install.sh | bash -``` +| Tool | Method | Time | Found | +|------|--------|------|-------| +| **KSubdomain** | pcap raw socket | **~30 sec** | 1397 | +| massdns | pcap/socket | ~3 min 29 sec | 1396 | +| dnsx | socket | ~5 min 26 sec | 1396 | -### Manual Installation - -1. **Download Pre-compiled Binary:** Visit the [Releases](https://github.com/boy-hack/ksubdomain/releases) page to download the latest version for your system. -2. **Install `libpcap` Dependency:** - * **Windows:** Download and install [Npcap](https://npcap.com/) driver (WinPcap may not work). - * **Linux:** Already statically compiled with `libpcap`, usually no additional action needed. If issues occur, try installing `libpcap-dev` or `libcap-devel` package. - * **macOS:** System comes with `libpcap`, no installation needed. -3. **Grant Execute Permission (Linux/macOS):** `chmod +x ksubdomain` -4. **Run!** +## 📦 Installation -### Docker +### Pre-built Binary -```bash -# Pull image -docker pull ksubdomain/ksubdomain:latest +Download the latest release from [Releases](https://github.com/boy-hack/ksubdomain/releases). -# Run -docker run --network host --privileged ksubdomain/ksubdomain enum -d example.com -``` +**Dependencies:** +- **Windows:** Install [Npcap](https://npcap.com/) +- **Linux:** Statically compiled — usually no extra steps; install `libpcap-dev` if needed +- **macOS:** libpcap built-in, no installation needed ### Build from Source -Ensure you have Go 1.23+ and `libpcap` environment installed. - ```bash git clone https://github.com/boy-hack/ksubdomain.git cd ksubdomain -go build -o ksubdomain ./cmd/ksubdomain +go build -ldflags "-X github.com/boy-hack/ksubdomain/v2/pkg/core/conf.Version=v2.x.y" \ + -o ksubdomain ./cmd/ksubdomain ``` -## 📖 Usage +Or via `go install`: ```bash -KSubdomain - Ultra-Fast Stateless Subdomain Enumeration Tool +go install github.com/boy-hack/ksubdomain/v2/cmd/ksubdomain@latest +``` -Usage: - ksubdomain [global options] command [command options] [arguments...] +> **Note:** Requires root / `CAP_NET_RAW` privilege for raw packet capture. -Version: - Check version: ksubdomain --version +## 📖 Usage + +``` +Usage: + ksubdomain [global options] [command options] Commands: - enum, e Enumeration mode: Provide root domain for brute-force - verify, v Verification mode: Provide domain list for verification - test Test local network adapter's maximum packet sending speed - help, h Show command list or help for a command + enum, e Enumerate subdomains via dictionary brute-force + verify, v Verify a domain list for DNS resolution + test Test maximum packet rate of local network adapter + device List available network interfaces + help, h Show help Global Options: - --help, -h Show help (default: false) - --version, -v Print version (default: false) + --help, -h Show help + --version, -v Print version ``` -### Verification Mode +### Verify Mode -Verification mode quickly checks the alive status of provided domain lists. +Check whether a list of domains resolves in DNS. + +``` +Options: + --domain, -d Target domain(s) (repeatable) + --filename, -f File containing domain list + --stdin Read domains from stdin + --bandwidth, -b Bandwidth limit, e.g. 5m, 10m (default: 3m) + --resolvers, -r DNS resolver(s), uses built-in defaults + --output, -o Output file path + --format Output format: txt (default), json, csv, jsonl + --silent, -s Silent mode: suppress banner/logs, output domains only + --only-domain, --od Output domains only (no IPs/records) + --retry Retry count; -1 = infinite (default: 3) + --timeout DNS query timeout in seconds (default: 6) + --interface, -e Network interface (repeatable for multi-NIC) + --wildcard-filter Wildcard filter: none (default), basic, advanced + --predict Enable subdomain prediction + --quiet, -q Suppress screen output (file only) + --color, -c Colorized output +``` ```bash -./ksubdomain verify -h # or ksubdomain v - -OPTIONS: - --filename value, -f value Domain file path - --domain value, -d value Domain - --band value, -b value Bandwidth downstream speed, e.g., 5M, 5K, 5G (default: "3m") - --resolvers value, -r value DNS servers (uses built-in DNS by default) - --output value, -o value Output filename - --output-type value, --oy value Output file type: json, txt, csv, jsonl (default: "txt") - --silent Only output domains to screen (default: false) - --retry value Retry count, -1 for infinite retry (default: 3) - --timeout value Timeout in seconds (default: 6) - --stdin Accept stdin input (default: false) - --not-print, --np Don't print domain results (default: false) - --eth value, -e value Specify network adapter name - --wild-filter-mode value Wildcard filtering mode: basic, advanced, none (default: "none") - --predict Enable domain prediction mode (default: false) - --only-domain, --od Only output domains, no IPs (default: false) - --help, -h Show help (default: false) - -# Examples: -# Verify multiple domains -./ksubdomain v -d xx1.example.com -d xx2.example.com - -# Read domains from file and save to output.txt +# Verify single/multiple domains +./ksubdomain v -d www.example.com -d mail.example.com + +# Read from file, save results ./ksubdomain v -f domains.txt -o output.txt -# Read from stdin with 10M bandwidth limit -cat domains.txt | ./ksubdomain v --stdin -b 10M +# Pipe to httpx (silent + domain-only) +cat domains.txt | ./ksubdomain v --stdin -b 10m -s --od | httpx -silent -# Enable prediction mode with advanced wildcard filtering, save as CSV -./ksubdomain v -f domains.txt --predict --wild-filter-mode advanced --oy csv -o output.csv +# Advanced wildcard filter, JSONL output +./ksubdomain v -f domains.txt --wildcard-filter advanced --format jsonl -o output.jsonl -# JSONL format for tool chaining -./ksubdomain v -f domains.txt --oy jsonl | jq '.domain' +# Multi-NIC parallel sending +./ksubdomain v -f domains.txt --interface eth0 --interface eth1 ``` -### Enumeration Mode +### Enum Mode -Enumeration mode brute-forces subdomains under specified domains based on dictionaries and prediction algorithms. +Brute-force subdomains using a dictionary and optional prediction. -```bash -./ksubdomain enum -h # or ksubdomain e - -OPTIONS: - --domain value, -d value Domain - --band value, -b value Bandwidth downstream speed (default: "3m") - --resolvers value, -r value DNS servers - --output value, -o value Output filename - --output-type value, --oy value Output type: json, txt, csv, jsonl (default: "txt") - --silent Only output domains (default: false) - --retry value Retry count (default: 3) - --timeout value Timeout in seconds (default: 6) - --stdin Accept stdin input (default: false) - --not-print, --np Don't print results (default: false) - --eth value, -e value Specify network adapter - --wild-filter-mode value Wildcard filter mode (default: "none") - --predict Enable prediction mode (default: false) - --only-domain, --od Only output domains (default: false) - --filename value, -f value Dictionary path - --ns Read domain NS records and add to resolvers (default: false) - --help, -h Show help (default: false) - -# Examples: -# Enumerate multiple domains -./ksubdomain e -d example.com -d hacker.com - -# Use dictionary file -./ksubdomain e -d example.com -f subdomain.txt -o output.txt - -# Read from stdin with 10M bandwidth -cat domains.txt | ./ksubdomain e --stdin -b 10M - -# Enable prediction with advanced wildcard filtering -./ksubdomain e -d example.com --predict --wild-filter-mode advanced --oy jsonl +``` +Additional Options (beyond verify): + --filename, -f Dictionary file (uses built-in wordlist by default) + --domain-list, --ds File containing root domains for batch enumeration + --use-ns-records Query domain NS records and add them as resolvers ``` -## ✨ Features & Tips +```bash +# Enumerate with built-in wordlist +./ksubdomain e -d example.com -* **Automatic Bandwidth Adaptation:** Just specify your public network downstream bandwidth with `-b` (e.g., `-b 10m`), and KSubdomain automatically optimizes packet sending rate. -* **Test Maximum Rate:** Run `./ksubdomain test` to test maximum theoretical packet rate in current environment. -* **Automatic Network Adapter Detection:** KSubdomain auto-detects available network adapters. -* **Progress Display:** Real-time progress bar showing Success / Sent / Queue / Received / Failed / Time Elapsed. -* **Parameter Tuning:** Adjust `--retry` and `--timeout` based on network quality and target domain count for best results. When `--retry` is -1, it will retry indefinitely until all requests succeed or timeout. -* **Multiple Output Formats:** Supports `txt` (real-time), `json` (on completion), `csv` (on completion), `jsonl` (streaming). Specify with `-o` and file extension (e.g., `result.json`). -* **Environment Variables:** - * `KSubdomainConfig`: Specify config file path. +# Custom dictionary +./ksubdomain e -d example.com -f subdomains.txt -o result.txt -## 🔗 Integration Examples +# Prediction + advanced wildcard filter +./ksubdomain e -d example.com --predict --wildcard-filter advanced -### With httpx -```bash -./ksubdomain enum -d example.com --od | httpx -silent -``` +# Batch enumeration from file +./ksubdomain e --domain-list roots.txt -b 10m --format jsonl -o result.jsonl -### With nuclei -```bash -./ksubdomain enum -d example.com --od | nuclei -l /dev/stdin +# Pipe to httpx +./ksubdomain e -d example.com -s --od | httpx -silent ``` -### With nmap -```bash -./ksubdomain enum -d example.com --od | nmap -iL - -``` +### Test / Device Commands -### Streaming processing with JSONL ```bash -./ksubdomain enum -d example.com --oy jsonl | \ - jq -r 'select(.type == "A") | .domain' | \ - httpx -silent +# Test max packet rate of current network adapter +./ksubdomain test + +# List available network interfaces +./ksubdomain device ``` -### In Python scripts -```python -import subprocess -import json +## ✨ Features -result = subprocess.run( - ['ksubdomain', 'enum', '-d', 'example.com', '--oy', 'jsonl'], - capture_output=True, text=True -) +### Dynamic Timeout Adaptation +No need to hand-tune `--timeout`. The engine measures real RTT samples and computes an adaptive timeout using EWMA (α=0.125, β=0.25 per RFC 6298), bounded between 1 s and 10 s. -for line in result.stdout.strip().split('\n'): - data = json.loads(line) - print(f"{data['domain']} => {data['records']}") -``` +### Multi-NIC Parallel Sending +Repeat `--interface` to distribute sending across multiple adapters. Each adapter runs an independent `sendCycleForIface` goroutine competing on the shared `domainChan`. -### In Go programs -```go -import "github.com/boy-hack/ksubdomain/v2/sdk" +```bash +./ksubdomain e -d example.com --interface eth0 --interface eth1 -b 20m +``` -scanner := sdk.NewScanner(&sdk.Config{ - Bandwidth: "5m", - Retry: 3, -}) +### Prediction Mode +With `--predict`, the tool uses discovered subdomains to predict related patterns, increasing coverage. -results, err := scanner.Enum("example.com") -for _, result := range results { - fmt.Printf("%s => %s\n", result.Domain, result.IP) -} -``` +### Wildcard Filtering +- `none` — no filtering (default) +- `basic` — remove obvious wildcard IPs +- `advanced` — multi-dimensional wildcard detection -## 🌟 Platform Notes +### Output Formats -### macOS Users +| Format | Description | Best For | +|--------|-------------|----------| +| `txt` | `domain => record`, real-time | Human review | +| `json` | Full JSON, written on completion | Programmatic parsing | +| `csv` | CSV table, written on completion | Data analysis | +| `jsonl` | One JSON per line, real-time | Streaming / tool chaining | -macOS uses BPF (Berkeley Packet Filter) with smaller default buffers: +## 🔗 Integration Examples ```bash -# Recommended: 5M bandwidth for stability -sudo ./ksubdomain e -d example.com -b 5m +# Pipe to httpx +./ksubdomain e -d example.com -s --od | httpx -silent -# If buffer errors occur -sudo ./ksubdomain e -d example.com -b 3m --retry 10 +# Pipe to nuclei +./ksubdomain e -d example.com -s --od | nuclei -l /dev/stdin -# System tuning (optional) -sudo sysctl -w net.bpf.maxbufsize=4194304 +# JSONL streaming: extract A records only +./ksubdomain e -d example.com --format jsonl | jq -r 'select(.type=="A") | .domain' + +# Batch + deduplicate +./ksubdomain e --domain-list roots.txt -s --od | sort -u > all_subs.txt ``` -### WSL/WSL2 Users +## 🌐 Platform Notes +**macOS** — BPF buffer is smaller; keep bandwidth conservative: ```bash -# Usually use eth0 -./ksubdomain e -d example.com --eth eth0 - -# If network adapter is not up -sudo ip link set eth0 up +sudo ./ksubdomain e -d example.com -b 5m ``` -### Windows Users - +**WSL/WSL2:** ```bash -# Must install Npcap driver first -# Download: https://npcap.com/ +./ksubdomain e -d example.com --interface eth0 +``` -# Run with administrator privileges +**Windows** — Must install Npcap first; run as Administrator: +```bash .\ksubdomain.exe enum -d example.com ``` -## 📊 Output Formats +## 🧩 Go SDK -### TXT (Default) -``` -www.example.com => 93.184.216.34 -mail.example.com => CNAME mail.google.com -api.example.com => 93.184.216.35 +```bash +go get github.com/boy-hack/ksubdomain/v2/sdk ``` -### JSON -```json -{ - "domains": [ - { - "subdomain": "www.example.com", - "answers": ["93.184.216.34"] - } - ] -} -``` +```go +import "github.com/boy-hack/ksubdomain/v2/sdk" -### CSV -```csv -subdomain,type,record -www.example.com,A,93.184.216.34 -mail.example.com,CNAME,mail.google.com -``` +scanner := sdk.NewScanner(&sdk.Config{ + Bandwidth: "5m", + Retry: 3, + WildcardFilter: "advanced", +}) + +// Blocking: collect all results +results, err := scanner.Enum("example.com") -### JSONL (JSON Lines) - **New!** 🆕 -```jsonl -{"domain":"www.example.com","type":"A","records":["93.184.216.34"],"timestamp":1709011200} -{"domain":"mail.example.com","type":"CNAME","records":["mail.google.com"],"timestamp":1709011201} +// Streaming: real-time callback +err = scanner.EnumStream(ctx, "example.com", func(r sdk.Result) { + fmt.Printf("%s => %v\n", r.Domain, r.Records) +}) ``` -Perfect for streaming processing and tool chaining! +See [SDK README](./sdk/README.md) and [API Reference](./docs/api.md). + +## 🧪 Testing -## 🛡️ Security & Ethics +```bash +# Unit tests (no root needed) +go test ./pkg/... -**Responsible Use:** -* Only scan domains you own or have permission to test -* Respect target systems and network resources -* Comply with local laws and regulations -* This tool is for security research and authorized testing only +# Regression tests (root + network required) +go build -o ksubdomain ./cmd/ksubdomain +sudo go test ./test/regression/... -tags regression -v -timeout 120s +``` ## 📚 Documentation -- [中文文档](./readme.md) - Chinese Documentation - [Quick Start Guide](./docs/quickstart.md) -- [API Documentation](./docs/api.md) +- [API Reference](./docs/api.md) - [Best Practices](./docs/best-practices.md) - [FAQ](./docs/faq.md) - -## 🤝 Contributing - -We welcome contributions! See [CONTRIBUTING.md](./CONTRIBUTING.md) for guidelines. - -- Report bugs: [GitHub Issues](https://github.com/boy-hack/ksubdomain/issues) -- Feature requests: [GitHub Discussions](https://github.com/boy-hack/ksubdomain/discussions) -- Submit PRs: Performance improvements, bug fixes, new features all welcome! +- [SDK README](./sdk/README.md) ## 💡 References -* Original KSubdomain: [https://github.com/knownsec/ksubdomain](https://github.com/knownsec/ksubdomain) -* From Masscan/Zmap Analysis to Practice: [https://paper.seebug.org/1052/](https://paper.seebug.org/1052/) -* KSubdomain Stateless Tool Introduction: [https://paper.seebug.org/1325/](https://paper.seebug.org/1325/) +- Original KSubdomain: [knownsec/ksubdomain](https://github.com/knownsec/ksubdomain) +- Stateless scanning theory: [paper.seebug.org/1052](https://paper.seebug.org/1052/) +- KSubdomain introduction: [paper.seebug.org/1325](https://paper.seebug.org/1325/) ## 📜 License -KSubdomain is released under the MIT License. See [LICENSE](LICENSE) for details. - -## 🙏 Acknowledgments - -Special thanks to all contributors and the open-source community! - ---- - -**Star ⭐ this repo if you find it useful!** - -Made with ❤️ by the KSubdomain Team +MIT License — see [LICENSE](LICENSE) diff --git a/docs/OUTPUT_FORMATS.md b/docs/OUTPUT_FORMATS.md index 76876c7..6a116ab 100644 --- a/docs/OUTPUT_FORMATS.md +++ b/docs/OUTPUT_FORMATS.md @@ -101,6 +101,8 @@ One JSON object per line, perfect for streaming. ### Usage ```bash +./ksubdomain enum -d example.com --format jsonl -o results.jsonl +# or using legacy alias ./ksubdomain enum -d example.com --oy jsonl -o results.jsonl ``` diff --git a/docs/api.md b/docs/api.md index eb0dcde..443ed40 100644 --- a/docs/api.md +++ b/docs/api.md @@ -13,7 +13,8 @@ type Config struct { Bandwidth string // bandwidth cap, e.g. "5m" (default "5m") Retry int // per-domain retry count, -1 = infinite (default 3) Resolvers []string // DNS resolver IPs; nil = built-in defaults - Device string // network interface; "" = auto-detect + Device string // single network interface; "" = auto-detect (backward-compat) + Devices []string // multi-NIC: takes precedence over Device when non-empty Dictionary string // wordlist file for Enum; "" = built-in list Predict bool // enable AI subdomain prediction WildcardFilter string // "none" | "basic" | "advanced" (default "none") diff --git a/docs/best-practices.md b/docs/best-practices.md index 5c90940..c8c21ee 100644 --- a/docs/best-practices.md +++ b/docs/best-practices.md @@ -131,7 +131,35 @@ For scans with millions of domains (e.g., combined wordlists): --- -## SDK usage tips +## Multi-NIC for higher throughput + +Repeat `--interface` to send from multiple adapters simultaneously. +Each interface spawns an independent goroutine; they share the same domain +channel, naturally load-balancing queries: + +```bash +sudo ksubdomain enum -d example.com --interface eth0 --interface eth1 -b 20m +``` + +Equivalently in the SDK: + +```go +scanner := sdk.NewScanner(&sdk.Config{ + Bandwidth: "20m", + Devices: []string{"eth0", "eth1"}, +}) +``` + +--- + +## Dynamic timeout + +The engine automatically adjusts the per-query timeout using an RTT +sliding window (EWMA α=0.125, β=0.25, RFC 6298), bounded between 1 s +and 10 s. You rarely need to override `--timeout`; only do so if your +network has unusually high or low latency. + +--- ### Cancellation diff --git a/docs/faq.md b/docs/faq.md index 1910199..32c03e7 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -172,9 +172,30 @@ brew install libpcap Use the `build.sh` script which sets the correct `CGO_ENABLED` and `CC` for each target platform. +### How do I use multiple network interfaces? + +Repeat `--interface` (or `--eth`) to send from multiple NICs simultaneously: + +```bash +sudo ksubdomain enum -d example.com --interface eth0 --interface eth1 -b 20m +``` + --- -## Exit codes +## Flags renamed in v2.4+ + +Several flag names changed for clarity. Old aliases still work for backward-compat. + +| Old flag | New recommended flag | +|----------|---------------------| +| `--band` / `-b` | `--bandwidth` | +| `--eth` | `--interface` | +| `--wild-filter-mode` | `--wildcard-filter` | +| `--not-print` / `--np` | `--quiet` / `-q` | +| `--output-type` | `--format` | +| `-s` | `--silent` (also `-s`) | + +--- | Code | Meaning | |---|---| diff --git a/docs/quickstart.md b/docs/quickstart.md index 0fd528a..fcbc17c 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -4,9 +4,10 @@ | Requirement | Notes | |---|---| -| OS | Linux, macOS, Windows (WSL2 recommended) | +| OS | Linux, macOS, Windows | | Privileges | **root** or `CAP_NET_RAW` — raw packet capture requires elevated access | | libpcap / npcap | Linux: `libpcap-dev`; macOS: built-in; Windows: [Npcap](https://npcap.com) | +| Go (build only) | 1.23+ | --- @@ -14,6 +15,8 @@ ### Download pre-built binary (recommended) +Visit [Releases](https://github.com/boy-hack/ksubdomain/releases) and download the binary for your platform. + ```bash # Linux x86_64 curl -L https://github.com/boy-hack/ksubdomain/releases/latest/download/ksubdomain_linux_amd64 \ @@ -26,19 +29,31 @@ chmod +x /usr/local/bin/ksubdomain ```bash git clone https://github.com/boy-hack/ksubdomain.git cd ksubdomain -go build -o ksubdomain ./cmd/ksubdomain -# or use the build script (cross-compile all platforms): -./build.sh +# Inject version via ldflags (recommended) +go build -ldflags "-X github.com/boy-hack/ksubdomain/v2/pkg/core/conf.Version=v2.x.y" \ + -o ksubdomain ./cmd/ksubdomain ``` --- ## Your first scan -### 1 — Enumerate subdomains (built-in wordlist) +### 1 — Check available network interfaces + +```bash +sudo ./ksubdomain device +``` + +### 2 — Test maximum packet rate + +```bash +sudo ./ksubdomain test +``` + +### 3 — Enumerate subdomains (built-in wordlist) ```bash -sudo ksubdomain enum -d example.com +sudo ./ksubdomain enum -d example.com ``` Sample output: @@ -49,93 +64,87 @@ mail.example.com => 93.184.216.50 api.example.com => 93.184.216.51 ``` -### 2 — Enumerate with a custom wordlist +### 4 — Verify a domain list ```bash -sudo ksubdomain enum -d example.com -f /path/to/wordlist.txt +sudo ./ksubdomain verify -f domains.txt -o output.txt ``` -### 3 — Verify a list of known subdomains +--- + +## Common workflows + +### Pipe to httpx for HTTP probing ```bash -cat domains.txt | sudo ksubdomain verify -# or -sudo ksubdomain verify -f domains.txt +sudo ./ksubdomain enum -d example.com --only-domain --silent | httpx -silent ``` -### 4 — Pipe into httpx +### Save as JSONL for streaming processing ```bash -sudo ksubdomain enum -d example.com --silent --only-domain | httpx -silent +sudo ./ksubdomain enum -d example.com --format jsonl -o result.jsonl ``` -`--only-domain` prints one clean domain per line with no extra characters, -making the output safe to pipe into any line-oriented tool. +### Batch enumeration of multiple root domains ---- +```bash +sudo ./ksubdomain enum --domain-list roots.txt -b 10m --format jsonl -o all.jsonl +``` -## Common flags - -| Flag | Short | Description | -|---|---|---| -| `--domain` | `-d` | Target domain (enum mode) | -| `--file` | `-f` | Input file (wordlist for enum, domain list for verify) | -| `--band` | `-b` | Bandwidth limit, e.g. `5m`, `100m` (default: `5m`) | -| `--retry` | `-r` | Max retries per domain (default: `3`) | -| `--resolvers` | | Custom DNS resolver IPs, comma-separated | -| `--output` | `-o` | Output file path | -| `--output-type` | `--ot` | Output format: `txt`, `json`, `csv`, `jsonl` | -| `--only-domain` | `--od` | Print only the domain name, no record values | -| `--silent` | | Suppress progress bar and informational logs | -| `--wild-filter-mode` | | Wildcard filter: `none` (default), `basic`, `advanced` | -| `--predict` | | Enable AI subdomain prediction | +### Predict + advanced wildcard filter ---- +```bash +sudo ./ksubdomain enum -d example.com --predict --wildcard-filter advanced -o result.txt +``` -## Output formats +### Multi-NIC parallel sending ```bash -# Plain text (default) -sudo ksubdomain enum -d example.com -o results.txt +sudo ./ksubdomain enum -d example.com --interface eth0 --interface eth1 -b 20m +``` + +--- -# JSON Lines (jq-compatible, one object per line) -sudo ksubdomain enum -d example.com -o results.jsonl --ot jsonl +## Platform notes -# Parse with jq -jq '.domain' results.jsonl -jq 'select(.type=="A") | .records[]' results.jsonl -``` +### Linux -See [docs/OUTPUT_FORMATS.md](./OUTPUT_FORMATS.md) for full format details. +```bash +# Grant CAP_NET_RAW instead of running as root (optional) +sudo setcap cap_net_raw+ep ./ksubdomain ---- +./ksubdomain enum -d example.com +``` -## Bandwidth tuning +### macOS -ksubdomain operates at the raw packet level and sends DNS queries at the -rate you specify. Start conservatively and increase: +macOS uses BPF with smaller default buffers. Keep bandwidth conservative: ```bash -# ~60 Mbit bandwidth cap (safe for most home connections) -sudo ksubdomain enum -d example.com -b 60m +sudo ./ksubdomain enum -d example.com -b 5m +``` -# Max out a Gigabit interface -sudo ksubdomain enum -d example.com -b 1000m +### WSL / WSL2 + +```bash +./ksubdomain enum -d example.com --interface eth0 ``` -See [docs/best-practices.md](./best-practices.md) for bandwidth and resolver advice. +### Windows + +1. Install [Npcap](https://npcap.com/) (WinPcap is not supported) +2. Run as Administrator + +```bat +.\ksubdomain.exe enum -d example.com +``` --- -## Troubleshooting quick reference +## Next steps -| Symptom | Fix | -|---|---| -| `permission denied` | Run with `sudo` or grant `CAP_NET_RAW` | -| `network device not found` | Specify interface with `--eth `; list with `ip link show` | -| `network device not active` | Bring interface up: `sudo ip link set up` | -| No results, no errors | Try `--wild-filter-mode none`; target domain may have wildcard DNS | -| macOS `ENOBUFS` | Lower bandwidth: `-b 5m` | -| WSL2 wrong interface | Add `--eth eth0` | - -For more, see [docs/faq.md](./faq.md). +- [API Reference](./api.md) +- [Best Practices](./best-practices.md) +- [FAQ](./faq.md) +- [SDK README](../sdk/README.md) diff --git a/readme.md b/readme.md index ed736c5..02acb3d 100755 --- a/readme.md +++ b/readme.md @@ -2,194 +2,265 @@ [![Release](https://img.shields.io/github/release/boy-hack/ksubdomain.svg)](https://github.com/boy-hack/ksubdomain/releases) [![Go Report Card](https://goreportcard.com/badge/github.com/boy-hack/ksubdomain)](https://goreportcard.com/report/github.com/boy-hack/ksubdomain) [![License](https://img.shields.io/github/license/boy-hack/ksubdomain)](https://github.com/boy-hack/ksubdomain/blob/main/LICENSE) -**KSubdomain 是一款基于无状态技术的子域名爆破工具,带来前所未有的扫描速度和极低的内存占用。** 告别传统工具的效率瓶颈,体验闪电般的 DNS 查询,同时拥有可靠的状态表重发机制,确保结果的完整性。 KSubdomain 支持 Windows、Linux 和 macOS,是进行大规模DNS资产探测的理想选择。 +[English](./README_EN.md) | **中文** -![](image.gif) +**KSubdomain 是一款基于无状态技术的子域名爆破工具,带来前所未有的扫描速度和极低的内存占用。** 采用原始套接字直接操作网络适配器,绕过系统内核协议栈,配合可靠的状态表重发机制,确保结果完整性。支持 Windows、Linux、macOS,是大规模 DNS 资产探测的首选工具。 ## 🚀 核心优势 -* **闪电般的速度:** 采用无状态扫描技术,直接操作网络适配器进行原始套接字发包,绕过系统内核的网络协议栈,实现惊人的发包速率。通过 `test` 命令可探测本地网卡的最大发送速度。 -* **极低的资源消耗:** 创新的内存管理机制,包括对象池和全局内存池,显著降低内存分配和 GC 压力,即使处理海量域名也能保持低内存占用。 -* **无状态设计:** 类似 Masscan 的无状态扫描,不从系统维护状态表,自建轻量状态表,从根本上解决了传统扫描工具的内存瓶颈和性能限制,以及解决了无状态扫描漏包问题。 -* **可靠的重发:** 内建智能重发机制,有效应对网络抖动和丢包,确保结果的准确性和完整性。 -* **跨平台支持:** 完美兼容 Windows, Linux, macOS。 -* **易于使用:** 简洁的命令行接口,提供验证 (verify) 和枚举 (enum) 两种模式,并内置常用字典。 - -## ⚡ 性能亮点 - -KSubdomain 在速度和效率上远超同类工具。以下是在 4 核 CPU、5M 带宽网络环境下,使用 10 万字典进行的对比测试: - -| 工具 | 扫描模式 | 发包方式 | 命令 | 耗时 | 成功个数 | 备注 | -| ------------ | -------- | ------------ | -------------------------------------------------------------------------- | -------------- | -------- | ------------------------- | -| **KSubdomain** | 验证 | pcap 网卡发包 | `time ./ksubdomain v -b 5m -f d2.txt -o k.txt -r dns.txt --retry 3 --np` | **~30 秒** | 1397 | `--np` 关闭实时打印 | -| massdns | 验证 | pcap/socket | `time ./massdns -r dns.txt -t A -w m.txt d2.txt --root -o L` | ~3 分 29 秒 | 1396 | | -| dnsx | 验证 | socket | `time ./dnsx -a -o d.txt -r dns.txt -l d2.txt -retry 3 -t 5000` | ~5 分 26 秒 | 1396 | `-t 5000` 设置 5000 并发 | - -**结论:** KSubdomain 的速度是 massdns 的 **7 倍**,是 dnsx 的 **10 倍** 以上! -## 🛠️ 技术革新 (v2.0) - -KSubdomain 2.0 版本引入了多项底层优化,进一步压榨性能潜力: - -1. **状态表优化:** - * **分片锁 (Sharded Lock):** 替代全局锁,大幅减少锁竞争,提高并发写入效率。 - * **高效哈希:** 优化键值存储,均匀分布域名,提升查找速度。 -2. **发包机制优化:** - * **对象池:** 复用 DNS 包结构体,减少内存分配和 GC 开销。 - * **模板缓存:** 为相同 DNS 服务器复用以太网/IP/UDP 层数据,减少重复构建开销。 - * **并行发送:** 多协程并行发包,充分利用多核 CPU 性能。 - * **批量处理:** 批量发送域名请求,减少系统调用和上下文切换。 -3. **接收机制优化:** - * **对象池:** 复用解析器和缓冲区,降低内存消耗。 - * **并行处理管道:** 接收 → 解析 → 处理三阶段并行,提高处理流水线效率。 - * **缓冲区优化:** 增加内部 Channel 缓冲区大小,避免处理阻塞。 - * **高效过滤:** 优化 BPF 过滤规则和包处理逻辑,快速丢弃无效数据包。 -4. **内存管理优化:** - * **全局内存池:** 引入 `sync.Pool` 管理常用数据结构,减少内存分配和碎片。 - * **结构复用:** 复用 DNS 查询结构和序列化缓冲区。 -5. **架构与并发优化:** - * **动态并发:** 根据 CPU 核心数自动调整协程数量。 - * **高效随机数:** 使用性能更优的随机数生成器。 - * **自适应速率:** 根据网络状况和系统负载动态调整发包速率。 - * **批量加载:** 批量加载和处理域名,降低单个域名处理的固定开销。 +- **闪电般的速度:** 无状态扫描直接操作网卡,速度是 massdns 的 **7 倍**,dnsx 的 **10 倍**以上 +- **极低资源消耗:** 对象池 + 全局内存池,海量域名处理依然低内存占用 +- **动态超时自适应:** 基于 TCP RFC 6298 RTT 滑动均值(EWMA)动态调整超时,无需手动调参 +- **多网卡并发发包:** 支持 `--interface` 重复指定多张网卡,叠加带宽、利用多个出口 IP +- **流式 SDK:** 提供 `EnumStream`/`VerifyStream` 回调 API,实时处理结果 +- **跨平台支持:** Windows、Linux、macOS 完美兼容 + +## ⚡ 性能对比 + +4 核 CPU、5M 带宽,10 万字典测试: + +| 工具 | 方式 | 耗时 | 成功数 | +|------|------|------|--------| +| **KSubdomain** | pcap 网卡发包 | **~30 秒** | 1397 | +| massdns | pcap/socket | ~3 分 29 秒 | 1396 | +| dnsx | socket | ~5 分 26 秒 | 1396 | ## 📦 安装 -1. **下载预编译二进制文件:** 前往 [Releases](https://github.com/boy-hack/ksubdomain/releases) 页面下载对应系统的最新版本。 -2. **安装 `libpcap` 依赖:** - * **Windows:** 下载并安装 [Npcap](https://npcap.com/) 驱动 (WinPcap 可能无效)。 - * **Linux:** 已静态编译打包 `libpcap`,通常无需额外操作。若遇问题,请尝试安装 `libpcap-dev` 或 `libcap-devel` 包。 - * **macOS:** 系统自带 `libpcap`,无需安装。 -3. **赋予执行权限 (Linux/macOS):** `chmod +x ksubdomain` -4. **运行!** +### 下载预编译二进制 -### 源码编译 (可选) +前往 [Releases](https://github.com/boy-hack/ksubdomain/releases) 下载对应系统的最新版本。 -确保您已安装 Go 1.23 版本和 `libpcap` 环境。 +**依赖安装:** +- **Windows:** 安装 [Npcap](https://npcap.com/) 驱动 +- **Linux:** 已静态编译,通常无需操作;如有问题安装 `libpcap-dev` +- **macOS:** 系统自带 libpcap,无需安装 + +### 源码编译 ```bash -go install -v github.com/boy-hack/ksubdomain/v2/cmd/ksubdomain@latest -# 二进制文件通常位于 $GOPATH/bin 或 $HOME/go/bin +git clone https://github.com/boy-hack/ksubdomain.git +cd ksubdomain +# 建议通过 ldflags 注入版本号 +go build -ldflags "-X github.com/boy-hack/ksubdomain/v2/pkg/core/conf.Version=v2.x.y" \ + -o ksubdomain ./cmd/ksubdomain ``` -## 📖 使用说明 +或使用 `go install`: ```bash -KSubdomain - 极速无状态子域名爆破工具 +go install github.com/boy-hack/ksubdomain/v2/cmd/ksubdomain@latest +``` -用法: - ksubdomain [全局选项] 命令 [命令选项] [参数...] +> **注意:** 需要 root / `CAP_NET_RAW` 权限运行(原始套接字抓包) -版本: - 查看版本信息: ksubdomain --version +## 📖 使用说明 + +``` +用法: + ksubdomain [全局选项] <命令> [命令选项] 命令: - enum, e 枚举模式: 提供主域名进行爆破 - verify, v 验证模式: 提供域名列表进行验证 - test 测试本地网卡最大发包速度 - help, h 显示命令列表或某个命令的帮助 + enum, e 枚举模式:对主域名进行子域名爆破 + verify, v 验证模式:验证域名列表是否解析 + test 测试本地网卡最大发包速度 + device 查看可用网卡信息 + help, h 显示帮助 全局选项: - --help, -h 显示帮助 (默认: false) - --version, -v 打印版本信息 (默认: false) + --help, -h 显示帮助 + --version, -v 打印版本信息 ``` -### 验证模式 (Verify) +### 验证模式(verify) -验证模式用于快速检查提供的域名列表的存活状态。 +验证提供的域名列表是否有 DNS 解析记录。 + +``` +选项: + --domain, -d 指定域名(可重复) + --filename, -f 从文件读取域名列表 + --stdin 从标准输入读取 + --bandwidth, -b 带宽限制,如 5m、10m、100m(默认 3m) + --resolvers, -r 指定 DNS 服务器(默认使用内置) + --output, -o 输出文件路径 + --format, -f 输出格式:txt(默认)、json、csv、jsonl + --silent, -s 安静模式,仅输出域名(不显示 banner 和日志) + --only-domain, --od 只输出域名,不显示 IP/记录 + --retry 重试次数,-1 表示无限重试(默认 3) + --timeout 单次查询超时秒数(默认 6) + --interface, -e 指定网卡名(可重复,多网卡并发) + --wildcard-filter 泛解析过滤:none(默认)、basic、advanced + --predict 启用子域名预测模式 + --quiet, -q 不打印结果到屏幕(仅写文件) + --color, -c 彩色输出 +``` ```bash -./ksubdomain verify -h # 查看验证模式帮助,可缩写 ksubdomain v - -USAGE: - ksubdomain verify [command options] [arguments...] - -OPTIONS: - --filename value, -f value 验证域名的文件路径 - --domain value, -d value 域名 - --band value, -b value 宽带的下行速度,可以5M,5K,5G (default: "3m") - --resolvers value, -r value dns服务器,默认会使用内置dns - --output value, -o value 输出文件名 - --output-type value, --oy value 输出文件类型: json, txt, csv (default: "txt") - --silent 使用后屏幕将仅输出域名 (default: false) - --retry value 重试次数,当为-1时将一直重试 (default: 3) - --timeout value 超时时间 (default: 6) - --stdin 接受stdin输入 (default: false) - --not-print, --np 不打印域名结果 (default: false) - --eth value, -e value 指定网卡名称 - --wild-filter-mode value 泛解析过滤模式[从最终结果过滤泛解析域名]: basic(基础), advanced(高级), none(不过滤ne") - --predict 启用预测域名模式 (default: false) - --help, -h show help (default: false) - -# 示例: -# 验证多个域名解析 -./ksubdomain v -d xx1.example.com -d xx2example.com - -# 从文件读取域名进行验证,保存为 output.txt +# 验证单个/多个域名 +./ksubdomain v -d www.example.com -d mail.example.com + +# 从文件读取,保存为 txt ./ksubdomain v -f domains.txt -o output.txt -# 从标准输入读取域名,带宽限制为 10M -cat domains.txt | ./ksubdomain v --stdin -b 10M +# 管道输入,带宽 10M,静默模式对接下游工具 +cat domains.txt | ./ksubdomain v --stdin -b 10m -s --od | httpx -silent -# 启用预测模式,泛解析过滤,保存为csv -./ksubdomain v -f domains.txt --predict --wild-filter-mode advanced --oy csv -o output.csv +# 高级泛解析过滤,输出 JSONL +./ksubdomain v -f domains.txt --wildcard-filter advanced --format jsonl -o output.jsonl + +# 多网卡并发(叠加带宽) +./ksubdomain v -f domains.txt --interface eth0 --interface eth1 ``` -### 枚举模式 (Enum) +### 枚举模式(enum) + +基于字典和预测算法爆破指定域名下的子域名。 -枚举模式基于字典和预测算法爆破指定域名下的子域名。 +``` +选项: + --domain, -d 目标主域名(可重复) + --domain-list, --ds 批量主域名文件 + --filename, -f 字典文件路径(默认使用内置字典) + --use-ns-records 读取域名 NS 记录并加入 DNS 解析器 + (其余选项同 verify 模式) +``` ```bash -./ksubdomain enum -h # 查看枚举模式帮助,可简写 ksubdomain e - -USAGE: - ksubdomain enum [command options] [arguments...] - -OPTIONS: - --domain value, -d value 域名 - --band value, -b value 宽带的下行速度,可以5M,5K,5G (default: "3m") - --resolvers value, -r value dns服务器,默认会使用内置dns - --output value, -o value 输出文件名 - --output-type value, --oy value 输出文件类型: json, txt, csv (default: "txt") - --silent 使用后屏幕将仅输出域名 (default: false) - --retry value 重试次数,当为-1时将一直重试 (default: 3) - --timeout value 超时时间 (default: 6) - --stdin 接受stdin输入 (default: false) - --not-print, --np 不打印域名结果 (default: false) - --eth value, -e value 指定网卡名称 - --wild-filter-mode value 泛解析过滤模式[从最终结果过滤泛解析域名]: basic(基础), advanced(高级), none(不过滤) (default: "none") - --predict 启用预测域名模式 (default: false) - --filename value, -f value 字典路径 - --ns 读取域名ns记录并加入到ns解析器中 (default: false) - --help, -h show help (default: false) - -# 示例: -# 枚举多个域名 -./ksubdomain e -d example.com -d hacker.com - -# 从文件读取字典枚举,保存为 output.txt -./ksubdomain e -f sub.dict -o output.txt - -# 从标准输入读取域名,带宽限制为 10M -cat domains.txt | ./ksubdomain e --stdin -b 10M - -# 启用预测模式枚举域名,泛解析过滤,保存为csv -./ksubdomain e -d example.com --predict --wild-filter-mode advanced --oy csv -o output.csv -``` - -## ✨ 特性与技巧 - -* **带宽自动适配:** 只需使用 `-b` 参数指定你的公网下行带宽 (如 `-b 10m`), KSubdomain 会自动优化发包速率。 -* **测试最大速率:** 运行 `./ksubdomain test` 测试当前环境的最大理论发包速率。 -* **自动网卡检测:** KSubdomain 会自动检测可用网卡。 -* **进度显示:** 实时进度条显示 成功数 / 发送数 / 队列长度 / 接收数 / 失败数 / 已耗时。 -* **参数调优:** 根据网络质量和目标域名数量,调整 `--retry` 和 `--timeout` 参数以获得最佳效果。当 `--retry` 为 -1 时,将无限重试直至所有请求成功或超时。 -* **多种输出格式:** 支持 `txt` (实时输出), `json` (完成后输出), `csv` (完成后输出)。通过 `-o` 指定文件名后缀即可 (如 `result.json`)。 -* **环境变量配置:** - * `KSubdomainConfig`: 指定配置文件的路径。 +# 枚举单个域名(使用内置字典) +./ksubdomain e -d example.com + +# 指定字典文件 +./ksubdomain e -d example.com -f subdomains.txt -o result.txt + +# 启用预测模式 + 高级泛解析过滤 +./ksubdomain e -d example.com --predict --wildcard-filter advanced + +# 批量枚举多个主域名 +./ksubdomain e --domain-list roots.txt -b 10m --format jsonl -o result.jsonl + +# 管道对接 httpx +./ksubdomain e -d example.com --od -s | httpx -silent +``` + +### 测试/设备命令 + +```bash +# 测试网卡最大发包速度 +./ksubdomain test + +# 查看可用网卡 +./ksubdomain device +``` + +## ✨ 功能特性 + +### 动态超时自适应 +无需手动设置 `--timeout`,引擎自动基于实测 RTT(指数加权移动平均,EWMA α=0.125)动态调整超时上下界,有效减少漏报同时避免不必要等待。 + +### 多网卡并发 +重复指定 `--interface` 即可使用多张网卡同时发包,多个 goroutine 共享同一 `domainChan` 竞争消费,天然实现负载均衡: + +```bash +./ksubdomain e -d example.com --interface eth0 --interface eth1 -b 20m +``` + +### 预测模式 +启用 `--predict` 后,工具会根据已发现的子域名预测可能存在的相关子域名,提升发现率。 + +### 泛解析过滤 +- `none`:不过滤(默认) +- `basic`:基础过滤,剔除明显泛解析 IP +- `advanced`:高级过滤,综合多维度判断 + +### 输出格式 + +| 格式 | 说明 | 适用场景 | +|------|------|---------| +| `txt` | `域名 => 记录`,实时输出 | 默认,人工查看 | +| `json` | 完整 JSON,完成后写入 | 程序解析 | +| `csv` | CSV 表格,完成后写入 | 数据分析 | +| `jsonl` | 每行一条 JSON,实时输出 | 流式处理、管道对接 | + +## 🔗 工具联动示例 + +```bash +# 联动 httpx 探测存活 Web +./ksubdomain e -d example.com -s --od | httpx -silent + +# 联动 nuclei 漏洞扫描 +./ksubdomain e -d example.com -s --od | nuclei -l /dev/stdin + +# JSONL 流式处理,只取 A 记录 +./ksubdomain e -d example.com --format jsonl | jq -r 'select(.type=="A") | .domain' + +# 多域名批量 + 去重 +./ksubdomain e --domain-list roots.txt -s --od | sort -u > all_subs.txt +``` + +## 🌐 平台注意事项 + +**macOS:** +```bash +# BPF 缓冲区较小,建议限制带宽 +sudo ./ksubdomain e -d example.com -b 5m +``` + +**WSL/WSL2:** +```bash +# 通常使用 eth0 +./ksubdomain e -d example.com --interface eth0 +``` + +**Windows:** +```bash +# 需先安装 Npcap,以管理员权限运行 +.\ksubdomain.exe enum -d example.com +``` + +## 🧩 Go SDK + +```bash +go get github.com/boy-hack/ksubdomain/v2/sdk +``` + +```go +import "github.com/boy-hack/ksubdomain/v2/sdk" + +scanner := sdk.NewScanner(&sdk.Config{ + Bandwidth: "5m", + Retry: 3, + WildcardFilter: "advanced", +}) + +// 阻塞式,收集所有结果 +results, err := scanner.Enum("example.com") + +// 流式回调,实时处理 +err = scanner.EnumStream(ctx, "example.com", func(r sdk.Result) { + fmt.Printf("%s => %v\n", r.Domain, r.Records) +}) +``` + +详见 [SDK 文档](./sdk/README.md) 和 [API 文档](./docs/api.md)。 + +## 🧪 测试 + +```bash +# 单元测试(无需 root) +go test ./pkg/... + +# 回归测试(需要 root + 网络) +go build -o ksubdomain ./cmd/ksubdomain +sudo go test ./test/regression/... -tags regression -v -timeout 120s +``` ## 💡 参考 -* 原 KSubdomain 项目: [https://github.com/knownsec/ksubdomain](https://github.com/knownsec/ksubdomain) -* 从 Masscan, Zmap 源码分析到开发实践: [https://paper.seebug.org/1052/](https://paper.seebug.org/1052/) -* KSubdomain 无状态域名爆破工具介绍: [https://paper.seebug.org/1325/](https://paper.seebug.org/1325/) -* KSubdomain 与 massdns 的对比分析: [微信公众号文章链接](https://mp.weixin.qq.com/s?__biz=MzU2NzcwNTY3Mg==&mid=2247484471&idx=1&sn=322d5db2d11363cd2392d7bd29c679f1&chksm=fc986d10cbefe406f4bda22f62a16f08c71f31c241024fc82ecbb8e41c9c7188cfbd71276b81&token=76024279&lang=zh_CN#rd) +- 原 KSubdomain 项目:[knownsec/ksubdomain](https://github.com/knownsec/ksubdomain) +- 无状态扫描原理:[paper.seebug.org/1052](https://paper.seebug.org/1052/) +- KSubdomain 介绍:[paper.seebug.org/1325](https://paper.seebug.org/1325/) + +## 📜 License + +MIT License — 详见 [LICENSE](LICENSE) diff --git a/sdk/README.md b/sdk/README.md index 9f8a234..471a255 100644 --- a/sdk/README.md +++ b/sdk/README.md @@ -1,6 +1,6 @@ # KSubdomain Go SDK -Simple and powerful Go SDK for integrating ksubdomain into your applications. +Simple, powerful Go SDK for integrating ksubdomain subdomain scanning into your applications. ## 📦 Installation @@ -8,14 +8,17 @@ Simple and powerful Go SDK for integrating ksubdomain into your applications. go get github.com/boy-hack/ksubdomain/v2/sdk ``` +> **Note:** Requires root / `CAP_NET_RAW` privilege (raw packet capture via pcap). + ## 🚀 Quick Start -### Basic Usage +### Basic Enumeration (blocking) ```go package main import ( + "errors" "fmt" "log" @@ -23,456 +26,194 @@ import ( ) func main() { - // Create scanner with default config scanner := sdk.NewScanner(sdk.DefaultConfig) - // Enumerate subdomains results, err := scanner.Enum("example.com") if err != nil { - log.Fatal(err) + switch { + case errors.Is(err, sdk.ErrPermissionDenied): + log.Fatal("run with sudo or grant CAP_NET_RAW") + case errors.Is(err, sdk.ErrDeviceNotFound): + log.Fatal("network device not found") + default: + log.Fatal(err) + } } - // Process results - for _, result := range results { - fmt.Printf("%s => %v\n", result.Domain, result.Records) + for _, r := range results { + fmt.Printf("%s [%s] %v\n", r.Domain, r.Type, r.Records) } } ``` -### Custom Configuration +### Streaming (real-time callback) ```go -scanner := sdk.NewScanner(&sdk.Config{ - Bandwidth: "10m", // 10M bandwidth - Retry: 5, // Retry 5 times - Timeout: 10, // 10 seconds timeout - Resolvers: []string{"8.8.8.8", "1.1.1.1"}, - Predict: true, // Enable prediction - WildcardFilter: "advanced", // Advanced wildcard filtering - Silent: true, // Silent mode -}) +ctx, cancel := context.WithCancel(context.Background()) +defer cancel() -results, err := scanner.Enum("example.com") +err := scanner.EnumStream(ctx, "example.com", func(r sdk.Result) { + fmt.Printf("%s => %v\n", r.Domain, r.Records) +}) ``` ### Verify Mode ```go -domains := []string{ - "www.example.com", - "mail.example.com", - "api.example.com", -} +domains := []string{"www.example.com", "mail.example.com", "api.example.com"} results, err := scanner.Verify(domains) -if err != nil { - log.Fatal(err) +for _, r := range results { + fmt.Printf("✓ %s [%s] %v\n", r.Domain, r.Type, r.Records) } -for _, result := range results { - fmt.Printf("✓ %s is alive\n", result.Domain) -} +// Streaming verify +err = scanner.VerifyStream(ctx, domains, func(r sdk.Result) { + fmt.Printf("%s is alive\n", r.Domain) +}) ``` -### With Context (Timeout/Cancellation) +## ⚙️ Configuration ```go -import "context" +scanner := sdk.NewScanner(&sdk.Config{ + // Bandwidth cap, e.g. "5m", "10m", "100m" (default: "5m") + Bandwidth: "10m", -// With timeout -ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) -defer cancel() + // Retry count per domain; -1 = infinite (default: 3) + Retry: 5, -results, err := scanner.EnumWithContext(ctx, "example.com") + // DNS resolvers; nil = built-in defaults + Resolvers: []string{"8.8.8.8", "1.1.1.1"}, -// With cancellation -ctx, cancel := context.WithCancel(context.Background()) + // Single network interface; "" = auto-detect + Device: "", -go func() { - time.Sleep(10 * time.Second) - cancel() // Stop scanning after 10 seconds -}() + // Multiple interfaces for parallel sending (takes precedence over Device) + Devices: []string{"eth0", "eth1"}, -results, err := scanner.EnumWithContext(ctx, "example.com") -``` + // Wordlist file for Enum; "" = built-in list + Dictionary: "/path/to/subdomains.txt", -## 📚 API Reference + // Enable AI-powered subdomain prediction + Predict: true, -### Config + // Wildcard filter: "none" (default), "basic", "advanced" + WildcardFilter: "advanced", -Configuration for the scanner. + // Suppress progress output + Silent: true, -```go -type Config struct { - Bandwidth string // Bandwidth (e.g., "5m", "10m", "100m") - Retry int // Retry count (-1 for infinite) - Timeout int // Timeout in seconds - Resolvers []string // DNS resolvers (nil for default) - Device string // Network adapter (empty for auto-detect) - Dictionary string // Dictionary file path - Predict bool // Enable prediction mode - WildcardFilter string // Wildcard filter: "none", "basic", "advanced" - Silent bool // Silent mode (no progress output) -} + // Inject custom output sinks (implement outputter.Output) + ExtraWriters: []outputter.Output{myWriter}, +}) ``` -**DefaultConfig:** +### DefaultConfig + ```go -var DefaultConfig = &Config{ +var DefaultConfig = &sdk.Config{ Bandwidth: "5m", Retry: 3, - Timeout: 6, - Resolvers: nil, - Device: "", - Dictionary: "", - Predict: false, WildcardFilter: "none", - Silent: false, } ``` -### Scanner +> **Timeout is not configurable.** The scanner uses a dynamic RTT-based timeout +> (RFC 6298 EWMA, α=0.125, β=0.25) bounded between 1 s and 10 s. +> This eliminates the need for manual tuning. -Main scanner interface. +## 📐 API Reference -#### NewScanner - -```go -func NewScanner(config *Config) *Scanner -``` - -Creates a new scanner with given configuration. If `config` is nil, uses `DefaultConfig`. - -#### Enum - -```go -func (s *Scanner) Enum(domain string) ([]Result, error) -``` - -Enumerates subdomains for the given domain. - -#### EnumWithContext - -```go -func (s *Scanner) EnumWithContext(ctx context.Context, domain string) ([]Result, error) -``` - -Enumerates subdomains with context support (timeout, cancellation). - -#### Verify - -```go -func (s *Scanner) Verify(domains []string) ([]Result, error) -``` - -Verifies a list of domains. - -#### VerifyWithContext - -```go -func (s *Scanner) VerifyWithContext(ctx context.Context, domains []string) ([]Result, error) -``` - -Verifies domains with context support. - -### Result - -Scan result structure. +### Types ```go +// Config holds scanner settings. +type Config struct { + Bandwidth string + Retry int + Resolvers []string + Device string // single NIC (backward-compat) + Devices []string // multi-NIC parallel sending + Dictionary string + Predict bool + WildcardFilter string + Silent bool + ExtraWriters []outputter.Output // custom sinks +} + +// Result is a single resolved subdomain. type Result struct { - Domain string // Subdomain - Type string // Record type (A, CNAME, NS, PTR, etc.) - Records []string // Record values -} -``` - -## 📖 Examples - -### Example 1: Simple Enumeration - -```go -package main - -import ( - "fmt" - "log" - "github.com/boy-hack/ksubdomain/v2/sdk" -) - -func main() { - scanner := sdk.NewScanner(nil) // Use default config - - results, err := scanner.Enum("example.com") - if err != nil { - log.Fatal(err) - } - - fmt.Printf("Found %d subdomains:\n", len(results)) - for _, r := range results { - fmt.Printf(" %s (%s)\n", r.Domain, r.Type) - } -} -``` - -### Example 2: Batch Verification - -```go -package main - -import ( - "bufio" - "fmt" - "log" - "os" - - "github.com/boy-hack/ksubdomain/v2/sdk" -) - -func main() { - // Read domains from file - file, _ := os.Open("domains.txt") - defer file.Close() - - var domains []string - scanner := bufio.NewScanner(file) - for scanner.Scan() { - domains = append(domains, scanner.Text()) - } - - // Verify - ksubScanner := sdk.NewScanner(&sdk.Config{ - Bandwidth: "10m", - Retry: 5, - }) - - results, err := ksubScanner.Verify(domains) - if err != nil { - log.Fatal(err) - } - - // Save results - outFile, _ := os.Create("alive.txt") - defer outFile.Close() - - for _, r := range results { - fmt.Fprintf(outFile, "%s => %s\n", r.Domain, r.Records[0]) - } + Domain string // e.g. "www.example.com" + Type string // "A", "CNAME", "NS", "PTR", "TXT", "AAAA" + Records []string // resolved values } ``` -### Example 3: High-Speed Enumeration - -```go -package main - -import ( - "fmt" - "log" - "github.com/boy-hack/ksubdomain/v2/sdk" -) +### Methods -func main() { - // High-speed configuration - scanner := sdk.NewScanner(&sdk.Config{ - Bandwidth: "20m", // High bandwidth - Retry: 1, // Fast mode: fewer retries - Timeout: 3, // Short timeout - Predict: true, // Enable prediction - WildcardFilter: "advanced", // Advanced filtering - Silent: true, // No progress output - }) - - results, err := scanner.Enum("example.com") - if err != nil { - log.Fatal(err) - } - - fmt.Printf("High-speed scan found %d subdomains\n", len(results)) -} -``` +| Method | Signature | Description | +|--------|-----------|-------------| +| `NewScanner` | `(config *Config) *Scanner` | Create scanner; nil uses DefaultConfig | +| `Enum` | `(domain string) ([]Result, error)` | Blocking subdomain enumeration | +| `EnumWithContext` | `(ctx, domain) ([]Result, error)` | Enum with context (timeout/cancel) | +| `EnumStream` | `(ctx, domain, func(Result)) error` | Streaming enumeration via callback | +| `Verify` | `(domains []string) ([]Result, error)` | Blocking domain verification | +| `VerifyWithContext` | `(ctx, domains) ([]Result, error)` | Verify with context | +| `VerifyStream` | `(ctx, domains, func(Result)) error` | Streaming verification via callback | -### Example 4: With Context and Timeout +### Sentinel Errors ```go -package main - -import ( - "context" - "fmt" - "log" - "time" - - "github.com/boy-hack/ksubdomain/v2/sdk" -) - -func main() { - scanner := sdk.NewScanner(sdk.DefaultConfig) - - // Set 30 seconds timeout - ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) - defer cancel() - - results, err := scanner.EnumWithContext(ctx, "example.com") - if err != nil { - if err == context.DeadlineExceeded { - fmt.Println("Scan timeout, partial results:") - } else { - log.Fatal(err) - } - } - - for _, r := range results { - fmt.Printf("%s => %v\n", r.Domain, r.Records) - } -} +sdk.ErrPermissionDenied // CAP_NET_RAW / root required +sdk.ErrDeviceNotFound // no matching network interface +sdk.ErrDeviceNotActive // interface is down +sdk.ErrPcapInit // pcap handle initialisation failed +sdk.ErrDomainChanNil // internal: nil domain channel ``` -### Example 5: Integration with Other Tools +## 🔌 Custom Output Sink (ExtraWriters) ```go -package main +import "github.com/boy-hack/ksubdomain/v2/pkg/runner/result" -import ( - "fmt" - "os/exec" - "strings" - - "github.com/boy-hack/ksubdomain/v2/sdk" -) +type MyWriter struct{} -func main() { - scanner := sdk.NewScanner(nil) - - // 1. Enum subdomains - results, _ := scanner.Enum("example.com") - - // 2. Extract domain names - var domains []string - for _, r := range results { - domains = append(domains, r.Domain) - } - - // 3. Pipe to httpx for HTTP probing - cmd := exec.Command("httpx", "-silent") - cmd.Stdin = strings.NewReader(strings.Join(domains, "\n")) - - output, err := cmd.Output() - if err == nil { - fmt.Printf("Live HTTP services:\n%s", output) - } -} -``` - -## 🎯 Use Cases - -### Web Application Scanning - -```go -// Discover all subdomains, then scan for vulnerabilities -results, _ := scanner.Enum("target.com") -for _, r := range results { - // Run nuclei, sqlmap, etc. on each subdomain - runVulnScan(r.Domain) -} -``` - -### Asset Discovery - -```go -// Monitor subdomain changes -oldResults := loadPreviousResults() -newResults, _ := scanner.Enum("company.com") - -for _, r := range newResults { - if !contains(oldResults, r.Domain) { - alert(fmt.Sprintf("New subdomain found: %s", r.Domain)) - } -} -``` - -### Automated Reconnaissance - -```go -// Periodic scanning with cron -func scanTask() { - scanner := sdk.NewScanner(sdk.DefaultConfig) - results, _ := scanner.Enum("target.com") - - saveToDatabase(results) - generateReport(results) - sendNotification(results) +func (w *MyWriter) WriteDomainResult(r result.Result) error { + fmt.Printf("custom: %s => %v\n", r.Subdomain, r.Answers) + return nil } -``` - -## 🔧 Advanced Usage - -### Custom DNS Resolvers +func (w *MyWriter) Close() error { return nil } -```go scanner := sdk.NewScanner(&sdk.Config{ - Resolvers: []string{ - "8.8.8.8", - "8.8.4.4", - "1.1.1.1", - "1.0.0.1", - }, + Bandwidth: "5m", + ExtraWriters: []outputter.Output{&MyWriter{}}, }) ``` -### Specify Network Adapter +## 🌐 Multi-NIC Parallel Sending ```go scanner := sdk.NewScanner(&sdk.Config{ - Device: "eth0", // or "en0" on macOS + Bandwidth: "20m", + Devices: []string{"eth0", "eth1"}, // two NICs, shared domain channel }) ``` -### Enable Prediction Mode +Each interface spawns an independent `sendCycleForIface` goroutine competing +on the same `domainChan`, naturally load-balancing without extra scheduling logic. -```go -scanner := sdk.NewScanner(&sdk.Config{ - Predict: true, // AI-powered subdomain prediction -}) -``` +## 📋 Examples -## 🐛 Error Handling +See [`examples/simple/main.go`](./examples/simple/main.go) and +[`examples/advanced/main.go`](./examples/advanced/main.go). -```go -results, err := scanner.Enum("example.com") -if err != nil { - switch { - case strings.Contains(err.Error(), "permission denied"): - log.Fatal("Need root permission. Run with sudo.") - - case strings.Contains(err.Error(), "device not found"): - log.Fatal("Network adapter not found. Try --device eth0") - - case strings.Contains(err.Error(), "network"): - log.Fatal("Network error. Check your connection.") - - default: - log.Fatal(err) - } -} -``` - -## 📝 Requirements - -- Go 1.23+ -- libpcap (automatically handled in most cases) -- Root/Administrator permission (for network adapter access) - -## 🔗 Links - -- [GitHub Repository](https://github.com/boy-hack/ksubdomain) -- [Documentation](https://github.com/boy-hack/ksubdomain/tree/main/docs) -- [Issues](https://github.com/boy-hack/ksubdomain/issues) - -## 📄 License - -MIT License. See [LICENSE](../LICENSE) for details. - ---- +```bash +# Run simple example (requires root) +cd sdk/examples/simple +sudo go run main.go -**Happy Scanning! 🚀** +# Run advanced example +cd sdk/examples/advanced +sudo go run main.go +```