Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 36 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,9 @@ firecrawl search "web data python" --categories github
firecrawl search "transformer architecture" --categories research
firecrawl search "machine learning" --categories github,research

# Developer search: GitHub issues, merged PRs, READMEs, and docs
firecrawl search "axum middleware ordering" --categories developer

# Time-based search
firecrawl search "AI announcements" --tbs qdr:d # Past day
firecrawl search "tech news" --tbs qdr:w # Past week
Expand All @@ -308,7 +311,7 @@ firecrawl search "AI data tools"
| ---------------------------- | ------------------------------------------------------------------------------------------- |
| `--limit <n>` | Maximum results (default: 5, max: 100) |
| `--sources <sources>` | Comma-separated: `web`, `images`, `news` (default: web) |
| `--categories <categories>` | Comma-separated: `github`, `research`, `pdf` |
| `--categories <categories>` | Comma-separated: `github`, `research`, `pdf`, `developer` |
| `--tbs <value>` | Time filter: `qdr:h` (hour), `qdr:d` (day), `qdr:w` (week), `qdr:m` (month), `qdr:y` (year) |
| `--location <location>` | Geo-targeting (e.g., "Germany", "San Francisco,California,United States") |
| `--country <code>` | ISO country code (default: US) |
Expand Down Expand Up @@ -337,6 +340,9 @@ firecrawl search "firecrawl documentation" --scrape --scrape-formats markdown --
# Find research papers
firecrawl search "large language models" --categories research --json

# Answer a programming question from issues, merged PRs, READMEs, and docs
firecrawl search "tokio select cancellation safety" --categories developer --json

# Search with location targeting
firecrawl search "best coffee shops" --location "Berlin,Germany" --country DE

Expand All @@ -346,6 +352,35 @@ firecrawl search "AI startups funding" --sources news --tbs qdr:w --limit 15

---

### `developer` - Search developer sources

Search an index built for coding agents: GitHub issues, merged pull requests, repository READMEs, and curated documentation sites. Use it for a programming question: code behaviour, a library or framework, an API contract, an error message, or a known bug.

```bash
firecrawl developer "axum middleware ordering"
```

#### Options

| Option | Description |
| --------------------- | ----------------------------------------- |
| `--limit <n>` | Number of results (default: 20, max: 100) |
| `-o, --output <path>` | Save to file |
| `--json` | Output as compact JSON |
| `--pretty` | Pretty print JSON output |

#### Examples

```bash
# Investigate a known bug
firecrawl developer "tokio spawn_blocking panics thread limit" --limit 10

# Keep the full passages for an agent
firecrawl developer "tokio select cancellation safety" --json -o results.json
```

---

### `feedback` - Send endpoint job feedback

Send concise feedback for a completed v2 `search`, `scrape`, `parse`, or `map`
Expand Down
51 changes: 39 additions & 12 deletions skills/firecrawl-search/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,22 +28,49 @@ firecrawl search "your query" --scrape -o .firecrawl/scraped.json --json

# News from the past day
firecrawl search "your query" --sources news --tbs qdr:d -o .firecrawl/news.json --json

# Programming question: search GitHub issues, merged PRs, READMEs, and docs
firecrawl search "your query" --categories developer -o .firecrawl/developer.json --json
```

## Developer search

`--categories developer` adds an index built for coding agents. It covers GitHub
issues, merged pull requests, repository READMEs, and curated documentation
sites. Use it for a programming question: an error message, an API contract, a
library behaviour, or a known bug.

The hits arrive in their own `data.developer` group beside `data.web`. Each hit
holds `url`, `title`, and `description`, where `description` is the matched
passage. Read the passages with
`jq -r '.data.developer[] | .url, .description' .firecrawl/developer.json`.

The dedicated `firecrawl developer` command searches only that index and keeps
the full matched passages:

```bash
# Developer search only, with full passages
firecrawl developer "your query" --limit 10 -o .firecrawl/developer.json --json
```

Each result holds `id`, `type` (`issue`, `pull_request`, `readme`, `doc`),
`url`, `title`, and `passages`. Read them with
`jq -r '.results[] | .url, .passages[].text' .firecrawl/developer.json`.

## Options

| Option | Description |
| ------------------------------------ | --------------------------------------------- |
| `--limit <n>` | Max number of results |
| `--sources <web,images,news>` | Source types to search |
| `--categories <github,research,pdf>` | Filter by category |
| `--tbs <qdr:h\|d\|w\|m\|y>` | Time-based search filter |
| `--location` | Location for search results |
| `--country <code>` | Country code for search |
| `--scrape` | Also scrape full page content for each result |
| `--scrape-formats` | Formats when scraping (default: markdown) |
| `-o, --output <path>` | Output file path |
| `--json` | Output as JSON |
| Option | Description |
| ---------------------------------------------- | --------------------------------------------- |
| `--limit <n>` | Max number of results |
| `--sources <web,images,news>` | Source types to search |
| `--categories <github,research,pdf,developer>` | Filter by category |
| `--tbs <qdr:h\|d\|w\|m\|y>` | Time-based search filter |
| `--location` | Location for search results |
| `--country <code>` | Country code for search |
| `--scrape` | Also scrape full page content for each result |
| `--scrape-formats` | Formats when scraping (default: markdown) |
| `-o, --output <path>` | Output file path |
| `--json` | Output as JSON |

## Tips

Expand Down
26 changes: 26 additions & 0 deletions src/__tests__/cli-argv.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,32 @@ describe('CLI argv parsing', () => {
const cliPath = resolve(process.cwd(), 'dist/index.js');
const testWithBuiltCli = existsSync(cliPath) ? it : it.skip;

testWithBuiltCli('lists the developer command in root help output', () => {
const result = spawnSync(process.execPath, [cliPath, '--help'], {
cwd: process.cwd(),
encoding: 'utf8',
});

expect(result.status).toBe(0);
expect(result.stdout).toMatch(/^\s*developer\b/m);
});

testWithBuiltCli('parses the developer command and shows its help', () => {
const result = spawnSync(
process.execPath,
[cliPath, 'developer', '--help'],
{
cwd: process.cwd(),
encoding: 'utf8',
}
);

expect(result.status).toBe(0);
expect(result.stdout).toContain('Usage: firecrawl developer');
expect(result.stdout).toContain('--limit');
expect(result.stderr).not.toContain('unknown command');
});

testWithBuiltCli(
'parses subcommands when a wrapper leaves the entry script path in argv',
() => {
Expand Down
177 changes: 177 additions & 0 deletions src/__tests__/commands/developer.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
/**
* Tests for developer command
*/

import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import { handleDeveloperSearchCommand } from '../../commands/developer';
import { getClient } from '../../utils/client';
import { initializeConfig } from '../../utils/config';
import { writeOutput } from '../../utils/output';
import { setupTest, teardownTest } from '../utils/mock-client';

vi.mock('../../utils/output', () => ({ writeOutput: vi.fn() }));

vi.mock('../../utils/client', async () => {
const actual = await vi.importActual('../../utils/client');
return {
...actual,
getClient: vi.fn(),
};
});

describe('handleDeveloperSearchCommand', () => {
let mockHttpGet: ReturnType<typeof vi.fn>;

// Wrap a payload in the axios envelope returned by `client.http.get`.
// Mirrors the `/v2/developer/search` response shape:
// { success, results: [{ id, type, url, title, passages: [{ text }] }] }
const mockDeveloperResponse = (results: any[]) => ({
data: { success: true, results },
});

const sampleResult = {
id: 'issue:tokio-rs/tokio#2309',
type: 'issue',
url: 'https://github.com/tokio-rs/tokio/issues/2309',
title: 'spawn_blocking panics when exceeding the thread limit',
passages: [{ text: 'It will panic if this limit is too low.' }],
};

beforeEach(() => {
setupTest();
initializeConfig({
apiKey: 'test-api-key',
apiUrl: 'https://api.firecrawl.dev',
});

mockHttpGet = vi.fn();
vi.mocked(getClient).mockReturnValue({
http: { get: mockHttpGet },
} as any);
});

afterEach(() => {
teardownTest();
vi.clearAllMocks();
});

describe('API call generation', () => {
it('calls /v2/developer/search with the query and integration tag', async () => {
mockHttpGet.mockResolvedValue(mockDeveloperResponse([sampleResult]));

await handleDeveloperSearchCommand({ query: 'tokio spawn_blocking' });

expect(mockHttpGet).toHaveBeenCalledTimes(1);
expect(mockHttpGet).toHaveBeenCalledWith(
'/v2/developer/search?query=tokio+spawn_blocking&integration=cli'
);
});

it('passes k when a limit is provided', async () => {
mockHttpGet.mockResolvedValue(mockDeveloperResponse([sampleResult]));

await handleDeveloperSearchCommand({
query: 'tokio spawn_blocking',
k: 5,
});

expect(mockHttpGet).toHaveBeenCalledWith(
'/v2/developer/search?query=tokio+spawn_blocking&k=5&integration=cli'
);
});

it('passes apiUrl and apiKey to getClient when provided', async () => {
mockHttpGet.mockResolvedValue(mockDeveloperResponse([]));

await handleDeveloperSearchCommand({
query: 'test',
apiKey: 'other-key',
apiUrl: 'http://localhost:3002',
});

expect(getClient).toHaveBeenCalledWith({
apiKey: 'other-key',
apiUrl: 'http://localhost:3002',
});
});
});

describe('output', () => {
it('renders id, type, title, url, and passage in readable output', async () => {
mockHttpGet.mockResolvedValue(mockDeveloperResponse([sampleResult]));

await handleDeveloperSearchCommand({ query: 'tokio spawn_blocking' });

const [content] = vi.mocked(writeOutput).mock.calls[0];
expect(content).toContain(
'## [issue:tokio-rs/tokio#2309] (issue) spawn_blocking panics when exceeding the thread limit'
);
expect(content).toContain(
'https://github.com/tokio-rs/tokio/issues/2309'
);
expect(content).toContain('It will panic if this limit is too low.');
});

it('joins multiple passages and clips long content', async () => {
mockHttpGet.mockResolvedValue(
mockDeveloperResponse([
{
...sampleResult,
passages: [{ text: 'first passage' }, { text: 'x'.repeat(5000) }],
},
])
);

await handleDeveloperSearchCommand({ query: 'tokio spawn_blocking' });

const [content] = vi.mocked(writeOutput).mock.calls[0] as [string];
expect(content).toContain('first passage\n---\nx');
const body = content.split('\n').slice(2).join('\n');
expect(body.length).toBeLessThanOrEqual(1200);
});

it('prints a placeholder when there are no results', async () => {
mockHttpGet.mockResolvedValue(mockDeveloperResponse([]));

await handleDeveloperSearchCommand({ query: 'no hits' });

const [content] = vi.mocked(writeOutput).mock.calls[0];
expect(content).toBe('(no results)');
});

it('outputs the full envelope as JSON with --json', async () => {
mockHttpGet.mockResolvedValue(mockDeveloperResponse([sampleResult]));

await handleDeveloperSearchCommand({
query: 'tokio spawn_blocking',
json: true,
});

const [content] = vi.mocked(writeOutput).mock.calls[0] as [string];
const parsed = JSON.parse(content);
expect(parsed.results[0].passages[0].text).toBe(
'It will panic if this limit is too low.'
);
});
});

describe('error handling', () => {
it('exits with code 1 when the request fails', async () => {
mockHttpGet.mockRejectedValue(new Error('boom'));
const exitSpy = vi
.spyOn(process, 'exit')
.mockImplementation((() => undefined) as any);
const errorSpy = vi
.spyOn(console, 'error')
.mockImplementation(() => undefined);

await handleDeveloperSearchCommand({ query: 'test' });

expect(errorSpy).toHaveBeenCalledWith('Error:', 'boom');
expect(exitSpy).toHaveBeenCalledWith(1);

exitSpy.mockRestore();
errorSpy.mockRestore();
});
});
});
Loading
Loading