Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ ClawScan turns that approach into a repeatable CLI. It includes a built-in `claw
| --- | --- |
| `clawscan <target> --scanner <id>` | Run one or more scanners against an explicit skill or OpenClaw plugin target. Omit `<target>` to scan child skill directories under `./skills`; plugins are never auto-discovered. |
| `clawscan scanners [list\|<scanner-id>]` | Discover supported scanner IDs, required env vars, upstream links, descriptions, and install guidance. |
| `clawscan profiles [-v]` | Inspect built-in plus nearest project-local profiles; `-v` prints the resolved profile catalog as YAML. |
| `clawscan profiles [-v]` | Inspect built-in profiles; `-v` prints the catalog as YAML. |
| `clawscan benchmark [list\|<benchmark-id>]` | Discover or run supported benchmarks through a selected scanner/profile/judge setup. |
| `clawscan install <scanner-id> [...]` | Install or verify local scanner dependencies where ClawScan has registry-backed install plans. |

Expand Down Expand Up @@ -173,8 +173,7 @@ The same profile accepts an explicit OpenClaw plugin directory (or its
`openclaw.plugin.json` manifest), runs both scanners, and renders the
bundled judge prompt with `packageRelease` target context.

Inspect the resolved profile catalog, including the nearest project
`.clawscan.yml` / `.clawscan.yaml` when present:
Inspect the built-in profile catalog:

```bash
clawscan profiles
Expand Down
5 changes: 3 additions & 2 deletions cmd/clawscan/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ func runProfiles(args []string) error {
if err != nil {
return err
}
catalog, err := profiles.InspectProfiles(cwd)
catalog, err := profiles.InspectProfiles()
if err != nil {
return err
}
Expand Down Expand Up @@ -566,6 +566,7 @@ Usage:
Core flags:
--profile <name> Profile to run. Use --profile clawhub for ClawHub parity.
--config <path> Load profiles from a specific .clawscan.yml file; omit --profile to run them all.
--discover-config Find and load the nearest .clawscan.yml/.clawscan.yaml in the current directory or a parent. Requires --profile. Off by default: config files can define scanners that run arbitrary commands, so ClawScan never loads config it was not explicitly told to trust.
--scanner <id> Scanner to run. Repeat for multiple scanners.
--scanner-result <id=path> Use a JSON fixture instead of running that scanner.
--context <path> Load profile runtime context from a JSON file.
Expand Down Expand Up @@ -596,7 +597,7 @@ Catalog commands:
clawscan scanners List supported scanners with required env vars.
clawscan scanners list Alias for clawscan scanners.
clawscan scanners <id> Show scanner repository, description, env vars, and install guidance.
clawscan profiles List built-in plus nearest project .clawscan.yml/.clawscan.yaml profiles.
clawscan profiles List built-in profiles.
clawscan profiles -v Print the resolved profile catalog as pasteable YAML.
clawscan benchmark list List supported benchmarks with source datasets and splits.

Expand Down
185 changes: 173 additions & 12 deletions cmd/clawscan/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ func TestRunCommandPrintsHelp(t *testing.T) {
"Install scanner dependencies without running scans.",
"--profile <name>",
"--config <path>",
"--discover-config",
"Benchmark command flags:",
"--split <name>",
"--ids <path-or-url>",
Expand Down Expand Up @@ -150,7 +151,7 @@ func TestRunCommandScannerDetailPrintsHumanReadableInfo(t *testing.T) {
}
}

func TestRunCommandProfilesPrintsMergedDiscoveredProfiles(t *testing.T) {
func TestRunCommandProfilesPrintsBuiltInProfilesOnly(t *testing.T) {
dir := t.TempDir()
removedProfile := "skills" + "-sh"
writeFile(t, filepath.Join(dir, ".clawscan.yml"), `version: 1
Expand All @@ -177,12 +178,10 @@ profiles:
"Source",
"Scanners",
"clawhub",
".clawscan.yml",
"clawscan-static",
"built-in",
"skillspector, clawscan-static",
"clawhub-aig",
"skillspector, aig",
"local-review",
"snyk",
} {
if !strings.Contains(stdout, want) {
t.Fatalf("profiles output missing %q:\n%s", want, stdout)
Expand All @@ -191,6 +190,9 @@ profiles:
if strings.Contains(stdout, removedProfile) {
t.Fatalf("profiles output should not include removed profile %q:\n%s", removedProfile, stdout)
}
if strings.Contains(stdout, "local-review") {
t.Fatalf("profiles output should not include project profile:\n%s", stdout)
}
}

func TestRunCommandProfilesVerbosePrintsResolvedYAML(t *testing.T) {
Expand All @@ -216,10 +218,8 @@ profiles:
"profiles:",
"clawhub:",
"clawhub-aig:",
"local-review:",
"- clawscan-static",
"- skillspector",
"- aig",
"json: true",
} {
if !strings.Contains(stdout, want) {
t.Fatalf("verbose profiles output missing %q:\n%s", want, stdout)
Expand All @@ -228,6 +228,9 @@ profiles:
if strings.Contains(stdout, removedProfile+":") {
t.Fatalf("verbose profiles output should not include removed profile %q:\n%s", removedProfile, stdout)
}
if strings.Contains(stdout, "local-review:") {
t.Fatalf("verbose profiles output should not include project profile:\n%s", stdout)
}
}

func TestRunCommandBenchmarkListPrintsCatalogTable(t *testing.T) {
Expand Down Expand Up @@ -675,7 +678,7 @@ profiles:
t.Chdir(dir)

stdout := captureStdout(t, func() {
if err := run([]string{target, "--profile", "local"}, []string{}); err != nil {
if err := run([]string{target, "--profile", "local", "--discover-config"}, []string{}); err != nil {
t.Fatal(err)
}
})
Expand All @@ -695,7 +698,111 @@ profiles:
}
}

func TestRunCommandConfigWithoutProfileRunsEveryConfigProfile(t *testing.T) {
func TestRunDefaultModeIgnoresConfigWithoutNotice(t *testing.T) {
dir := t.TempDir()
target := filepath.Join(dir, "skill")
writeSkill(t, target, "# Ignored config\n")
config := filepath.Join(dir, ".clawscan.yml")
writeFile(t, config, "version: [\n")
t.Chdir(dir)

stdout, stderr := captureOutput(t, func() {
if err := run([]string{target, "--scanner", "clawscan-static", "--json"}, []string{}); err != nil {
t.Fatal(err)
}
})

var artifact struct {
ConfigSource *string `json:"configSource"`
}
if err := json.Unmarshal([]byte(stdout), &artifact); err != nil {
t.Fatalf("stdout is not artifact JSON: %v\n%s", err, stdout)
}
if artifact.ConfigSource != nil {
t.Fatalf("config source = %q, want nil", *artifact.ConfigSource)
}
if !strings.Contains(stdout, `"configSource": null`) {
t.Fatalf("stdout lacks explicit null config source: %s", stdout)
}
if stderr != "" {
t.Fatalf("stderr = %q, want empty", stderr)
}
}

func TestRunWithDiscoverConfigLoadsConfig_NoNotice(t *testing.T) {
dir := t.TempDir()
target := filepath.Join(dir, "skill")
writeSkill(t, target, "# Discovered config\n")
config := filepath.Join(dir, ".clawscan.yml")
writeFile(t, config, `version: 1
profiles:
custom:
scanners:
- clawscan-static
`)
t.Chdir(dir)

stdout, stderr := captureOutput(t, func() {
if err := run([]string{target, "--profile", "custom", "--discover-config", "--json"}, []string{}); err != nil {
t.Fatal(err)
}
})

var artifact struct {
ConfigSource *string `json:"configSource"`
}
if err := json.Unmarshal([]byte(stdout), &artifact); err != nil {
t.Fatal(err)
}
if artifact.ConfigSource == nil {
t.Fatalf("config source = nil, want %q", config)
}
if *artifact.ConfigSource != config {
t.Fatalf("config source = %q, want %q", *artifact.ConfigSource, config)
}
if stderr != "" {
t.Fatalf("stderr = %q", stderr)
}
}

func TestRunWithExplicitConfigLoadsConfig_NoNotice(t *testing.T) {
dir := t.TempDir()
target := filepath.Join(dir, "skill")
writeSkill(t, target, "# Explicit config\n")
writeFile(t, filepath.Join(dir, ".clawscan.yml"), "version: [\n")
explicit := filepath.Join(dir, "explicit.yml")
writeFile(t, explicit, `version: 1
profiles:
p:
scanners:
- clawscan-static
`)
t.Chdir(dir)

stdout, stderr := captureOutput(t, func() {
if err := run([]string{target, "--config", explicit, "--profile", "p", "--json"}, []string{}); err != nil {
t.Fatal(err)
}
})

var artifact struct {
ConfigSource *string `json:"configSource"`
}
if err := json.Unmarshal([]byte(stdout), &artifact); err != nil {
t.Fatal(err)
}
if artifact.ConfigSource == nil {
t.Fatalf("config source = nil, want %q", explicit)
}
if *artifact.ConfigSource != explicit {
t.Fatalf("config source = %q, want %q", *artifact.ConfigSource, explicit)
}
if stderr != "" {
t.Fatalf("stderr = %q", stderr)
}
}

func TestBatchArtifactRuns_EachHasConfigSource(t *testing.T) {
dir := t.TempDir()
writeSkill(t, filepath.Join(dir, "skills", "foo"), "# Foo\n")
writeSkill(t, filepath.Join(dir, "skills", "bar"), "# Bar\n")
Expand All @@ -720,8 +827,9 @@ profiles:
var artifact struct {
SchemaVersion string `json:"schemaVersion"`
Runs []struct {
Profile string `json:"profile"`
Scanners map[string]interface{} `json:"scanners"`
Profile string `json:"profile"`
ConfigSource *string `json:"configSource"`
Scanners map[string]interface{} `json:"scanners"`
} `json:"runs"`
}
if err := json.Unmarshal([]byte(stdout), &artifact); err != nil {
Expand All @@ -737,6 +845,12 @@ profiles:
t.Fatalf("profiles = %q", got)
}
for _, run := range artifact.Runs {
if run.ConfigSource == nil {
t.Fatalf("config source = nil, want %q", config)
}
if *run.ConfigSource != config {
t.Fatalf("config source = %q, want %q", *run.ConfigSource, config)
}
if _, ok := run.Scanners["clawscan-static"]; !ok {
t.Fatalf("missing clawscan-static scanner for %s: %#v", run.Profile, run.Scanners)
}
Expand Down Expand Up @@ -864,3 +978,50 @@ func captureStdout(t *testing.T, fn func()) string {
os.Stdout = original
return string(out)
}

func captureOutput(t *testing.T, fn func()) (string, string) {
t.Helper()

originalStdout := os.Stdout
originalStderr := os.Stderr
stdoutRead, stdoutWrite, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
stderrRead, stderrWrite, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
os.Stdout = stdoutWrite
os.Stderr = stderrWrite
t.Cleanup(func() {
os.Stdout = originalStdout
os.Stderr = originalStderr
})

fn()

if err := stdoutWrite.Close(); err != nil {
t.Fatal(err)
}
if err := stderrWrite.Close(); err != nil {
t.Fatal(err)
}
stdout, err := io.ReadAll(stdoutRead)
if err != nil {
t.Fatal(err)
}
stderr, err := io.ReadAll(stderrRead)
if err != nil {
t.Fatal(err)
}
if err := stdoutRead.Close(); err != nil {
t.Fatal(err)
}
if err := stderrRead.Close(); err != nil {
t.Fatal(err)
}
os.Stdout = originalStdout
os.Stderr = originalStderr
return string(stdout), string(stderr)
}
2 changes: 1 addition & 1 deletion docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,6 @@ ClawScan turns that approach into a repeatable CLI. It includes a built-in `claw
| --- | --- |
| `clawscan <target> --scanner <id>` | Run one or more scanners against an explicit target. Omit `<target>` to scan child skill directories under `./skills`. |
| `clawscan scanners [list\|<scanner-id>]` | Discover supported scanner IDs, required env vars, upstream links, descriptions, and install guidance. |
| `clawscan profiles [-v]` | Inspect built-in plus nearest project-local profiles; `-v` prints the resolved profile catalog as YAML. |
| `clawscan profiles [-v]` | Inspect built-in profiles; `-v` prints the catalog as YAML. |
| `clawscan benchmark [list\|<benchmark-id>]` | Discover or run supported benchmarks through a selected scanner/profile/judge setup. |
| `clawscan install <scanner-id> [...]` | Install or verify local scanner dependencies where ClawScan has registry-backed install plans. |
35 changes: 33 additions & 2 deletions docs/profiles.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,39 @@ The same profile accepts an explicit OpenClaw plugin directory (or its
`openclaw.plugin.json` manifest), runs both scanners, and renders the
bundled judge prompt with `packageRelease` target context.

Inspect the resolved profile catalog, including the nearest project
`.clawscan.yml` / `.clawscan.yaml` when present:
## Config discovery

By default, ClawScan does not auto-discover `.clawscan.yml` or `.clawscan.yaml`
files from the current directory or parent directories.

ClawScan never loads a config file it was not explicitly pointed at. A
`.clawscan.yml` can define user-defined scanners whose commands execute with
your credentials in the environment, so silently loading one from the current
directory or a parent (for example, from inside a repository you are scanning)
would let an untrusted target execute commands. Pass `--config <path>` or opt in
with `--discover-config`.

To load a discovered config file, use one of these flags:

- `--config <path>` - Explicitly specify a config file path
- `--discover-config` - Search upward from the current directory and load the nearest `.clawscan.yml` or `.clawscan.yaml`

Mixing `--config` and `--discover-config` is an error. `--discover-config`
also requires `--profile`: without a profile the run would record the
discovered file as its config source while applying none of its settings.
Use `--config <path>` without `--profile` to run every profile in a config.

```bash
clawscan ./my-skill --config ./security/clawscan.yml --profile review
clawscan ./my-skill --profile review --discover-config
```

Without either flag, ClawScan uses built-in profiles and CLI flags only. The
`clawscan profiles` catalog command lists built-in profiles only.

## Inspect available profiles

Inspect the built-in profile catalog:

```bash
clawscan profiles
Expand Down
7 changes: 2 additions & 5 deletions internal/profiles/registry.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,11 +53,8 @@ func ProfileIDs() []string {
return DefaultProfileRegistry().IDs()
}

func InspectProfiles(cwd string) (ProfileCatalog, error) {
registry, err := loadConfigs(cwd, "")
if err != nil {
return ProfileCatalog{}, err
}
func InspectProfiles() (ProfileCatalog, error) {
registry := DefaultProfileRegistry()
catalog := ProfileCatalog{profiles: map[string]ProfileInfo{}}
for _, id := range registry.IDs() {
resolved, _ := registry.Profile(id)
Expand Down
Loading
Loading