Claude code Integration Feature and Integration Tests#2059
Claude code Integration Feature and Integration Tests#2059adamwrose wants to merge 8 commits intodanielmiessler:mainfrom
Conversation
|
@adamwrose Did Claude write this? What is the problem this is solving? How would I run it and test it? |
…ns` in tests - Replace custom `contains()` with stdlib `slices.Contains` in tests - Remove redundant `contains` helper function from test file - Add `slices` package import to test file
ksylvan
force-pushed
the
feat/claudecode-integration-tests
branch
from
b6e8cdc to
6ab52ea
Compare
|
@adamwrose Your PR description has a piece of a conversation (with some large language model?) in it....
|
|
I have not forgotten about this. I have been doing research on this for the last couple of weeks. There are opensource projects that expose claude code as a service that fabric and openclaw can talk to. What do you think. Want me to clean this up where AI helped me on the test cases and commit or do you want me to document a way to use claude code with an open source API? |
|
@adamwrose Thanks for your response. I'd love for us to be able to use Tell me what you're thinking, let's discuss! |
|
|
||
| binary := c.getBinary() | ||
| debuglog.Debug(debuglog.Detailed, "ClaudeCode SendStream launching: %s %s\n", binary, strings.Join(args, " ")) | ||
| cmd := exec.Command(binary, args...) |
There was a problem hiding this comment.
[Major] SendStream uses exec.Command but Send uses exec.CommandContext. If the claude subprocess hangs or the caller disconnects, this process cannot be killed — it will orphan.
Suggestion: Use exec.CommandContext with an internal context.WithCancel or context.WithTimeout, consistent with Send() at line 183.
| if userPrompt == "" { | ||
| return "", nil | ||
| } |
There was a problem hiding this comment.
[Major] When userPrompt is empty after message extraction, Send returns ("", nil) — a successful result with no content. Callers cannot distinguish "model returned nothing" from "no input was provided."
Suggestion: Return an error instead:
return "", fmt.Errorf("claude: no prompt content after message extraction")| func (c *Client) ListModels() ([]string, error) { | ||
| return []string{ | ||
| "claude-opus-4-6", | ||
| "claude-sonnet-4-6", | ||
| "claude-opus-4-5-20251101", | ||
| "claude-opus-4-5", | ||
| "claude-haiku-4-5", | ||
| "claude-haiku-4-5-20251001", | ||
| "claude-sonnet-4-5", | ||
| "claude-sonnet-4-5-20250929", | ||
| "claude-opus-4-1-20250805", | ||
| "claude-sonnet-4-20250514", | ||
| "claude-sonnet-4-0", | ||
| "claude-4-sonnet-20250514", | ||
| "claude-opus-4-0", | ||
| "claude-opus-4-20250514", | ||
| "claude-4-opus-20250514", | ||
| }, nil | ||
| } |
There was a problem hiding this comment.
[Suggestion] This static model list will become outdated as Anthropic releases new models. Consider querying claude --list-models dynamically (if the CLI supports it), falling back to this static list on error.
| func (c *Client) buildArgs(opts *domain.ChatOptions, system string) []string { | ||
| args := []string{"--print", "--no-session-persistence"} | ||
| if opts.Model != "" { | ||
| args = append(args, "--model", opts.Model) | ||
| } | ||
| if system != "" { | ||
| args = append(args, "--system-prompt", system) | ||
| } | ||
| switch opts.Thinking { | ||
| case domain.ThinkingLow: | ||
| args = append(args, "--effort", "low") | ||
| case domain.ThinkingMedium: | ||
| args = append(args, "--effort", "medium") | ||
| case domain.ThinkingHigh: | ||
| args = append(args, "--effort", "high") | ||
| } | ||
| if opts.ImageFile != "" { | ||
| args = append(args, "--add-dir", filepath.Dir(opts.ImageFile)) | ||
| } | ||
| return args | ||
| } |
There was a problem hiding this comment.
[Minor] ChatOptions fields like Temperature, TopP, MaxTokens, and Seed are silently dropped. Consider passing --max-tokens if the CLI supports it, and logging a debug warning for unsupported options so users aren't confused when their settings have no effect.
| channel <- domain.StreamUpdate{ | ||
| Type: domain.StreamTypeContent, | ||
| Content: text, | ||
| } | ||
| } |
There was a problem hiding this comment.
[Minor] If the channel consumer stops reading, this send blocks the goroutine forever. A select with a ctx.Done() channel would enable graceful shutdown (this ties into the exec.CommandContext suggestion above).
| args := c.buildArgs(opts, system) | ||
| args = append(args, userPrompt) | ||
| binary := c.getBinary() | ||
| debuglog.Debug(debuglog.Detailed, "ClaudeCode Send launching: %s %s\n", binary, strings.Join(args, " ")) |
There was a problem hiding this comment.
[Minor] Full command args (including the user prompt) are logged at debug level. For long prompts this could be noisy or leak sensitive content into logs. Consider truncating the prompt portion.
| @@ -0,0 +1,293 @@ | |||
| package claudecode | |||
There was a problem hiding this comment.
[Suggestion] Missing package-level doc comment. A brief // Package claudecode ... would help with go doc and IDE hover.
| for scanner.Scan() { | ||
| line := strings.TrimSpace(scanner.Text()) | ||
| if line == "" { | ||
| continue | ||
| } | ||
| text, ok := parseStreamDelta(line) | ||
| if !ok { | ||
| continue | ||
| } | ||
| channel <- domain.StreamUpdate{ | ||
| Type: domain.StreamTypeContent, | ||
| Content: text, | ||
| } | ||
| } |
There was a problem hiding this comment.
[Suggestion] The message event from Claude's stream-json output contains token usage metadata (input_tokens, output_tokens). Currently this is discarded. Forwarding it as a StreamTypeUsage update would give callers visibility into costs.
| func buildGoBinary(t *testing.T, relativeDir string) string { | ||
| t.Helper() | ||
|
|
||
| entryAny, _ := binaryCache.LoadOrStore(relativeDir, &builtBinary{}) | ||
| entry := entryAny.(*builtBinary) | ||
|
|
||
| entry.once.Do(func() { | ||
| outputDir, err := os.MkdirTemp("", "fabric-binary-*") | ||
| if err != nil { | ||
| entry.err = err | ||
| return | ||
| } | ||
|
|
||
| binaryName := filepath.Base(relativeDir) | ||
| if runtime.GOOS == "windows" { | ||
| binaryName += ".exe" | ||
| } | ||
| entry.path = filepath.Join(outputDir, binaryName) | ||
|
|
||
| cmd := exec.Command("go", "build", "-o", entry.path, "./"+filepath.ToSlash(relativeDir)) | ||
| cmd.Dir = repoRoot(t) | ||
|
|
||
| var stderr bytes.Buffer | ||
| cmd.Stderr = &stderr | ||
| entry.err = cmd.Run() | ||
| entry.log = stderr.String() | ||
| }) | ||
|
|
||
| if entry.err != nil { | ||
| t.Fatalf("build %s: %v\n%s", relativeDir, entry.err, entry.log) | ||
| } | ||
|
|
||
| return entry.path | ||
| } |
There was a problem hiding this comment.
[Suggestion] buildGoBinary compiles Go binaries at test time. While the sync.Once caching is smart, this still adds significant CI wall time. Consider a TestMain that pre-builds all needed binaries, or a Makefile target so CI can cache them between runs.
| return []string{"CLAUDECODE_BINARY_PATH=" + wrapperPath} | ||
| } | ||
|
|
||
| func TestIntegrationClaudeCodePatternExecution(t *testing.T) { |
There was a problem hiding this comment.
[Minor] Integration tests don't set per-test timeouts. If a Claude API call hangs, the entire CI job blocks until the global timeout. Consider adding t.Deadline() checks or explicit context.WithTimeout for each test (e.g. 2-5 minutes).
2 participants
What this Pull Request (PR) does
Add an integration to claude code.
I was introduced to your project from NetworkChuck and became facinated with the work of fabric. I currently have an github copilot and claude (Anthropic pro) subscription. I noticed that the anthropic API was charged differently and claude code could use the Subscription vs paying $5 per x many tokens. I hope this will make it to where this is useful for more users with a tighter budget. If you like this, I can make one for opencode as well.
prerequisites
Related issues
Please reference any open issues this PR relates to in here.
If it closes an issue, type
closes #[ISSUE_NUMBER].Screenshots
I have written a lot of tests to validate my work and done some manual testing as well.
Great Work,
Adam