diff --git a/.gitignore b/.gitignore index 1a96847..ad9c4c1 100644 --- a/.gitignore +++ b/.gitignore @@ -19,3 +19,4 @@ coverage.xml venv/ broadside_ai_output/ pytest-cache-files-*/ +*.pdf diff --git a/AUDIT_REPORT.md b/AUDIT_REPORT.md deleted file mode 100644 index 6472c16..0000000 --- a/AUDIT_REPORT.md +++ /dev/null @@ -1,274 +0,0 @@ -# Broadside-AI Project Audit Report - -**Date:** 2026-04-03 -**Version Audited:** 0.1.0 (pre-release alpha) -**Repository:** HuginnIndustries/Broadside-AI - ---- - -## 1. Executive Summary - -Broadside-AI is a CLI-first Python tool for **parallel LLM orchestration using a scatter/gather architecture**. It fans a prompt out to N independent LLM calls, collects the results, and synthesizes them into a single output. The project targets automation and CI/CD scenarios where running a task multiple times and aggregating signal is valuable without the overhead of a full workflow framework. - -**Overall Assessment:** The project demonstrates strong engineering fundamentals for an alpha-stage tool. Architecture is clean and intentionally scoped, code quality tooling is strict, security posture is solid, and CI/CD is comprehensive. A handful of minor issues and missing features are noted below. - -| Dimension | Rating | Notes | -|-----------|--------|-------| -| Architecture | Strong | Clean scatter/gather pipeline, well-separated concerns | -| Code Quality | Strong | Strict mypy, Ruff, no dead code, consistent patterns | -| Security | Strong | No hardcoded secrets, safe parsing, validated inputs | -| Testing | Adequate | 39 tests cover core paths; no coverage reporting | -| CI/CD | Strong | Multi-platform matrix, smoke tests, package validation | -| Documentation | Strong | 6 markdown docs, task library, examples | -| Dependencies | Good | Minimal core deps, optional extras, modern build system | - ---- - -## 2. Architecture & Design - -### Core Pipeline - -``` -Task -> Scatter -> [Agent 1, Agent 2, ..., Agent N] -> Gather -> Synthesize -> Output -``` - -- **Task** (`task.py`, 43 lines) - Pydantic model: prompt + optional context + optional output_schema -- **Scatter** (`scatter.py`, 120 lines) - Fans task to N independent calls (parallel or sequential) -- **Gather** (`gather.py`, 74 lines) - Normalizes results, parses JSON, computes stats -- **Synthesize** (`synthesize.py`, 103 lines) - Merges results via pluggable strategies -- **Run** (`run.py`, 94 lines) - Convenience orchestrator for the full pipeline - -### Design Patterns - -| Pattern | Where | Purpose | -|---------|-------|---------| -| Strategy | `strategies/` | Pluggable synthesis algorithms (llm, consensus, voting, weighted_merge) | -| Plugin/Registry | `backends/__init__.py` | Dynamic backend loading with `register()` / `get_backend()` | -| Stateless Pipeline | Core modules | No shared state between stages or runs | -| Circuit Breaker | `budget.py` | Token budget enforcement with thread-safe tracking | - -### Scope Boundaries (By Design) - -The project explicitly excludes: inter-agent messaging, workflow DAGs, persistent state, crew/role hierarchies, and autonomous long-running agents. This is documented in `ARCHITECTURE.md` and enforced in `CONTRIBUTING.md` (what-won't-be-merged section). - -### Codebase Size - -- **Source:** ~2,400 lines across 24 files in `src/broadside_ai/` -- **Tests:** ~500 lines across 9 test files + 79-line conftest -- **Total repository:** Compact and navigable - ---- - -## 3. Code Quality - -### Tooling - -| Tool | Configuration | Status | -|------|--------------|--------| -| **mypy** | `strict = true`, Python 3.10 target | Enabled, CI-enforced | -| **Ruff** | Line length 99, select E/F/I/N/W/UP | Enabled, CI-enforced | -| **Type hints** | Throughout all source files | Consistent | - -### Observations - -- **No dead code detected.** All modules are used in the pipeline. -- **No TODO/FIXME/HACK comments** in the source. -- **No hardcoded secrets** anywhere in the codebase. -- **Consistent naming conventions** across all modules. -- **Clean async patterns** using `asyncio.gather()` for parallel execution. -- **Pydantic validation** with `extra = "forbid"` prevents silent misconfiguration. -- **Thread-safe budget tracking** using `threading.Lock`. - -### Minor Style Notes - -- Backend constructors contain long multi-line error messages with shell commands (`backends/anthropic.py:36-41`, `backends/openai.py:37-42`). Functional but could use `textwrap.dedent` for readability. - ---- - -## 4. Security - -### Strengths - -| Area | Implementation | Risk | -|------|---------------|------| -| API key handling | Environment variables only (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`), never logged | Low | -| YAML parsing | `yaml.safe_load()` used exclusively (prevents code injection) | Low | -| Input validation | Pydantic with `extra: forbid`; `n >= 1` enforced; budget limits | Low | -| Code injection surface | No `eval()`, `exec()`, `pickle`, or shell command construction | Low | -| HTTP requests | Via httpx and official SDK clients only | Low | -| Dependency surface | 5 core deps, all well-maintained libraries | Low | - -### Minor Observations - -1. **Model name path construction** (`cli.py:419`): Model names are sanitized with `.replace(":", "-").replace("/", "-")` before use in directory paths. Adequate for the threat model (CLI user controls input), but a `pathlib` canonicalization would be more defensive. - -2. **Prompt injection via task context**: Context is directly interpolated into prompts (`task.py:28-42`). This is expected behavior for a CLI tool where the caller controls all inputs, not a vulnerability. - -3. **Conflict detection severity** (`conflicts.py:84-92`): Always returns `severity: "hard"` regardless of LLM classification. Not a security issue, but the feature is incomplete. - ---- - -## 5. Testing - -### Overview - -| Metric | Value | -|--------|-------| -| Test framework | pytest + pytest-asyncio | -| Test files | 9 | -| Test cases | 39 | -| Test LOC | ~500 | -| Async mode | Auto | -| Mock infrastructure | MockBackend, JsonMockBackend in conftest.py | - -### Coverage by Module - -| Test File | Module Tested | Tests | -|-----------|--------------|-------| -| `test_budget.py` | Budget circuit breaker | 4 | -| `test_cli.py` | CLI commands and output contracts | 4 | -| `test_execution.py` | Parallel/sequential mode resolution | 5 | -| `test_gather.py` | Result normalization and stats | 4 | -| `test_integration.py` | Full scatter-gather-synthesize pipeline | 2 | -| `test_quality.py` | Early stop and agreement detection | 6 | -| `test_synthesize.py` | Synthesis strategy routing | 3 | -| `test_task.py` | Task model validation | 5 | -| `test_weighted_merge.py` | Weighted merge algorithm | 6 | - -### Gaps - -- **No coverage reporting** - `pytest-cov` is not in dev dependencies. No way to measure untested paths. -- **Integration tests are minimal** - Only 2 tests cover the full pipeline. -- **Backend implementations untested directly** - Covered only via mocks. This is acceptable given they wrap SDK clients, but edge cases (network errors, rate limits) are not exercised. - ---- - -## 6. CI/CD - -### GitHub Actions Workflows - -| Workflow | Trigger | Purpose | -|----------|---------|---------| -| `ci.yml` | Push to main, all PRs | Full quality gate | -| `publish.yml` | GitHub Release event | PyPI publication (Trusted Publishing) | -| `publish-testpypi.yml` | Manual dispatch | TestPyPI dry-run | - -### CI Pipeline (`ci.yml`) - -1. Checkout code -2. Setup Python (matrix: **3.10 + 3.13** on **Ubuntu + Windows**) -3. Install package with dev dependencies -4. Run pytest -5. Run Ruff lint + format checks -6. Run mypy type checking -7. CLI smoke tests (`--help`, `validate-task`) -8. Build package with `build` -9. Validate distribution with `twine check` -10. Smoke-test built package (Python 3.13 only) - -**Assessment:** Comprehensive. Multi-platform matrix catches OS-specific issues (a recent Windows fix confirms this catches real bugs). Package build validation prevents broken releases. - ---- - -## 7. Documentation - -### Files - -| File | Size | Purpose | -|------|------|---------| -| `README.md` | 10 KB | Install, quickstart, CLI/API examples, dev setup | -| `ARCHITECTURE.md` | 4.3 KB | Design philosophy, data flow, extension points, non-goals | -| `CONTRIBUTING.md` | 2.3 KB | Dev setup, contribution rules, what won't be merged | -| `RELEASE.md` | 2.1 KB | Release process, PyPI Trusted Publishing setup | -| `ROADMAP.md` | 2.1 KB | v1 priorities, longer-term ideas, non-goals | -| `SECURITY.md` | 565 B | Security policy, vulnerability reporting | -| `tasks/README.md` | - | Task library documentation | -| `benchmarks/README.md` | - | Benchmark documentation | -| `.github/PULL_REQUEST_TEMPLATE.md` | - | PR template | - -**Assessment:** Unusually thorough for an alpha project. Architecture decisions are well-documented, contribution boundaries are explicit, and the release process is step-by-step. - -### Missing - -- **CHANGELOG.md** - No changelog file. Git history is clean but a changelog is standard practice before public release. - ---- - -## 8. Dependencies - -### Core (Required) - -| Package | Version | Purpose | -|---------|---------|---------| -| httpx | >=0.27.0 | Async HTTP client (Ollama backend) | -| pydantic | >=2.0.0 | Data validation and serialization | -| rich | >=13.0.0 | Terminal formatting | -| click | >=8.0.0 | CLI framework | -| pyyaml | >=6.0 | YAML task parsing | - -### Optional (Backend Extras) - -| Package | Extra | Purpose | -|---------|-------|---------| -| anthropic | `[anthropic]` | >=0.40.0, Claude API | -| openai | `[openai]` | >=1.50.0, OpenAI-compatible APIs | - -### Dev - -| Package | Purpose | -|---------|---------| -| pytest >=8.0.0 | Testing | -| pytest-asyncio >=0.24.0 | Async test support | -| ruff >=0.8.0 | Linting and formatting | -| mypy >=1.13.0 | Static type checking | -| build >=1.2.2 | Package building | -| twine >=6.1.0 | Distribution validation | - -### Notes - -- **No lock file** - Acceptable for a library (applications pin, libraries don't). -- **Build system:** Hatchling (modern, PEP 517 compliant). -- **Python support:** 3.10, 3.11, 3.12, 3.13 (tested in CI for 3.10 and 3.13). - ---- - -## 9. Issues & Recommendations - -### Issues Found - -| # | Issue | Severity | Location | -|---|-------|----------|----------| -| 1 | PDF file checked into repository | Low | Root directory | -| 2 | Conflict detection incomplete - always returns `severity: "hard"` | Low | `conflicts.py:84-92` | -| 3 | Voting strategy makes N extra LLM calls for label extraction | Low | `strategies/voting.py:53-62` | -| 4 | No test coverage reporting configured | Low | `pyproject.toml` | -| 5 | Benchmark results committed to repo could bloat wheel builds | Low | `benchmarks/results/` | -| 6 | No CHANGELOG.md | Low | Root directory | - -### Recommendations - -1. **Add the PDF to `.gitignore`** or move it to an external location. Binary files in Git repos increase clone size permanently. - -2. **Add `pytest-cov`** to dev dependencies and configure a coverage target. This provides visibility into untested code paths and can be enforced in CI. - -3. **Add a `CHANGELOG.md`** before the first public PyPI release. Even a simple keep-a-changelog format helps users track breaking changes. - -4. **Exclude `benchmarks/results/`** from the wheel distribution via `pyproject.toml` build configuration to keep package size minimal. - -5. **Consider batching voting label extraction** into a single LLM call to reduce cost and latency of the voting strategy. - -6. **Finish or remove conflict detection** (`conflicts.py`). The current implementation is partially functional - it should either be completed with proper severity parsing or explicitly marked as experimental. - ---- - -## 10. Summary - -Broadside-AI is a well-engineered, narrowly-scoped tool that does one thing cleanly: parallel LLM scatter/gather with synthesis. For a v0.1.0 alpha, the project demonstrates mature engineering practices: - -- **Architecture** is clean, documented, and deliberately constrained -- **Code quality** is enforced by strict tooling (mypy strict, Ruff) -- **Security** posture is solid with no identified vulnerabilities -- **CI/CD** is comprehensive with multi-platform testing -- **Documentation** is thorough and includes design rationale - -The issues identified are all low-severity and relate to missing features or polish rather than fundamental problems. The project is well-positioned for a public release after addressing the recommendations above. diff --git a/AUDIT_REPORT_CO.md b/AUDIT_REPORT_CO.md deleted file mode 100644 index 9295acb..0000000 --- a/AUDIT_REPORT_CO.md +++ /dev/null @@ -1,386 +0,0 @@ -# Broadside-AI Audit Report - -Audit date: 2026-04-03 - -## Executive summary - -Broadside-AI is a well-scoped, CLI-first Python project with a clear product thesis, a readable core pipeline, solid packaging hygiene, and working quality gates. The repository currently presents as release-aware pre-1.0 software rather than a rough prototype. During this audit, the local quality gates all passed: `py -3 -m pytest tests -q` reported `39 passed`, `py -3 -m ruff check src tests benchmarks examples` passed cleanly, `py -3 -m ruff format --check src tests benchmarks examples` reported files already formatted, `py -3 -m mypy src/broadside_ai` succeeded with no issues, and `py -3 -m build` produced both sdist and wheel artifacts successfully. - -The main concerns are not general code quality problems. They are contract-level mismatches between the behavior documented for automation users and the behavior actually enforced in the runtime. The three most important findings are: - -1. Parallel budget enforcement behaves like a soft cap, not the hard upper bound described in architecture and exception messaging. -2. `output_schema` currently drives JSON parsing and prompting, but not true schema validation of types or extra fields. -3. `weighted_merge` list behavior does not fully match the README claim of "majority presence" in tie cases. - -Overall assessment: strong project foundation, good release posture, and good evaluator-facing documentation, but some machine-contract claims should be tightened before this should be treated as fully trustworthy infrastructure for strict automation pipelines. - -## Project overview - -Broadside-AI positions itself as a narrow scatter/gather engine for CLI automation rather than a general agent framework. The README defines that boundary clearly and consistently: no inter-agent messaging, no DAG workflows, no persistent multi-step state, and no role hierarchy (`README.md:3-16`). The architecture document reinforces the same thesis and keeps the dominant flow small and understandable: `Task -> Scatter -> Gather -> Synthesize -> Output` (`ARCHITECTURE.md:14-25`). - -This focus is a strength. The repo is small, cohesive, and aligned around a single product shape: - -- Primary package: `src/broadside_ai/` -- Focused docs: `README.md`, `ARCHITECTURE.md`, `RELEASE.md`, `SECURITY.md`, `ROADMAP.md` -- Example task library: `tasks/` -- Benchmarks with committed snapshots: `benchmarks/` -- Test suite centered on public behavior: `tests/` - -The project is already set up like something intended to ship, not just to demonstrate ideas. - -## Architecture review - -### What is working well - -- The architecture is coherent and intentionally narrow. The system explicitly avoids workflow-engine complexity and preserves branch independence (`ARCHITECTURE.md:3-12`, `ARCHITECTURE.md:25-27`). -- The function-first API is appropriate for the product shape. Exposing `scatter()`, `gather()`, `synthesize()`, and `run()` keeps the public surface small and easy to understand (`ARCHITECTURE.md:28-42`). -- The backend abstraction is minimal and healthy. Backends only need `complete()` and `name()`, which is enough for the product without introducing lifecycle complexity (`ARCHITECTURE.md:70-79`). -- The CLI contract is thoughtfully designed for scripts. `run` defaults to stdout, artifacts are opt-in, and a JSON mode is available for automation (`ARCHITECTURE.md:44-55`, `src/broadside_ai/cli.py:47-72`, `src/broadside_ai/cli.py:82-103`). -- Execution defaults are sensible. The code distinguishes between cloud and local Ollama usage, defaulting cloud backends to parallel execution and local Ollama to sequential (`ARCHITECTURE.md:81-89`, `src/broadside_ai/execution.py` reviewed during audit). - -### Risks or gaps - -- The documented budget model is stronger than the actual runtime behavior. The architecture says `ScatterBudget` provides an upper bound on token usage (`ARCHITECTURE.md:91-97`), but parallel scatter launches all branches before token usage is recorded (`src/broadside_ai/scatter.py:41-67`). -- Structured-output handling is described as part of the main automation path (`ARCHITECTURE.md:57-68`), but the runtime only parses JSON dictionaries rather than validating them against `output_schema` (`src/broadside_ai/gather.py:52-60`, `src/broadside_ai/task.py:19-24`). -- The architecture is intentionally simple, but that simplicity also means some higher-trust contracts are currently implemented as conventions rather than enforced invariants. - -### Evidence - -- Product boundary and flow: `ARCHITECTURE.md:3-12`, `ARCHITECTURE.md:14-25` -- Function-first API: `ARCHITECTURE.md:28-42` -- CLI contract: `ARCHITECTURE.md:44-55`, `src/broadside_ai/cli.py:47-72` -- Budget claim: `ARCHITECTURE.md:91-97` -- Parallel scatter implementation: `src/broadside_ai/scatter.py:41-67` - -## Code quality review - -### What is working well - -- The core code is small and readable. The main CLI and orchestration paths are easy to follow, with limited indirection. -- Type hygiene is good for a pre-1.0 project. The repository uses strict mypy mode (`pyproject.toml:93-99`) and passed `py -3 -m mypy src/broadside_ai` during this audit. -- Modules are sensibly separated by responsibility: task definition, scatter, gather, synthesis, backends, validation, benchmarking, and CLI. -- The package surface is clean and predictable. `pyproject.toml` defines a straightforward console script, base dependencies, extras, and build exclusions (`pyproject.toml:28-78`). - -### Risks or gaps - -- Some code and docstrings imply guarantees that the implementation does not enforce. The clearest case is `Task.output_schema`, which is described as "Used to validate agent responses" (`src/broadside_ai/task.py:19-24`) even though gather only parses JSON objects (`src/broadside_ai/gather.py:52-60`). -- `weighted_merge` accepts and merges any parsed dictionary keys, including keys not declared in `output_schema`, because the schema parameter is not used in the merge logic (`src/broadside_ai/strategies/weighted_merge.py:12-40`, `src/broadside_ai/strategies/weighted_merge.py:58-82`). -- Budget accounting is thread-safe in isolation (`src/broadside_ai/budget.py:24-56`) but not protective enough to uphold a true hard limit in parallel mode because accounting happens after completions return (`src/broadside_ai/scatter.py:52-56`). - -### Evidence - -- Strict typing: `pyproject.toml:93-99` -- CLI payload model: `src/broadside_ai/cli.py:33-72` -- Output schema description: `src/broadside_ai/task.py:19-24` -- Gather parsing behavior: `src/broadside_ai/gather.py:52-60` -- Weighted merge ignoring schema: `src/broadside_ai/strategies/weighted_merge.py:12-40`, `src/broadside_ai/strategies/weighted_merge.py:58-82` - -## Security review - -### What is working well - -- The repository includes a clear security policy with private disclosure instructions and a supported-version statement (`SECURITY.md:1-26`). -- Provider credentials are expected from environment variables rather than checked into config files. Anthropic and OpenAI backends explicitly require `ANTHROPIC_API_KEY` and `OPENAI_API_KEY` (`src/broadside_ai/backends/anthropic.py:25-45`, `src/broadside_ai/backends/openai.py:25-49`). -- The base dependency set is small and appropriate for the product (`pyproject.toml:28-48`). -- Build exclusions intentionally omit caches, benchmark result snapshots, and generated artifacts from published distributions (`pyproject.toml:63-78`), reducing accidental package leakage. -- Ollama defaults to local-first behavior with explicit helpful connection errors (`src/broadside_ai/backends/ollama.py:13-29`, `src/broadside_ai/backends/ollama.py:41-66`). - -### Risks or gaps - -- There is no dependency-vulnerability scanning or SBOM generation in CI. The CI workflow runs tests, Ruff, mypy, build, `twine check`, and wheel smoke tests, but no `pip-audit`, `safety`, or similar step (`.github/workflows/ci.yml:25-52`). -- Prompt and context content are forwarded directly to providers without redaction or sensitive-data screening. This is not inherently wrong for a CLI tool, but it is an important trust boundary for users handling proprietary material. -- The project does not yet offer explicit safeguards around prompt injection or untrusted context in structured tasks. Given the product's purpose, some guidance in docs would help. -- Security posture is adequate for an alpha CLI tool, but not yet especially mature beyond credential handling and disclosure process. - -### Evidence - -- Security policy: `SECURITY.md:1-26` -- Anthropic key handling: `src/broadside_ai/backends/anthropic.py:34-45` -- OpenAI key handling: `src/broadside_ai/backends/openai.py:35-49` -- Minimal dependency set and package exclusions: `pyproject.toml:28-78` -- CI workflow scope: `.github/workflows/ci.yml:25-52` - -## Testing review - -### What is working well - -- The current suite exercises the main product paths: CLI behavior, execution defaults, gather behavior, budget primitives, early stop behavior, synthesis, integration, and weighted merge. -- Test feedback is fast. During this audit, `py -3 -m pytest tests -q` completed successfully with `39 passed in 1.07s`. -- The test suite does a good job of validating public contracts rather than only internals. Examples include CLI JSON output, validate-task behavior, and end-to-end synthesis (`tests/test_cli.py`, `tests/test_integration.py:12-40`). -- Weighted merge has direct tests for numeric aggregation, confidence weighting, list merging, and LLM fallback (`tests/test_weighted_merge.py:33-111`). - -### Risks or gaps - -- The budget tests validate the `ScatterBudget` class in isolation, but they do not validate the actual behavior of parallel `scatter()` under a low token budget (`tests/test_budget.py:8-40`). -- The weighted-merge tests cover the happy path but do not cover schema-invalid JSON, extra fields, or two-way list ties (`tests/test_weighted_merge.py:33-111`). -- There is no coverage reporting or minimum threshold enforced in CI (`.github/workflows/ci.yml:28-52`). -- The existing integration coverage confirms that weighted merge works when upstream JSON is well-formed, but not when it is malformed or partially invalid (`tests/test_integration.py:20-40`). - -### Evidence - -- Budget unit tests: `tests/test_budget.py:8-40` -- Weighted merge tests: `tests/test_weighted_merge.py:33-111` -- Integration test scope: `tests/test_integration.py:12-40` -- CI test scope: `.github/workflows/ci.yml:28-52` -- Observed local result: - -```text -py -3 -m pytest tests -q -....................................... [100%] -39 passed in 1.07s -``` - -## CI/CD and release review - -### What is working well - -- CI is practical and release-oriented. It runs on both Ubuntu and Windows across Python 3.10 and 3.13 (`.github/workflows/ci.yml:9-16`). -- The workflow covers tests, lint, format, typing, CLI smoke checks, build, metadata validation, and built-wheel smoke testing (`.github/workflows/ci.yml:25-52`). -- Release documentation is unusually thorough for a project at this stage. It includes local release gates, pre-publish review, TestPyPI guidance, publish flow, and post-publish verification (`RELEASE.md:18-87`). -- The publish workflow follows a good two-stage pattern: build and validate artifacts first, then publish via PyPI trusted publishing (`.github/workflows/publish.yml:8-56`). -- The built-wheel smoke script validates both console-script and module entrypoints in a clean venv (`tools/smoke_dist_install.py:37-50`). - -### Risks or gaps - -- CI is strong on packaging correctness but currently weaker on supply-chain and dependency audit checks. -- Publish automation assumes release publication rather than enforcing any additional post-build policy checks beyond what build/test already did. -- Release docs explicitly warn that `README.md` must match runtime behavior (`RELEASE.md:45-54`), and the current structured-output and budget issues show why that warning matters. - -### Evidence - -- CI matrix and steps: `.github/workflows/ci.yml:9-52` -- Release checklist: `RELEASE.md:18-54`, `RELEASE.md:66-87` -- Publish workflow: `.github/workflows/publish.yml:8-56` -- Clean-venv smoke test: `tools/smoke_dist_install.py:37-50` -- Observed local results: - -```text -py -3 -m ruff check src tests benchmarks examples -All checks passed! - -py -3 -m ruff format --check src tests benchmarks examples -37 files already formatted - -py -3 -m mypy src/broadside_ai -Success: no issues found in 24 source files - -py -3 -m build -Successfully built broadside_ai-0.1.0.tar.gz and broadside_ai-0.1.0-py3-none-any.whl -``` - -## Documentation review - -### What is working well - -- Documentation quality is a strong point. The README is clear about product scope, install paths, workflows, structured output, and JSON automation use cases (`README.md:1-40`, `README.md:223-325`). -- Architecture and release docs are aligned with the project's intended audience: users automating CLI workflows and maintainers preparing packages for distribution. -- The repository includes supporting docs beyond the minimum expected set: contributing guidance, roadmap, benchmarks, release process, and security policy. - -### Risks or gaps - -- The most important documentation issue is doc/code drift, not lack of content. -- The README says `weighted_merge` merges lists by majority presence and that `confidence` is only metadata (`README.md:288-294`), but implementation details can include tied list items and accept undeclared keys during merge (`src/broadside_ai/strategies/weighted_merge.py:58-118`). -- The architecture doc says budget provides an upper bound (`ARCHITECTURE.md:91-97`), which is stronger than what parallel execution currently guarantees (`src/broadside_ai/scatter.py:63-67`, `src/broadside_ai/budget.py:35-40`). -- `Task.output_schema` is described as validation-oriented in code comments (`src/broadside_ai/task.py:19-24`) and implied similarly in docs, but runtime enforcement is parsing-only (`src/broadside_ai/gather.py:52-60`). - -### Evidence - -- README scope and automation framing: `README.md:1-40`, `README.md:223-325` -- Architecture claims: `ARCHITECTURE.md:44-68`, `ARCHITECTURE.md:91-97` -- Weighted merge docs: `README.md:288-294` -- Weighted merge code: `src/broadside_ai/strategies/weighted_merge.py:58-118` -- Schema description vs runtime: `src/broadside_ai/task.py:19-24`, `src/broadside_ai/gather.py:52-60` - -## Dependencies and packaging review - -### What is working well - -- The dependency set is lean: `httpx`, `pydantic`, `rich`, `click`, and `pyyaml` in the base install, with Anthropic/OpenAI separated into extras (`pyproject.toml:28-48`). -- Packaging metadata is complete enough for an alpha release: project URLs, supported Python versions, console script, and build backend are all defined (`pyproject.toml:1-58`). -- Build exclusions are thoughtful and reduce the chance of shipping transient or bulky non-package files (`pyproject.toml:63-78`). -- The project successfully builds both sdist and wheel artifacts locally. - -### Risks or gaps - -- Dependencies use lower bounds but no upper bounds, and there is no lockfile. This is common for libraries but increases exposure to upstream breaking changes. -- Optional dependency imports are handled cleanly, but compatibility will depend on continued API stability in provider SDKs. -- There is no automated dependency-review or vulnerability-scanning gate in CI. - -### Evidence - -- Build and dependency config: `pyproject.toml:1-99` -- Publish and smoke build verification: `.github/workflows/ci.yml:44-52`, `tools/smoke_dist_install.py:30-50` - -## Prioritized findings - -### P1: Parallel budget enforcement is softer than documented - -Impact: High for automation users who interpret `max_tokens` as a hard safety boundary for provider cost. - -The architecture document says `ScatterBudget` provides an upper bound on token usage (`ARCHITECTURE.md:91-97`), and the exception text says remaining branches are cancelled (`src/broadside_ai/budget.py:18-20`). In practice, parallel scatter instantiates all branch tasks before any token usage is recorded (`src/broadside_ai/scatter.py:63-67`). That means a budget exceed can be detected only after multiple calls have already been launched. - -Audit reproduction: - -```text -Observed with a local reproduction backend: -{'completed_results': 1, 'budget_used': 300, 'calls': 3} -``` - -This was produced with a 150-token budget and three parallel branches returning 100 tokens each. The runtime kept the result list short, but actual recorded usage still doubled the configured cap. - -### P1: `output_schema` is parse-oriented, not validation-oriented - -Impact: High for downstream tools relying on typed or schema-constrained structured output. - -`Task.output_schema` is described as validating responses (`src/broadside_ai/task.py:19-24`), but gather only attempts to parse a JSON object (`src/broadside_ai/gather.py:52-60`). `weighted_merge` then merges any parsed dictionaries and ignores its `output_schema` argument (`src/broadside_ai/strategies/weighted_merge.py:12-40`). - -Audit reproduction: - -```text -Observed with schema {'label': 'string', 'tags': 'list'}: -{ - 'n_parsed': 2, - 'parsed_outputs': [ - {'label': 123, 'tags': ['a'], 'unexpected': 'x'}, - {'label': 'spam', 'tags': ['b']} - ], - 'merged': {'label': '123', 'tags': ['a', 'b'], 'unexpected': 'x'} -} -``` - -This shows three problems: - -- wrong types are accepted as parsed -- undeclared keys are accepted and merged -- merge output can drift away from the declared schema - -### P2: List merging does not strictly implement majority presence - -Impact: Medium, because it changes the semantics of structured aggregation in ambiguous cases. - -The README claims list merging works by majority presence (`README.md:288-294`). The implementation uses `counts[key] >= len(values) / 2` (`src/broadside_ai/strategies/weighted_merge.py:108-117`), which includes tied items for even-numbered result sets. - -Audit reproduction: - -```text -_merge_lists([['a'], ['b']]) -> ['a', 'b'] -``` - -This is not a strict majority. It is a "half or more" rule. - -### P3: Test coverage is good for healthy-path behavior but thin on contract regressions - -Impact: Medium, because the current suite gives confidence in general stability but not in the exact contracts most important to automation users. - -The current tests are strong enough to keep the project stable day-to-day, but they do not yet pin the most important contract mismatches found in this audit. In particular, there is no regression test for parallel budget overshoot, no test proving extra schema keys are rejected, and no tie-case test for list merging (`tests/test_budget.py:8-40`, `tests/test_weighted_merge.py:33-111`). - -## Actionable recommendations - -### Immediate contract fixes - -1. Reclassify budget behavior honestly. - - Either implement true hard-cap semantics or update docs and error messaging to describe best-effort/soft-cap behavior in parallel mode. - - Minimum acceptable fix: align `ARCHITECTURE.md`, README wording, and `BudgetExceeded` messaging with actual runtime semantics. - -2. Add real schema validation for structured outputs. - - Validate types and reject unexpected keys after JSON parse. - - Exclude invalid parsed outputs from `n_parsed`, early-stop structured agreement checks, and `weighted_merge`. - -3. Bring list merge semantics in line with documentation. - - If "majority presence" is the intended contract, change the threshold to strict majority. - - If current behavior is intentional, update docs to say "half or more" instead. - -### Near-term reliability improvements - -1. Add regression tests for the identified contract issues. - - Parallel scatter under a low budget - - Schema-invalid structured outputs - - Extra keys in structured outputs - - Two-way list ties in weighted merge - -2. Extend CI with a dependency-audit step. - - `pip-audit` is the obvious starting point for this repository. - -3. Strengthen machine-readable status reporting. - - Consider including whether structured outputs were fully validated, partially valid, or fell back to freeform synthesis in JSON output. - -4. Add lightweight security guidance to the README. - - Warn users that prompts and context may be sent directly to third-party providers. - - Recommend caution with secrets, proprietary code, and sensitive datasets. - -### Lower-priority polish and future enhancements - -1. Add coverage reporting or at least track coverage locally in CI artifacts. -2. Consider documenting provider timeout/retry expectations more explicitly. -3. Add a small compatibility matrix for supported provider SDK versions as the project matures. -4. If structured outputs become a core selling point, consider moving from "JSON-schema-like dict" wording to a fully defined internal schema contract. - -## Verification appendix - -### Workspace and branch state before writing - -```text -git branch --show-current -codex/Audit - -git status --short -(no output) - -Test-Path AUDIT_REPORT_CO.md -False -``` - -### Quality gates observed during this audit - -```text -py -3 -m pytest tests -q -....................................... [100%] -39 passed in 1.07s -``` - -```text -py -3 -m ruff check src tests benchmarks examples -All checks passed! -``` - -```text -py -3 -m ruff format --check src tests benchmarks examples -37 files already formatted -``` - -```text -py -3 -m mypy src/broadside_ai -Success: no issues found in 24 source files -``` - -```text -py -3 -m build -Successfully built broadside_ai-0.1.0.tar.gz and broadside_ai-0.1.0-py3-none-any.whl -``` - -### Files reviewed directly during the audit - -- `README.md` -- `ARCHITECTURE.md` -- `pyproject.toml` -- `.github/workflows/ci.yml` -- `.github/workflows/publish.yml` -- `RELEASE.md` -- `SECURITY.md` -- `src/broadside_ai/cli.py` -- `src/broadside_ai/scatter.py` -- `src/broadside_ai/gather.py` -- `src/broadside_ai/task.py` -- `src/broadside_ai/budget.py` -- `src/broadside_ai/strategies/weighted_merge.py` -- `src/broadside_ai/backends/ollama.py` -- `src/broadside_ai/backends/anthropic.py` -- `src/broadside_ai/backends/openai.py` -- `tools/smoke_dist_install.py` -- `tests/test_budget.py` -- `tests/test_weighted_merge.py` -- `tests/test_integration.py` - -### Scope note - -This audit was evidence-based and static-analysis-heavy, with local quality gates and targeted runtime reproductions. It did not include live calls to Anthropic, OpenAI, or Ollama services, external dependency vulnerability scanning, or dynamic fuzz testing against untrusted prompts. diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..49f4014 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,16 @@ +# Changelog + +## 0.1.0 + +Initial public release. + +- CLI-first scatter/gather with `broadside-ai run` and `broadside-ai validate-task` +- Backends: Ollama (base install), Anthropic, OpenAI-compatible +- Synthesis strategies: llm, consensus, voting, weighted_merge +- Structured output parsing with `output_schema` +- Early-stop controls (`--early-stop`, `--agreement`) +- Stable JSON output mode (`--json-output`, schema version 1) +- Context file support (`--context-file`) +- Python API: `run()`, `run_sync()`, `Task`, `EarlyStop` +- Task library with community-contributed YAML definitions +- Benchmark suite with committed result snapshots diff --git a/README.md b/README.md index 1c0b023..a17db61 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,10 @@ # Broadside-AI +[![CI](https://github.com/HuginnIndustries/Broadside-AI/actions/workflows/ci.yml/badge.svg)](https://github.com/HuginnIndustries/Broadside-AI/actions/workflows/ci.yml) +[![PyPI](https://img.shields.io/pypi/v/broadside-ai)](https://pypi.org/project/broadside-ai/) +[![Python](https://img.shields.io/pypi/pyversions/broadside-ai)](https://pypi.org/project/broadside-ai/) +[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) + Broadside-AI is a CLI-first Python tool for parallel LLM aggregation. It fans a task out to multiple independent runs, gathers the outputs, and produces one final result that works well in scripts, CI, and other automation. @@ -32,84 +37,66 @@ Committed benchmark snapshots in `benchmarks/results/` currently show: ## Install -Broadside-AI gives you two equivalent ways to run it: - -- `broadside-ai ...` -- `py -3 -m broadside_ai ...` - -If you are on Windows and the `broadside-ai` command is not recognized, use the -`py -3 -m broadside_ai ...` form. It works even when the user Scripts -directory is not on `PATH`. - -On macOS or Linux, replace `py -3` with `python3` in the commands below. - -### From PyPI - ```bash -py -3 -m pip install broadside-ai +pip install broadside-ai ``` -Recommended for CLI users: +Or, to install it as an isolated CLI tool: ```bash pipx install broadside-ai ``` -Optional extras: +Optional extras for cloud backends: ```bash -py -3 -m pip install "broadside-ai[anthropic]" -py -3 -m pip install "broadside-ai[openai]" -py -3 -m pip install "broadside-ai[all]" +pip install "broadside-ai[anthropic]" +pip install "broadside-ai[openai]" +pip install "broadside-ai[all]" ``` -### From a downloaded ZIP or cloned repo - -If you prefer to install from source: +After installing, verify the CLI works: -Windows PowerShell, if you downloaded the GitHub ZIP: - -```powershell -cd .\Broadside-AI-main -py -3 -m pip install . +```bash +broadside-ai --help ``` -Windows PowerShell, if you cloned the repo: +
+Windows: if broadside-ai is not on PATH + +Use the module entrypoint instead: -```powershell -cd .\Broadside-AI -py -3 -m pip install . +```cmd +py -3 -m broadside_ai --help ``` -macOS or Linux: +This works even when the user Scripts directory is not on `PATH`. You can use +`py -3 -m broadside_ai` anywhere this README shows `broadside-ai`. -```bash -cd Broadside-AI-main -python3 -m pip install . -``` +If you prefer the console-script form, install with `pipx` so the command is +added to a CLI-friendly location. -To install backend extras from the repo checkout, use: +
-```powershell -py -3 -m pip install ".[anthropic]" -py -3 -m pip install ".[openai]" -py -3 -m pip install ".[all]" -``` +
+Install from source -### First success on Windows +Clone the repo or download the GitHub ZIP, then install: -If you want the most reliable first-run path on Windows, use this sequence: +```bash +cd Broadside-AI +pip install . +``` -1. Install the package from the repo or from PyPI. -2. Verify the module entrypoint works: +To install backend extras from a source checkout: -```cmd -py -3 -m broadside_ai --help +```bash +pip install ".[anthropic]" +pip install ".[openai]" +pip install ".[all]" ``` -3. If `broadside-ai --help` also works, you can use either form. -4. If `broadside-ai` is not recognized, keep using `py -3 -m broadside_ai ...` - or install with `pipx` so the command is added to a CLI-friendly location. +
## Quick start @@ -124,22 +111,10 @@ way to avoid a frustrating first try is: #### Step 1: Confirm the install worked -Start here: - -```cmd -py -3 -m broadside_ai --help -``` - -If you want to try the console-script form too: - -```cmd +```bash broadside-ai --help ``` -If that second command says `'broadside-ai' is not recognized`, nothing is -wrong with Broadside-AI itself. It just means the script location is not on -your `PATH` yet. Use `py -3 -m broadside_ai ...` and keep going. - #### Step 2: Pick one backend Broadside-AI supports Ollama, Anthropic, and OpenAI-compatible APIs. For a @@ -149,14 +124,14 @@ first run, Ollama local is the least setup-heavy option. Install Ollama, then pull a local model: -```cmd +```bash ollama pull gemma3:1b ``` Now run Broadside-AI: -```cmd -py -3 -m broadside_ai run --prompt "Write a pitch for a dotfile manager" --n 3 --model gemma3:1b +```bash +broadside-ai run --prompt "Write a pitch for a dotfile manager" --n 3 --model gemma3:1b ``` That should print one synthesized result to stdout. @@ -167,8 +142,7 @@ That should print one synthesized result to stdout. easy to compose with other tools: ```bash -python -m broadside_ai run --prompt "Summarize this changelog" --n 3 > summary.txt -py -3 -m broadside_ai run --prompt "Write a pitch for a dotfile manager" --n 3 --model gemma3:1b +broadside-ai run --prompt "Summarize this changelog" --n 3 > summary.txt broadside-ai run --prompt "Write a pitch for a dotfile manager" --n 3 --model gemma3:1b ``` @@ -198,7 +172,7 @@ Install Ollama, sign in, and pull the default cloud model: ```bash ollama signin ollama pull nemotron-3-super:cloud -py -3 -m broadside_ai run --prompt "Write a pitch for a dotfile manager" --n 3 +broadside-ai run --prompt "Write a pitch for a dotfile manager" --n 3 ``` Execution defaults are tuned for user success: @@ -212,7 +186,7 @@ Override with `--parallel` or `--sequential` when needed. ```bash ollama pull gemma3:1b -py -3 -m broadside_ai run --prompt "Write a pitch for a dotfile manager" --n 3 --model gemma3:1b +broadside-ai run --prompt "Write a pitch for a dotfile manager" --n 3 --model gemma3:1b ``` ### Anthropic @@ -220,8 +194,8 @@ py -3 -m broadside_ai run --prompt "Write a pitch for a dotfile manager" --n 3 - Set `ANTHROPIC_API_KEY` in your shell first, then run: ```bash -py -3 -m pip install "broadside-ai[anthropic]" -py -3 -m broadside_ai run --prompt "Review this design" --n 3 --backend anthropic +pip install "broadside-ai[anthropic]" +broadside-ai run --prompt "Review this design" --n 3 --backend anthropic ``` ### OpenAI-compatible APIs @@ -229,8 +203,8 @@ py -3 -m broadside_ai run --prompt "Review this design" --n 3 --backend anthropi Set `OPENAI_API_KEY` in your shell first, then run: ```bash -py -3 -m pip install "broadside-ai[openai]" -py -3 -m broadside_ai run --prompt "Compare these options" --n 3 --backend openai +pip install "broadside-ai[openai]" +broadside-ai run --prompt "Compare these options" --n 3 --backend openai ``` For OpenAI-compatible providers, set `OPENAI_BASE_URL` and pass `--model`. @@ -249,7 +223,7 @@ Choose the synthesis strategy based on the kind of output you want: Use `--json-output` for scripts and subprocess integrations: ```bash -py -3 -m broadside_ai run tasks/ticket_classification.yaml --n 5 --synthesis weighted_merge --json-output +broadside-ai run tasks/ticket_classification.yaml --n 5 --synthesis weighted_merge --json-output ``` The JSON payload always includes: @@ -276,8 +250,8 @@ The JSON payload always includes: ### Save artifacts when you want them ```bash -py -3 -m broadside_ai run tasks/code_review.yaml --n 3 --save -py -3 -m broadside_ai run tasks/code_review.yaml --n 3 --output artifacts/review-run +broadside-ai run tasks/code_review.yaml --n 3 --save +broadside-ai run tasks/code_review.yaml --n 3 --output artifacts/review-run ``` Saved runs go under: @@ -289,7 +263,6 @@ broadside_ai_output/{model}/{topic}_{timestamp}/ ### Validate task files ```bash -py -3 -m broadside_ai validate-task tasks/ticket_classification.yaml broadside-ai validate-task tasks/ticket_classification.yaml ``` @@ -311,7 +284,7 @@ That enables `weighted_merge`, an algorithmic synthesis strategy that: Example: ```bash -py -3 -m broadside_ai run tasks/ticket_classification.yaml --n 5 --synthesis weighted_merge --json-output +broadside-ai run tasks/ticket_classification.yaml --n 5 --synthesis weighted_merge --json-output ``` You can also stop early when enough branches have arrived or agreed: @@ -337,7 +310,7 @@ broadside-ai run tasks/ticket_classification.yaml --n 5 --synthesis weighted_mer ### CI validation for task files ```bash -python -m broadside_ai validate-task tasks/_template.yaml +broadside-ai validate-task tasks/_template.yaml ``` ## Python API @@ -409,7 +382,8 @@ Repository docs: ## Status -Broadside-AI is pre-1.0 software, but the repo is intended to be usable and -releasable as-is: package metadata, CLI behavior, task validation, and release -checks are kept in sync in-repo. The markdown docs in this repository are the -source of truth. +Broadside-AI is at **v0.1.0** (first public release). The CLI interface, +JSON output schema, and Python API (`run`, `run_sync`, `Task`, `EarlyStop`) +are considered stable for this release. Synthesis strategies and backend +options may expand in future versions. Breaking changes before v1.0 will +be noted in [release notes](https://github.com/HuginnIndustries/Broadside-AI/releases). diff --git a/ROADMAP.md b/ROADMAP.md index 24c2ee1..839220b 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -47,7 +47,7 @@ The project now has: ### 5. Community readiness -- add issue templates and a PR template +- ~~add issue templates and a PR template~~ (done) - label good first issues - document benchmark contribution expectations @@ -73,4 +73,4 @@ These remain out of scope unless the product identity changes: The repository markdown files are part of the product surface. If behavior changes, update the docs in the same PR. -Last updated: 2026-04-03 +Last updated: 2026-04-05 diff --git a/Research Results Scatter_Gather Has a Real Gap in the AI Agent Framework Market.pdf b/Research Results Scatter_Gather Has a Real Gap in the AI Agent Framework Market.pdf deleted file mode 100644 index 10ac0f0..0000000 Binary files a/Research Results Scatter_Gather Has a Real Gap in the AI Agent Framework Market.pdf and /dev/null differ diff --git a/llms.txt b/llms.txt index d050277..000ec28 100644 --- a/llms.txt +++ b/llms.txt @@ -24,8 +24,16 @@ Primary CLI: - `broadside-ai run` - `broadside-ai validate-task` +- `broadside-ai --version` - `python -m broadside_ai` +Key flags: + +- `--context-file` appends local files as grounding context for the task +- `--json-output` emits a stable machine-readable JSON payload +- `--early-stop` / `--agreement` for cost and latency control +- `--save` / `--output` to persist run artifacts + Synthesis strategies: - `llm`: one direct final answer using an LLM diff --git a/pyproject.toml b/pyproject.toml index 4855f72..648c420 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -73,6 +73,8 @@ exclude = [ "/pytest-cache-files-*", "/benchmarks/results", "/Research Results*.pdf", + "/AUDIT_REPORT*.md", + "/CHANGELOG.md", "/python", "**/__pycache__", "**/*.py[cod]", diff --git a/src/broadside_ai/cli.py b/src/broadside_ai/cli.py index 57d2928..c0e5ed4 100644 --- a/src/broadside_ai/cli.py +++ b/src/broadside_ai/cli.py @@ -427,9 +427,13 @@ def _resolve_model_display(backend: str, backend_kwargs: dict[str, Any]) -> str: return f"{_DEFAULT_MODEL} (default)" if backend == "anthropic": - return "claude-sonnet-4-20250514 (default)" + from broadside_ai.backends.anthropic import _DEFAULT_MODEL as _ANTHROPIC_DEFAULT + + return f"{_ANTHROPIC_DEFAULT} (default)" if backend == "openai": - return "gpt-4o-mini (default)" + from broadside_ai.backends.openai import _DEFAULT_MODEL as _OPENAI_DEFAULT + + return f"{_OPENAI_DEFAULT} (default)" return "(backend default)" diff --git a/src/broadside_ai/py.typed b/src/broadside_ai/py.typed new file mode 100644 index 0000000..e69de29