Skip to content

WIP: dockerize audited repository - #19

Open
sinachaichi wants to merge 1 commit into
mainfrom
feat/dockerize-audited-repo
Open

sinachaichi wants to merge 1 commit into
mainfrom
feat/dockerize-audited-repo

Conversation

@sinachaichi

Copy link
Copy Markdown
Owner

Work in progress, not ready to merge.

Implements extra objective:
generating and verifying a Dockerfile/compose for the advisory's repository when missing.

Verified end to end against real repositories (Grafana, n8n, pyLoad):
deterministic classification, LLM tie-breaker for ambiguous cases, sandboxed generation-verify-repair loop all work as designed. Output validation correctly rejects malformed generations rather than passing them to Docker.

Known limitations, documented but not yet resolved:

  • Classifier signals cover common Python/Node/Go conventions but miss several frameworks (Next.js, Rails, Phoenix, and others).
  • Generation is occasionally unstable on large monorepos, producing garbled structured output a few times during testing; validation catches this and reports GENERATION_FAILED rather than passing it through, but the underlying success rate of generation itself has not been measured systematically.
  • Verification requires the application to actually start and answer HTTP; real-world repositories needing secrets, external services, or database migrations to boot will fail verification even with a correct Dockerfile, which is inherent to what "successfully runs" requires, not a bug in this implementation.

Next step before merging:
measure success rate (application detection accuracy, generation success, build success, verification success) across a larger sample of real advisories, similar to the eval harness already built for the main tracing pipeline.

Generate and verify a Dockerfile/compose for an advisory's repository
when missing: resolve the vulnerable commit, classify application vs
library (rule-based, LLM tie-breaker when ambiguous), generate with an
agent restricted to reading the checkout, then build/run/health-check
in a sandboxed, network-isolated compose project with a bounded repair
loop on failure. Not merged yet; see PR description for known
limitations and next steps.
@sinachaichi sinachaichi added the help wanted Extra attention is needed label Jul 6, 2026
@sinachaichi
sinachaichi force-pushed the feat/dockerize-audited-repo branch 3 times, most recently from 6571b87 to a493813 Compare July 14, 2026 15:34
@sinachaichi
sinachaichi force-pushed the feat/dockerize-audited-repo branch from a493813 to 149735f Compare August 24, 2026 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

help wanted Extra attention is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant