Skip to content

feat: add Atlas Cloud WRBench first-frame provider - #10

Open
binyangzhu000-sudo wants to merge 1 commit into
OpenEnvision:mainfrom
binyangzhu000-sudo:codex/atlas-cloud-firstframe-provider
Open

feat: add Atlas Cloud WRBench first-frame provider#10
binyangzhu000-sudo wants to merge 1 commit into
OpenEnvision:mainfrom
binyangzhu000-sudo:codex/atlas-cloud-firstframe-provider

Conversation

@binyangzhu000-sudo

Copy link
Copy Markdown

Summary

Adds Atlas Cloud as an optional WRBench first-frame image provider. The implementation follows Atlas Cloud's asynchronous image flow: one generation submission, bounded prediction polling, then output download.

User-Visible Behavior

wrbench firstframe --provider atlascloud can generate Natural-25 first frames with an Atlas Cloud image model. Existing bundled frames and providers are unchanged.

Affected Pipeline / Benchmark

  • Pipeline(s): WRBench first-frame generation
  • Benchmark(s): WRBench Natural-25
  • Runtime profile(s): N/A
  • Public entrypoint(s): wrbench firstframe

Change Type

  • Bug fix
  • Model integration
  • Benchmark integration
  • Pipeline/runtime change
  • Documentation only
  • Test/QA tooling

Asset / API / GPU Requirements

  • Downloads or large assets: None
  • API keys or quota: Atlas Cloud API key and image-generation quota for live use
  • GPU / simulator requirements: None
  • Official repository or checkpoint assumptions: None

Commands Run

Command Result Notes
python3 -m pytest -q worldfoundry/evaluation/tasks/execution/runners/wrbench/test_atlas_cloud_firstframe.py worldfoundry/evaluation/tasks/execution/runners/wrbench/test_wrbench_generation_bridge.py Pass 4 tests passed
python3 -m ruff check --select E4,E7,E9,F --ignore F401 ... Pass Changed provider and test files
python3 -m py_compile ... Pass Changed Python files
bash scripts/docs/build.sh --skip-bootstrap Pass 1,257 static pages generated
Live Atlas Cloud smoke Pass Seedream v5.0 Lite; one POST; completed image downloaded

Checkpoint / API Key Needs

  • Checkpoints / weights: None
  • Required environment variables: ATLASCLOUD_API_KEY (passed to --api-key)
  • API providers: Atlas Cloud
  • Local cache assumptions: None

Sample Artifact Evidence

  • Artifact path or link: Local smoke output generated during validation; not committed
  • What it demonstrates: Submission, prediction polling, URL extraction, and image download complete successfully
  • Known limitations: Atlas Cloud currently returns one output per first-frame request, so --n must be 1

Validation Matrix Status

Area Status Evidence / Command
Unit or focused regression Pass 4 focused tests
Real inference validation Pass Live Seedream v5.0 Lite generation
Streaming or multi-turn path N/A Image generation is asynchronous polling
Benchmark runner / metric path Pass Existing WRBench generation bridge tests
Docs build or link check Pass Fumadocs production build
GPU/API-key dependent path Pass Live API-key smoke; no GPU required

Leaderboard Validity Impact

  • Readiness status changed: No
  • leaderboard_valid impact: None
  • Scorecard/preflight evidence: N/A
  • Demo or contract-only limitations: This only adds an optional first-frame source and does not change scoring

Compatibility And Risk

  • Backward compatibility impact: None; provider selection remains explicit
  • Expected resource cost: One billed image generation per requested frame
  • Failure modes or rollout concerns: Missing credentials, rejected model parameters, provider failure, or prediction timeout surfaces as an exception; generation POSTs are not retried

Checklist

  • I kept the change narrowly scoped.
  • I did not commit secrets, API keys, large checkpoints, or generated cache files.
  • I reused fixtures from data/test_cases/ or assets from data/benchmarks/ where practical.
  • I updated docs when public behavior, install steps, or entry commands changed.
  • I documented API/GPU/checkpoint/official-repo requirements or confirmed none are needed.
  • I did not promote demo, contract-only, normalizer-only, API-blocked, or partial evidence to leaderboard validity.

Signed-off-by: binyangzhu000-sudo <224954946+binyangzhu000-sudo@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant