Skip to content

Use the configured FFN drop path for ViT residuals - #9

Open
zjn20030811 wants to merge 1 commit into
OpenEnvision:mainfrom
zjn20030811:fix-vit-ffn-drop-path
Open

Use the configured FFN drop path for ViT residuals#9
zjn20030811 wants to merge 1 commit into
OpenEnvision:mainfrom
zjn20030811:fix-vit-ffn-drop-path

Conversation

@zjn20030811

Copy link
Copy Markdown

Summary

  • Route the FFN residual through the block's configured drop_path2 in the moderate stochastic-depth branch.
  • Preserve the historical drop_path1 fallback for direct callers that omit drop_path2.
  • Add focused regression coverage for distinct path selection and the compatibility fallback.

User-Visible Behavior

Transformer training now applies the independently configured stochastic-depth module to the FFN residual. Evaluation behavior and callers that omit the optional second path are unchanged.

Affected Pipeline / Benchmark

  • Pipeline(s): N/A (shared ViT transformer primitive)
  • Benchmark(s): N/A
  • Runtime profile(s): N/A
  • Public entrypoint(s): worldfoundry.core.nn.vit_block.apply_prenorm_transformer_residuals

Change Type

  • Bug fix
  • Model integration
  • Benchmark integration
  • Pipeline/runtime change
  • Documentation only
  • Test/QA tooling

Asset / API / GPU Requirements

  • Downloads or large assets: None
  • API keys or quota: None
  • GPU / simulator requirements: None; validation is CPU-only
  • Official repository or checkpoint assumptions: None

Commands Run

Command Result Notes
python -m pytest -q test/eval_core/test_vit_block.py Pass 2 passed
python -m py_compile worldfoundry/core/nn/vit_block.py test/eval_core/test_vit_block.py Pass Both changed Python files compile
git diff --check Pass No whitespace errors

Checkpoint / API Key Needs

  • Checkpoints / weights: None
  • Required environment variables: None
  • API providers: None
  • Local cache assumptions: None

Sample Artifact Evidence

  • Artifact path or link: None
  • What it demonstrates: Deterministic CPU regression output and invocation counts
  • Known limitations: Full model training and GPU kernels were not run

Validation Matrix Status

Area Status Evidence / Command
Unit or focused regression Pass python -m pytest -q test/eval_core/test_vit_block.py (2 passed)
Real inference validation N/A Shared residual helper only; no checkpoint path changed
Streaming or multi-turn path N/A Not applicable
Benchmark runner / metric path N/A Not applicable
Docs build or link check N/A No documentation changed
GPU/API-key dependent path N/A CPU-only change

Leaderboard Validity Impact

  • Readiness status changed: No
  • leaderboard_valid impact: None
  • Scorecard/preflight evidence: Not applicable
  • Demo or contract-only limitations: None

Compatibility And Risk

  • Backward compatibility impact: drop_path2=None continues to use drop_path1; evaluation and high-ratio stochastic-depth branches are unchanged.
  • Expected resource cost: None beyond the existing FFN drop-path operation.
  • Failure modes or rollout concerns: The fix changes training-time residual masking to honor the module already constructed for the FFN branch.

Checklist

  • I kept the change narrowly scoped.
  • I did not commit secrets, API keys, large checkpoints, or generated cache files.
  • I reused fixtures from data/test_cases/ or assets from data/benchmarks/ where practical.
  • I updated docs when public behavior, install steps, or entry commands changed.
  • I documented API/GPU/checkpoint/official-repo requirements or confirmed none are needed.
  • I did not promote demo, contract-only, normalizer-only, API-blocked, or partial evidence to leaderboard validity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant