Skip to content

Arm backend: Support sampler-backed VGF grid_sample for RIFE#20329

Merged
bdemirb merged 4 commits into
pytorch:mainfrom
bdemirb:baris_mletorch-2218_rife_sampler_align_corners
Jun 17, 2026
Merged

Arm backend: Support sampler-backed VGF grid_sample for RIFE#20329
bdemirb merged 4 commits into
pytorch:mainfrom
bdemirb:baris_mletorch-2218_rife_sampler_align_corners

Conversation

@bdemirb

@bdemirb bdemirb commented Jun 17, 2026 •
edited by pytorch-bot Bot
Loading

Copy link
Copy Markdown
Collaborator

Add VGF grid_sample custom-shader improvements needed by the RIFE optical-flow workload.

This stack enables sampler-backed grid_sample payloads where supported, adds support for quantised int8/SNORM image payloads, handles align_corners=True, and updates VGF test infrastructure so multi-delegate exports can be validated.

Changes

  • Add sampler/image payload selection for VGF grid_sample.
  • Keep unsupported cases on the existing storage-buffer fallback path.
  • Allow safe C3-to-C4 padding so RIFE-style 3-channel inputs can use sampler-compatible image payloads.
  • Add int8/SNORM image payload support for quantised grid_sample.
  • Keep grid coordinates in float for range and precision.
  • Add align_corners=True sampler shader variants.
  • Allow VGF tests to check an expected delegate count instead of assuming one delegate.
  • Add focused tests for sampler, int8/SNORM, fallback, and aligned shader selection.

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani

bdemirb added 4 commits June 16, 2026 22:31
@bdemirb
Arm backend: Use sampler path for VGF grid sampler
8d3e801 
Enable the VGF grid_sampler custom payload to select an image
sampler shader for NCHW float32 C4 inputs, while keeping the
existing storage-buffer shader for unsupported layouts.

Pad static C3 inputs to C4 in the rewrite pass so RIFE feature
warps can use texture samplers, then slice outputs back to C3.

Add shader resources and tests covering sampler and fallback paths.

Signed-off-by: Baris Demir <baris.demir@arm.com>
Change-Id: I8737436d2b5920e74804103dae2a609ce2d24183
@bdemirb
Arm backend: Allow VGF delegate count checks
056d58d 
Allow VgfPipeline callers to set the expected number of
executorch_call_delegate nodes while keeping the default at one.
This is a test-framework-only change for multi-delegate VGF tests.
It does not affect Arm backend lowering or runtime behavior.

Signed-off-by: Baris Demir <baris.demir@arm.com>
Change-Id: Iac60bb75bd40cab424ccb9841a2d02ee3767f11c
@bdemirb
Arm backend: Use int8 grid sampler payload
587e438 
Add an int8 sampler shader payload for quantized grid_sample
boundaries using fixed SNORM-compatible qparams.

Annotate grid_sampler in the Arm quantizer with the same fixed int8
image qparams so PT2E places the expected Q/DQ boundary before the VGF
custom-op rewrite.

The rewrite pass consumes that quantized boundary when selecting the
int8 image payload, avoiding a float image payload and the extra
runtime Q/DQ around the custom shader.

Signed-off-by: Baris Demir <baris.demir@arm.com>
Change-Id: I7c85dde518f871a7572386ca6cfe67e865ee6745
@bdemirb
Arm backend: Support aligned VGF grid sampler
1d9b2c9 
Add sampler shader variants for grid_sample with align_corners=True.

Select the aligned sampler shader from the VGF custom payload when the
operator requests align_corners=True, while keeping bicubic on the
storage-buffer fallback path.

Allow C3 inputs to use the sampler padding path for both align modes so
RIFE warps can use the int8 SNORM sampler payload without changing the
model's grid_sample semantics.

Signed-off-by: Baris Demir <baris.demir@arm.com>
Change-Id: I83f8cdb8024d3d77c8706278c9b12c9dbe77b58f
@bdemirb bdemirb requested a review from digantdesai as a code owner June 17, 2026 09:40
@pytorch-bot

pytorch-bot Bot commented Jun 17, 2026 •
edited
Loading

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20329

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 44 Pending

As of commit 1d9b2c9 with merge base 551e90e (image):

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 17, 2026
@github-actions github-actions Bot added ciflow/trunk module: arm Issues related to arm backend labels Jun 17, 2026
@bdemirb bdemirb changed the title Baris mletorch 2218 rife sampler align corners Arm backend: Support sampler-backed VGF grid_sample for RIFE Jun 17, 2026
@bdemirb

bdemirb commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator Author

@pytorchbot label "partner: arm"

@bdemirb

bdemirb commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator Author

@pytorchbot label "release notes: arm"

@pytorch-bot pytorch-bot Bot added the partner: arm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Arm label Jun 17, 2026
@bdemirb

bdemirb commented Jun 17, 2026

Copy link
Copy Markdown
Collaborator Author

@claude review this PR

@pytorch-bot pytorch-bot Bot added the release notes: arm Changes to the ARM backend delegate label Jun 17, 2026
@claude

claude Bot commented Jun 17, 2026 •
edited
Loading

Copy link
Copy Markdown

Claude encountered an error after 3m 51s —— View job

I'll analyze this and get back to you.

zingo
zingo approved these changes Jun 17, 2026
View reviewed changes

@zingo zingo left a comment •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK to merged if Arm tests pass OK (Still running)

@bdemirb bdemirb merged commit cf59749 into pytorch:main Jun 17, 2026
483 of 507 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zingo zingo zingo approved these changes

@digantdesai digantdesai Awaiting requested review from digantdesai digantdesai is a code owner

Assignees

No one assigned

Labels

ciflow/trunk CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. module: arm Issues related to arm backend partner: arm For backend delegation, kernels, demo, etc. from the 3rd-party partner, Arm release notes: arm Changes to the ARM backend delegate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@bdemirb @zingo