fix(ffmpeg): fix vulkan encode feedback flags check before capabilities query

[neatnoise]

Description

The feedback flags check added in FFmpeg commit 735c84b57f ("vulkan: fix encode feedback query handling") reads ctx->enc_caps.supportedEncodeFeedbackFlags at line 775, but GetPhysicalDeviceVideoCapabilitiesKHR does not populate enc_caps until line 863. Since the struct is zero-initialized, the bitmask test always fails and all Vulkan encoders (h264_vulkan, hevc_vulkan, av1_vulkan) are broken on every driver:

[h264_vulkan @ 0x561bfc4d0140] Driver does not support required encode feedback flags (BUFFER_OFFSET and BYTES_WRITTEN).

The patch moves the check to after the capabilities query where enc_caps is valid.

Tested with AMD Radeon RX 9070 XT (RADV GFX1201) and AMD Radeon 780M (RADV PHOENIX), Mesa 26.2.0-devel. Both work after the fix.

This bug exists in upstream FFmpeg on both master and release/8.1 — reported as https://code.ffmpeg.org/FFmpeg/FFmpeg/issues/22967.

Screenshot

Issues Fixed or Closed

Roadmap Issues

Type of Change

• feat: New feature (non-breaking change which adds functionality)
• fix: Bug fix (non-breaking change which fixes an issue)
• docs: Documentation only changes
• style: Changes that do not affect the meaning of the code (white-space, formatting, missing semicolons, etc.)
• refactor: Code change that neither fixes a bug nor adds a feature
• perf: Code change that improves performance
• test: Adding missing tests or correcting existing tests
• build: Changes that affect the build system or external dependencies
• ci: Changes to CI configuration files and scripts
• chore: Other changes that don't modify src or test files
• revert: Reverts a previous commit
• BREAKING CHANGE: Introduces a breaking change (can be combined with any type above)

Checklist

• Code follows the style guidelines of this project
• Code has been self-reviewed
• Code has been commented, particularly in hard-to-understand areas
• Code docstring/documentation-blocks for new or existing methods/components have been added or updated
• Unit tests have been added or updated for any new or modified functionality

AI Usage

• None: No AI tools were used in creating this PR
• Light: AI provided minor assistance (formatting, simple suggestions)
• Moderate: AI helped with code generation or debugging specific parts
• Heavy: AI generated most or all of the code changes

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[ReenigneArcher]

Don't worry about the whitespace lint