Suggestion: List of possible improvements / work

[DatCaptainHorse]

Doing this list for some nice-to-have's on my side, but anyone is obviously free to do them before me, putting this as issue here may help potential contributors 🙂

Improvements

• Add decoding support.

• Improve code re-use across encoders. (Some work towards this was done in feat: encode pipelining with "pushification" of outputs #21)

• Add VkVideoEncodeTuningModeKHR, VkVideoEncodeContentFlagBitsKHR and VkVideoEncodeUsageFlagBitsKHR support for tuning encoding latency along other tunables. - feat: Add encoding hints and tuning modes #20

• B-frame support for potentially better encoding efficiency.

• Video Encode Intra Refresh, reduces bitrate-spikes, adds error resiliency and potentially lowers the decode latency by gradually constructing frame-data instead of having periodic and fat I-frame being encoded and sent.

• Video Encode Quantization Maps, allow to specify video regions that are potentially higher or lower quality, known as ROI (Region Of Interest) encoding on VA-API. Cool example is Valve using this with their upcoming VR headset, when streaming content from a PC they use eye-tracking to focus bitrate where the person is looking, leaving peripheral area at lower quality.

• Update Rust edition from 2021 to 2024, will require updating unsafe usages along possibly other project-wide changes. Benefit is access to improved usage patterns along other niceties.

• Update to latest working ash commit, the current used package is 6 months old, Vulkan evolves fast and getting access to latest extensions can open up some niceties (i.e. VK_KHR_video_encode_feedback2 was recent addition to the spec).

[urwrstkn8mare]

B-frame support for potentially better encoding efficiency.

I'm not really an export on encoding but just out of interest to learn would B-frame support increase latency? I'm assuming this would an optional feature for those who really need lower bitrate at the cost of latency.

Also how would ROI be used for game streaming outside of VR? Sounds really cool but I'm curious on what use case you're seeing.

[DatCaptainHorse]

B-frame support for potentially better encoding efficiency.

I'm not really an export on encoding but just out of interest to learn would B-frame support increase latency? I'm assuming this would an optional feature for those who really need lower bitrate at the cost of latency.

Also how would ROI be used for game streaming outside of VR? Sounds really cool but I'm curious on what use case you're seeing.

B-frames would increase latency yes, but pixelforge isn't targeted at just low-latency streaming so it's a nice-to-have for those that want or need better encoding efficiency with it. So for low-latency streaming no much use.

ROI is in the same bandwagon somewhat as a nice-to-have. Personally I'd like to test how good you can get with machine-learning based eye-tracking, using plain webcam as input - latency wouldn't be as good but for slower-paced game content it could be interesting to see the effect.

[urwrstkn8mare]

Oh yeah true. Latency isn't everything having the stream quality almost as good as it is when you're playing directly on your computer is almost as important.

[hgaiser]

B-frames would increase latency yes, but pixelforge isn't targeted at just low-latency streaming so it's a nice-to-have for those that want or need better encoding efficiency with it. So for low-latency streaming no much use.

I agree 👍 this is also why I opted to make it a standalone library. I currently only need it for low-latency streaming, but I imagine there'll be other usecases in the future.