Supporting custom draw pipelines in imaging?

[dwuggh]

I would like to discuss whether imaging should support drawing through user-defined/custom pipelines, similar in spirit to custom shaders in wgpu or skia's sksL.

the design in my mind right now would be like a generalized version of bevy's SpecializedRenderPipeline:

trait PipelineDesc {
    /// the data type for this pipeline
    type State;
    fn draw(&self, state: &State);
    // ...
}
// pipeline with data, store as draw commands
struct PipeLine<P: PipelineDesc> {
    desc: P,
    data: P::State,
}

trait PipelineSpecializer {
    type Key: Clone + Hash + PartialEq + Eq;

    type Desc: PipelineDesc;
    /// Construct a new render pipeline based on the provided key.
    fn specialize(&self, key: Self::Key) -> Desc;
}

pub struct PipelineRef<'a> {
    pub transform: Affine,
    pipeline: &'a dyn Pipeline,
    // ...
}

I believe we can do runtime type checking, to match the type of Pipeline and its caller, though I don't have a clear thought now.

Will custom pipeline draw command be supported, or it is not in plan? If so, I'd be happy to contribute.

[jrmoulton]

I personally don't think this belongs in imaging. Or at least not of the current crates as we have them now. I have custom drawing hooked up in Floem.

lapce/floem#1075

There is a draft here that you might find interesting although this is a big change and not exactly easy to see where the custom drawing is happening. But if you look at the external-surface-cube example you can see the api that I am using

This isn't an api for custom pipelines but I think it does cover at least a wide range of use cases for UI.

Do you specifically want custom pipelines for something?

[dwuggh]

I took a look at the example, please correct me if I understand wrong:
the example render a cube to a texture(with wgpu), sample it to a image, and use the image as a brush.

I'm not familiar with subduction, I imagine it as a sort of compositor, that can mix different surfaces in layers. This works great for UI use cases, but would be much less effcient if I have a lot of tiny objects to render, in arbitrary orders with imaging objects. Or render something like interactive fog effects that can interact with UI elements. I would like to extend imaging to a general purpose 2d graphics engine interface, not only vector graphics rendering.

Do you specifically want custom pipelines for something?

I'm writing a calligraphy stroke renderer, simply speaking, rendering variable width strokes. This is easier and more effcient to do with shaders. I want to use a 2d graphics engine to build this, and let the engine to take on rendering normal strokes, images and texts. this lead me to the vello-masonry-xilem tech stack, but both vello the engine and masonry the ui toolkit is using imaging as its graphics interface. my current solution is to customize a widget in masonry and draw my content there using imaging's API by expanding strokes into outlines, this could work but is far less flexible than rendering with shaders. So in my case it would really be great to draw with customizable pipelines.

another use case is rendering CSG shapes. they can be effciently directly rendered without generating an outline(check this paper and also fidget). It is also possible in vello-cpu and vello-hybrid by directly generating sparse strips. all these can be abstracted into a custom rendering pipeline.

I also found my previous example sksL is a bit inproprate, sksL is a different level of abstraction, like adding a middleware in rendering process, not by defining a new pipeline. However a custom pipeline is general enough to include it.

maybe something like PaintExt is better?

[jrmoulton]

I think that example I linked to is actually closer than you're seeing. The compositor stuff isn't relevant. What is relevant is the usage of shaders as a custom brush. What I'm doing there is intercepting that at paint time and then using a custom pipeline to render it. I could even ensure it's rendered all into the same command encoder. You could do the same thing, use a custom calligraphy setting on the stroke and ensure that it's rendered with the pipeline you want. There is probably room for a crate that abstracts that but I think it should use imaging not be a core part of imaging.

[jrmoulton]

I would be really interested in a general API that exposes the ability to have custom pipelines at the draw command level that builds on top of imaging so I am interested in the idea

[waywardmonkeys]

Yes, imaging should be part of a stack rather than the whole thing. There's a lot to sort out and figure out.