Implement Drawable Textures

[reduz]

Describe the project you are working on

Godot

Describe the problem or limitation you are having in your project

One feature that is widely requested in Godot is the ability to easily have a texture and just draw to it (or even run a custom shader).
Examples of things you want to do:

• Animated textures
• Smoke and fire using ping-pong transfers
• Water plane effects
• Effects such as painting
• etc.

Traditionally this has be done with Viewports, but the whole thing is still relatively limited, as you can only write to a single image, writing to alpha is more complex, and you can't run multiple writing iterations per frame (ping-pong).

While in Godot 4.0+ users can access compute and RenderingDevice, this is still hard and beyond the level of experience of what most users can do, plus it won't work on GLES3 (compatibility) renderer.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Ideally, it should be possible to expose a very simple API to do this via RenderingServer, and an higher level DrawableTexture2D exposed for regular usage.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

At low level, the following API should be exposed for RenderingServer:

enum TextureDrawableFormat {
   TEXTURE_DRAWABLE_FORMAT_RGBA8,
   TEXTURE_DRAWABLE_FORMAT_RGBA8_SRGB, // Use this if you want to read the result from both 2D (non-hdr) and 3D.
   TEXTURE_DRAWABLE_FORMAT_RGBAH,
   TEXTURE_DRAWABLE_FORMAT_RGBAF,
};

// Create
RID texture_drawable_create(const Size2i& p_size, TextureDrawableFormat p_format,bool p_with_mipmaps = false);

// Draw

// Blit a rect
void texture_drawable_blit_rect(const Vector<RID> &p_textures, const Rect2i& p_rect, RID p_material,const Color& p_modulate,const Vector<RID>& p_source_textures,int p_to_mipmap=0); 
// Blit a polygon, 1 = point, 2 = line, 3 = polygon (will be tesselated). If UV provided, use these, otherwise UV is from dest texture.
void texture_drawable_blit_polygon(const Vector<RID> &p_textures, const PackedVector2Array& p_points, const PackedVector2Array& p_uvs,const PackedVector2Array& p_modulate_colors, RID p_material,const Vector<RID>& p_source_textures,int p_to_mipmap=0); 

// Utility functions
void texture_drawable_generate_mipmaps(RID p_texture); // Update mipmaps if modified
RID texture_drawable_get_default_material(); // To use with simplified functions in DrawableTexture2D

And that's it! We also need to add a new type of shader in Godot: SHADER_TYPE_TEXTURE_BLIT

Example of how to use:

shader_type texture_blit
// Optional render modes:
render_mode blend_mix /* default , also available: blend_add, blend_sub, blend_mul */;

uniform texture2D source_texture : hint_blit_source;
uniform texture2D source_texture2 : hint_blit_source2;
// up to 4 sources

void blit() {
     // Read sources and blit
     COLOR = texture(source_texture,UV) * MODULATE;
     COLOR2 = texture(source_texture2,UV);
     // up to COLOR4
}

Then put in a material and use.

Higher Level Abstraction

A resource can be provided:

class DrawableTexture2D : public Texture2D {
//...//
   enum DrawableFormat {
       DRAWABLE_FORMAT_RGBA8,
       DRAWABLE_FORMAT_RGBA8_SRGB,
       DRAWABLE_FORMAT_RGBAH,
       DRAWABLE_FORMAT_RGBAF,
   };
    void setup(Size2i p_size, DrawableFormat p_format,bool p_use_mipmaps=false);
    // Simple version without needing to use materials (uses a built-in material). Single source texture.
    void blit_rect(const Rect2i p_rect, const Ref<Texture2D>& p_source, const Color& p_modulate,int p_mipmap=0);
    void blit_polygon(const Rect2i p_rect, const Ref<Texture2D>& p_source, const PackedVector2Array& p_points const PackedVector2Array& p_uvs, const PackedColorArray& p_modulate,int p_mipmap=0);

    void blit_rect_multi(const Rect2i p_rect, const TypedArray<Ref<Texture2D>>& p_sources, const TypedArray<Ref<DrawableTexture2D>>& p_extra_targets, const Color& p_modulate,int p_mipmap=0);
    void blit_polygon_multi(const Rect2i p_rect,  const TypedArray<Ref<Texture2D>>& p_sources, const TypedArray<Ref<DrawableTexture2D>>& p_extra_targets,, const PackedVector2Array& p_points const PackedVector2Array& p_uvs, const PackedColorArray& p_modulate,int p_mipmap=0);

   void generate_mipmaps();
};

If this enhancement will not be used often, can it be worked around with a few lines of script?

N/A

Is there a reason why this should be core and not an add-on in the asset library?

N/A

[clayjohn]

I think this supercedes #5272

[lyuma]

Is there a way to support blitting an arbitrary Mesh object and/or expand the Vector2 points to allow for some vector3 data, such as vertices or normals. Clearly this wouldn't have a depth buffer or anything, just access to the mesh data.

As an example, a baking process in which we want to paint from a source texture onto a target texture, using the mesh's UV layout. In this case, baking texture data might need a mesh's UV texture layout, but also with access to vertex, normal and tangent data.

(And if 3D meshes are ruled out due to vertex layout constraints and performance tradeoffs, I could ask a related question for meshes in 2D mode which ought to be compatible with this vertex layout)

[reduz]

@lyuma Yes, but to be honest, my way of thinking is that this is intended to just be a simple API, so if you want to do something considerably more complex, its best to just use RenderingDevice directly.

[TokisanGames]

If I understand it, this would supersede many devs' needs for texture partial updates and high bit viewports as we can now do partial updates directly on the texture.

If so, do the same limitations that have hindered implementation of both of those hinder this? Can this be implemented faster than implementing the other two? Is this proposal just a nice looking API but still has to be figured out underneath, or is it something doable sooner rather than later?

[reduz]

@TokisanGames It should not be hard to implement, but its probably the better way to go forward, as otherwise this means more and more hacks have to be piled up on Viewport, which it is not meant for.

[KoBeWi]

Sounds like it would allow to rework AnimatedTexture instead of removing it.

[Zylann]

Would this API be blocking or asynchronous? I think there are two cases where this can be used (at least from what I would need):

• Editor tools and occasional computations: the renderer will likely stall, but it's ok since the context allows for it. Although, if we do 30 erosion steps on a heightmap, it wouldn't be wise to stall 30 times.
• Post-processing effects: done every frame (or more than once per frame). The renderer must not stall, and instead it may even be scheduled in order at particular steps of main rendering (example: accessing opaque depth of the currently-rendering frame of a certain viewport, so the result of the drawable texture can be used in a post-processing effect of that same frame).

Or would the two still require completely different APIs?

In fact it seems that either way it should not be synchronous until needed (like download back), but in the second case the time to run it matters more than the first case.

[bitsawer]

Instead of limiting texture_drawable_blit_rect() by forcing users to give explicit p_source_textures and modulate arguments, why not leverage existing ShaderMaterial so that you can call set_shader_parameter() to declare and set any parameters you need? I believe that would make the API quite flexible for most uses.

So, for example texture_drawable_blit_rect() would then be something like this:

void texture_drawable_blit_rect(const Vector<RID> &p_textures, const Rect2i& p_rect, RID p_material,int p_to_mipmap=0);

where RID p_material should be a ShaderMaterial. This way you can define and give whatever arguments you need by calling ShaderMaterial.set_shader_parameter() beforehand. We would not need an explicit modulate argument, either. If you need it, just define an uniform for it and set it using set_shader_parameter(). Easy to add any other custom data. This should also be easy to understand as other shader rendering works this way, too.

I think both this and godotengine/godot#75436 are complementary and have different use cases, although they can overlap and many will be happy with just this API. For example, godotengine/godot#75436 would be very useful for batch rendering octahedral impostors in my current project during runtime.

[reduz]

@bitsawer I think you misunderstood the proposal. If you want to do that, nobody stops you. That by sole definition of using a material in the API should work. The idea that you have the option to pass custom textures is for the very fact that it can get quite annoying to have to create a ShaderMaterial for every combination of textures you can pass.

[bitsawer]

Sound good, the proposal just didn't directly mention any way to pass custom data other than textures and modulate so I was a bit worried. But if using a ShaderMaterial with it is possible when you need some extra parameters and customization, it sounds good to me.

[reduz]

@Zylann Its synchronous, if you want to get the texture to CPU you can but you will stall the render until that part is done. Implementing asynchronous texture retrieval is planned, so eventually I guess that could work too.