Fix segfault in software renderer when malloc fails for large meshes

[zsoerenm]

Summary

• Fix NULL pointer dereference in draw_mesh_softwarerendered when malloc fails for very large surface meshes
• Replace assert-only checks with proper RETURN_ERROR(GR3_ERROR_OUT_OF_MEM), matching the pattern used elsewhere (e.g. gr3_createsurfacemesh)

Problem

When rendering very large surface meshes (e.g. a 44,445 × 3,334 grid from a GNSS signal acquisition plot), the vertices_fp allocation in draw_mesh_softwarerendered requires ~78 GB of memory:

number_of_indices = (44445 - 1) * (3334 - 1) * 6 = 888,791,112
allocation = 888,791,112 * sizeof(vertex_fp) = 888,791,112 * 88 = 78.2 GB

When malloc returns NULL, the assert(draw->vertices_fp[draw_id]) at line 2384 is compiled out in release builds (NDEBUG defined), so the code proceeds to write through the NULL pointer, causing a segfault:

[149418] signal 11 (1): Segmentation fault
draw_mesh_softwarerendered.constprop.0.isra.0 at libGR3.so
gr3_getpixmap_softwarerendered at libGR3.so
gr3_getimage at libGR3.so
...
gr3_surface at libGR3.so

GDB confirms the crash writes to address 0x0:

=> movss  %xmm14,-0x58(%rdx)    # rdx=0x58, effective addr = 0x0

Fix

Replace assert-only NULL checks with proper error returns using RETURN_ERROR(GR3_ERROR_OUT_OF_MEM) in two locations in gr3_sr.c:

1. draw->vertices_fp allocation in gr3_draw_softwarerendered (line 2166)
2. draw->vertices_fp[draw_id] allocation in draw_mesh_softwarerendered (line 2383)

Test plan

• Verify the fix compiles (syntax-checked with gcc -fsyntax-only)
• Test with a large surface mesh that exceeds available memory — should now return an error instead of crashing
• Test with normal-sized surface meshes — should continue to work as before

🤖 Generated with Claude Code