Updates for package parity pillow, numpy etc.... - #419
Merged
smatula merged 7 commits intoSep 16, 2026
Merged
Conversation
Reviewer's GuideThe PR broadens manylinux package parity by adding checksum-verified native builds for Pillow, NumPy/SciPy, OpenCV, and Arrow dependencies, wiring their build order and artifacts through builder/runtime images, and enabling the corresponding Arrow, codec, image, font, and threading features. Sequence diagram for Arrow build using source-built dependenciessequenceDiagram
participant Builder as build_arrow
participant Curl as build_curl
participant XML as build_libxml2
participant Zlib as build_zlib
participant Arrow as build-arrow.sh
participant Artifact as manylinux-buildfs
Curl-->>Builder: /manylinux-buildfs
XML-->>Builder: /manylinux-buildfs
Zlib-->>Builder: /manylinux-buildfs
Builder->>Arrow: cmake with cloud and format features
Arrow->>Artifact: install Arrow libraries and headers
Artifact-->>Builder: staged Arrow build and runtime files
Flow diagram for Pillow image and font dependency chainflowchart LR
Zlib[zlib] --> FreeType[FreeType\nPNG Brotli, no HarfBuzz]
Brotli[Brotli] --> FreeType
FreeType --> HarfBuzz[HarfBuzz\nFreeType integration]
LibPNG[libpng] --> HarfBuzz
Brotli --> HarfBuzz
JPEG[libjpeg turbo] --> TIFF[libtiff\nJPEG Zstandard]
Zstd[zstd] --> TIFF
TIFF --> WebP[libwebp]
TIFF --> OpenJPEG[OpenJPEG]
AOM[AOM] --> AVIF[libavif]
AVIF --> Runtime[Builder and runtime images]
WebP --> Runtime
OpenJPEG --> Runtime
HarfBuzz --> Runtime
Flow diagram for OpenCV multimedia and AVIF supportflowchart LR
Zlib[zlib] --> FFmpeg[FFmpeg\nshared codec libraries]
AOM[AOM\nAV1 codec] --> LibAVIF[libavif\nAVIF support]
FFmpeg --> Runtime[Builder and runtime images]
LibAVIF --> Runtime
Runtime --> OpenCV[OpenCV package compatibility]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
smatula
changed the title
jvulgan
reviewed
Sep 14, 2026
jvulgan
left a comment
jvulgan
left a comment
Contributor
There was a problem hiding this comment.
I think the change looks good. I'd just remove references to ADR-0020 in the commit messages, without context it's not clear what ADR that references to and we can't just link to it since it's in a private git instance
Addresses feature-parity gap where Pillow and opencv wheels were built without optional image codec libraries, causing silent feature drops (e.g., PIL.ImageFont, WebP support). Changes: - Add build-freetype.sh (FreeType 2.14.3) for text rendering support - Add build-libwebp.sh (libwebp 1.4.0) for WebP image format support - Wire both into Containerfile build stages and final image layers Next steps: - Rebuild affected packages (pillow, opencv-python, opencv-python-headless) to pick up the new libraries - Additional libs (lcms2, openjpeg, libavif) can be added in follow-up PRs Related: docs/wheel-feature-parity-issue.md investigation Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
Completes the image codec library additions to fix Pillow feature gaps. Changes: - Add build-lcms2.sh (Little CMS 2.16) for color management (PIL.ImageCms) - Add build-openjpeg.sh (OpenJPEG 2.5.4) for JPEG2000 support - Add build-aom.sh (libaom 3.12.1) as a source-built AV1 codec - Add build-libavif.sh (libavif 1.4.2) for AVIF image format, using the source-built aom (AVIF_CODEC_AOM=SYSTEM) so no binaries or build-time git clones are introduced - Wire all four into Containerfile build stages and final image layers, and expose their headers/libs under /usr/local for pkg-config discovery Together with the previous freetype and libwebp additions, this provides Pillow with all major optional codecs that were silently missing. Remaining: ffmpeg (for opencv video I/O) - deferred as separate work due to its size and complexity. Related: docs/wheel-feature-parity-issue.md Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
Addresses numerical divergence from upstream PyPI numpy wheels and drops
the bundled OpenMP runtime.
Change:
- build-openblas.sh: USE_OPENMP=1 -> USE_OPENMP=0 (pthreads)
Why:
- Upstream numpy/scipy deliberately ship pthreads OpenBLAS, not OpenMP.
- The OpenMP variant changes parallel FP reduction order, so results
differ from PyPI numpy in the low bits (fails strict == / tight
allclose).
- The bundled OpenMP runtime makes a process forked after multithreaded
BLAS work segfault in the child.
Measured with numpy 2.5.2 in three venvs inside a single container, so
CPU, ISA, core count and libc are constant and the wheel is the only
variable. 13 fixed-seed BLAS/LAPACK operations, compared as sha256 of
the raw float64 bytes:
threads=1 threads=4
before 11/13 9/13
after 13/13 13/13
before: qr and svd differ at 1 thread; inv, solve, qr and svd at 4
after: bitwise identical to PyPI at both thread counts
Repeated on the builder image and on python:3.12-slim with identical
results. Divergence growing with thread count is the signature of
reduction order rather than a kernel or ISA difference.
Worth recording: the pre-change wheel's bundled numpy.libs/libgomp-*.so.1
is not GNU libgomp but LLVM's OpenMP runtime wearing the libgomp SONAME
(655 __kmpc_* symbols, kmp_affinity.cpp in its strings, and it honours
KMP_AFFINITY at runtime). It decodes x2APIC topology during import. After
this change no OpenMP runtime is mapped at all, matching PyPI.
The fork() failure reported against these wheels reproduces, and this
change fixes it. On python:3.12-slim, parent does a 1024x1024 matmul,
solve and inv, then forks a plain Process:
forked child
before SIGSEGV (exit -11), 4 of 4 configs
after clean, 4 of 4
PyPI clean, 4 of 4
configs: default, taskset -c 0,3,7, taskset -c 0,8, OMP_NUM_THREADS=64
after and PyPI also return identical result hashes in every config
An earlier round of testing on this change reported that the fork failure
did NOT reproduce. That was wrong, and it was wrong for two reasons that
both bias the test toward a false pass:
- it warmed BLAS with a 256x256 matmul, which stays under OpenBLAS's
threading threshold, so no OpenMP pool is ever created and the fork is
genuinely harmless;
- it forked via multiprocessing.Pool, where a segfaulting worker surfaces
as a hang or a generic error rather than as an exit signal.
Fix both -- 1024x1024 operands and a plain Process -- and the failure is
immediate and unambiguous. Anyone re-running this should check those two
things before concluding the wheel is clean.
The same mechanism shows up in the opencv-python wheel currently on the
index (5.0.0.93), which bundles the same OpenMP OpenBLAS: SIGSEGV after
fork in 5 of 6 configs, while the PyPI wheel of that version and an
opencv rebuilt on the fixed builder image are both clean and map no
OpenMP runtime. This follows the OpenBLAS build, not the package.
Still not reproduced: the "OMP: Error calungaproject#13" affinity assertion, on any
platform, and the x86_64-on-ARM emulation case, which cannot be exercised
on an x86_64 host. What reproduces is a segfault rather than the hang the
report describes; both are consistent with OpenMP runtime state not
surviving fork(), but only the segfault is claimed here.
After this merges and the builder rebuilds, affected packages (numpy,
scipy, opencv) must be rebuilt to pick up the pthreads OpenBLAS. The numpy
wheel will drop the bundled OpenMP runtime and match upstream's threading
model.
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
Close the remaining feature gaps between our source-built Pillow wheel and the
official PyPI wheel, verified by diffoscope/readelf/PIL.features comparison.
Tier 1 -- enable codecs using libs the builder already builds:
- libtiff: point CMake find_package at our libjpeg-turbo and zstd and force
-Djpeg=ON -Dzstd=ON, so libtiff.so gains JPEG and ZSTD codecs (previously
silently absent -- headers were not on the default include path).
- freetype: force -DFT_REQUIRE_PNG=ON so embedded-PNG (color bitmap / emoji)
glyphs work.
Tier 2 -- add brotli (WOFF2 fonts) and harfbuzz (complex-script shaping):
- New build-brotli.sh and build-harfbuzz.sh (source-built, hash-verified,
CMake -- meson is unavailable since CPython is built later).
- freetype gains Brotli (WOFF2) support and is built WITHOUT harfbuzz
(-DFT_DISABLE_HARFBUZZ=ON), matching PyPI's --with-harfbuzz=no.
- harfbuzz is built AFTER freetype with -DHB_HAVE_FREETYPE=ON so hb-ft.h and
the hb_ft_* symbols exist -- required by Pillow's vendored raqm. Building
freetype-without-harfbuzz then harfbuzz-with-freetype breaks the dependency
cycle (the same order Pillow's own wheel build uses).
- Expose both libs' headers/libs in the finalized image so Pillow's build
discovers harfbuzz/hb.h and libharfbuzz and links it into _imagingft.
Verified: our rebuilt Pillow now matches the PyPI wheel -- _imagingft DT_NEEDEDs
libharfbuzz (bundled), libtiff carries zstd+jpeg, freetype carries png+brotli,
and raqm complex-script shaping activates at runtime (fribidi-shim dlopens the
host libfribidi, exactly as PyPI's wheel does). Only remaining delta is zlib_ng
(performance-only, out of scope).
Note: fully closing the harfbuzz gap for Pillow also requires the package to be
built with config-settings "raqm=enable raqm=vendor fribidi=vendor" (as PyPI
does); this builder change is the prerequisite for that.
Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
The builder was pinned to OpenBLAS 0.3.31 as a point-in-time snapshot of the pypa/manylinux fork (commit 9d71ca3); there was no numpy-specific reason for that version. PyPI's numpy wheels now bundle 0.3.34, which Max Andersen's wheel-compare analysis (CALUNGA-401) flagged as a version drift. This is a routine 0.3.x patch bump (same soname/ABI). Hash verified against the upstream release tarball. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
…n, S3/GCS/Azure pyarrow's trusted-index wheel was missing 9 extension modules vs the PyPI wheel (S3/GCS/Azure filesystems, Flight, ORC, Parquet encryption, Substrait), because the Arrow C++ library it links was built without those components. A missing extension module is a hard parity-gate failure: the wheel's .cpython-*.so set must be a superset of PyPI's. Enable the corresponding Arrow C++ components in build-arrow.sh. pyarrow auto-detects components from the installed Arrow C++ config, so no pyarrow per-package setting change is needed. The cloud-storage SDKs are built BUNDLED by Arrow but link system libcurl (S3/GCS/Azure) and libxml2 (Azure), and pull in zlib; stage those source-built dev trees into the build_arrow Containerfile stage so CMake can find them. build-curl.sh previously exported only the runtime .so; make it additionally emit a full /manylinux-buildfs (headers, linkable .so, pkgconfig) and add a dedicated build_curl stage so build_arrow can link libcurl at compile time. Requires a full builder rebuild to validate the bundled Arrow + gRPC + aws-sdk-cpp + google-cloud-cpp + azure-sdk compile. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
PyPI's opencv-python wheels bundle FFmpeg, giving VideoCapture/VideoWriter (Video I/O) support. Our builder lacked any FFmpeg, so opencv built with FFMPEG: NO — silently dropping all video decode/encode, a feature the PyPI wheel has and ours does not. Add a build_ffmpeg stage that source-builds FFmpeg 8.0.3 (avcodec 62 series, matching the PyPI wheel) as LGPL shared libraries and stages them into both the build and runtime filesystems. Configure notes: - --disable-x86asm: the UBI8 base ships no assembler (aom is built the same way with ENABLE_NASM=OFF). Functionally complete, no hand-tuned x86 asm. - --disable-autodetect + --enable-zlib: never silently link a system library; only zlib is re-enabled, so the libs stay self-contained and gate-clean (libz.so.1 is manylinux_2_28-allowlisted). Verified: opencv-python-headless 4.14.0.94 rebuilt on the new image reports FFMPEG: YES (avcodec 62.11.103, avformat 62.3.103, avutil 60.8.103, swscale 9.1.103) and AVIF: YES via getBuildInformation(). The parity gate passes: the repaired wheel's external deps are a subset of PyPI's, all allowlisted. Assisted-by: Claude Opus 4.8 <noreply@anthropic.com>
smatula
force-pushed
the
add-builder-libs-freetype-webp
branch
from
September 15, 2026 14:40
3e8b070 to
847b704
Compare
Contributor
Author
Done — the ADR-0020 references are out of the commit messages as of The branch is also rebased onto So the pyarrow change is confirmed against a real build rather than a local rebuild: the bundled Arrow + gRPC + aws-sdk-cpp + google-cloud-cpp + azure-sdk compile works, which the commit message flagged as needing a full builder rebuild to validate. Image from that run if you want to poke at it: One heads-up for future runs: freetype is fetched from |
smatula
marked this pull request as ready for review
September 15, 2026 19:05
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 2 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="builder/build_scripts/build-curl.sh" line_range="21-27" />
<code_context>
# On UBI images, curl-devel is not available, so we need to build from source
if dnf list available curl-devel &>/dev/null; then
echo "skipping installation of ${CURL_ROOT} - using system curl-devel"
+ # Ensure the buildfs dir always exists so `COPY --from=build_curl
+ # /manylinux-buildfs /` succeeds even when we skip the source build
+ # (system curl-devel already provides headers/pkgconfig to consumers).
+ mkdir -p /manylinux-buildfs
exit 0
fi
</code_context>
<issue_to_address>
**issue (broader_impact):** When `curl-devel` is available in the package repositories, `build_curl` exits without building or staging any curl headers or libraries, so `build_arrow` links against the system curl-devel installation instead of the required source-built curl. The resulting Arrow build and runtime therefore depend on the host UBI curl version, defeating the reproducible source-built dependency contract.
**Triggers:** When the UBI repository exposes `curl-devel`, which is also installed into `build_base` by `install-build-packages.sh`.
**Suggested fix:** Either always build and stage the pinned curl for Arrow, or explicitly install and stage the system curl development files and make the Arrow/runtime dependency choice intentional and verifiable.
</issue_to_address>
### Comment 2
<location path="builder/build_scripts/build-libwebp.sh" line_range="44" />
<code_context>
+ -DWEBP_BUILD_IMG2WEBP=OFF \
+ -DWEBP_BUILD_VWEBP=OFF \
+ -DWEBP_BUILD_WEBPINFO=OFF \
+ -DWEBP_BUILD_WEBPMUX=OFF \
+ -DWEBP_BUILD_EXTRAS=OFF \
+ > /dev/null
</code_context>
<issue_to_address>
**issue (bug_risk):** The libwebp build disables `WEBP_BUILD_WEBPMUX`, so the finalized image has no libwebpmux support. Pillow can therefore build only basic WebP support and silently omits WebP animation/mux functionality that depends on WebPMux, leaving a feature gap despite the stated Pillow parity goal.
**Triggers:** When Pillow is rebuilt against this libwebp and callers use animated WebP or WebP metadata/mux operations.
**Suggested fix:** Build and stage WebPMux by removing `-DWEBP_BUILD_WEBPMUX=OFF` or enabling it explicitly, then ensure Pillow discovers the resulting libwebpmux package.
</issue_to_address>Sourcery assessment
Approval pending. 2 findings to address first.
Blocking findings: builder/build_scripts/build-curl.sh:27, builder/build_scripts/build-libwebp.sh:44
jvulgan
approved these changes
Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by Sourcery
Expand the manylinux native dependency stack to improve package parity and enable richer image, numerical, multimedia, and Arrow functionality.
New Features:
Bug Fixes:
Enhancements:
Build: