Harden core read contracts and release validation - #1
Merged
Conversation
Restrict supported Ray versions to final releases in >=2.49.2,<2.57, while allowing local build suffixes and rejecting pre/dev/post releases. Balance tablet splits into deterministic adjacent groups whose sizes differ by at most one. Separate normal MySQL EOF cleanup from consumer abort so unread SSCursor results are detached without draining, and share column and nullability validation across MySQL and Flight. Validate datasource configuration with cloudpickle before network access, and prevent filters, server error text, and sensitive exception chains from reaching repr, logs, or public errors. Validate release tags, package versions, and origin/master ancestry with tools/check_release.py. Add the core slow-integration profile and document Flight SQL and auto as experimental and Python 3.9 as an Alpha legacy target. Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
This PR hardens the core read path of
ray-dorisso that planning, streaming, termination, and error reporting follow one documented contract:>=2.49.2,<2.57before network access. Local build suffixes are accepted; pre-release, development, and post-release versions are rejected. The dependency bound inpyproject.tomland the runtime guard stay in sync.group_tablets()computes the target task count asmin(parallelism, ceil(tablet_count / tablet_size))and distributes adjacent tablet IDs with at most one-tablet difference between groups. Every tablet appears in exactly one split.DorisDatasourceverifies the actual configuration is cloudpickle-serializable before opening any Doris connection. Filters, server error text, and sensitive exception chains no longer reachrepr(), logs, or public errors.tools/check_release.pyverifies the tag matches the package version and the release commit belongs toorigin/masterbefore any artifact is built or published.transport="auto"are now documented as experimental, Python 3.9 as an Alpha legacy compatibility target, and the slow suite gains acoreprofile for tablet task-count and three-worker distribution changes.Why
The previous implementation derived a fixed group size with
max(tablet_size, ceil(count / parallelism)), which could create uneven groups and sometimes fewer tasks than the intended target; closing an activeSSCursoron abort drained the remaining result; and several error paths included server-provided text or chained exceptions that could expose internal detail.User-facing changes
Existing callers on supported final Ray releases require no API changes.
Reads may now produce a different number of more evenly balanced tasks.
Unsupported Ray versions fail during datasource construction. MySQL remains
the production-candidate transport; Flight SQL and
autoremain experimental,and Python 3.9 remains an Alpha legacy target.
How tested
git diff --check, strict Sphinx build, spelling, and link checks all passed.tools/check_release.pypasses for a matching tag onorigin/masterand fails for a mismatched tag or a commit outsidemaster; the verification ran locally and published no artifacts.This PR does not add TLS enforcement, credential environment references, failover, worker retry, or the HA/failure injection scenarios; those remain out of scope for the core contract.