Skip to content

Harden core read contracts and release validation - #1

Merged
jiangxt2 merged 1 commit into
masterfrom
fix/core-read-hardening
Aug 11, 2026
Merged

Harden core read contracts and release validation#1
jiangxt2 merged 1 commit into
masterfrom
fix/core-read-hardening

Conversation

@jiangxt2

Copy link
Copy Markdown
Owner

What changes

This PR hardens the core read path of ray-doris so that planning, streaming, termination, and error reporting follow one documented contract:

  • Ray compatibility window. Construction now rejects any installed Ray outside the final releases >=2.49.2,<2.57 before network access. Local build suffixes are accepted; pre-release, development, and post-release versions are rejected. The dependency bound in pyproject.toml and the runtime guard stay in sync.
  • Balanced tablet splits. group_tablets() computes the target task count as min(parallelism, ceil(tablet_count / tablet_size)) and distributes adjacent tablet IDs with at most one-tablet difference between groups. Every tablet appears in exactly one split.
  • MySQL reader lifecycle. Normal EOF closes the cursor and then the connection. Consumer abort closes the connection first and detaches the active server-side cursor result so PyMySQL never drains the unread result on a closed socket. Consumer-visible generator termination cleans up exactly once and never masks the primary exception.
  • Shared schema contract. MySQL and Flight now use the same result-column validation and non-nullable NULL rejection through the same public error classification.
  • Serialization and redaction. DorisDatasource verifies the actual configuration is cloudpickle-serializable before opening any Doris connection. Filters, server error text, and sensitive exception chains no longer reach repr(), logs, or public errors.
  • Release validation. A local tools/check_release.py verifies the tag matches the package version and the release commit belongs to origin/master before any artifact is built or published.
  • Documentation. Flight SQL and transport="auto" are now documented as experimental, Python 3.9 as an Alpha legacy compatibility target, and the slow suite gains a core profile for tablet task-count and three-worker distribution changes.

Why

The previous implementation derived a fixed group size with max(tablet_size, ceil(count / parallelism)), which could create uneven groups and sometimes fewer tasks than the intended target; closing an active SSCursor on abort drained the remaining result; and several error paths included server-provided text or chained exceptions that could expose internal detail.

User-facing changes

Existing callers on supported final Ray releases require no API changes.
Reads may now produce a different number of more evenly balanced tasks.
Unsupported Ray versions fail during datasource construction. MySQL remains
the production-candidate transport; Flight SQL and auto remain experimental,
and Python 3.9 remains an Alpha legacy target.

How tested

  • Unit suites: 155 passed on each of Python 3.9 / Ray 2.49.2, Python 3.10 / Ray 2.55.1, and Python 3.12 / Ray 2.56.1 (93% coverage on the primary baseline).
  • Required Doris 4.0.6 integration suite: 18 passed; the real cancellation test confirms zero unbuffered-result drain calls.
  • Core distributed slow profile: 1 passed, covering 48 tablets, 10,000 rows, and three independent Ray workers.
  • Static and documentation gates: Ruff, mypy, pre-commit, git diff --check, strict Sphinx build, spelling, and link checks all passed.
  • Packaging: final wheel/sdist build, Twine check, clean install, and import all passed.
  • Release verification: tools/check_release.py passes for a matching tag on origin/master and fails for a mismatched tag or a commit outside master; the verification ran locally and published no artifacts.

This PR does not add TLS enforcement, credential environment references, failover, worker retry, or the HA/failure injection scenarios; those remain out of scope for the core contract.

Restrict supported Ray versions to final releases in >=2.49.2,<2.57,
while allowing local build suffixes and rejecting pre/dev/post releases.

Balance tablet splits into deterministic adjacent groups whose sizes
differ by at most one. Separate normal MySQL EOF cleanup from consumer
abort so unread SSCursor results are detached without draining, and
share column and nullability validation across MySQL and Flight.

Validate datasource configuration with cloudpickle before network
access, and prevent filters, server error text, and sensitive exception
chains from reaching repr, logs, or public errors.

Validate release tags, package versions, and origin/master ancestry with
tools/check_release.py. Add the core slow-integration profile and
document Flight SQL and auto as experimental and Python 3.9 as an Alpha
legacy target.

Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>
@jiangxt2
jiangxt2 merged commit cb4cd48 into master Aug 11, 2026
7 checks passed
@jiangxt2
jiangxt2 deleted the fix/core-read-hardening branch August 11, 2026 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant