Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 16 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,9 @@ print(client.get_status(job_id))

[Read the full quickstart](https://tributo.readthedocs.io/en/latest/getting-started/quickstart/).

For a sealed Ray runtime containing the first-party dependency closure, see
the [runtime image guide](https://tributo.readthedocs.io/en/latest/how-to/runtime-images/).

---

## Architecture
Expand Down Expand Up @@ -88,6 +91,9 @@ uv sync
# With XGBoost training + ONNX export
uv sync --extra training

# With BayesOpt search for Ray Tune
uv sync --extra tune

# With data formats (Lance / Iceberg)
uv sync --extra data

Expand Down Expand Up @@ -117,14 +123,15 @@ dialect or backend you use:
| Local/S3 Iceberg and Lance | Ray Data / Daft public readers | `tributo[data,data-daft]` | Alpha; real dual-engine Conformance |
| PostgreSQL structured table | Ray Data / Daft SQL readers | `tributo[postgresql,data-daft]` | Alpha; real PostgreSQL Conformance |
| HDFS Parquet/CSV | Ray Data + PyArrow Hadoop filesystem | Ray runtime with HDFS libraries | Adapter present; cluster gate pending |
| ClickHouse | independent local `daft-clickhouse` wheel | `tributo[clickhouse]` plus the connector wheel | Adapter present; package/infrastructure gates pending |
| Doris | independent `ray-doris` / local `daft-doris` wheel | `tributo[mysql]` or `tributo[doris-flight]` plus the connector wheel | Adapters present; package/infrastructure gates pending |
| ClickHouse | `daft-clickhouse==1.0` | `tributo[clickhouse]` or `uv sync --extra clickhouse` | Adapter only; the full image contains the v1.0 package, while real-database Conformance remains the support gate |
| Doris | `ray-doris==1.0` / `daft-doris==1.0` | `tributo[mysql]` or `tributo[doris-flight]` | Adapter only; Ray routes require `ray-doris`, Daft routes require `daft-doris`, and real-database Conformance remains the support gate |
| ORC / Hive external tables | no locked public reader path | — | Unsupported |

Provider/binding presence is not a support claim. ClickHouse, Doris, HDFS, and Hive are
reported as available only after their locked external dependencies and real
infrastructure gates pass. Tributo never installs optional providers or bindings at
runtime.
Provider/binding presence is not a support claim. The canonical full image
contains the locked v1.0 ClickHouse and Doris connector packages, but those
paths remain adapter-only until their real-database Conformance gates pass.
HDFS and Hive still require their own external dependency and infrastructure
gates. Tributo never installs optional providers or bindings at runtime.

---

Expand Down Expand Up @@ -204,6 +211,9 @@ uv run tributo serve streaming status
### Hyperparameter tuning with Ray Tune

Random search / BayesOpt with FIFO / ASHA / HyperBand schedulers.
BayesOpt requires the optional `tune` extra (`uv sync --extra tune` or
`python -m pip install "tributo[tune]"`); the full runtime image already
contains it.
Tune trials execute setup and fit only: they report the configured metric and
checkpoint without publishing production Bundles. After selecting parameters,
run the Trainer explicitly to publish the single production Bundle.
Expand Down
48 changes: 48 additions & 0 deletions ci/test-suites.json
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,15 @@
"patterns": ["src/**"],
"domains": ["code", "docs", "storage"]
},
{
"name": "runtime-image",
"patterns": [
"docker/**",
"tools/tributo-runtime-full.json",
"tools/build_tributo_image.py"
],
"domains": ["code", "storage"]
},
{
"name": "tests",
"patterns": ["tests/**", "scripts/**", "tools/**"],
Expand Down Expand Up @@ -160,6 +169,45 @@
"log_contract": "pytest terminal output; skips and missing JUnit evidence fail the suite",
"rationale": "Dedicated bounded shard for integration-named modules that exercise local contracts without external infrastructure."
},
{
"id": "runtime-image",
"owner": "core",
"domain": "runtime-image",
"tier": "manual_external",
"workflow": "external",
"owns_tests": true,
"entrypoint": ["bash", "scripts/run_runtime_image_it.sh"],
"args": [],
"extras": ["dev"],
"test_paths": [
"docker/tributo-runtime/Dockerfile",
"scripts/run_runtime_image_it.sh",
"tests/integrations/docker-compose.runtime-image.yml",
"tests/integrations/jobs/runtime_image_gate_job.py",
"tests/tools/test_build_tributo_image.py",
"tools/build_tributo_image.py",
"tools/tributo-runtime-full.json"
],
"trigger_domains": [],
"trigger_paths": [
"docker/tributo-runtime/**",
"scripts/run_runtime_image_it.sh",
"tests/integrations/docker-compose.runtime-image.yml",
"tests/integrations/jobs/runtime_image_gate_job.py",
"tests/integrations/runtime-profiles.json",
"tests/tools/test_build_tributo_image.py",
"tools/build_tributo_image.py",
"tools/tributo-runtime-full.json",
"pyproject.toml",
"uv.lock"
],
"requires": ["docker", "multi_node_ray", "ray_jobs"],
"budget_seconds": 1800,
"ci_allowed": false,
"forbid_skips": true,
"log_contract": "/tmp/tributo-runtime-image-it-*/compose.log and image attestations",
"rationale": "The pinned full image and its Ray Jobs worker closure require one isolated multi-node Docker gate."
},
{
"id": "documentation-api",
"owner": "docs",
Expand Down
118 changes: 118 additions & 0 deletions docker/tributo-runtime/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
ARG BASE_IMAGE=tributo-ray-base:2.55.1-py312
ARG UV_IMAGE=tributo-uv:0.11.23
ARG TRIBUTO_BASE_IMAGE=rayproject/ray:2.55.1-py312@sha256:911245f2478ad2e9f67ac13978dc2a75bcae0498b9f188b10bba703324b78379
ARG TRIBUTO_PLATFORM

FROM ${UV_IMAGE} AS uv
FROM ${BASE_IMAGE}

ARG BASE_IMAGE
ARG TRIBUTO_BASE_IMAGE
ARG TRIBUTO_PLATFORM
ARG TRIBUTO_MANIFEST_SHA256=unsealed
ARG TRIBUTO_RUNTIME_EXTRAS
ARG TRIBUTO_VERSION=1.0.0

LABEL org.opencontainers.image.title="tributo-runtime-full" \
org.opencontainers.image.version="${TRIBUTO_VERSION}" \
org.tributo.base-image="${TRIBUTO_BASE_IMAGE}" \
org.tributo.ray-version="2.55.1" \
org.tributo.python-version="3.12" \
org.tributo.platform="${TRIBUTO_PLATFORM}" \
org.tributo.runtime-extras="${TRIBUTO_RUNTIME_EXTRAS}" \
org.tributo.manifest-sha256="${TRIBUTO_MANIFEST_SHA256}"

COPY --from=uv /uv /uvx /usr/local/bin/

USER root
RUN mkdir -p /opt/tributo /opt/tributo-image /workspace && \
chown -R ray:users /opt/tributo /opt/tributo-image /workspace

# Some locked Alpha dependencies, notably econml on linux/arm64, do not ship
# a compatible wheel and must build their C extensions in the image.
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install --yes --no-install-recommends \
build-essential && \
rm -rf /var/lib/apt/lists/*

USER ray

ENV PATH="/opt/tributo/.venv/bin:${PATH}" \
PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
UV_HTTP_RETRIES=10 \
UV_HTTP_TIMEOUT=600 \
UV_LINK_MODE=copy \
UV_PROJECT_ENVIRONMENT=/opt/tributo/.venv

COPY --chown=ray:users pyproject.toml uv.lock README.md /opt/tributo/
COPY --chown=ray:users src /opt/tributo/src

WORKDIR /opt/tributo

RUN --mount=type=cache,target=/home/ray/.cache/uv,uid=1000,gid=100 \
env -u HTTP_PROXY -u HTTPS_PROXY -u ALL_PROXY \
-u http_proxy -u https_proxy -u all_proxy \
uv sync \
--extra data \
--extra data-daft \
--extra vector-index \
--extra postgresql \
--extra clickhouse \
--extra mysql \
--extra doris-flight \
--extra s3 \
--extra model-export \
--extra model-export-torch \
--extra hf \
--extra model-export-hf \
--extra training \
--extra tune \
--extra explainability \
--extra identity \
--extra streaming \
--extra grpc \
--extra registry \
--extra graph \
--extra causal \
--extra streaming-inference \
--no-default-groups \
--no-dev \
--locked

# Ray runtime_env may need pip when an algorithm uses the offline wheelhouse
# distribution mode. Bootstrap it in the image without contacting an index.
RUN python -m ensurepip --upgrade

# The builder always supplies this named context, including an empty directory
# when no external connector wheels were requested.
COPY --from=external-wheelhouse / /opt/tributo-image/external-wheelhouse/

# Keep the manifest value in a RUN instruction as well as an OCI label. This
# makes the sealing value part of the BuildKit cache key for the second build.
RUN printf '%s\n' "${TRIBUTO_MANIFEST_SHA256}" > /opt/tributo-image/manifest-seal

RUN if find /opt/tributo-image/external-wheelhouse -type f -name '*.whl' -print -quit | grep -q .; then \
python -m pip install --no-index --no-deps \
/opt/tributo-image/external-wheelhouse/*.whl; \
fi && \
pip_check_output=""; \
pip_check_status=0; \
pip_check_output="$(python -m pip check 2>&1)" || pip_check_status=$?; \
if [ "${pip_check_status}" -ne 0 ]; then \
if [ "${TRIBUTO_PLATFORM}" != "linux/arm64" ] || \
! printf '%s\n' "${pip_check_output}" | grep -Eq \
'^nvidia-cusparselt-cu13 0[.]8[.]1 is not supported on this platform[[:space:]]*$'; then \
printf '%s\n' "${pip_check_output}"; \
exit "${pip_check_status}"; \
fi; \
fi

COPY --chown=ray:users tools/generate_distributions.py /opt/tributo-image/

RUN python /opt/tributo-image/generate_distributions.py

WORKDIR /workspace

# Keep an explicit final non-root boundary if later instructions add a root step.
USER ray
8 changes: 8 additions & 0 deletions docker/tributo-runtime/Dockerfile.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
**
!pyproject.toml
!uv.lock
!README.md
!src/
!src/**
!tools/
!tools/generate_distributions.py
12 changes: 12 additions & 0 deletions docs/STABILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,18 @@ This page provides module-level guidance and deprecation notes.
| `tributo.exceptions` — `EngineNotAvailableError` | `alpha` | Candidate bounded-ingestion error |
| `tributo.cli` | `beta` | Command-line interface |

### Runtime image tooling

The image builder and emitted files are repository tooling rather than public
Python API symbols. Their contract is intentionally Alpha and is
covered by the external `runtime-image` suite.

| Surface | Level | Notes |
|--------|-------|-------|
| `tools/build_tributo_image.py` | `alpha` | JSON-only Buildx builder for the pinned image validated for CPU execution by default, with explicit `linux/amd64`/`linux/arm64` targeting; it performs dependency-closure discovery, manifest sealing, and fail-closed import checks |
| `tools/tributo-runtime-full.json` | `alpha` | Multi-architecture pinned Ray/uv image references, native-platform default, complete first-party runtime-extra closure including locked v1.0 ClickHouse/Doris connectors, and optional external wheelhouse support |
| `manifest.json` / `image-profile.json` | `alpha` | Build attestations consumed for immutable image selection and algorithm artifact compatibility; not a registry or deployment API |

### Training (tributo.training.*)

Callbacks without a public `failure_policy` are best-effort in every normal
Expand Down
8 changes: 5 additions & 3 deletions docs/architecture/call-chain-inventory.md
Original file line number Diff line number Diff line change
Expand Up @@ -279,9 +279,11 @@ credential-safe descriptor validation and atomic registration
WriteBinding selection; native dependency import occurs at factory/execute
```

Selected optional integrations (`ray-doris`, `daft-doris`, `daft-clickhouse`) also have
thin built-in descriptors and explicit install diagnostics. Their adapters are
not support claims until their external packages and infrastructure gates pass.
Selected optional integrations (`ray-doris==1.0`, `daft-doris==1.0`,
`daft-clickhouse==1.0`) also have thin built-in descriptors and explicit install
diagnostics. The canonical full runtime locks these packages into the image,
but their adapters are not support claims until database infrastructure gates
pass. Ray and Daft routes remain explicit and are not interchangeable.

---

Expand Down
7 changes: 4 additions & 3 deletions docs/data/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,14 +56,15 @@ selects `binding_id` explicitly.
| Local/S3 Lance | Native reader | Native reader | Verified |
| PostgreSQL structured table | Native SQL reader | Native SQL reader | Verified |
| HDFS Parquet/CSV | Native reader with PyArrow HDFS | No locked public reader | Adapted; cluster gate pending |
| ClickHouse | No selected Binding | External `daft-clickhouse` wheel | Adapter only; external package and database gates pending |
| Doris | `ray-doris` | External `daft-doris` wheel | Adapter only; external package and database gates pending |
| ClickHouse | No selected Binding | `daft-clickhouse==1.0` | Adapter only; install with `tributo[clickhouse]` or `uv sync --extra clickhouse`; real-database Conformance remains the support gate |
| Doris | `ray-doris==1.0` | `daft-doris==1.0` | Adapter only; Ray routes use `ray-doris`, Daft routes use `daft-doris`, and the full runtime image contains both v1.0 packages |
| ORC or Hive external table | No locked public reader | No locked public reader | Unsupported, fail-closed |

“Verified” means the current combination has semantic Conformance and real
storage or database evidence. It does not turn every engine/source combination
into a supported path. See the [support matrix](../reference/support-matrix.md)
for the exact boundary.
for the exact boundary. Daft and Ray Doris are explicit engine routes; a
missing `ray-doris` package does not silently fall back to Daft.

Credentials belong to runtime configuration. They must not appear in dataset
identifiers, logical plans, receipts, logs, or public errors. Bounded providers
Expand Down
1 change: 1 addition & 0 deletions docs/getting-started/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ Choose one execution boundary and follow its guide from start to finish.

installation
quickstart
../how-to/runtime-images
```

- Use the local quickstart to validate a formal algorithm and Bundle without
Expand Down
13 changes: 12 additions & 1 deletion docs/getting-started/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,11 @@ Pydantic, ONNX Runtime, PyArrow, pandas, and S3 filesystem support.
| Ray Data table formats | `python -m pip install "tributo[data]"` |
| Daft ingestion | `python -m pip install "tributo[data,data-daft]"` |
| PostgreSQL ingestion | `python -m pip install "tributo[postgresql]"` |
| ClickHouse via Daft | `python -m pip install "tributo[clickhouse]"` |
| Doris via Daft/Ray Data | `python -m pip install "tributo[mysql]"` |
| Doris Flight via Daft/Ray Data | `python -m pip install "tributo[doris-flight]"` |
| Distributed training | `python -m pip install "tributo[training]"` |
| BayesOpt search for Ray Tune | `python -m pip install "tributo[tune]"` |
| Explainability | `python -m pip install "tributo[explainability]"` |
| Lance vector indexing | `python -m pip install "tributo[vector-index]"` |
| Torch model export | `python -m pip install "tributo[model-export-torch]"` |
Expand All @@ -30,7 +34,14 @@ Pydantic, ONNX Runtime, PyArrow, pandas, and S3 filesystem support.

An extra installs dependencies. It does not turn a protocol, adapter, or
reserved problem type into a verified implementation. Check the
[support matrix](../reference/support-matrix.md) before deployment.
[support matrix](../reference/support-matrix.md) before deployment. Ray Tune
itself is included by the core Ray dependency; the `tune` extra adds the
optional BayesOpt search implementation. The `clickhouse` extra installs
`daft-clickhouse==1.0`; `mysql` installs `daft-doris==1.0` and
`ray-doris==1.0` for their explicit engine routes, while `doris-flight` adds
their Flight dependencies. The equivalent uv commands are
`uv sync --extra clickhouse`, `uv sync --extra mysql`, and
`uv sync --extra doris-flight`.

## Prepare a source checkout

Expand Down
Loading
Loading