Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
117 changes: 117 additions & 0 deletions docs/superpowers/plans/2026-05-18-pyg-uv-flat-index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
# PyG uv Flat Index Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Make generated uv-based projects with PyTorch Geometric resolve PyG wheel pages correctly by emitting a flat uv index and binding PyG extension packages to that explicit source.

**Architecture:** Keep the current template structure intact and change only the generated uv configuration for PyG: mark the wheel page as a flat index and map PyG extension packages to that explicit index. Validate by regenerating projects that enable PyG and rerunning dependency resolution.

**Tech Stack:** Copier, Jinja2, uv, PyTorch Geometric

---

### Task 1: Add flat index metadata and source mappings for PyG uv packages

**Files:**
- Modify: `pyproject.toml.jinja`
- Test: generated `pyproject.toml` in a temporary copied project

- [ ] **Step 1: Write the failing test**

Use the existing reproduction as the failing test: generate a project with uv + PyG enabled and run `uv lock`, expecting failure before the template change.

```bash
uvx copier copy --trust --defaults \
--data project_name="tmp-pyg-flat" \
--data package_name="tmp_pyg_flat" \
--data python_version="3.12" \
--data package_manager="uv" \
--data pytorch_cuda_preset="pytorch-2.8.0-cuda-12.6" \
--data use_pytorch_geometric=true \
--data use_lightning=true \
--data use_hydra=true \
--data logger_choice="none" \
. /tmp/tmp-pyg-flat
```

- [ ] **Step 2: Run test to verify it fails**

Run:

```bash
rtk uv lock --project /tmp/tmp-pyg-flat
```

Expected: dependency resolution fails while probing the PyG wheel page or while resolving PyG extension wheels, often by falling back to a source build for `torch-cluster`.

- [ ] **Step 3: Write minimal implementation**

Update the PyG-specific uv configuration in `pyproject.toml.jinja`.

```toml
[[tool.uv.index]]
name = "pytorch-geometric"
url = "https://data.pyg.org/whl/torch-{{ pytorch_version }}+cu{{ cuda_version | replace('.', '') }}.html"
format = "flat"
explicit = true

[tool.uv.sources]
torch-scatter = { index = "pytorch-geometric" }
torch-sparse = { index = "pytorch-geometric" }
torch-cluster = { index = "pytorch-geometric" }
```

- [ ] **Step 4: Run test to verify it passes**

Regenerate the same project, then run:

```bash
rtk uv lock --project /tmp/tmp-pyg-flat
```

Expected: `uv lock` succeeds and resolves PyG dependencies.

- [ ] **Step 5: Commit**

```bash
git add pyproject.toml.jinja
git commit -m "fix: mark PyG uv source as flat index"
```

### Task 2: Sanity-check generated configuration

**Files:**
- Modify: none
- Test: generated `/tmp/tmp-pyg-flat/pyproject.toml`

- [ ] **Step 1: Write the failing test**

Check that the generated file lacks the required flat format and source mappings before the change.

```bash
grep -n 'format = "flat"' /tmp/tmp-pyg-flat/pyproject.toml
grep -n '\[tool\.uv\.sources\]\|torch-scatter\|torch-sparse\|torch-cluster' /tmp/tmp-pyg-flat/pyproject.toml
```

- [ ] **Step 2: Run test to verify it fails**

Expected: no matching lines before the template fix.

- [ ] **Step 3: Write minimal implementation**

No additional implementation beyond Task 1.

- [ ] **Step 4: Run test to verify it passes**

Regenerate the project and run:

```bash
grep -n 'format = "flat"' /tmp/tmp-pyg-flat/pyproject.toml
grep -n '\[tool\.uv\.sources\]\|torch-scatter\|torch-sparse\|torch-cluster' /tmp/tmp-pyg-flat/pyproject.toml
```

Expected: the generated PyG uv configuration contains `format = "flat"` and the three source mappings.

- [ ] **Step 5: Commit**

No additional commit beyond Task 1 unless follow-up changes are needed.
36 changes: 36 additions & 0 deletions docs/superpowers/specs/2026-05-18-pyg-uv-flat-index-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# PyG uv Flat Index Design

## Goal
Fix generated uv-based projects that enable PyTorch Geometric so `uv lock` can resolve PyG extension wheels from `data.pyg.org` without failing on index probing or falling back to source builds.

## Problem
The template currently emits a PyG-specific `[[tool.uv.index]]` entry pointing at a `data.pyg.org` HTML wheel listing, but it does not mark that source as a flat index and it does not bind PyG extension packages to that explicit index. `uv` treats normal indexes as Simple API-style repositories unless told otherwise, and indexes marked `explicit = true` are only used for packages explicitly mapped to them. As a result, generated projects can fail during `uv lock` with 403 errors while probing the PyG URL or by attempting to build packages like `torch-cluster` from source instead of resolving wheels.

## Chosen Approach
Update the generated uv configuration for PyG by adding `format = "flat"` to the `pytorch-geometric` `[[tool.uv.index]]` block in `pyproject.toml.jinja` and add `[tool.uv.sources]` mappings for `torch-scatter`, `torch-sparse`, and `torch-cluster` so those packages resolve from the explicit PyG index.

## Why This Approach
- It matches `uv`'s documented handling of find-links style package pages.
- It matches a known-good working configuration already validated in another project.
- It is the smallest possible template change that addresses the root cause.
- It preserves the current package-selection model and does not require changing Copier questions or post-generation installation steps.

## Scope
### In scope
- Add `format = "flat"` to the generated PyG uv index block.
- Add `[tool.uv.sources]` mappings for `torch-scatter`, `torch-sparse`, and `torch-cluster`.
- Regenerate uv + PyG projects and verify `uv lock` succeeds.

### Out of scope
- Changing available PyTorch/CUDA presets.
- Reworking Pixi behavior.
- Replacing the uv index approach with custom install commands.

## Testing
1. Generate projects with `package_manager=uv` and `use_pytorch_geometric=true`.
2. Confirm the generated `pyproject.toml` includes both `format = "flat"` and `[tool.uv.sources]` mappings for the PyG extension packages.
3. Run `uv lock` in the generated projects.
4. Confirm dependency resolution succeeds for PyG packages such as `torch-cluster` without source build fallback.

## Risks
The only meaningful risk is that some PyG wheel pages may still have upstream availability gaps for specific torch/CUDA combinations. That would be a separate compatibility issue, not a misconfiguration of `uv` index format.
6 changes: 6 additions & 0 deletions pyproject.toml.jinja
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,13 @@ explicit = true
[[tool.uv.index]]
name = "pytorch-geometric"
url = "https://data.pyg.org/whl/torch-{{ pytorch_version }}+cu{{ cuda_version | replace('.', '') }}.html"
format = "flat"
explicit = true

[tool.uv.sources]
torch-scatter = { index = "pytorch-geometric" }
torch-sparse = { index = "pytorch-geometric" }
torch-cluster = { index = "pytorch-geometric" }
{% endif -%}

# ============================================================================
Expand Down
Loading