Skip to content

combine overlapping suggestions #181

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
saikonen wants to merge 1 commit into
base: master
Choose a base branch
from
Open

combine overlapping suggestions #181

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 1 addition & 29 deletions docs/internals/gsoc-2026.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -385,34 +385,6 @@

---

## Metadata service request improvements

**Difficulty:** Easy

**Duration:** 175 hours (Medium project)

**Technologies:** Python, Docker, PostgreSQL

**Mentors:** Sakari

### Description

The current metadata service for Metaflow does not provide paginated responses for its endpoints. Introducing pagination is required for some backfill-patterns that need to iterate over existing resources, in order to keep the resource requirements of these operations limited. Currently the payloads returned over the wire are not capped, and can be significant in size with more established deployments.

Resources can also be filtered by tags in the Metaflow client. This is currently still happening in-memory over the response payload, as the API does not support filtering. Being able to apply filters on the request level would also cut down on the resource use.

### Goals

1. Being able to return filtered, paginated responses from metadata-service

2. Backwards compatibility with older Metaflow clients that do not support pagination. Possibly by feature-gating via client version in request headers.

3. Handling paginated responses in Metaflow client

4. handling filtering by tag in Metaflow client on the request level, not in-memory.

---

## Jupyter-Native Metaflow

**Difficulty:** Medium
Expand Down Expand Up @@ -522,15 +494,15 @@

### Description

Metaflow supports various compute backends for executing steps remotely: `@kubernetes`, `@batch` (AWS Batch), and community extensions like [`@slurm`](https://github.com/outerbounds/metaflow-slurm) for HPC clusters. However, many organizations use [HashiCorp Nomad](https://www.nomadproject.io/) as their workload orchestrator — a lightweight alternative to Kubernetes that's simpler to operate and supports diverse workload types (containers, VMs, binaries).

Check warning on line 497 in docs/internals/gsoc-2026.md

View workflow job for this annotation

GitHub Actions / Run linters 

Line length: Expected: 100; Actual: 459

Nomad is particularly popular in organizations already using HashiCorp's stack (Vault, Consul) and in edge computing scenarios where Kubernetes' complexity is overkill. Despite this, there's currently no way to run Metaflow steps on Nomad clusters.

Check warning on line 499 in docs/internals/gsoc-2026.md

View workflow job for this annotation

GitHub Actions / Run linters 

Line length: Expected: 100; Actual: 248

This project aims to implement a `@nomad` decorator that executes Metaflow steps as Nomad jobs, bringing Metaflow's workflow capabilities to the Nomad ecosystem. The [`@slurm` extension](https://github.com/outerbounds/metaflow-slurm) provides a reference implementation for integrating custom compute backends.

Check warning on line 501 in docs/internals/gsoc-2026.md

View workflow job for this annotation

GitHub Actions / Run linters 

Line length: Expected: 100; Actual: 310

### Goals

1. **`@nomad` decorator** - Execute Metaflow steps as Nomad batch jobs with basic resource configuration (CPU, memory).

Check warning on line 505 in docs/internals/gsoc-2026.md

View workflow job for this annotation

GitHub Actions / Run linters 

Line length: Expected: 100; Actual: 119
2. **Docker task driver support** - Run steps in Docker containers, similar to how `@kubernetes` and `@batch` work.
3. **Job submission and monitoring** - Submit jobs to Nomad, poll for completion, and retrieve exit codes.
4. **Log streaming** - Capture and display stdout/stderr from Nomad allocations in the Metaflow CLI.
Expand Down Expand Up @@ -590,7 +562,7 @@

**Technologies:** Python, Metaflow Client API, Metaflow Metadata Service

**Mentors:** Valay Dave
**Mentors:** Valay Dave, Sakari Ikonen

### Description

Expand Down
Loading