Garage does not get configured.

[alanscodelog]

Describe the bug
Garage buckets do not get configured. I tried debugging a copy of the service about locally but am still stumped. I think the post ready configure task isn't running for some reason... but the ready probe definitely fires. I tried setting "after" on the configure task instead of as a before on the process, but still nothing... idk how to better debug if the dependencies between tasks is getting correctly interpreted...

I had filed #2786 before.

To reproduce
I would expect just this to work.

services.garage.enable = true;
services.garage.buckets = [ "assets" ]

devenv up in one terminal, in another in a devenv shell, run garage -c $GARAGE_CONFIG_FILE bucket list
It errors with Error: ListBuckets returned InternalError (500): Internal error: Layout not ready when it should be ready (the configure task should be applying the layout afaik).

Version
devenv 2.1.1+b7d73aa (x86_64-linux)

Logs:

``` 2026-05-19T23:02:23.658392Z INFO garage::server: Loading configuration from /nix/store/fy8bcsd91ccfnra4ns6xcdfarrf5c0bc-garage.toml... 2026-05-19T23:02:23.658750Z INFO garage::server: Initializing Garage main data store... 2026-05-19T23:02:23.658807Z INFO garage_model::garage: Opening database... 2026-05-19T23:02:23.658829Z INFO garage_db::lmdb_adapter: Opening LMDB database at: /home/alan/code/=clones/garage-bug/.devenv/state/garage/meta/db.lmdb 2026-05-19T23:02:23.659204Z INFO garage_model::garage: Initializing RPC... 2026-05-19T23:02:23.659245Z INFO garage_model::garage: Initialize background variable system... 2026-05-19T23:02:23.659250Z INFO garage_model::garage: Initialize membership management system... 2026-05-19T23:02:23.659275Z INFO garage_rpc::system: Generating new node key pair. 2026-05-19T23:02:23.659443Z INFO garage_rpc::system: Node ID of this node: b88a8e18fe0ea8f4 2026-05-19T23:02:23.659543Z INFO garage_rpc::layout::manager: No valid previous cluster layout stored (IO error: No such file or directory (os error 2)), starting fresh. 2026-05-19T23:02:23.659584Z INFO garage_rpc::layout::helper: ack_until updated to 0 2026-05-19T23:02:23.659642Z INFO garage_model::garage: Initialize block manager... 2026-05-19T23:02:23.660040Z INFO garage_model::garage: Initialize admin_token_table... 2026-05-19T23:02:23.660254Z INFO garage_model::garage: Initialize bucket_table... 2026-05-19T23:02:23.660374Z INFO garage_model::garage: Initialize bucket_alias_table... 2026-05-19T23:02:23.660479Z INFO garage_model::garage: Initialize key_table_table... 2026-05-19T23:02:23.660584Z INFO garage_model::garage: Initialize block_ref_table... 2026-05-19T23:02:23.660706Z INFO garage_model::garage: Initialize version_table... 2026-05-19T23:02:23.660818Z INFO garage_model::garage: Initialize multipart upload counter table... 2026-05-19T23:02:23.660935Z INFO garage_model::garage: Initialize multipart upload table... 2026-05-19T23:02:23.661040Z INFO garage_model::garage: Initialize object counter table... 2026-05-19T23:02:23.661153Z INFO garage_model::garage: Initialize object_table... 2026-05-19T23:02:23.661311Z INFO garage_model::garage: Load lifecycle worker state... 2026-05-19T23:02:23.661336Z INFO garage_model::garage: Initialize K2V counter table... 2026-05-19T23:02:23.661487Z INFO garage_model::garage: Initialize K2V subscription manager... 2026-05-19T23:02:23.661491Z INFO garage_model::garage: Initialize K2V item table... 2026-05-19T23:02:23.661626Z INFO garage_model::garage: Initialize K2V RPC handler... 2026-05-19T23:02:23.661666Z INFO garage::server: Initializing background runner... 2026-05-19T23:02:23.661782Z INFO garage::server: Spawning Garage workers... 2026-05-19T23:02:23.661829Z INFO garage_model::s3::lifecycle_worker: Starting lifecycle worker for 2026-05-19 2026-05-19T23:02:23.661836Z INFO garage::server: Initialize Admin API server and metrics collector... 2026-05-19T23:02:23.662279Z INFO garage_model::s3::lifecycle_worker: Lifecycle worker finished for 2026-05-19, objects expired: 0, mpu aborted: 0 2026-05-19T23:02:23.710989Z INFO garage::server: Launching internal Garage cluster communications... 2026-05-19T23:02:23.711068Z INFO garage::server: Initializing S3 API server... 2026-05-19T23:02:23.711084Z INFO garage::server: Launching Admin API server... 2026-05-19T23:02:23.711221Z INFO garage_net::netapp: Listening on [::]:3902 2026-05-19T23:02:23.711383Z INFO garage_api_common::generic_server: S3 API server listening on http://127.0.0.1:3900 2026-05-19T23:02:23.711404Z INFO garage_api_common::generic_server: Admin API server listening on http://127.0.0.1:3903 Exec probe succeeded - process ready 2026-05-19T23:02:33.672933Z INFO garage_api_common::generic_server: 127.0.0.1:60280 GET /v1/health 2026-05-19T23:02:52.032352Z INFO garage_net::netapp: Accepted connection from da118d1b1ec2d7d9 at [::ffff:127.0.0.1]:55112 2026-05-19T23:02:52.074068Z INFO garage_api_admin::api_server: Proxied admin API request: ListBuckets 2026-05-19T23:02:52.074563Z INFO garage_net::netapp: Connection from da118d1b1ec2d7d9 closed ```

[linusfri]

It seems like there is a bigger problem related to tasks. I have the same issues as you but with other services.

[alanscodelog]

@linusfri I was starting to suspect. Next thing I was going to try was creating a service from scratch. Looking back I don't think I noticed because everything else was setup in my project already so they didnt need configuration.

Reading #2843 maybe I misinterpreted the docs but they give me the impression one can do a task dependent on a service.

[domenkozar]

Thanks for the detailed report and debugging, that pointed right at it. Confirmed and reproduced.

Root cause: devenv up defaults to --mode before, which starts each process plus the things it depends on upstream. The garage module wires its post ready setup (apply the cluster layout, create the buckets) as a task that runs after the process via processes.garage.before = [ "devenv:garage:configure" ], which is downstream. So under devenv up that configure step is never scheduled, the layout is never applied, and you get Layout not ready with no buckets. Your hunch that the configure step was not running was correct.

This also explains why it looked fine in our tests: devenv test runs with --mode all, which does include downstream tasks, so CI was exercising a different scheduling path than real devenv up. The same latent issue affects services.mysql initial databases and users.

Workaround for now (verified to create the bucket):

devenv up --mode all

That schedules the downstream configure step, so the layout gets applied and assets is created.

Possible ways to fix it

The constraint is that whatever we do must not bring back #2721, where devenv up postgres also started unrelated processes that merely declared after = postgres.

1. Bare up uses all, named up <name> stays before. For a no argument devenv up every process is already a root, so all cannot over start anything, and Processes: unexpected dependencies running #2721 only happens with a named process. Smallest possible change, fixes the reported case, but does not fix devenv up garage (naming a single process).
2. Include downstream setup tasks under before, stop at processes. Schedule downstream oneshot tasks of a process (the configure step) but never downstream processes. Fixes garage and mysql for the usual conventions, but it is a heuristic based on task vs process type, so it has edge cases.
3. Membership by ownership (systemd style). Record who declared each dependency. devenv up X then runs what X itself declared in its own before/after, and never something that merely declared a dependency on X. One rule that covers garage, mysql, and Processes: unexpected dependencies running #2721 correctly, and lets up and test share the same scheduling so this class of bug cannot hide again. This is the direction we prefer for the real fix.
4. Model the setup as a process (the approach in bug: improve garage service and startup + tests #2846), with the service ready probe waiting for setup to finish. Works without a scheduler change and adds nice extras for garage, but every service has to adopt the pattern individually.

Current thinking: do 1 as a quick correctness fix, and land 3 as the proper long term fix. #2846 remains a good improvement to the garage module itself on top of that.