switch to upstream hydra, new queue runner

[zowoq]

Need x86_64-freebsd support for the hydra builder before we can do this. It builds but haven't tested it yet. Going to wait for h.n.o to deploy the queue runner and for hydra upstream to stabilise a bit as rebuilding the freebsd vm image isn't cheap for the cachix cache.

[zowoq]

@Mic92 With tribuchet and the new hydra queue runner we'd need to have both deployed on each machine, I was wondering if running a single hydra queue builder with all of our supported systems on the tribuchet hub might work? e.g.

graph TB

    subgraph BUILD03["build03"]
        nixbot["nixbot"]
        hydra["hydra"]
        HQR["hydra-queue-runner"]
        HQB["hydra-queue-builder
        - aarch64-linux
        - aarch64-darwin
        - x86_64-freebsd
        - x86_64-linux
        "]
        TH["tribuchet-hub"]
        build03["x86_64-linux"]
        ND["nix-daemon"]

        nixbot <--> ND
        ND <--> build03
        hydra <--> HQR
        HQR <--> HQB
        HQB <--> ND
        ND <--> TH
    end

    subgraph build04["build04"]
        TW-B04["tribuchet-worker"]
        aarch64-linux["aarch64-linux"]
    end

    subgraph darwin02["darwin02"]
        TW-D02["tribuchet-worker"]
        aarch64-darwin["aarch64-darwin"]
    end

    subgraph build03-freebsd["build03-freebsd"]
        TW-B03F["tribuchet-worker"]
        x86_64-freebsd["x86_64-freebsd"]
    end

    TH <--> TW-B04
    TH <--> TW-D02
    TW-B04 <--> aarch64-linux
    TW-D02 <--> aarch64-darwin
    TH <--> TW-B03F
    TW-B03F <--> x86_64-freebsd

Loading

[Mic92]

When queue builder and runner are collocated on the same machine, they should skip most transfers. I haven't tested that scenario but if it's not already a test case maybe it should be.

I would maybe wait a little bit longer with deploying tribuchet. It's getting more stable now but I am still discovering bugs through my own CI. I don't have a MacOS instance outside of the github runner test. I planning also on adding a different remote builder client so it doesn't have to displace all builders on a machine and can be more selectively enabled in a CI which would allow to collocate it better with queue builder.

Otherwise it's probably worth thinking of what frontent features are still needed to make nixbot a hydra replacement at least for the non NixOS build infra usecase. The UI scales very well with larger build graphs. It might need an option to show build regressions as a differential few and switch to fully control all derivations scheduled in order to cache failures on all levels - also I am still thinking where the build scheduler should live since tribuchet could also take over this task. I stress tested the build scheduler and it would more likely bottlenecked on external components like tribuchet than it's ability to push out a build graph.