v3 ideas, remove the ".gomake" folder

[brad-jones]

While gomake by and large solves the issues I set out to solve there are some new ideas I have that I would like to explore.

• The .gomake folder name in some cases (IDEs / gopls) tends to cause issues because of it's "hidden" name
• Starting a brand new project it would be nice to create just 2 files, go.mod & makefile.go in the root of the project.
• Instead of using the main package we have the user create their makefile with whatever package seems relevant. It could be makefile or projectfoo, or microservicea, etc... this has the added benefit of allowing any exported members to be used in other "makefiles" and no errors returned from IDEs when makefile_generated.go does not exist.
• Now instead of having one big .gomake folder that does everything for an entire monorepo we have many smaller makefile.go files
• Similar to mage we will store makefile_generated.go and the built binary in the users home directory out of the way somewhere

[StevenACoffman]

Just curious, but can you describe the differences between your tool and mage?

[brad-jones]

Basically I started gomake to solve this issue magefile/mage#24

Essentially it generates a https://github.com/spf13/cobra app

[brad-jones]

I have actually done a bit more dev on v3 in the last couple of days and after much deliberation I don't think this project has a viable future. It's basically a catch 22 situation, let me explain.

While gomake has successfully solved magefile/mage#24 allowing targets to accept parameters from the command line.

func Foo(bar string) error {
    fmt.Println("bar: ", bar)
    return nil
}

gomake foo --bar "baz"

It can not easily support the mg.Deps / mg.SerialDeps APIs.

func Foo(bar string) error {
    fmt.Println("bar: ", bar)
    return nil
}

func Baz() error {
    mg.Deps(Foo("abc")) // this is invalid
}

We could resolve this by using closures and it's the compromise v1 & v2 of gomake made.

func Baz() error {
    return run.Serial(
        func() error { return Task1("abc") },
        func() error { return Task2("xyz") },
        run.Parallel(
            func() error { return Task3("123") },
            func() error { return Task3("456") },
        ),
        func() error { return Task4("xyz") },
    )()
}

This works I guess but after using it in some production code bases for a while I still wasn't happy with it.

You say ok, just don't use those APIs and call the functions directly. Which of course you can certainly do and for serial workflows it works just fine. But when you want to run things concurrently you create some "goroutines", then you need to wait from them to finish somehow, probably a wait group, maybe some channels. You need to handle errors as well and before you know it a simple target that calls 2 other targets asynchronously is full of go's bloated concurrency primitives.

Don't get me wrong I think go has a fantastic concurrency API and it's very powerful but for this particular job it makes things difficult.

I then went and created https://github.com/brad-jones/goasync and while I think this has merit in it's own right. It still didn't really help me define that nice and clear DSL I am looking for.

[brad-jones]

v3 of gomake could look something like:

type Task interface {
	Run() error
}

type SerialTasks struct {
	tasks []Task
}

func (st *SerialTasks) Run() error {
	return nil
}

func Serial(tasks ...Task) Task {
	return &SerialTasks{tasks}
}

type ParallelTasks struct {
	tasks []Task
}

func (st *ParallelTasks) Run() error {
	return nil
}

func Parallel(tasks ...Task) Task {
	return &ParallelTasks{tasks}
}

// Build docs
//
// Structs with the "Run" method become cli sub commands just
// like functions currently do in the current version of gomake.
type Build struct {
	// Foo docs
	//
	// Struct fields become cli options / flags just like
	// function arguments currently do in the current
	// version of gomake.
	Foo string
}

// Run executes the actual task code
func (b *Build) Run() error {
	fmt.Println(b.Foo)
	return nil
}

type BuildAll struct{}

func (ba *BuildAll) Run() error {
	// We can now construct pipelines of tasks like this
	return Parallel(
		&Build{Foo: "abc"},
		&Build{Foo: "xyz"},
		Serial(
			&Build{Foo: "123"},
			&Build{Foo: "456"},
		),
	).Run()
}

[brad-jones]

But in all honesty I think I prefer https://github.com/brad-jones/drun

[brad-jones]

Some more thoughts... Any task runner worth using should de-duplicate the tasks. So in this example buildDomain should only run once (at least by default).

All my task runners that I have built to date (this & drun - maybe even an older TypeScript incarnation) have some form of "Once" functionality however it tends to only apply to a single anonymous function.

For example in drun we can do something like:

Future foo(String bar) => task((drun) => drun.once(() => drun.log(bar)));

But if we call foo('a') & then foo('b') the task only ever runs once, even though the input is different, this is clearly non-intuitive.

Other imperative based task runners like say https://magefile.org/ get away with this because their tasks don't allow input. They are strict targets like in a Makefile.

But the whole reason we started building these task runners was because we ran into issues when we wanted "parametrize" a target/task. We would either have a collection of buildX targets, which doesn't scale. Or we would use an environment variable which in turn causes issues with the "Once" functionality. You see we just went in a big circle there. :)

Anyway the real answer is we need to build a declarative task runner that is type safe (no magic strings) using an imperative language. AKA like the AWS CDK

package main

import (
	"fmt"

	"github.com/brad-jones/goasync/v2/await"
	"github.com/brad-jones/goasync/v2/task"
)

func deps(awaitables ...await.Awaitable) *task.Task {
	return await.AllOrErrorAsync(awaitables...)
}

func build() *task.Task {
	return deps(buildApi("Debug"), buildServiceHost("Debug"))
}

func buildDomain(configuration string) *task.Task {
	return task.New(func(t *task.Internal) {
		fmt.Println("building domain... config = ", configuration)
	})
}

func buildApi(configuration string) *task.Task {
	return deps(buildDomain(configuration)).Then(func(t *task.Internal) {
		fmt.Println("building api... config = ", configuration)
	})
}

func buildServiceHost(configuration string) *task.Task {
	return deps(buildDomain(configuration)).Then(func(t *task.Internal) {
		fmt.Println("building service host... config = ", configuration)
	})
}

func main() {
	await.All(build())
}

[brad-jones]

Latest ideas, think I finally found the relatively clean yet powerful API/DSL I was looking for.

You can probably see where I am going with this, the first task/job is modeled after Github Actions.
I want to build my task runner once and have it generate my CI/CD config, similar to: https://nuke.build/docs/authoring-builds/ci-integration.html

The plan will be to support Github Actions and Teamcity out of the box with others to be provided by the community / as needed.

The important thing is what we have here is TypeSafe and has deferred execution so the task deduplication can happen correctly.

var SomeArtifact = gomake.Artifact("./foo/bar/**")

func BuildAll() *gomake.J {
	return gomake.Job(func(j *gomake.J) {
		j.RunsOn("ubuntu-latest")
		j.Deps(BuildFoo(), BuildBar("v2"))
		j.Steps(func(s *gomake.S) {
			s.Action("actions/checkout@v2", nil)
			s.Action("actions/cache@v2", map[string]string{
				"path":         "~/go/pkg/mod",
				"key":          "${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}",
				"restore-keys": "${{ runner.os }}-go-",
			})
			s.GoAction(func() {})
			s.Run("ping 1.1.1.1")
		})
	})
}

func BuildFoo() *gomake.J {
	return gomake.Job(func(j *gomake.J) {
		j.Deps(BuildBaz("v1.0.0"))
		t.Action(func() {
			fmt.Println("BuildFoo")
		})
	})
}

func BuildBar(baseVersion string) *gomake.J {
	return gomake.Job(func(j *gomake.J) {
		j.Deps(BuildBaz(baseVersion + ".0.0"))
		t.Consumes(SomeArtifact)
		t.Action(func() {
			fmt.Println("BuildBar")
		})
	})
}

func BuildBaz(version string) *gomake.J {
	return gomake.Job(func(j *gomake.J) {
		t.Produces(SomeArtifact)
		t.Action(func() {
			fmt.Println("BuildBaz", version)
		})
	})
}

[brad-jones]

An interesting observation, the way I have tended to build a task runner or rather the way I execute a task runner is from the top down whereas a pipeline tends to be from the bottom up.

jobs:
  buildBaz:
    steps:
      - run: gomake build-baz --version "???"

  buildFoo:
    needs:
      - buildBaz
    steps:
      - run: gomake build-foo --no-deps

  buildBar:
    needs:
      - buildBaz
    steps:
      - run: gomake build-bar --no-deps

  buildAll:
    needs:
      - buildFoo
      - buildBar
    steps:
      - run: gomake build-all --no-deps

[brad-jones]

Locally if I execute gomake build-all the BuildAll() function will execute first. Where as in the pipeline gomake build-baz runs first and thus loses the additional context that is provided by BuildAll() & other tasks.

[brad-jones]

Something more like this is what we would need to generate.

jobs:
  buildAllPre:
    outputs:
      baseVersion: ${{ steps.1.outputs.test }}
    steps:
      - run: echo "::set-output name=baseVersion::v2"
  
  buildFoo:
    outputs:
      version: ${{ steps.1.outputs.test }}
    steps:
      - run: gomake build-foo --no-deps
      - run: echo "::set-output name=version::v1.0.0"

  buildBar:
    needs:
      - buildAllPre
    outputs:
      version: ${{ steps.0.outputs.test }}
    steps:
      - run: gomake build-bar --no-deps
      - run: echo "::set-output name=version::${{needs.buildAll.outputs.baseVersion}}.0.0"
  
  buildBazFoo:
    needs:
      - buildFoo
    steps:
      - run: gomake build-baz --version "${{needs.buildFoo.outputs.version}}"
  
  buildBazBar:
    needs:
      - buildBar
    steps:
      - run: gomake build-baz --version "${{needs.buildBar.outputs.version}}"
  
  buildAllPost:
    needs:
      - buildBazFoo
      - buildBazBar
    steps:
      - run: gomake build-all --no-deps

[brad-jones]

I guess the other thing is that there are 2 levels of concurrency. That provided by the pipeline running multiple jobs/agents and that provided by the task runner/operating system with goroutines/threads/processes. Perhaps we can better model this in our DSL design.

[brad-jones]

Seriously I need sleep, I got all excited a few hours ago when I thought I had found my perfect DSL but it still suffers from not being able to deduplicate tasks because the task parameters are part of the closure not the struct returned from the function. We would have to something something like this to capture the arguments into the struct which just sucks.

func Foo(bar string, baz int) *gomake.T {
	return gomake.Task(func(t *gomake.T) {
		t.Props(bar, baz)
		t.Action(func() {
			fmt.Println("Foo", bar, baz)
		})
	})
}

Considering we are already doing AST parsing to generate the cobra app, I mean I guess we could do it through that way but thats fraught with issues too and would only work for static cases. The only way is the Class/struct way like earlier on in this issue or here brad-jones/drun#1

[brad-jones]

So the reason I built gomake in the first place was to be able to do something like this.

func Build() {
    BuildGeneric("foo")
    BuildGeneric("bar")
    BuildGeneric("baz")
}

func BuildGeneric(someArg) {}

Or even

func Build() {
    for _, v := range someDynamicThing() {
        BuildGeneric(v)
    }
}

func BuildGeneric(someArg) {}

In the context of a monorepo I needed the ability to execute any of the tasks either individually or collectively as required. For example if I wanted to build the entire solution I would run gomake build but if just wanted to build the single project I was working on at the time in isolation I would execute gomake build-generic --some-arg baz to help speed up the process.

The alternative (more traditional approach) being something like.

func Build() {
    BuildFoo()
    BuildBar()
    BuildBaz()
}

func BuildGeneric(someArg) {}

func BuildFoo() {
    BuildGeneric("foo")
}

func BuildBar() {
    BuildGeneric("bar")
}

func BuildBaz() {
    BuildGeneric("baz")
}

But then you can't generate tasks like in the above example with someDynamicThing(). Now in most projects (read not a monorepo) this is probably fine but at the time when I first built gomake I had a whole collection of BuildX targets that were continuing to grow as new services and features were spun up.

And now we are wanting to use a task runner definition to generate CI/CD pipelines, it seemed like a nice idea because that's what I have tended do in the past. Build a task runner and then build a pipeline that executes that task runner so I don't have to repeat myself, at least not as much as I would if those 2 things were completely separate.

But I have discovered that the way a task runner (at least an imperative one like gomake) is built and executed Vs the way a pipeline is built and executed are almost polar opposites. Of course in the past I have subconsciously accounted for this difference many times over by using flags like --no-deps & other methods but it has taken until now to actual consciously realize it.

If we have a look at https://github.com/nektos/act you can see that the local runner looks nothing like a task runner, which came as a surprise after reading more than the first few lines of the readme. Eg: I can't call act test or act buildX I have to run the entire workflow. The best I can do is tell act what trigger to execute.

We could probably make act do something like being able to execute a specific job but it could never be called like act build-generic --some-arg foo as in the above examples. It can only have input from it's dependencies (outputs & artifacts) or from global parameters like environment variables.

This is exactly how https://nuke.build/ works, it's tasks/targets have no function parameters & they are able to generate CI/CD config.

When you stop to think about it a pipeline essentially has static input to it's first job. There is no human there that's entering a dynamic value for some parameter. The only dynamic data is the git metadata, what branch, tags, what code changed, etc... so it makes total sense that act works the way it does.

The more I think about it the more I think that so long as we do all the nice things like build caching locally as we do in the pipeline then there should be no reason for a developer not to want to run the entire pipeline or at least "up to" a certain job/stage.

It's also now so much more apparent as to why pretty much every single other task runner I have used, https://www.gnu.org/software/make/manual/make.html, https://github.com/microsoft/just, https://gulpjs.com, https://magefile.org, etc... don't have parameters for individual tasks. Some might say I have learnt that the hard way, best way to learn right :)