Considered making decoupled parse, coercion, validate, etc fns available in API

[lread]

Use Case

Babashka cli seems to assume that input is only coming from the command line.
This seems reasonable, given, well, its name is cli!

But, input can also come from, for example:

• clojure -X invocation
• values from .edn file
• maybe properties
• maybe env vars

In these cases, the values have been parsed by other means, but I still want to coerce and validate them. And likely, in the case of an .edn file (and maybe properties and env vars), I might want to validate after merging with parsed and coerced cli values before dispatch. I cannot do this today because behaviour is coupled (and sometimes intertwined).

A Look at Current Behaviour

If I've understood:

parse-opts

• parses args and coerces values at same time
• applies restrict validation
• applies require validation
• applies other validation

dispatch

• parses args (effectively via parse-opts)
• calls error-fn on error
• calls cmd fn

A Rough Draft of Decoupled fns

I'm wondering if we could tease out (without breaking any existing API) something like:

• parse-cli-args (converts cli args to map of option/raw-value(s))
• coerce-opts (converts option/raw-value(s) to option/coerced-value)
• validate-opts (restrict, require, other validation)
• dispatch-lookup (looks up dispatch cmd)
• dispatch-invoke (invokes dispatch fn)
• dipatch-? (I have to think more about dispatch decoupling)

There might be 2 types of coercion. One type of coercion accommodates the limitation of the command line.
:collect comes to mind, and also coercing multiple values to a vector. Another type of coercion might be thought more of as user-kindnesses, like coercing a string to a keyword, for example. It might be useful to separate the first from the 2nd, not sure yet.

Is this worth digging deeper?

This would not be a tiny effort, so before I dig any deeper, do you also find the idea interesting?

[borkdude]

@lread I have a first draft here that's ready to merge. Can you review this?

https://github.com/babashka/cli/tree/expose

[borkdude]

@lread Or we should / could add a default-opts fn that fetchs default options from other places? Or squint cli fetches the .edn file first and provides this input as default-opts?

[lread]

supporting defaults from an .edn file

That might satisfy the squint use case.
I expect we can assume that the .edn values don't need coercion (maybe?) but are not validated.

Side thought: When values are also read from .edn, it can be nice for user to know:

• that .edn file was found and is being used (this can help greatly with the forgotten .edn file scenario)
• when a command line opt is overriding .edn file opt.

supporting both -X/-T and -M

Another use case I've stumbled on is supporting both -X/-T and -M for some projects.
I really dislike the usability of -X/-T and try to avoid it, but I contribute to projects that support both -M and -X/-T.
For those projects, I'd like to reuse the -M specs for -X/-T.
This made me think of decoupling as a solution.

For -M things work fine as is.
For -X/-T, the args are already parsed to an opts map by Clojure.
And they now need to be coerced and validated before dispatch.

For these projects I also support presenting help in the -X/-T syntax, but that detail, I think, is straying from our focus.

[lread]

I realize that I proposed a solution with this discussion rather than presenting problems/opportunities.
My last comment, I think, presents problems/opportunities.

[borkdude]

I'm not sure where the -X/-T -M comment suddenly comes from. Is that about bb.cli?

Anyway, for the .edn file use case, a user could first read the edn file, coerce + validate it using the API fns and then supply it to the spec as default opts, then parse like normally.

[lread]

Yeah, the -X/-T is about bb cli and was another problem I thought would be helped by decoupling bb cli fns. But we can scope this to the squint edn file use case to avoid overwhelming ourselves.

Anyway, for the .edn file use case, a user could first read the edn file, coerce + validate it using the API fns and then supply it to the spec as default opts, then parse like normally.

The .edn validate can't apply :require validation because some opts may still be coming from the command line. But it could be nice to use :restrict validation on edn, I suppose, to catch typos in edn file. This all seems doable with your expose branch.

I'll take another crack at this with your idea on that old squint PR soon.

[lread]

I am playing with the expose branch and noticed a tiny difference from main branch.

(let [errors (atom [])]
  (cli/parse-args ["--foo"] {:spec {:foo {:coerce [:keyword]}}
                             :error-fn #(swap! errors conj %)})
  @errors)
;; main branch
;; => [{:spec {:foo {:coerce [:keyword]}},
;;      :type :org.babashka/cli,
;;      :cause :coerce,
;;      :msg "Coerce failure: cannot transform (implicit) true to keyword",
;;      :option :foo,
;;      :value "true",
;;      :opts {}}]


;; expose branch
;; => [{:spec {:foo {:coerce [:keyword]}},
;;      :type :org.babashka/cli,
;;      :cause :coerce,
;;      :msg "Coerce failure: cannot transform (implicit) true to keyword",
;;      :option :foo,
;;      :value ["true"],
;;      :opts {}}]

Notice the difference in the :value for the expose branch.
I was relying on :value "true" to detect a missing opt.
See #137. Example of my technique from squint PR:
https://github.com/lread/squint/blob/a1e6fe22a2e512902797c24bc03e164219fe002b/src/squint/internal/cli.cljs#L289-L290
Perhaps my technique is hacky? Brittle?

[lread]

I'm thinking about edn file value coercion.

It is appropriate to be as forgiving and kind when coercing from edn file values as we are when coercing from command line args to opts?

An edn file allows exact specification of typed values.
Are we doing folks a favour by coercing when they have not gotten values exactly right?

(cli/coerce-opts {:foo "blarp"} ;; this is not in a vector
                 {:spec {:foo {:coerce [:keyword]}}})
;; => {:foo [:blarp]}

I suppose there is no one right answer.
But my inclination might be to fail fast if the edn file values do not match post-coerced cli expectations.

That said, I've experimented with coercing edn file values in my old squint PR (against your bb cli expose branch): lread/squint@0eef0f3

[borkdude]

But my inclination might be to fail fast if the edn file values do not match post-coerced cli expectations.

That sounds good.