Document how to use duct with tools.deps

[RickMoynihan]

I've just ported a duct project to work with tools.deps, and I've worked around some of the issues that come up.

Below are some notes on the issues and those which I have found solutions for. At some point they may be a useful basis for forming some documentation on the subject, or perhaps changing the duct project templates themselves.

The main issue I encountered is how to generate duct uberjars. With lein duct requires the addition of the lein-duct plugin to ensure that duct_hierarchy.edn files from dependencies are merged together into a single duct_hierarchy.edn at the top of the root of the classpath. The main reason this is necessary is because the lein uberjar process flattens all jars into one classpath root. It's worth noting that this is not what typically happens at a REPL. At a REPL the classpath is typically composed of your dependency jars, so the files are indepedently addressable via the classloader's getResources which returns all matches for a given file on the classpath appropriately disambiguated by their jar/classloader. duct.core correctly iterates over this which means providing you don't destroy this context in your uberjar step, you shouldn't need a plugin to handle merge conflicts (lein uses :uberjar-merge-with).

So how can you generate uberjars without destroying that context, and do so from tools.deps? The answer is pack.alpha and specifically their support for creating capsule uberjars. Essentially Capsule provides us a way to package an uberjar of jars instead of a munged uberjar of classes; capsule then transparently manages the extraction process and the rebuilding of the classpath. The upshot is that your uberjar is then a much more similar environment to your repl, and duct will naturally handle the resolution of duct_hierarchy.edn itself.

The only other thing I know duct does that I have not yet looked at, is :aot compiling the dependencies specified in your duct system by calling :duct/compile. I expect for those that require this, it should be relatively easy to add.

[weavejester]

The uberjar issue with duct_hierarchy.edn was what stopped me from porting Duct over to clj-new. It's good to hear that capsule uberjars are a potential solution.

[RickMoynihan]

Well, I'm glad to be helpful.

It's early days but they seem to work quite well.

Though saying this capsule's currently seem to create reflective access warnings on JDK9+, though there are open issues and it looks like people are trying to fix it.

Other things to note about capsules are that they do the double JVM kickstart dance a little like leiningen. Like leiningen you can also alegedly trampoline them. This extra complexity might be a reason some people want to avoid them; though so far they seem to work quite well.

I've not yet looked into pack.alpha's other uberjarring options, such as skinny jars or OneJar, but these might also be good candidates.

[zerg000000]

This library provides all the small building blocks for customising the whole package process. It might be a good candidate, if no other pre-build option could fully satisfy the needs.

https://github.com/EwenG/badigeon.git

[ianbishop]

You can also use lein-tools-deps and lein uberjar to build your jar. We wanted to produce an aot'd jar which was not possible with Capsule. This allows you to keep using your existing deps.edn normally.

Here's a sample project.clj:

(defproject project "0.1.0-SNAPSHOT"
  :main ^:skip-aot project.main
  :plugins [[duct/lein-duct "0.12.1"]
            [lein-tools-deps "0.4.5"]]
  :middleware [lein-duct.plugin/middleware
               lein-tools-deps.plugin/resolve-dependencies-with-deps-edn]
  :lein-tools-deps/config {:config-files [:install :user :project]}
  :prep-tasks ["javac" "compile" ["run" ":duct/compiler"]]
  :resource-paths ["resources"]
  :profiles {:uberjar {:aot :all}})

[WhittlesJr]

Pack + Capsule has worked beautifully for me out of the box. +1.

[rynkowsg]

HERE is an example of deps.tools project with duct and integration of both solutions for creating uberjars:

• pack + capsule,
• lein uberjar + lein-tools-deps.

[jimpil]

I realise this is an old ticket/issue, but I recently had to deal with this at work, and I am documenting the full solution here. Producing an uberjar from a duct-based project goes as follows:

1. Port dependencies & aliases from project.clj to deps.edn (ignore the duct leiningen middleware/compiler).
2. Add a :build alias to your deps.edn which adds the tools.build extra dependency.
3. Create a build script, similar to the one below.

#!/usr/bin/env -S clojure -J-Dclojure.main.report=stderr -A:build -M

(require
 '[clojure.pprint :as pprint]
 '[clojure.tools.build.api :as build])

(defn- conflict-duct-hierarchy
  "The same as `clojure.tools.build.tasks.uber/conflict-data-readers`,
   but uses `(partial merge-with into)` as the merging-fn."
  [{:keys [path in existing]}]
  (binding [*read-eval* false]
    (let [in-str       (-> (.readAllBytes ^java.io.InputStream in)
                           (String. "UTF-8"))
          existing-str (slurp existing)
          existing-map (read-string
                        {:read-cond :preserve :features #{:clj}}
                        existing-str)
          append-map   (read-string
                        {:read-cond :preserve :features #{:clj}}
                        in-str)
          final-str    (with-out-str (pprint/pprint (merge-with into existing-map append-map)))]
      {:write {path {:string final-str}}})))

(defn uberjar
  "Build an uberjar in target/foo.jar"
  {:flags ["--target-dir" {:doc     "Directory to build in"
                           :default "target"}]}
  [{:keys [target-dir]
    :or   {target-dir "target"}}]
  (let [basis (build/create-basis {:project "./deps.edn"})]
    (println "· copying resources/")
    (build/copy-dir {:src-dirs   ["resources"]
                     :target-dir target-dir})

    (build/copy-file {:src "src/duct_hierarchy.edn" :target (str target-dir "/duct_hierarchy.edn")}) ;; <===
    
    (println "· AOT compiling")
    (build/compile-clj {:basis     basis
                        :src-dirs  ["src"]
                        :class-dir target-dir
                        :compile-opts
                        {:elide-meta     [:doc]
                         :direct-linking true}})

    (println "· uberjarring" (str target-dir "/foo.jar"))
    (build/uber {:class-dir         target-dir
                 :uber-file         (str target-dir "/foo.jar")
                 :basis             basis
                 :main              'foo.main
                 :conflict-handlers {"duct_hierarchy.edn"    conflict-duct-hierarchy ;; <===
                                     "^META-INF/versions/.*" :append
                                     :default                :warn}
                 :exclude           [".well-known/jwks.json"
                                     "jwk_public.pem"
                                     "jwk_private.pem"
                                     "META-INF/license/LICENSE.*.txt"]})

    (println "done.")))

Hope that helps someone :)