Can't put module config in a profile

[WhittlesJr]

I have some module config that varies between :duct.profile/dev and :duct.profile/prod. But the module is not initialized when it exists only in those profiles.

config.edn:

:duct.profile/dev
{:duct.module/foo {:bar :dev-value}}

:duct.profile/prod
{:duct.module/foo {:bar :prod-value}}

Now if I forget about profiles and just put the module config directly in the top-level of config.edn, the module works as expected.

Is this supposed to work and am I probably missing something?

[WhittlesJr]

I'll add that :duct.module/foo does show up in the system as a function, but it does not apply that function to the system config.

[weavejester]

Unfortunately this currently isn't possible without adding a fourth layer to how the system is processed.

Currently Duct turns a module configuration into a component configuration into a system:

modules -> components -> system

At the moment profiles are implemented as a type of module. Separating modules and profiles would require a fourth layer, e.g.

profiles -> modules -> components -> system

That's almost how 0.10 worked, except the profiles, modules and component layers were all mixed together in a haphazard way which caused a number of issues when the different pieces interacted.

Could you give a little more information as to what you're trying to do?

[WhittlesJr]

OK, understood! So for configuration that differs between profiles, should I just be changing top-level keys instead of module options, like below?

config.edn

:duct.profile/dev
{:foo/bar :dev-value}

:duct.profile/prod
{:foo/bar :prod-value}

foo.clj:

(defmethod ig/init-key :duct.module/foo [_ _]
  #(duct/merge-configs
    %
    {::some-component (ig/ref :foo/bar)}))

I was essentially doing the following instead, using module options rather than top-level keys:

foo.clj (previously):

(defmethod ig/init-key :duct.module/foo [_ {:keys [bar] :as options}]
  #(duct/merge-configs
    %
    {::some-component bar}))

[weavejester]

OK, understood! So for configuration that differs between profiles, should I just be changing top-level keys instead of module options

Right. To me that seems the simpler solution, since it doesn't mean separating profiles from modules. However, if it turns out to be particularly clunky, I'd be willing to think about other solutions.

I'd be very interested in some real-world examples, since it's hard to get a good feeling of what is most suitable when there's only small "foobar" code snippets.

[WhittlesJr]

Got it, thank you!

My use case really is not any more sophisticated than the example I provided, but I sadly can't share any real code until I can convince my employer to open-source the project (some day maybe). Right now it's not really any better or worse to do it the suggested way, but it might be helpful to have some kind of disclaimer about not being able to configure modules via profiles in your documentation. (Unless you have that somewhere and I missed it?)

I did see that your web module has some kind of reference to the "environment" but I couldn't follow the logic... it's hard enough for me to keep the other two distinct notions of "profiles" straight in my head (leiningen profiles vs duct profiles).

[WhittlesJr]

Ah actually I lied, this does turn out to be slightly inconvenient, because my :foo/bar key has some simple logic attached to it (like a default value, or casting the value, or some such). Of course, you could do this with a (defmethod ig/init-key :foo/bar [_ v] (or v :default-value)), but it seemed convenient to just throw that into the module itself. So here's a closer example to what I'm doing:

(defmethod ig/init-key :duct.module/foo [_ {:keys [bar baz] :as options}]
  #(duct/merge-configs
    %
    {[:duct/const :foo/bar] (or bar :default-value)
     [:duct/const :foo/baz] (boolean baz)

     ::other-stuff-already-warranting-a-foo-module {}}))

So yeah I could just use init-keys for :foo/bar and :foo/baz, and it's all pretty trivial anyway. But my first instinct was to do it as above to keep it all in one place, and it was surprising that it didn't work.

[weavejester]

Have you taken a look at ig/prep-key? That covers some of the cases that used to require a module.

[WhittlesJr]

I had, and I just reviewed the information I could find on it. It sounds like, as you said, modules can sometimes be replaced entirely by profiles+prep-key? But I'm unclear on some things:

So, consider my module (:duct.module/foo) that adds some keys (my :foo/bar and :foo/baz) above. You could have a profile that adds those keys too (call it :duct.profile/foo). But say I want them to be added by including :duct.profile/foo and then customized in other profiles (:duct.profile/dev vs :duct.profile/prod). To my knowledge, there's no notion of profiles overriding each other in a hierarchy, right? So dev and prod can't just override foo predictably, can they?

Assuming that's not the approach you intended for profiles, that leaves me using a module (:duct.module/foo). It could add the keys as above. But to get the default-value (or similar) behavior above, I'd have to use prep-key, because I want dev and prod to be able to override the values (and as you said, I can't customize a module qua the module in profiles, I'd have to use top-level keys.)

So I'm left with the following, I think:

config.edn:

{:duct.profile/dev  {:foo/bar :dev-value}
 :duct.profile/prod {:foo/bar :prod-value-1
                     :foo/baz :prod-value-2}
 :duct.module/foo   {}
 }

foo.clj:

(defmethod ig/prep-key ::bar [_ v] (or v :default-value))
(defmethod ig/prep-key ::baz [_ v] (boolean v))

(defmethod ig/init-key :duct.module/foo [_ _]
  #(duct/merge-configs
    %
    {[:duct/const ::bar] nil
     [:duct/const ::baz] nil}))

Is that correct? It feels awkward and verbose. Is there a better way?

[weavejester]

It's really hard to suggest a solution without knowing more about the problem. Is there any way you can talk about your use case in slightly more specific terms, without revealing any code? Using "foo" keys tells me roughly what you're trying to do, but not why.

[WhittlesJr]

I really have been striving to convey the "why"... I'll try to boil it down to a list of needed features:

• I need a central thing to include that adds keys to the system. (At one point I used :duct/include, now I use modules, it sounds like you can also use profiles)
• The "central thing" is a modular feature that could be included in other configurations by itself
• The added keys are referenced by other modular features
• Those keys have some kind of behavior on them (defaulting / casting / etc.) but they must be in the system whether I explicitly customize them or not
• The included thing must be able to have those keys overridden based on dev vs prod environment

Does that help?

[WhittlesJr]

This seems like it would be a totally normative use case concerning how modules are to be developed... again, your web module seems to do some sort of magic to reference the dev vs. prod environment beyond the above stated mechanisms but I can't grasp it yet.

[weavejester]

That helps a little, and I can at least give you a few options, though without further information I can't say which is the best solution for your problem.

Option 1

The first option is to do the same thing as Duct's middleware, which is to look for a key in the configuration called :duct.core/environment. This key is set to :production in production, and :development in development. So you could write:

(defmethod ig/init-key :foo.module/example [_ _]
  (fn [config]
    (if (= :development (:duct.core/environment config))
      ;; add dev setup to config
      ;; add prod setup to config
      )))

Option 2

If that's too coarse-grained, and you specifically need options in your config, then you can add a key that is just for passing per-profile options to the module:

(defmethod ig/init-key :foo.module/example [_ top-level-options]
  (fn [config]
    (let [profile-options (:foo.module.profile-options/example config)
          all-options     (duct.core/merge-configs top-level-options profile-options)
      ...)))

Option 3

The third option is not to use #duct/include at all, but instead manually merge the profiles. So in your dev.clj file, you override the read-config to:

(defn read-config []
  (duct/merge-config
   (duct/read-config (io/resource "todo/config.edn"))
   (duct/read-config (io/resource "dev"))))

Then remove these lines from your config:

 :duct.profile/dev   #duct/include "dev"
 :duct.profile/local #duct/include "local"

And instead have a dev.edn file that looks like:

{:duct.profile/local #duct/include "local"
 :duct.profile/dev
 {...}
 :foo.module/example {...}}

[weavejester]

Additionally, I'll give this problem a little more thought and try to come up with a more general solution.

[WhittlesJr]

Thank you! Here are my questions:

Option 1:

This seemingly moves the dev/prod decision-making to the module itself, rather than the configuration. So the module developer can build in dev/prod differentiation, but the module user can't? Unless I included a module option like so:

config.edn:

{:foo.module/example {:dev  {:option-a :dev-value}
                      :prod {:option-a :prod-value}}}

foo.clj:

(defmethod ig/init-key :foo.module/example [_ {:keys [dev prod] :as options}]
  (fn [config]
    (if (= :development (:duct.core/environment config))
      (duct.core/merge-configs config dev) ;; obviously it wouldn't be a straight merge but you get the point
      (duct.core/merge-configs config prod))))

Doable but pretty awkward, given that we already have a mechanism for differentiating dev vs prod config (the profiles).

Option 2:

As I consider it more, I think this is how you would address the concern above, right?

Option 3:

This seems like the most straightforward path, and I may end up going with it for now...

If what I originally assumed was going to work could be made to work (modules configured in a profile), that would be better than any of these workarounds. It was hard enough for me to grok the concept of how modules and profiles and duct itself all work. Once I got there though, I had such a nice, tidy mental model of the system, because duct is really elegant on its surface once you get a wide enough perspective of it. But then I hit this issue and my mental model had to suddenly become more complex and nuanced.

[WhittlesJr]

Is it more than just resolving all profiles first, and then taking that config and resolving all non-profile modules next?