Support for running with `--no-start`

[raysegantii]

My project's application starts with a few workers that shouldn't start during testing. When I run mix white_bread.run --no-start, the context fails to find any module in the project.

Referring to https://github.com/elixir-lang/elixir/blob/master/lib/mix/lib/mix/tasks/test.ex#L195 I've to fix it by:

feature_starting_state fn ->
  Mix.Project.compile([])
  Application.ensure_all_started :postgrex
  Application.ensure_all_started :ecto
  ...
end

I wonder why white_bread.run actually doesn't go through test/test_helper.exs, and the use WhiteBread.Context only do import ExUnit.Assertions rather than use ExUnit.Case as in: https://github.com/drewolson/ex_spec/blob/master/lib/ex_spec.ex#L44 .

[meadsteve]

@raysegantii The initial reason I didn't use test/test_helper.exs, and use ExUnit.Case is that I wanted to keep it separate from exunit where as ex_spec is intended to wrap around exunit.

I'll take a look and see why running --no-start is causing problems

[mgwidmann]

I think it would make the most sense to go through the test/test_helper.exs as this project can't possibly know how to properly setup the environment (which is what that helper file is for). Instead since the contexts are loaded first, each context must describe startup procedures such as this. I'd prefer it integrated with exunit more than it currently does.

[meadsteve]

@mgwidmann from the general feedback I've had I think closer integration with exunit is the way forward. The current master is going to be a BC breaking v3 release so this is a good time for me to rework this.

[mgwidmann]

Yeah in my mind, a .feature file should basically compile down to an exunit test IMO... That way something like:

Feature: Test Stuff

  Scenario: Link to the test page
    Given I navigate to "/"
    When I click the link "test"
    Then I am taken to "/test"

With:

defmodule TestStuff.Context do
  use WhiteBread.Context
  use Hound.Helpers

  given ~r/^I navigate to "(?<path>[^"])"$/, _state, %{path: path} do
    navigate_to("http://localhost:4000#{path}")
  end

  when ~r/^I click the link "(?<link_text>[^"])"$/, _state, %{link_text: text} do
    find_element(:link_text, text) |> click()
  end

  then ~r/I am taken to "(?<path>[^"])"$/, _state, %{path: path} do
    assert current_path() == path
  end
end

Would use macros to compile to:

defmodule MyApp.TestStuff do
  use ExUnit.Case

  @tag :integration
  test "Link to the test page" do
    navigate_to("http://localhost:4000#{path}")
    find_element(:link_text, text) |> click()
    assert current_path() == path
  end
end

As a nice to have it may even benefit to see this output in some respect. Testers get their Gherkin files, we get plain code to deal with... Win win for everyone.

[meadsteve]

I've actually been thinking about this kind of thing for a while (though you've expressed it very eloquently). I was considering doing a bit of a spike rewrite of the tool that had a two step process (bit of a break from what I've seen in other cucumber implementations):

mix white_bread compile

would use the contexts to build the test cases as you've shown from the features. These could then be committed to source control.

mix white_bread run

Would then execute these tests (checking that they still match the features and context) using a normal exunit run (but with some tag like @cucumber or @white_bread).

I'm still not convinced about it as an idea but the main reason I've not ended up trying this out is I haven't had a lot of spare time recently so I've stepped back from open source a little.

[mgwidmann]

Yeah the only issue I can see from basically generating source files is that when you want to override the generated source it may get annoying to discard the changes it removes (your overrides). Which is why I am leaning to using macros to generate it and not generate the source.

I'd imagine instead of what you proposed, letting ExUnit take care of it all...

test
├── features
│   ├── test_stuff.feature # ExUnit would ignore this file since its not an elixir file
│   └── contexts
│       └── test_stuff_context.exs # This would compile into the exunit test
...

Then the user need only exclude them like:

$ mix test --exclude integration

or in their test_helper.exs like:

ExUnit.start
ExUnit.configure exclude: [:integration], trace: true

If you'd like, I can submit an issue with a proposal to discuss this...