diff --git a/COMPARISON.md b/COMPARISON.md index 5ee30c9..24aa7ed 100644 --- a/COMPARISON.md +++ b/COMPARISON.md @@ -19,11 +19,22 @@ Testo focuses on extensibility through plugins and stays a thin layer over usual Other frameworks may be a better fit if you need BDD scenarios or some unique features "out of the box". Testo can support BDD-style tests through plugins. -[^1]: Ginkgo runs parallel tests in [separate processes](https://onsi.github.io/ginkgo/#mental-model-how-ginkgo-runs-parallel-specs) with its own runner (not available through `go test`). This is _less performant_ than native `go test` parallelization based on goroutines. +## Why we moved off [Allure-Go] + +Testo was created at Ozon as the successor to [Allure-Go]. +In Allure-Go, the test framework and Allure report generation are one +piece: you can't swap the reporting or extend the framework. Testo +splits them. The framework is a dependency-free layer over +`testing.T`, and Allure reporting is +[one plugin among many](https://github.com/ozontech/testo-allure). +If you are on Allure-Go today, see the +[migration guide](./docs/migration.md#migrating-from-allure-go). + +[^1]: Ginkgo runs parallel tests in [separate processes](https://onsi.github.io/ginkgo/#mental-model-how-ginkgo-runs-parallel-specs) with its own runner (not available through `go test`). It costs more startup time and memory than goroutine-based `go test` parallelization, but you get process isolation in return. [^2]: [Not supported](https://github.com/smartystreets/goconvey/issues/360). [^3]: See [issue #934](https://github.com/stretchr/testify/issues/934). -[^4]: DSL — Domain-Specific Language. Requires describing tests in a specific way, different from usual Go tests. Not necessarily a bad thing, but has a learning curve and is less flexible. -[^5]: Whether it has any dependencies. Not necessarily a bad thing, but fewer dependencies mean a smaller footprint, faster build times and avoiding potential vulnerabilities. +[^4]: DSL: Domain-Specific Language. Requires describing tests in a specific way, different from usual Go tests. Not necessarily a bad thing, but has a learning curve and is less flexible. +[^5]: "Yes" means no third-party dependencies: only the standard library is used. [^6]: Any report format is achievable through plugins, but none is baked into Testo by default. See [testo-allure](https://github.com/ozontech/testo-allure). [^7]: "Native `go test`" means _all features_ are supported using only the `go test` command without any other CLIs. diff --git a/README.md b/README.md index d461db8..8f16f91 100644 --- a/README.md +++ b/README.md @@ -13,30 +13,29 @@ with an extensive plugin system. > Testo (/tɛstɒ/) is a play on words "test" and "тесто", meaning "dough". > Just like you can cook anything from dough, you can test anything with Testo! -Add some flavor to your tests with -[toppings - a collection of small, miscellaneous plugins for Testo framework.](https://github.com/ozontech/testo-toppings) +See also [toppings](https://github.com/ozontech/testo-toppings) - +a collection of small optional plugins for Testo. ## Features -- [Plugins](./examples/04_plugins/main_test.go) - adapt your tests to any scenario with features you need. -- [Parametrized tests](./examples/03_parametrized/main_test.go) - describe a test once, repeat it with different parameters. -- [Parallel tests](./examples/05_parallel/main_test.go) - make your tests faster by running them all at once. -- [Lifecycle hooks](./examples/02_hooks/main_test.go) - before and after any suite, test & sub-test. -- [Test annotations](./examples/07_annotations/main_test.go) - attach static options to any test. -- [Informative errors and traces](./examples/06_errors/main_test.go) - no need to guess what went wrong. -- Sub-tests & sub-suites - support for nested tests and nested suites. -- Test reflection - deeply inspect test's meta-information. -- Caching - key-value storage persistent between test runs. +- [Plugins](./docs/plugins.md) - hook, filter and extend `T` without forking the framework ([example](./examples/04_plugins/main_test.go)). +- [Parametrized tests](./docs/how-to.md#how-to-write-parametrized-tests) - describe a test once, repeat it with different parameters ([example](./examples/03_parametrized/main_test.go)). +- [Parallel tests](./examples/05_parallel/main_test.go) - run independent tests concurrently. +- [Lifecycle hooks](./docs/tutorial.md#suite-hooks) - before and after any suite, test & sub-test ([example](./examples/02_hooks/main_test.go), [parallel caveat](./docs/how-to.md#hooks-and-parallel-sub-tests)). +- [Test annotations](./docs/how-to.md#how-to-annotate-tests) - attach static options to any test ([example](./examples/07_annotations/main_test.go)). +- [Test filtering](./docs/how-to.md#how-to-run-and-skip-specific-tests) - `-run`, `-testo.m`. +- [Informative errors and traces](./examples/06_errors/main_test.go) - error messages name the exact method and type that caused them. +- [Sub-tests & sub-suites](./examples/08_subsuites/main_test.go) - support for nested tests and nested suites. +- [Test reflection](https://pkg.go.dev/github.com/ozontech/testo/testoreflect) - deeply inspect test's meta-information. +- [Caching](./docs/how-to.md#how-to-use-persistent-cache) - key-value storage persistent between test runs. - [Zero dependencies](./go.mod). ## Why Testo -At Ozon, Testo powers thousands of end-to-end tests daily in production. +Ozon runs thousands of end-to-end tests on Testo every day. -With plugins, it is flexible enough to adapt to diverse requirements, -without leaving the Go ecosystem - just a layer over `testing.T`. - -If your needs are outgrowing standard `testing` package, Testo is a great choice. +Plugins let teams add what they need: reporting, retries, custom `T` +methods. Tests stay plain `go test` tests. See [comparison with other frameworks](./COMPARISON.md). @@ -70,33 +69,51 @@ And run it with `go test` as usual: go test . ``` -But there is more! -Testo supports suites, parametrized tests & plugins, see [Next steps](#next-steps). +Testo can do more. For example, run parametrized tests in suite: + +```go +type Suite struct{ testo.Suite[*testo.T] } + +func (Suite) CasesWord() []string { + return []string{"dough", "bread"} +} + +func (Suite) TestLen(t *testo.T, p struct{ Word string }) { + if len(p.Word) == 0 { + t.Error("word must not be empty") + } +} + +func TestSuite(t *testing.T) { + testo.RunSuite(t, new(Suite)) +} +``` -See also [VS Code extension for Testo](#vs-code-extension). +`TestLen` runs once per word from `CasesWord`. ### Next steps - Take [a guided tour of Testo](./docs/tutorial.md) by making simple plugins and running the tests using various features. - See [test examples](./examples). -- Learn [how to use various Testo features](./docs/how-to.md). -- Read a [brief description and technical overview](./docs/technical-overview.md) of Testo. +- Learn [how to use Testo features](./docs/how-to.md). +- Migrating from testify or allure-go? See the [migration guide](./docs/migration.md). +- Read the [technical overview](./docs/technical-overview.md) - lifecycle, panics, plugin internals. - View [API documentation](https://pkg.go.dev/github.com/ozontech/testo). ## Plugins -Testo features a powerful plugin system. - Plugins can: -- Provide `BeforeAll`/`AfterAll`, `BeforeEach`/`AfterEach` & `BeforeSubEach`/`AfterSubEach` hooks. +- Provide `BeforeAll`/`AfterAll`, `BeforeEach`/`AfterEach` & `BeforeEachSub`/`AfterEachSub` hooks. - Plan tests for execution - filter, duplicate & reorder. -- Override built-in `T` methods, such as `Log`, `Error` and _etc._ +- Override built-in `T` methods, such as `Log` and `Error`. - Extend `T` by adding new methods. - Allow users to configure their behavior through options. - Communicate with other plugins. - Add command line flags for `go test` command. +See [the guide on writing plugins](./docs/plugins.md). + Examples: - [Testo Allure Plugin](https://github.com/ozontech/testo-allure) - enhance your tests with automatically generated [Allure Reports](https://allurereport.org/). @@ -108,7 +125,7 @@ Examples: Testo has its own [VS Code extension](./vscode-extension). -Makes it easier to run and debug individual suite tests and adds helpful snippets. +The extension adds run/debug buttons for individual suite tests, plus snippets. ![VSCode extension screenshot showing codelens buttons for running and debugging a test](./vscode-extension/example.png) @@ -116,7 +133,7 @@ Makes it easier to run and debug individual suite tests and adds helpful snippet Testo guarantees to support at least **3 latest major** [Go releases](https://go.dev/doc/devel/release). -Currently, minimum supported Go version is **1.24** +Currently, the minimum supported Go version is **1.24**. ## Contributing @@ -127,9 +144,3 @@ See [contributing guidelines](./CONTRIBUTING.md). ## License This project is released under the [Apache-2.0 license](./LICENSE). - ---- - -> [!TIP] -> If you find Testo useful, [consider giving this repository a star](https://github.com/ozontech/testo) to help it reach more people. -> Thank you! diff --git a/docs/README.md b/docs/README.md index d9f022a..0034790 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,7 +1,11 @@ # Documentation -- [Tutorial](./tutorial.md) -- [How to use Testo features](./how-to.md) -- [Persistent cache](./how-to.md#how-to-use-persistent-cache) -- [Technical Overview](./technical-overview.md) +Suggested reading order: + +- [Tutorial](./tutorial.md) - a guided tour from a plain Go test to a full Testo suite. +- [Examples](../examples) - small runnable projects, one per feature. +- [How to use Testo features](./how-to.md) - goal-oriented recipes. +- [Writing plugins](./plugins.md) - the full plugin API: hooks, overrides, planning, options, flags. +- [Migration guide](./migration.md) - moving from testify or allure-go. +- [Technical overview](./technical-overview.md) - lifecycle, panics, plugin internals. - [API documentation](https://pkg.go.dev/github.com/ozontech/testo). diff --git a/docs/how-to.md b/docs/how-to.md index 44977cb..a44ef20 100644 --- a/docs/how-to.md +++ b/docs/how-to.md @@ -1,13 +1,15 @@ -# How to +# How to use Testo features -Learn how to use the features of Testo. +Snippets on this page are fragments: they assume a project set up as +in the [tutorial](./tutorial.md), with a suite and a `T` type already +defined. ## How to write parametrized tests Parametrized tests are defined as regular tests with a second argument: ```go -func (*Suite) TestFoo(t *testo.T, p struct{ Name string; Age int }) { +func (*Suite) TestFoo(t T, p struct{ Name string; Age int }) { t.Logf("Using name=%q and age=%d", p.Name, p.Age) } ``` @@ -27,56 +29,87 @@ func (*Suite) CasesAge() []int { > [!TIP] > `CasesXxx` are invoked *after* `BeforeAll` hook. -Field names used in a `struct{ Name string; Age int}` must be equal to existing `CasesXxx` functions. +Each parameter field must have a matching `CasesXxx` method. Given that, test `TestFoo` will be invoked with all possible combinations of names and ages: -```python -TestFoo(name=John, age=18) -TestFoo(name=John, age=60) -TestFoo(name=John, age=6) -TestFoo(name=Joe, age=18) -TestFoo(name=Joe, age=60) -TestFoo(name=Joe, age=6) +```txt +TestFoo with p = {Name: "John", Age: 18} +TestFoo with p = {Name: "John", Age: 60} +TestFoo with p = {Name: "John", Age: 6} +TestFoo with p = {Name: "Joe", Age: 18} +TestFoo with p = {Name: "Joe", Age: 60} +TestFoo with p = {Name: "Joe", Age: 6} ``` -If for at least one required parameter function `CasesXxx` -returns zero values Testo will log a warning with similar message: +Parameter values do not appear in test names. +In `go test -v` output the runs are named `TestFoo`, `TestFoo#01`, `TestFoo#02` and so on, +following the standard `go test` convention for repeated names. +Log the parameters at the start of the test, so a failing `TestFoo#03` +identifies its case - or let a +[ten-line plugin](./plugins.md#reading-test-metadata) do it for every +test automatically. + +If a `CasesXxx` method returns an empty slice, Testo logs a warning +(visible with `go test -v`): ```txt -main_test.go:15: testo: (*main.Suite).CasesName returned empty slice, (*main.Suite).TestFoo will not run +main_test.go:15: testo: warning: (*main.Suite).CasesName returned empty slice, (*main.Suite).TestFoo will not run ``` -To turn this log into fatal error and do not proceed with further execution -pass flag `-testo.strict` to the `go test` command invocation: +To make this warning fatal, enable +[strict mode](#how-to-enable-strict-mode). -```bash -go test ./... -testo.strict +### Table tests (correlated parameters) + +Separate parameters always produce the Cartesian product of their values. +When values are correlated - a classic table test, where each case is one +row with its own input and expected output - use a single struct-typed +parameter instead: + +```go +type Case struct { + Input string + Want int +} + +func (*Suite) CasesCase() []Case { + return []Case{ + {Input: "one", Want: 1}, + {Input: "two", Want: 2}, + } +} + +func (*Suite) TestParse(t T, p struct{ Case Case }) { + if got := Parse(p.Case.Input); got != p.Case.Want { + t.Errorf("Parse(%q) = %d, want %d", p.Case.Input, got, p.Case.Want) + } +} ``` -> [!TIP] -> You can also set `TESTO_STRICT` environment variable to `true` -> for the same effect. -> -> Flags have higher priority than environment variables. +The test runs once per element of the slice, with no cross-combination. ## How to write parallel tests -You can use your regular `t.Parallel` method to mark a test as parallel. +Mark a test as parallel with the standard `t.Parallel`: ```go -func (*Suite) TestFoo(t *testo.T) { +func (*Suite) TestFoo(t T) { t.Parallel() // your test here } ``` -You can expect all `AfterEach` and `AfterAll` hooks to execute at the end of each test properly. +Standard `go test` flags such as `-parallel`, `-count` and `-timeout` +apply unchanged - Testo tests are regular Go tests underneath. +With `-count=N`, `BeforeAll` and `AfterAll` run once per iteration. + +### Hooks and parallel sub-tests -Please note, that `AfterEach` (only applies to it) hook is deferred to run at the end of the test. -If that test has sub-tests marked as parallel, -this hook will run BEFORE those sub-tests are finished. +`AfterEach` and `AfterAll` still run for parallel tests, but +`AfterEach` is deferred to the end of the test body. If the test has +parallel sub-tests, the hook runs **before** they finish: ```go func (*Suite) Test(t T) { @@ -99,12 +132,16 @@ func (*Suite) Test(t T) { } ``` -Unless you need to run sub-tests during this hook, -it is recommended to use t.Cleanup during BeforeEach. +If your teardown must run after all parallel sub-tests finish, +register it with `t.Cleanup` inside `BeforeEach` instead of using the `AfterEach` hook - +cleanups run after all parallel sub-tests of the test are done: ```go func (*Suite) BeforeEach(t T) { - t.Cleanup(t.afterEach) + t.Cleanup(func() { + // Teardown logic here. + // Runs after the test AND all its parallel sub-tests finish. + }) } ``` @@ -149,66 +186,75 @@ func Test(t *testing.T) { Passing to `testo.Run`: ```go -func (s *Suite) TestFoo(t *testo.T) { - testo.Run(t, "my sub-test", func(t *testo.T) { +func (s *Suite) TestFoo(t T) { + testo.Run(t, "my sub-test", func(t T) { // ... }, myplugin.SomeOption()) } ``` -> Options can be propagated to inner sub-tests, but it's up to a plugin author -> to decide whether an option should be propagated or not. -> -> If needed, you can change propagation for certain options -> by setting `.Propagate` field to `false` (disable) or `true` (enable). -> However, it is highly discouraged to do so. +> [!NOTE] +> The plugin author decides if an option is passed to inner +> sub-tests. You can force this with the `.Propagate` field, +> but usually you should not. ## How to use persistent cache -`testocache` stores key-value data between `go test` runs. -By default it uses `.testo_cache` in the test working directory. +`testocache` stores key-value data between `go test` runs: -```bash -go test ./path/to/package -cache.dir /tmp/my-testo-cache +```go +import "github.com/ozontech/testo/testocache" + +err := testocache.Set("token", []byte("abc")) +value, err := testocache.Get("token") ``` -When testing multiple packages with `./...`, use the environment variable so -packages that do not import `testocache` do not receive an unknown test flag: +By default the cache lives in `.testo_cache` in the test working +directory: ```bash +# change the directory for one package +go test ./path/to/package -cache.dir /tmp/my-testo-cache + +# for ./... use the env var, so packages that don't import +# testocache don't fail on an unknown flag TESTO_CACHE_DIR=/tmp/my-testo-cache go test ./... -``` -```bash +# disable the cache entirely TESTO_CACHE_DISABLE=true go test ./... ``` -For plugin state, prefer a namespace: +> [!NOTE] +> Cache flags use the `-cache.` prefix, not `-testo.` - +> there is no `-testo.cache.dir`. + +For plugin state, prefer a namespace. It is isolated from other +namespaces and from the package-level functions, and has the same +`Get`, `Set`, `Keys` and `Remove` methods: ```go var cache = testocache.Namespace("myplugin") ``` -Namespaces are isolated from each other and from package-level -`testocache.Get`, `Set`, `Keys`, and `Remove`. -The scoped cache provides the same `Get`, `Set`, `Keys`, and `Remove` methods. -`Keys` accepts the same glob syntax as `path.Match`. +Behavior details: -If cache is disabled, operations return `testocache.ErrDisabled`. -If a key is missing, `Get` and `Remove` return `testocache.ErrNotFound`. -Cache writes use a temporary file followed by `os.Rename`; atomic replacement -follows the guarantees of `os.Rename` on the host platform. -Malformed entries and entries whose stored key does not match the requested key -are treated as missing. Concurrent operations are synchronized inside one test -process; Testo does not provide cross-process locking for multiple `go test` -processes sharing the same cache directory. +- `Get` and `Remove` return `testocache.ErrNotFound` for missing keys. +- All operations return `testocache.ErrDisabled` when the cache is off. +- `Keys` accepts the same glob syntax as `path.Match`. +- Writes are atomic (temporary file + `os.Rename`); malformed entries + are treated as missing. +- Operations are synchronized within one process. There is no locking + between separate `go test` processes sharing a cache directory. -## How to structure tests +> [!NOTE] +> `go test` caches successful test results, and a cached pass skips +> execution - so `testocache` state will not refresh. Pass `-count=1` +> to force a run. -Testo does not enforce any particular file structure to work. -However, some patterns are proved to be useful. +## How to structure tests -### Standalone suites +Testo does not enforce any particular file structure. +Here is one pattern we find useful - standalone suite packages: ```txt go.mod @@ -233,18 +279,16 @@ import ( "github.com/ozontech/testo/testoplugin" ) -// Here we define a global plugin common for all tests. -// -// It is highly advised to make it even if you don't use any plugins yet, -// as in the future it won't require any further changes in other files if you -// decide to add plugins. +// A plugin shared by all tests. +// Define it even if you have no plugins yet. When you add +// a plugin later, you will change only this file. type PluginCommon struct { *testo.T } // This method implements Plugin interface. func (*PluginCommon) Plugin(testoplugin.Plugin, ...testoplugin.Option) testoplugin.Spec { - return testolpugin.Spec{} + return testoplugin.Spec{} } ``` @@ -253,9 +297,14 @@ Contents of `tests/suite/suitefoo/t.go`: ```go package suitefoo -import "example/tests/testcommon" +import ( + "example/tests/testcommon" + + "github.com/ozontech/testo" +) -// Global (common) T used by all suites. +// T for this suite. With several suites, +// hoist it into a shared package. type T struct { *testo.T *testcommon.PluginCommon @@ -269,7 +318,7 @@ package suitefoo import "github.com/ozontech/testo" -// An actual test logic for this suite goes here. +// The test logic for this suite goes here. type Suite struct{ testo.Suite[T] } @@ -293,12 +342,9 @@ import ( "github.com/ozontech/testo" ) -// In this file we actually run our suites. -// The reason for starting tests here is that -// single file makes it easier to see what suites will we run or skip. -// -// Moreover, if you add some build tag, like we do here ("integration" on the top), -// specifying it only once makes it less typo-prone. +// All suites are started here. +// One file shows at a glance which suites run, +// and the build tag ("integration") is declared once. func TestSuiteFoo(t *testing.T) { testo.RunSuite(t, new(suitefoo.Suite)) @@ -307,11 +353,30 @@ func TestSuiteFoo(t *testing.T) { ## How to run and skip specific tests -Testo works with default go test flags, such as `-run` and `-skip`. +Testo supports the standard `go test` flags, such as `-run` and `-skip`. + +See `go help testflag` for the full flag reference. + +To make hooks work correctly with parallel tests, Testo inserts a hidden +`testo!` level into every suite. The full name of a suite test is: -> See `go help testflag` for detailed flags description. +```txt +TestFunc/SuiteName/testo!/TestMethod[/sub-test...] +``` + +> [!WARNING] +> A `-run` pattern without the `testo!` segment, such as +> `-run 'Test/MySuite/TestFoo'`, matches **zero tests** - and `go test` +> still exits 0 (at best you get an easy-to-miss `[no tests to run]` +> note). Suite hooks (`BeforeAll`/`AfterAll`) still run even when zero +> tests match. +> +> Either include the segment (`-run 'Test/MySuite/testo!/TestFoo'`) +> or, better, use the `-testo.m` flag below. In VS Code, the +> [Testo extension](../vscode-extension) generates correct commands. -Testo also provides its own flag: `-testo.m regexp` to run specific suite tests. +Testo provides its own flag `-testo.m regexp` to select suite tests by +method name, without worrying about the `testo!` segment. For example, given the following suite: @@ -325,20 +390,108 @@ func (MySuite) TestFoo(t T) { func (MySuite) TestBar(t T) { // ... } + +func Test(t *testing.T) { + testo.RunSuite(t, new(MySuite)) +} ``` -We can run only `TestFoo` like that: +To run only `TestFoo`: ```shell -go test . -run ./MySuite -testo.m TestFoo +go test . -run 'Test/MySuite' -testo.m TestFoo +``` + +Here `-run 'Test/MySuite'` selects the suite and `-testo.m TestFoo` +selects the method inside it. + +> [!NOTE] +> `t.Name()` returns the name without `testo!`, e.g. `Test/MySuite/TestFoo`. +> The real `testing.T` name (the one in `go test -v` output and in `-run` +> patterns) still contains `testo!`. Remember this when you parse test +> output or build `-run` patterns. + +## How to enable strict mode + +Strict mode turns every Testo warning into a fatal error. +Currently two warnings exist - a suite with no tests, and a +`CasesXxx` method returning an +[empty slice](#how-to-write-parametrized-tests) - and the list +may grow: + +```bash +# one package +go test ./path/to/package -testo.strict + +# for ./... use the env var - packages that don't import +# Testo would fail on the unknown flag +TESTO_STRICT=true go test ./... ``` > [!TIP] -> See also [Visual Studio Code extension](../vscode-extension) which does just that for you. +> Flags have higher priority than environment variables. + +## How to annotate tests + +Annotations attach static plugin options to a specific test, so +plugins can see them before the test runs - for example to plan +retries or add report labels. + +Use `testo.For` for regular tests and `testo.ForEach` for parametrized ones: + +```go +var _ = testo.For(MySuite.TestFoo, myplugin.WithRetry()) + +func (MySuite) TestFoo(t T) { + // ... +} + +var _ = testo.ForEach(MySuite.TestBar, myplugin.WithRetry()) + +func (MySuite) TestBar(t T, p struct{ N int }) { + // ... +} +``` + +Multiple annotation calls for the same test append options. + +An option is just a `testoplugin.Option` value wrapping any type your +plugin knows how to interpret: + +```go +func WithRetry() testoplugin.Option { + return testoplugin.Option{Value: retryOption{}} +} +``` + +Plugins see annotations in two places: during planning, via +`PlannedTest.Annotations()` in `Plan.Prepare`, and per test, merged +into the `options` argument of the `Plugin` method. In both cases the +plugin type-asserts the `Value` field. + +See [options in the plugins guide](./plugins.md#options), +the [annotations example](../examples/07_annotations/main_test.go) +(its `plugin.go` defines the options) +and [API documentation](https://pkg.go.dev/github.com/ozontech/testo#For). + +## How to integrate with CI + +Testo output is standard `go test` output. `go test -json`, `test2json`, +[gotestsum](https://github.com/gotestyourself/gotestsum) and similar tools +work unchanged. + +Notes: + +- Read [how to run and skip specific tests](#how-to-run-and-skip-specific-tests). +- The hidden `testo!` node appears in reports as an extra nesting level + (e.g. JUnit converters render it as an empty intermediate node). +- For Allure reports, use the [testo-allure plugin](https://github.com/ozontech/testo-allure). +- For rerunning only failed tests (flaky-test handling), see the + [rerun plugin](https://github.com/ozontech/testo-toppings/tree/main/rerun). ## How to run sub-suites -There a `testo.RunSubSuite` function for that: +There is a `testo.RunSubSuite` function for that: ```go type OuterSuite struct{ testo.Suite[T] } @@ -354,5 +507,8 @@ func (InnerSuite) Test(t T) { } ``` +See the [sub-suites example](../examples/08_subsuites/main_test.go). + > [!WARNING] -> Running the same suite as sub-suite may cause infinite loop. +> Running a suite inside itself causes an infinite loop. +> Keep sub-suite nesting acyclic. diff --git a/docs/migration.md b/docs/migration.md new file mode 100644 index 0000000..ed12cbd --- /dev/null +++ b/docs/migration.md @@ -0,0 +1,143 @@ +# Migration guide + +How to move existing tests to Testo - from +[testify](#migrating-from-testify) or from +[allure-go](#migrating-from-allure-go). + +Both migrations can be done incrementally: Testo tests are regular +`go test` tests, so old and new suites run side by side in the same +package with no interference. Migrate one suite at a time. + +## Migrating from Testify + +### Suites and hooks + +| Testify | Testo | +| :--------------------------------------------------- | :---------------------------------------------------------------------- | +| `suite.Suite` (embedded) | `testo.Suite[T]` (embedded) | +| `suite.Run(t, new(S))` | `testo.RunSuite(t, new(S))` | +| `func (s *S) SetupSuite()` | `func (*S) BeforeAll(t T)` | +| `func (s *S) SetupTest()` | `func (*S) BeforeEach(t T)` | +| `func (s *S) TearDownTest()` | `func (*S) AfterEach(t T)` | +| `func (s *S) TearDownSuite()` | `func (*S) AfterAll(t T)` | +| `func (s *S) TestFoo()` | `func (*S) TestFoo(t T)` | +| `s.T()` | `t` (passed to every method) | +| `s.Run(name, func())` | `testo.Run(t, name, func(t T))` | +| `func (s *S) BeforeTest(suiteName, testName string)` | `BeforeEach(t T)` + `t.Name()` (same for `AfterTest`) | +| `SetupSubTest()` / `TearDownSubTest()` | `BeforeEachSub`/`AfterEachSub` plugin hooks (no suite-level equivalent) | +| `go test -run TestSuite -testify.m TestFoo` | `go test -run TestSuite -testo.m TestFoo` | + +> [!NOTE] +> `t.Name()` returns the full path (`Test/MySuite/TestFoo`), +> not the bare method name - compare with `strings.HasSuffix`, not `==`. +> And if a suite uses both `SetupTest` and `BeforeTest`, +> merge them into one `BeforeEach` (testify runs `SetupTest` first; keep that order). + +One caveat: testify method filters like `-run 'TestMySuite/TestFoo'` +stop matching after migration, because Testo inserts a hidden `testo!` +level into test names. Update CI filters to use `-testo.m`. See +[how to run and skip specific tests](./how-to.md#how-to-run-and-skip-specific-tests). + +`T` here is your own type built on `*testo.T` - in the simplest case +just an alias: + +```go +type T = *testo.T +``` + +See the [tutorial](./tutorial.md) for defining a `T` with plugins. + +### Assertions + +testify's `require` and `assert` packages work with Testo unchanged, +because `testo.T` implements `testing.TB`: + +```go +// testify/suite: +func (s *MySuite) TestFoo() { + s.Require().NoError(err) + s.Equal(want, got) // shorthand from the embedded suite.Suite +} + +// Testo: +func (*MySuite) TestFoo(t T) { + require.NoError(t, err) + assert.Equal(t, want, got) +} +``` + +Testo only replaces testify's `suite` package. Keep using `require` +and `assert`. + +### Suite state + +Testify suites keep per-test state in suite struct fields. +A Testo suite is also a single instance, so shared fields still work. +But writes to shared fields from parallel tests are a data race - +run migrated packages with `-race` before enabling `t.Parallel()`. +For per-test state, prefer [fixtures](./tutorial.md#fixtures) +(methods on your `T`) or plugin fields. Both get a fresh instance +for each test and are safe with parallel tests. + +### Parallelism + +Testify's `suite` does not support `t.Parallel()` ([issue #934](https://github.com/stretchr/testify/issues/934)). +With Testo, call `t.Parallel()` in any test - hooks handle parallel tests correctly. +See [how to write parallel tests](./how-to.md#how-to-write-parallel-tests). + +## Migrating from Allure-Go + +Testo [replaces Allure-Go at Ozon](../COMPARISON.md#why-we-moved-off-allure-go). +Reporting is now a separate plugin, not part of the framework. +So you need **two** pieces: `github.com/ozontech/testo` and the +[testo-allure](https://github.com/ozontech/testo-allure) plugin. + +```go +import ( + "github.com/ozontech/testo" + + allure "github.com/ozontech/testo-allure" +) + +type T struct { + *testo.T + *allure.PluginAllure +} +``` + +### Concept mapping + +| allure-go | Testo (+ testo-allure) | +| :------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `suite.Suite` | `testo.Suite[T]` | +| `suite.RunSuite(t, new(S))` (or `runner.NewSuiteRunner(...)`) | `testo.RunSuite(t, new(S))` | +| `BeforeAll(t provider.T)` | `BeforeAll(t T)` (same for the other hooks) | +| `func (s *S) TestFoo(t provider.T)` | `func (*S) TestFoo(t T)` | +| `t.WithNewStep("name", func(ctx provider.StepCtx) {...})` | `allure.Step(t, "name", func(t T) {...})` | +| `t.NewStep("name")` | `allure.Step(t, "name", func(T) {})` | +| `t.Title`, `t.Epic`, `t.Feature`, `t.Story`, `t.Tags`, `t.Severity`, `t.Owner`, `t.ID` | same-named methods added to `T` by `PluginAllure` | +| `t.Link(...)`, `t.TmsLink(...)`, `t.TmsLinks(...)` | `t.Links(...)` with the `allure.NewLink`/`allure.TMS`/`allure.Issue` constructors | +| `t.WithNewAttachment(name, mimeType, content)`, `t.WithAttachments(...)` | `t.Attach(name, allure.Bytes(...))` or `t.Attach(name, allure.File(...))` | +| table tests (`ParametrizedSuite` / `TableTestXxx` methods) | single struct-typed parameter, see [table tests](./how-to.md#table-tests-correlated-parameters) | +| `t.XSkip()` | `t.XFail()` via the [xfail plugin](https://github.com/ozontech/testo-toppings/tree/main/xfail) from testo-toppings (semantics differ slightly - check its README) | + +### Steps + +Steps are sub-tests. Two ways to create them: + +- `testo.Run(t, "name", func(t T) {...})` - the framework primitive. + `t.Fatal` inside stops only the sub-test, not the outer test. +- `allure.Step(t, "name", func(t T) {...})` - the testo-allure + helper. Same as `testo.Run`, but a fatal failure propagates to the + parent test and stops it - matching `WithNewStep` semantics from + allure-go. + +Use `allure.Step` instead of `WithNewStep`. +Nested steps (`sCtx.WithNewStep`, `sCtx.NewStep`) become nested +`allure.Step(t, ...)` calls. + +Assertions `sCtx.Assert()` and `sCtx.Require()` become `t.Assert()` & `t.Require()`, which `PluginAllure` adds to `T`. + +For the full testo-allure API (output directory, labels, attachments +and so on), see the +[testo-allure documentation](https://github.com/ozontech/testo-allure). diff --git a/docs/plugins.md b/docs/plugins.md new file mode 100644 index 0000000..a2be206 --- /dev/null +++ b/docs/plugins.md @@ -0,0 +1,264 @@ +# Writing plugins + +This page assumes you finished the [tutorial](./tutorial.md), where a +first plugin (the timer) is built step by step. + +A plugin is a struct that users embed into their `T`: + +```go +type T struct { + *testo.T + *PluginShout +} +``` + +Testo collects the plugins from the `T` type, creates a fresh +instance of each for every test and sub-test, and asks each one what +it wants to do by calling its `Plugin` method: + +```go +func (p *PluginShout) Plugin(parent testoplugin.Plugin, options ...testoplugin.Option) testoplugin.Spec +``` + +The returned `testoplugin.Spec` has three parts, all optional: + +| Part | What it does | +| :---------- | :---------------------------------------------------- | +| `Hooks` | run code around suites, tests and sub-tests | +| `Overrides` | wrap built-in `T` methods (`Log`, `Fail`, ...) | +| `Plan` | filter, reorder or duplicate the tests before the run | + +A plugin usually embeds `*testo.T`. Testo fills it with the same `T` +as the current test, so the plugin can log, fail, or inspect the test +it runs in. Pin the interface at compile time: + +```go +var _ testoplugin.Plugin = (*PluginShout)(nil) +``` + +The `parent` argument is the plugin instance of the enclosing scope: +the suite root for a test, the test for a sub-test. Only at the suite +root itself is it a typed-nil pointer. A non-nil interface can hold a +nil pointer, so `parent != nil` would lie - assert and nil-check the +pointer instead: + +```go +prev, _ := parent.(*PluginShout) +if prev != nil { + // not the suite root: prev belongs to the enclosing test or suite +} +``` + +Snippets in the Hooks, Overrides and Planning sections live inside +a `Plugin` method with a named return, like the tutorial's timer: +Go creates `spec` empty, the snippets fill its fields. + +## Hooks + +`BeforeAll`/`AfterAll` run once per suite, `BeforeEach`/`AfterEach` +around each test, `BeforeEachSub`/`AfterEachSub` around each +sub-test. The first four mirror the suite hook methods but are a +separate mechanism - both run, side by side. The `Sub` pair exists +only for plugins. + +```go +spec.Hooks.BeforeEach = testoplugin.Hook{ + Priority: testoplugin.TryFirst, + Func: func() { + p.Logf("starting %s", p.Name()) + }, +} +``` + +`Priority` is an `int` that orders hooks across plugins: lower +values run earlier, any value works, `TryFirst` and `TryLast` are +the extremes, zero keeps declaration order. Logs from hooks point at +the plugin's own file, which is expected. + +`AfterEach` and `AfterEachSub` are deferred, so with parallel +sub-tests they run before those sub-tests finish - same caveat as +[suite hooks](./how-to.md#hooks-and-parallel-sub-tests). + +## Overrides + +Overrides wrap built-in `T` methods. Testo hands your function the +current implementation (`next`) and installs whatever you return in +its place. When the test calls `t.Log`, your replacement runs and +decides if and when to call `next`. With several plugins the +replacements nest, like HTTP middleware. + +```go +spec.Overrides.Log = func(next testoplugin.FuncLog) testoplugin.FuncLog { + return func(args ...any) { + p.Helper() // keep the caller's file:line in the output + next(append([]any{"[quiet]"}, args...)...) + } +} +``` + +Call `t.Helper()` inside the wrapper, or every log line will point +at the wrapper's own file and line. + +Overridable methods: `Log`, `Error`, `Fatal`, `Fail`, `FailNow`, +`Failed`, `Skip`, `SkipNow`, `Skipped`, `Parallel`, `Cleanup`, +`Context`, `Deadline`, `TempDir`, `Setenv`, `Chdir`. +Each method `X` has a matching `testoplugin.FuncX` signature type. + +Methods call each other underneath - `Error` is `Log` + `Fail`, +`Fatal` is `Log` + `FailNow` - so overriding `Log` also affects +`Error` and `Fatal`. `Overrides.Priority` orders stacks across +plugins: the lowest priority becomes the outermost wrapper, so its +code runs first. For a real example, the +[xfail plugin](https://github.com/ozontech/testo-toppings/tree/main/xfail) +overrides `Fail` and `FailNow` to turn expected failures into skips. + +## Planning tests + +`Plan.Prepare` receives the collected tests before the run and may +filter, reorder or duplicate them in place: + +```go +import "github.com/ozontech/testo/testoreflect" + +spec.Plan.Prepare = func(suite testoreflect.SuiteInfo, tests *[]testoplugin.PlannedTest) { + slices.Reverse(*tests) +} +``` + +Each `PlannedTest` exposes `Annotations()` - the options attached to +that test with [`testo.For`](./how-to.md#how-to-annotate-tests) - and +`Info()`, which returns a `testoreflect.TestInfo` interface: + +```go +name := t.Info().GetName() +``` + +`GetName()` returns the full run path without the `testo!` segment, +e.g. `Test/Suite/TestFoo#01` - match with `strings.HasSuffix`, not `==`. + +Parametrized tests arrive already expanded, one `PlannedTest` per +case. The initial order is regular tests first (alphabetical), then +parametrized cases. One display quirk: after reordering, `go test` +reassigns the `#01` suffixes by run order, while `t.Name()` keeps the +original case index. + +The [rerun plugin](https://github.com/ozontech/testo-toppings/tree/main/rerun) +uses `Prepare` to drop every test that passed in the previous run. + +## Adding methods to T + +Any method on the plugin becomes a method on `T` through embedding. +If methods are all your plugin does, you don't even need a `Plugin` +method: + +```go +type PluginGreet struct{ *testo.T } + +func (g *PluginGreet) Greet() { g.Log("hello") } +``` + +```go +func (Suite) TestFoo(t T) { + t.Greet() +} +``` + +## Options + +An option is a `testoplugin.Option` wrapping any value your plugin +recognizes. Users pass options to `testo.RunSuite`, `testo.Run` or +`testo.Options` - see +[how to use plugin options](./how-to.md#how-to-use-plugin-options). +The common pattern is an unexported function type: + +```go +type shoutOption func(*PluginShout) + +func WithPrefix(prefix string) testoplugin.Option { + return testoplugin.Option{ + Value: shoutOption(func(p *PluginShout) { p.prefix = prefix }), + } +} +``` + +Set `Propagate: true` only when sub-tests need the option too; +leave it unset otherwise. + +Consume options at the top of the `Plugin` method. All user-supplied +options come through the variadic argument (including per-test +[annotations](./how-to.md#how-to-annotate-tests)); ignore values that +are not yours: + +```go +func (p *PluginShout) Plugin(_ testoplugin.Plugin, options ...testoplugin.Option) (spec testoplugin.Spec) { + for _, o := range options { + if o, ok := o.Value.(shoutOption); ok { + o(p) + } + } + // ... + return spec +} +``` + +## Command line flags + +Plugins register flags with the standard `flag` package at package +level. The `go test` machinery parses them like any other test flag: + +```go +var flagShout = flag.Bool("shout.enabled", false, "print a banner before each test") +``` + +```bash +go test . -shout.enabled +``` + +Read the value inside `Plugin` or a hook, after flags are parsed. +Prefix flag names with your plugin name to avoid collisions. Real +examples: `-rerun.failed` in the rerun plugin, `-allure.dir` in +testo-allure. + +Flags only exist in packages that import the plugin, so `go test +./... -shout.enabled` fails in packages that don't. Pair each flag +with an environment variable if your plugin must be configurable +across a whole repo (Testo does this with `TESTO_STRICT` and +`TESTO_CACHE_DIR`). + +## Reading test metadata + +`testo.Reflect` works on any `T` - including the plugin itself, via +its embedded `*testo.T`: + +```go +spec.Hooks.BeforeEach.Func = func() { + info, ok := testo.Reflect(p).Test.(testoreflect.ParametrizedTestInfo) + if ok { + p.Logf("params: %v", info.Params) + } +} +``` + +That snippet answers "which parameters did `TestFoo#01` run with": +the plugin logs them for every parametrized test. The `Reflection` struct also carries the suite info, failure +kind and source, and panic details - which is how +[testo-allure](https://github.com/ozontech/testo-allure) fills its +reports. See the +[testoreflect reference](https://pkg.go.dev/github.com/ozontech/testo/testoreflect). + +## Talking to other plugins + +Plugins reference each other by embedding; Testo reuses one instance +per test across all references. See +[cross-plugin communication](./technical-overview.md#plugins). + +## Plugins worth reading + +- [examples/04_plugins](../examples/04_plugins/main_test.go) - hooks, + overrides and planning in one small file. +- [examples/07_annotations](../examples/07_annotations/plugin.go) - a + plugin that defines options and consumes annotations in `Plan`. +- [testo-toppings](https://github.com/ozontech/testo-toppings) - four + small production plugins (rerun, xfail, parallel, async). +- [testo-allure](https://github.com/ozontech/testo-allure) - a large + reporting plugin using every part of the `Spec`. diff --git a/docs/technical-overview.md b/docs/technical-overview.md index 836aff3..eece09f 100644 --- a/docs/technical-overview.md +++ b/docs/technical-overview.md @@ -1,6 +1,26 @@ # Technical overview -A technical overview of Testo. +How Testo works under the hood. Read this if you write plugins or +debug hook ordering. + +## Mechanism + +Testo works through runtime reflection over suite method sets and `T` +types. There is no code generation and no separate CLI. +This puts a few constraints on your types, checked before any +test runs: + +- Plugins must be embedded as pointers. More generally, every + exported field of `T` and of plugin structs must be a pointer type. +- Recursive plugin type references are detected and rejected. + +Signature and `CasesXxx` mismatches are reported as test errors. +Violations of the type constraints above panic before any test runs, +aborting the test binary. + +Plugins are re-instantiated for every test and sub-test. This is what +makes writing to plugin fields safe without synchronization, at the +cost of a constructor call per (sub-)test. ## Lifecycle @@ -10,25 +30,24 @@ When you call `testo.RunSuite` the following happens: A root test named the same as a suite is run { Suite tests are collected and verified. - Plugins are collected and initialized with ".Plugin(parent, options)" method call, if implemented. Innermost plugins are initialized first. - At this top level, parent is a typed-nil instance of the plugin's own type: the interface is non-nil, but the concrete pointer inside is nil. + Plugins are initialized (innermost first; see note below). "BeforeAll" plugin hooks are called. "BeforeAll" suite hook is called. - "CasesXXX" functions are called and parametrized tests are collected. + "CasesXxx" functions are called and parametrized tests are collected. Test plan from plugins is applied to the final test collection. A test named "testo!" is run { For each test in collection { - Plugins are collected and initialized with ".Plugin(parent: parent, options)" method call, if implemented. Innermost plugins are initialized first. + Plugins are initialized (innermost first). "BeforeEach" plugin hooks are called. "BeforeEach" suite hook is called. Actual test is run { - For each sub-test in test (ran through "testo.Run") { - Plugins are collected and initialized with ".Plugin(parent: parent, options)" method call, if implemented. Innermost plugins are initialized first. + For each sub-test in test (run via "testo.Run") { + Plugins are initialized (innermost first). "BeforeEachSub" plugin hooks are called. @@ -48,17 +67,32 @@ A root test named the same as a suite is run { } ``` +Plugin initialization calls the `.Plugin(parent, options)` method, +if implemented, innermost plugins first: a plugin embedded by another +plugin initializes before the plugin embedding it. At the suite root, +`parent` is a typed-nil instance of the plugin's own type: the +interface is non-nil, but the concrete pointer inside is nil. Below +the root, `parent` is the plugin instance of the enclosing scope. + ## Panics -Testo **will** catch panics from tests, including `BeforeEach`, `BeforeEachSub`, `AfterEachSub` & `AfterEach` hooks. -Other tests will run even if some tests are panicking. +Unlike plain `go test`, a panicking test does not abort the whole +binary: Testo catches panics from tests and from all hooks. +Other tests keep running even if some panic. + +A panic in `BeforeAll` prevents the suite's tests from running. +A panic in `AfterAll` is caught like any other hook panic and fails +the suite. + +If the suite-level `T` is skipped (for example, `t.Skip` inside +`BeforeAll`), the `AfterAll` hooks are skipped too. -Testo **will** catch panics from `BeforeAll` & `AfterAll`. -Panic in these hooks will result in suite tests not running. +See also [suite hooks in the tutorial](./tutorial.md#suite-hooks) and +[hook behavior with parallel tests](./how-to.md#how-to-write-parallel-tests). ## Plugins -Testo uses dependency-injection-like mechanism to enable cross-plugin communication. +Testo uses a dependency-injection-like mechanism for cross-plugin communication. For example, assume we have plugin `X` and plugin `Y`. Plugin `X` needs to interact with plugin `Y`. To make it possible, `X` needs to embed `Y`: diff --git a/docs/tutorial.md b/docs/tutorial.md index d145dd4..3be4124 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -1,363 +1,700 @@ # Tutorial -## Create a New Go Project +In this tutorial we take a plain Go test and grow it, step by step, +into a full Testo suite with a plugin, parametrization, hooks, +fixtures and steps. + +Each section changes only a small piece of code, and the file compiles +and passes after every change. The complete final file is listed +[at the end](#the-finished-test-file). + +## Setting Up + +Create a new project and add Testo to it: ```bash -mkdir testo-tutorial -cd testo-tutorial -go mod init testo-tutorial +mkdir testo-bakery +cd testo-bakery +go mod init testo-bakery +go get github.com/ozontech/testo ``` -Create `main.go`: +We are going to test a tiny bakery. Create `main.go`: ```go package main -// Add returns sum of the given integers. -func Add(a, b int) int { return a + b } +// Pastry is something a bakery bakes. +type Pastry struct { + Name string + Tasty bool + Ingredients string +} + +var recipes = map[string]string{ + "honey cake": "honey, flour, eggs, sour cream", + "tiramisu": "mascarpone, coffee, ladyfingers", +} + +// Bake bakes a pastry, if the bakery knows the recipe. +func Bake(name string) (Pastry, bool) { + ingredients, ok := recipes[name] + if !ok { + return Pastry{}, false + } + + return Pastry{Name: name, Tasty: true, Ingredients: ingredients}, true +} + +// Eat disposes of a pastry in the most pleasant way possible. +func Eat(Pastry) {} + +// Oven turns the bakery oven on or off. +func Oven(on bool) error { return nil } func main() {} ``` -## Writing Tests +## A Plain Go Test -Testo uses `testo.T` - a wrapper around `testing.T` with extended features. -All methods available in `testing.T` are also available in `testo.T` (except `t.Run`, more on that later in [Sub-tests chapter](#sub-tests)). -Also, `testo.T` implements the `testing.TB` interface. +We start without Testo. Create `main_test.go`: -**Important**: We'll create an alias to avoid repetition. +```go +package main + +import "testing" + +func TestBake(t *testing.T) { + t.Run("HoneyCake", func(t *testing.T) { + pastry, ok := Bake("honey cake") + if !ok { + t.Fatal("the bakery must know how to bake a honey cake") + } + + // no reason to waste it after the test + t.Cleanup(func() { Eat(pastry) }) -Create file `main_test.go`: + if !pastry.Tasty { + t.Error("honey cake must be tasty") + } + }) +} +``` + +```bash +go test . -v +``` + +```txt +=== RUN TestBake +=== RUN TestBake/HoneyCake +--- PASS: TestBake (0.00s) + --- PASS: TestBake/HoneyCake (0.00s) +PASS +``` + +So far this is a plain Go test. Let's bring in Testo. + +## Switching to Testo + +Replace the `t.Run` call with `testo.RunTest`: ```go package main import ( - "testing" + "testing" - "github.com/ozontech/testo" + "github.com/ozontech/testo" ) -type T = *testo.T +func TestBake(t *testing.T) { + testo.RunTest(t, func(t *testo.T) { + pastry, ok := Bake("honey cake") + if !ok { + t.Fatal("the bakery must know how to bake a honey cake") + } + + // no reason to waste it after the test + t.Cleanup(func() { Eat(pastry) }) + + if !pastry.Tasty { + t.Error("honey cake must be tasty") + } + }) +} ``` -Now we need a Suite. A Suite must "inherit" `testo.Suite[T]` by embedding it. +The body of the test did not change - only the type of `t`. +This swap lets everything below (plugins, suites, parametrization) +attach to a plain test. +The `HoneyCake` name is gone, though: `RunTest` numbers its tests +instead. Suites (below) give tests proper names again. -> [!NOTE] -> It's possible to run tests without suites, more on that in [later](#running-tests-without-suites). +`testo.T` wraps `testing.T` and keeps its interface, with one +exception: sub-tests are started with `testo.Run` instead of `t.Run` +(covered [later](#steps-sub-tests)). It also implements `testing.TB`, +so assertion libraries and mocks built for standard tests keep +working. + +Run the test again: + +```txt +=== RUN TestBake +=== RUN TestBake/#00 +=== RUN TestBake/#00/testo! +=== RUN TestBake/#00/testo!/TestBake +--- PASS: TestBake (0.00s) + --- PASS: TestBake/#00 (0.00s) + --- PASS: TestBake/#00/testo! (0.00s) + --- PASS: TestBake/#00/testo!/TestBake (0.00s) +PASS +``` + +Two technical levels appeared in the names: + +- `#00` is the index of the test inside this `RunTest` call. +- `testo!` is a special test Testo inserts so hooks run correctly + around parallel tests. It does not affect your tests, but it + does affect `-run` patterns - see + [how to run specific tests](./how-to.md#how-to-run-and-skip-specific-tests). + +So far Testo has given us nothing new. Plugins are where it gets +interesting. + +## Plugins + +A plugin is a struct that describes what it does through the +`testoplugin.Plugin` interface: + +```go +type Plugin interface { + Plugin(parent Plugin, options ...Option) Spec +} + +type Spec struct { + // Plan tests for execution: filter, duplicate, reorder. + Plan Plan + + // Hooks around suites, tests and sub-tests. + Hooks Hooks + + // Middleware for built-in T methods, + // such as Fail, Log, Skip and others. + Overrides Overrides +} +``` + +Let's write a plugin that measures how long each test takes. +In `main_test.go`, update the import block as shown below and add +the plugin after it: ```go -type Suite struct{ testo.Suite[T] } +import ( + "testing" + "time" + + "github.com/ozontech/testo" + "github.com/ozontech/testo/testoplugin" +) -// Tests in Suites are regular methods, following the same -// naming rules as regular tests in Go. -// That means they must have the "Test" prefix. -// They also must use the same type T as specified in `testo.Suite[T]`. +type PluginTimer struct { + *testo.T -func (*Suite) TestAdd(t T) { - if Add(2, 2) != 4 { - t.Fatal("2 + 2 must equal 4") - } + start time.Time +} + +func (p *PluginTimer) Plugin(testoplugin.Plugin, ...testoplugin.Option) (spec testoplugin.Spec) { + spec.Hooks.BeforeEach.Func = func() { + p.start = time.Now() + } + + spec.Hooks.AfterEach.Func = func() { + p.Logf("test %q took %s", p.Name(), time.Since(p.start)) + } + + return spec } ``` -## Running Tests +Two Go details in the signature above. The parameters are unnamed +because this plugin ignores them. And `(spec testoplugin.Spec)` is a +named return: Go creates `spec` as an empty value, the method fills +its fields and returns it. + +The plugin embeds `*testo.T`. Testo fills it with the same `T` as +the current test, so the plugin sees everything the test sees. +The unused first argument of the `Plugin` method is the parent plugin +instance - see the +[technical overview](./technical-overview.md#lifecycle) if you need it. -To connect Testo with `go test` we must run it from a regular test: +The `Plugin` method is called for each test and sub-test, and each +gets its own plugin instance. That is why writing to plugin fields +is safe. + +To use the plugin, define your own `T` type that embeds `*testo.T` +together with the plugins you want: ```go -func Test(t *testing.T) { - testo.RunSuite(t, new(Suite)) +type T struct { + *testo.T + *PluginTimer } ``` -Now we can run tests: +(Both fields contain a `*testo.T`, but the direct one is shallower, +so calls like `t.Log` stay unambiguous.) -```bash -go test . -v +And change the test function to take this `T` - no `*` this time, +since our `T` is a struct that already contains the pointers: + +```go +func TestBake(t *testing.T) { + testo.RunTest(t, func(t T) { + // ... body unchanged ... + }) +} ``` +That is the whole installation. Testo looks at the `T` type a test +accepts, then collects and initializes the plugins listed in it: + ```txt -=== RUN Test -=== RUN Test/Suite -=== RUN Test/Suite/testo! -=== RUN Test/Suite/testo!/TestAdd ---- PASS: Test (0.00s) - --- PASS: Test/Suite (0.00s) - --- PASS: Test/Suite/testo! (0.00s) - --- PASS: Test/Suite/testo!/TestAdd (0.00s) +=== RUN TestBake +=== RUN TestBake/#00 + main_test.go:35: testo: plugins collected: 1: main.PluginTimer +=== RUN TestBake/#00/testo! +=== RUN TestBake/#00/testo!/TestBake + main_test.go:23: test "TestBake/#00/TestBake" took 98.125µs +--- PASS: TestBake (0.00s) +... PASS ``` -You may notice a special test `testo!` - Testo creates it "under the hood" -to guarantee correct work of hooks with parallel tests. +Your timings will differ. The logged name has no `testo!` in it: +`t.Name()` returns the logical test name +([details](./how-to.md#how-to-run-and-skip-specific-tests)). -Don't worry too much about this, as its existence doesn't affect your tests. - -## Suite Hooks +> [!NOTE] +> Plugins must be embedded as pointers. Pointers let plugins share +> state with each other by pointing to the same memory. -Suites can have the following hooks: +Plugins can do much more than hooks: reorder or filter the test plan, +wrap built-in methods like `t.Log`, add new methods to `T`, accept +[options](./how-to.md#how-to-use-plugin-options) and command line +flags. The [writing plugins guide](./plugins.md) covers the full +API with an example for each part. -- `BeforeAll(T)` - called _once before_ all tests. The passed `T` refers to the top test, i.e. `Test/Suite`. -- `BeforeEach(T)` - called _before each_ test. The passed `T` refers to the same test that the hook runs before. -- `AfterEach(T)` - called _after each_ test, but before `t.Cleanup`. The passed `T` refers to the same test that the hook runs after. -- `AfterAll(T)` - called _once after_ all tests. The hook waits for all (parallel) tests to finish. The passed `T` refers to the top test, i.e. `Test/Suite`. +Ready-made plugins: +[testo-allure](https://github.com/ozontech/testo-allure) for Allure +reports, and [testo-toppings](https://github.com/ozontech/testo-toppings) - +a collection of small utility plugins. -Hooks are defined as Suite methods: +For instance, here is a complete plugin that makes every test parallel: ```go -func (*Suite) BeforeEach(t T) { - t.Logf("Starting: %s", t.Name()) +type PluginParallel struct{ *testo.T } + +func (p *PluginParallel) Plugin(testoplugin.Plugin, ...testoplugin.Option) (spec testoplugin.Spec) { + spec.Hooks.BeforeEach.Func = func() { + p.Parallel() + } + + return spec } +``` + +Don't add this one to your file, or it will change the outputs +below. A more complete version is available as the +[parallel plugin](https://github.com/ozontech/testo-toppings/tree/main/parallel) +in testo-toppings. + +## Suites -func (*Suite) AfterEach(t T) { - t.Logf("Finished: %s", t.Name()) +When tests share setup and plugins, it is convenient to group them +into a suite. A suite embeds `testo.Suite[T]`, where `T` is the type +we defined above - i.e. the set of plugins every test in the suite +gets: + +```go +type Bakery struct{ testo.Suite[T] } +``` + +Tests become methods of the suite. They follow the usual Go naming +rules (the `Test` prefix) and must accept the same `T` as specified in +`testo.Suite[T]`. Value and pointer receivers both work. + +Replace the `TestBake` function with: + +```go +func (Bakery) TestBake(t T) { + pastry, ok := Bake("honey cake") + if !ok { + t.Fatal("the bakery must know how to bake a honey cake") + } + + t.Cleanup(func() { Eat(pastry) }) + + if !pastry.Tasty { + t.Error("honey cake must be tasty") + } } ``` -
-Output: +Go has no native notion of suites, so we launch the suite from one +regular test function: + +```go +func Test(t *testing.T) { + testo.RunSuite(t, new(Bakery)) +} +``` ```txt === RUN Test -=== RUN Test/Suite -=== RUN Test/Suite/testo! -=== RUN Test/Suite/testo!/Add - main_test.go:20: Starting: Test/Suite/TestAdd - main_test.go:28: Test/Suite/TestAdd - main_test.go:24: Finished: Test/Suite/TestAdd +=== RUN Test/Bakery + main_test.go:50: testo: plugins collected: 1: main.PluginTimer +=== RUN Test/Bakery/testo! +=== RUN Test/Bakery/testo!/TestBake + main_test.go:23: test "Test/Bakery/TestBake" took 66.583µs --- PASS: Test (0.00s) - --- PASS: Test/Suite (0.00s) - --- PASS: Test/Suite/testo! (0.00s) - --- PASS: Test/Suite/testo!/TestAdd (0.00s) + --- PASS: Test/Bakery (0.00s) + --- PASS: Test/Bakery/testo! (0.00s) + --- PASS: Test/Bakery/testo!/TestBake (0.00s) PASS ``` -
- ## Parametrized Tests -Parametrized tests let you run the same tests with different input parameters. +Our bakery bakes more than honey cake, and the test scenario is the +same for every dessert. Instead of copying the test, we make the +dessert a parameter. -Parametrized tests follow the same naming rules as regular tests, but take -a second argument after `T`. -This argument must be a struct containing the required parameters. +A parametrized test takes a second argument after `T`: a struct whose +fields are the parameters. Replace `TestBake` with: ```go -func (*Suite) TestAddButParametrized(t T, p struct{ A, B int }) { - if Add(p.A, p.B) != Add(p.B, p.A) { - t.Errorf("%[1]d + %[2]d != %[2]d + %[1]d", p.A, p.B) - } +func (Bakery) TestBake(t T, p struct{ Dessert string }) { + pastry, ok := Bake(p.Dessert) + if !ok { + t.Fatalf("the bakery must know how to bake %s", p.Dessert) + } + + t.Cleanup(func() { Eat(pastry) }) + + if !pastry.Tasty { + t.Errorf("%s must be tasty", p.Dessert) + } } ``` -We also need to declare the parameter values with which the test will run. -This is done by declaring special methods with names like `CasesXXX`, where -`XXX` is the parameter name. +Parameter values are declared with `CasesXxx` methods, where `Xxx` +matches the field name: ```go -func (*Suite) CasesA() []int { - return []int{1, 2, 3, 4, 5} +func (Bakery) CasesDessert() []string { + return []string{"honey cake", "tiramisu"} } +``` -func (*Suite) CasesB() []int { - return []int{11, 1000, 13} -} +```txt +=== RUN Test +=== RUN Test/Bakery + main_test.go:54: testo: plugins collected: 1: main.PluginTimer +=== RUN Test/Bakery/testo! +=== RUN Test/Bakery/testo!/TestBake + main_test.go:23: test "Test/Bakery/TestBake" took 108.791µs +=== RUN Test/Bakery/testo!/TestBake#01 + main_test.go:23: test "Test/Bakery/TestBake#01" took 15.042µs +--- PASS: Test (0.00s) + --- PASS: Test/Bakery (0.00s) + --- PASS: Test/Bakery/testo! (0.00s) + --- PASS: Test/Bakery/testo!/TestBake (0.00s) + --- PASS: Test/Bakery/testo!/TestBake#01 (0.00s) +PASS ``` -Parametrized tests are called with the [Cartesian product](https://en.wikipedia.org/wiki/Cartesian_product) -of all values obtained from `CasesXXX` functions. -For the example above, this is 15 different pairs of values. +The test ran once per dessert. The name `TestBake#01` does not say +which dessert it was, so log `p.Dessert` at the top of the test if +that matters to you. With several parameters, the test runs +with the [Cartesian product](https://en.wikipedia.org/wiki/Cartesian_product) +of all their values. For correlated values (classic table tests), use +a single struct-typed parameter - see +[how to write parametrized tests](./how-to.md#how-to-write-parametrized-tests). + +If a test declares a parameter with no matching `CasesXxx` method, or +the types mismatch, Testo reports an informative error before running +any tests. The [errors example](../examples/06_errors/main_test.go) +demonstrates these messages (that example fails on purpose). -If a test specifies a parameter for which no corresponding `Cases` function is found, -Testo will print an informative error before running any tests. -The same applies to type mismatches. +## Suite Hooks -See [error examples](../../examples/07_errors/main_test.go). +One day the whole bakery stopped producing pastry. It turned out +somebody had switched off the oven. The suite should manage the oven +itself. -## Sub-tests +Suites can define these hooks as methods: -Running sub-tests is a bit different from the usual `t.Run` call. +- `BeforeAll(T)` - called _once before_ all tests. Its `T` refers to + the top-level test, i.e. `Test/Bakery`. +- `BeforeEach(T)` - called _before each_ test, with that test's `T`. +- `AfterEach(T)` - called _after each_ test, before `t.Cleanup` + callbacks, with that test's `T`. +- `AfterAll(T)` - called _once after_ all tests, including parallel + ones. Its `T` refers to the top-level test. -Sub-tests must be started using the `testo.Run` function: +We need the oven on once before all tests and off once after them: ```go -func (*Suite) CasesC() []int { - return []int{-4, -99, 9} +func (Bakery) BeforeAll(t T) { + if err := Oven(true); err != nil { + t.Fatalf("failed to turn the oven on: %v", err) + } } -func (*Suite) TestAddButParametrized(t T, p struct{ A, B, C int }) { - testo.Run(t, "commutative", func(t T) { - if Add(p.A, p.B) != Add(p.B, p.A) { - t.Errorf("%[1]d + %[2]d != %[2]d + %[1]d", p.A, p.B) - } - }) - - testo.Run(t, "associative", func(t T) { - if Add(Add(p.A, p.B), p.C) != Add(p.A, Add(p.B, p.C)) { - t.Errorf("(%[1]d + %[2]d) + %[3]d != %[1]d + (%[2]d + %[3]d)", p.A, p.B, p.C) - } - }) +func (Bakery) AfterAll(t T) { + if err := Oven(false); err != nil { + t.Errorf("failed to turn the oven off: %v", err) + } } ``` -## Plugins +The run output looks the same as before: these hooks are silent. +Add a `t.Log` inside one if you want to see it fire. -One of the main features of Testo is the plugin system. +If `BeforeAll` fails, the suite's tests do not run at all. +For hook behavior with parallel sub-tests and panics, see +[how to write parallel tests](./how-to.md#how-to-write-parallel-tests) +and the [technical overview](./technical-overview.md#panics). -### Writing Plugins +## Fixtures -Plugin examples: +The test currently creates a pastry and remembers to clean it up. +With more resources this gets repetitive, so we move the +create-and-cleanup logic out of the test. -1. A plugin that runs tests in reverse order. -2. A plugin that extends the `t.Log` method. -3. A plugin that adds new methods for `T` (fixtures). -4. A plugin that measures the time of each test. +`T` is our own type, which means we can add methods to it. A method +that creates a resource and schedules its cleanup is a fixture: ```go -import ( - "github.com/ozontech/testo" - "github.com/ozontech/testo/plugin" -) +// Bake is a fixture: it bakes a pastry and +// schedules the cleanup for the end of the test. +func (t T) Bake(name string) (Pastry, bool) { + pastry, ok := Bake(name) + + if ok { + t.Cleanup(func() { Eat(pastry) }) + } -// We can embed testo.T in plugins - it will point to the same T -// as in the current test. -type ReverseTestsOrder struct{ *testo.T } - -// Plugins must implement this function -// so that testo understands it's a plugin and can handle it correctly. -func (*ReverseTestsOrder) Plugin(testoplugin.Plugin, ...testoplugin.Option) testoplugin.Spec { - return testoplugin.Spec{ - Plan: testoplugin.Plan{ - Modify: func(tests *[]testoplugin.PlannedTest) { - slices.Reverse(*tests) - }, - }, - } -} - -type OverrideLog struct { *testo.T } - -func (*OverrideLog) Plugin(testoplugin.Plugin, ...testoplugin.Option) testoplugin.Spec { - return testoplugin.Spec{ - Overrides: testoplugin.Overrides{ - Log: func(f testoplugin.FuncLog) testoplugin.FuncLog { - return func(args ...any) { - // will be called each time t.Log is called - fmt.Println("Inside log override") - f(args...) - } - }, - }, - } -} - -type AddNewMethods struct{ *testo.T } - -func (*AddNewMethods) Plugin(testoplugin.Plugin, ...testoplugin.Option) testoplugin.Spec { - // We have nothing to use here, so we return an empty specification - return testoplugin.Spec{} -} - -// Later we will see how we can call this method from tests. -func (a *AddNewMethods) Explode() { a.Fatal("BOOM") } - -type Timer struct { - *testo.T - start time.Time -} - -func (t *Timer) Plugin(testoplugin.Plugin, ...testoplugin.Option) testoplugin.Spec { - return testoplugin.Spec{ - Hooks: testoplugin.Hooks{ - BeforeEach: testoplugin.Hook{ - Priority: testoplugin.TryLast, - Func: func() { - // the .Plugin method is called for each (sub-)test, - // so we can modify fields without risk of synchronization errors. - t.start = time.Now() - }, - }, - AfterEach: testoplugin.Hook{ - Priority: testoplugin.TryFirst, - Func: func() { - elapsed := time.Since(t.start) - - fmt.Printf("Test %q took %s\n", t.Name(), elapsed) - }, - }, - }, - } + return pastry, ok } ``` -### Using Plugins - -Recall the alias for `T` defined earlier: +The `Bake(name)` inside the method is the package function from +`main.go` - methods don't shadow package names, so this is not +recursion. The test no longer deals with cleanup: ```go -type T = *testo.T +func (Bakery) TestBake(t T, p struct{ Dessert string }) { + pastry, ok := t.Bake(p.Dessert) + if !ok { + t.Fatalf("the bakery must know how to bake %s", p.Dessert) + } + + if !pastry.Tasty { + t.Errorf("%s must be tasty", p.Dessert) + } +} ``` -Now we can add (install) plugins to it: +A fixture in Testo is an ordinary method. There is nothing to +register, and every test of the suite can use it. -```go -type T struct{ - *testo.T - *ReverseTestsOrder - *OverrideLog - *AddNewMethods - *Timer -} -``` +## Steps (Sub-tests) -That's the only change. -Testo will take care of the rest. +Right now a failure tells us *which* test failed, but not *which +check* inside it. We can structure the test into named steps using +sub-tests. -And since `AddNewMethods` is now embedded in `T`, we can use its methods without magic: +Sub-tests are started with the `testo.Run` function (this is the one +place where Testo differs from `t.Run`). Replace `TestBake` with: ```go -func (*Suite) TestBoom(t T) { - t.Explode() +func (Bakery) TestBake(t T, p struct{ Dessert string }) { + pastry, ok := t.Bake(p.Dessert) + + testo.Run(t, "check the bakery can bake it", func(t T) { + if !ok { + t.Fatalf("the bakery must know how to bake %s", p.Dessert) + } + }) + + testo.Run(t, "check it is tasty", func(t T) { + if !pastry.Tasty { + t.Errorf("%s must be tasty", p.Dessert) + } + }) } ``` -> [!NOTE] -> Plugins must be installed as pointers. -> -> Pointers allow plugins to share their state with other plugins, -> by pointing to the same memory location through pointers. +```txt +=== RUN Test/Bakery/testo!/TestBake +=== RUN Test/Bakery/testo!/TestBake/check_the_bakery_can_bake_it +=== RUN Test/Bakery/testo!/TestBake/check_it_is_tasty +... +--- PASS: Test/Bakery/testo!/TestBake (0.00s) + --- PASS: Test/Bakery/testo!/TestBake/check_the_bakery_can_bake_it (0.00s) + --- PASS: Test/Bakery/testo!/TestBake/check_it_is_tasty (0.00s) +``` + +> [!WARNING] +> `t.Fatal` inside a sub-test stops only that sub-test, not the outer +> test. That is standard `go test` behavior. If a failed step must stop the +> whole test, check the sub-test result in the outer test, or use a +> plugin that propagates the failure, such as `allure.Step` from +> [testo-allure](https://github.com/ozontech/testo-allure). + +Plugins get their own hooks around every sub-test: +`BeforeEachSub` and `AfterEachSub`. +Reporting plugins use sub-tests as steps - with +[testo-allure](https://github.com/ozontech/testo-allure) installed, +each `testo.Run` above becomes a step in the Allure report. -## Running tests without suites +## Running Tests Without Suites -It's possible: +Suites are optional. You have already seen `testo.RunTest`: ```go -type T struct{ - *testo.T - *ReverseTestsOrder - *OverrideLog - *AddNewMethods - *Timer +func TestFoo(t *testing.T) { + testo.RunTest(t, func(t T) { + t.Log("Hello from Testo!") + }) } +``` +And to run several tests from a single test function, there is the +`testo.Test` adapter for `t.Run`: + +```go func TestFoo(t *testing.T) { - testo.RunSuite(t, func(t T) { - t.Log("Hello from testo!") - }) + t.Run("FirstTest", testo.Test(func(t T) { + t.Log("1!") + })) + + t.Run("SecondTest", testo.Test(func(t T) { + t.Log("2!") + })) } ``` -Or, if you need to run several tests from a single "real" test: +Each `testo.Test` call gets its own technical levels, so the names +run deep - that is normal: + +```txt +=== RUN TestFoo/FirstTest +=== RUN TestFoo/FirstTest/#00 +=== RUN TestFoo/FirstTest/#00/testo! +=== RUN TestFoo/FirstTest/#00/testo!/FirstTest +... +``` + +## The Finished Test File + +The complete `main_test.go` we built in this tutorial: ```go -func TestFoo(t *testing.T) { - t.Run("FirstTest", testo.Test(func(t T) { - t.Log("1!") - })) +package main + +import ( + "testing" + "time" + + "github.com/ozontech/testo" + "github.com/ozontech/testo/testoplugin" +) + +type PluginTimer struct { + *testo.T + + start time.Time +} + +func (p *PluginTimer) Plugin(testoplugin.Plugin, ...testoplugin.Option) (spec testoplugin.Spec) { + spec.Hooks.BeforeEach.Func = func() { + p.start = time.Now() + } + + spec.Hooks.AfterEach.Func = func() { + p.Logf("test %q took %s", p.Name(), time.Since(p.start)) + } + + return spec +} + +type T struct { + *testo.T + *PluginTimer +} + +// Bake is a fixture: it bakes a pastry and +// schedules the cleanup for the end of the test. +func (t T) Bake(name string) (Pastry, bool) { + pastry, ok := Bake(name) + + if ok { + t.Cleanup(func() { Eat(pastry) }) + } + + return pastry, ok +} + +type Bakery struct{ testo.Suite[T] } + +func (Bakery) BeforeAll(t T) { + if err := Oven(true); err != nil { + t.Fatalf("failed to turn the oven on: %v", err) + } +} - t.Run("SecondTest", testo.Test(func(t T) { - t.Log("2!") - })) +func (Bakery) AfterAll(t T) { + if err := Oven(false); err != nil { + t.Errorf("failed to turn the oven off: %v", err) + } +} + +func (Bakery) CasesDessert() []string { + return []string{"honey cake", "tiramisu"} +} + +func (Bakery) TestBake(t T, p struct{ Dessert string }) { + pastry, ok := t.Bake(p.Dessert) + + testo.Run(t, "check the bakery can bake it", func(t T) { + if !ok { + t.Fatalf("the bakery must know how to bake %s", p.Dessert) + } + }) + + testo.Run(t, "check it is tasty", func(t T) { + if !pastry.Tasty { + t.Errorf("%s must be tasty", p.Dessert) + } + }) +} + +func Test(t *testing.T) { + testo.RunSuite(t, new(Bakery)) } ``` + +## Next Steps + +- Learn [how to use various Testo features](./how-to.md) - filtering + tests, plugin options, persistent cache, project structure and more. +- Browse the [examples](../examples) - each is a small runnable + project with its expected output. +- Read the [technical overview](./technical-overview.md) for the full + lifecycle of a suite run. +- Migrating from testify or allure-go? See the + [migration guide](./migration.md). +- View the [API documentation](https://pkg.go.dev/github.com/ozontech/testo). diff --git a/examples/05_parallel/main_test.go b/examples/05_parallel/main_test.go index 27de900..a253914 100644 --- a/examples/05_parallel/main_test.go +++ b/examples/05_parallel/main_test.go @@ -26,6 +26,8 @@ func (Suite) AfterAll(t T) { } func (Suite) BeforeEach(t T) { + // AfterEach fires before parallel sub-tests finish; + // Cleanup runs after them. That is why teardown goes here. t.Cleanup(func() { t.Logf("cleanup %q", t.Name()) }) diff --git a/examples/05_parallel/output.golden b/examples/05_parallel/output.golden index 89cb061..f9ab823 100644 --- a/examples/05_parallel/output.golden +++ b/examples/05_parallel/output.golden @@ -3,21 +3,21 @@ main_test.go:21: starting suite === RUN Test/Suite/testo! === RUN Test/Suite/testo!/TestBar - main_test.go:33: before "Test/Suite/TestBar" + main_test.go:35: before "Test/Suite/TestBar" === PAUSE Test/Suite/testo!/TestBar === RUN Test/Suite/testo!/TestFoo - main_test.go:33: before "Test/Suite/TestFoo" + main_test.go:35: before "Test/Suite/TestFoo" === PAUSE Test/Suite/testo!/TestFoo === CONT Test/Suite/testo!/TestBar - main_test.go:62: inside test bar - main_test.go:37: after "Test/Suite/TestBar" - main_test.go:30: cleanup "Test/Suite/TestBar" + main_test.go:64: inside test bar + main_test.go:39: after "Test/Suite/TestBar" + main_test.go:32: cleanup "Test/Suite/TestBar" === CONT Test/Suite/testo!/TestFoo - main_test.go:44: inside test foo + main_test.go:46: inside test foo === RUN Test/Suite/testo!/TestFoo/sub-test === PAUSE Test/Suite/testo!/TestFoo/sub-test === NAME Test/Suite/testo!/TestFoo - main_test.go:37: after "Test/Suite/TestFoo" + main_test.go:39: after "Test/Suite/TestFoo" === CONT Test/Suite/testo!/TestFoo/sub-test === RUN Test/Suite/testo!/TestFoo/sub-test/inner_0 === PAUSE Test/Suite/testo!/TestFoo/sub-test/inner_0 @@ -40,27 +40,27 @@ === RUN Test/Suite/testo!/TestFoo/sub-test/inner_9 === PAUSE Test/Suite/testo!/TestFoo/sub-test/inner_9 === CONT Test/Suite/testo!/TestFoo/sub-test/inner_0 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_0" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_0" === CONT Test/Suite/testo!/TestFoo/sub-test/inner_5 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_5" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_5" === CONT Test/Suite/testo!/TestFoo/sub-test/inner_9 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_9" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_9" === CONT Test/Suite/testo!/TestFoo/sub-test/inner_8 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_8" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_8" === CONT Test/Suite/testo!/TestFoo/sub-test/inner_7 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_7" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_7" === CONT Test/Suite/testo!/TestFoo/sub-test/inner_6 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_6" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_6" === CONT Test/Suite/testo!/TestFoo/sub-test/inner_3 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_3" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_3" === CONT Test/Suite/testo!/TestFoo/sub-test/inner_4 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_4" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_4" === CONT Test/Suite/testo!/TestFoo/sub-test/inner_2 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_2" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_2" === CONT Test/Suite/testo!/TestFoo/sub-test/inner_1 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_1" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_1" === NAME Test/Suite/testo!/TestFoo - main_test.go:30: cleanup "Test/Suite/TestFoo" + main_test.go:32: cleanup "Test/Suite/TestFoo" === NAME Test/Suite main_test.go:25: finishing suite --- PASS: Test (0.00s) @@ -80,4 +80,4 @@ --- PASS: Test/Suite/testo!/TestFoo/sub-test/inner_2 (0.00s) --- PASS: Test/Suite/testo!/TestFoo/sub-test/inner_1 (0.00s) PASS -ok github.com/ozontech/testo/examples/05_parallel 0.314s +ok github.com/ozontech/testo/examples/05_parallel 0.364s diff --git a/examples/07_annotations/main_test.go b/examples/07_annotations/main_test.go index 96b2096..6fd0fc2 100644 --- a/examples/07_annotations/main_test.go +++ b/examples/07_annotations/main_test.go @@ -11,7 +11,9 @@ import ( type T struct { *testo.T - // This is an example plugin that supports annotations + // This is an example plugin that supports annotations. + // It (and the WithOrder & WithRuns options used below) + // is defined in plugin.go next to this file. *PluginUtils } diff --git a/examples/07_annotations/output.golden b/examples/07_annotations/output.golden index 18b6088..fd4bb8a 100644 --- a/examples/07_annotations/output.golden +++ b/examples/07_annotations/output.golden @@ -1,6 +1,6 @@ === RUN Test === RUN Test/Suite - main_test.go:32: testo: plugins collected: 1: main.PluginUtils + main_test.go:34: testo: plugins collected: 1: main.PluginUtils === RUN Test/Suite/testo! === RUN Test/Suite/testo!/TestFizz === RUN Test/Suite/testo!/TestBar @@ -16,4 +16,4 @@ --- PASS: Test/Suite/testo!/TestFoo#01 (0.00s) --- PASS: Test/Suite/testo!/TestFoo#02 (0.00s) PASS -ok github.com/ozontech/testo/examples/07_annotations 0.319s +ok github.com/ozontech/testo/examples/07_annotations 0.272s diff --git a/examples/README.md b/examples/README.md index 8d0a055..fbf8f56 100644 --- a/examples/README.md +++ b/examples/README.md @@ -1,17 +1,27 @@ # Examples -Showcase of various Testo features. +Small runnable projects, one per Testo feature. -Examples are sorted by their simplicity in ascending order from basic to advanced. -You may want to start from the simplest -examples such as [01_suite](./01_suite/main_test.go) or [01_suiteless](./01_suiteless/main_test.go). +| Example | What it shows | +| :------ | :------------ | +| [01_suite](./01_suite/main_test.go) | A minimal suite with a few tests. | +| [01_suiteless](./01_suiteless/main_test.go) | Running tests without a suite. | +| [02_hooks](./02_hooks/main_test.go) | Suite hooks: `BeforeAll`, `BeforeEach`, `AfterEach`, `AfterAll`. | +| [03_parametrized](./03_parametrized/main_test.go) | Parametrized tests with `CasesXxx` methods. | +| [04_plugins](./04_plugins/main_test.go) | Writing plugins: hooks, method overrides, test planning, new `T` methods. | +| [05_parallel](./05_parallel/main_test.go) | Parallel tests and how hooks behave around them. | +| [06_errors](./06_errors/main_test.go) | Error messages for common mistakes. **Fails on purpose.** | +| [07_annotations](./07_annotations/main_test.go) | Attaching static options to tests with `testo.For` & `testo.ForEach`. | +| [08_subsuites](./08_subsuites/main_test.go) | Nesting suites with `testo.RunSubSuite`. | -To run each test execute the following command in the example directory: +`01_` has two variants on purpose: the same starting point, with and +without a suite. + +To run an example, execute the following command in its directory +(or run `make`, which does the same): ```bash go test . -v -tags example -count=1 ``` -Or, if you can run `make` to do the same. - -Each example includes its output in `output.golden` file. +Each example includes its expected output in an `output.golden` file. diff --git a/examples_test.go b/examples_test.go index a31d540..a6ee931 100644 --- a/examples_test.go +++ b/examples_test.go @@ -17,7 +17,7 @@ import ( var ignoreLineOrder = []string{ "02_hooks", - "06_parallel", + "05_parallel", } func TestExamples(t *testing.T) { diff --git a/testoplugin/plugin.go b/testoplugin/plugin.go index c9b15b0..552d09c 100644 --- a/testoplugin/plugin.go +++ b/testoplugin/plugin.go @@ -5,15 +5,17 @@ // Plugins can implement [Plugin] interface to be registered as such. // // Method "Plugin" will be called for each plugin before running a suite. -// For sub-tests, parent refers to the plugin instance of the parent test. -// For top-level tests, parent is a typed-nil instance of the plugin's own -// type: the interface itself is non-nil, but the concrete pointer is nil, -// so an unconditional type assertion like parent.(*PluginFoo) is safe. -// To detect the top level, nil-check the asserted pointer, not the interface: +// The parent is the plugin instance of the enclosing scope: +// the suite root for a test, the test for a sub-test. +// Only at the suite root itself is parent a typed-nil instance of the +// plugin's own type: the interface itself is non-nil, but the concrete +// pointer is nil, so an unconditional type assertion like +// parent.(*PluginFoo) is safe. +// To detect the suite root, nil-check the asserted pointer, not the interface: // // prev, _ := parent.(*PluginFoo) // if prev != nil { -// // sub-test: use prev +// // not the suite root: prev belongs to the enclosing test or suite // } // // It is encouraged to ensure a plugin implements [Plugin] interface with the following line: