From cb46780778942fc4d9ca3f1505fd8c413ebdd3a0 Mon Sep 17 00:00:00 2001 From: metafates Date: Sat, 25 Jul 2026 16:32:23 +0300 Subject: [PATCH 1/8] update docs --- COMPARISON.md | 15 +- README.md | 13 +- docs/README.md | 11 +- docs/how-to.md | 186 ++++++- docs/migration.md | 144 +++++ docs/technical-overview.md | 22 + docs/tutorial.md | 773 ++++++++++++++++++-------- examples/07_annotations/main_test.go | 4 +- examples/07_annotations/output.golden | 4 +- examples/README.md | 29 +- examples_test.go | 2 +- 11 files changed, 929 insertions(+), 274 deletions(-) create mode 100644 docs/migration.md diff --git a/COMPARISON.md b/COMPARISON.md index 5ee30c9..d1fdd76 100644 --- a/COMPARISON.md +++ b/COMPARISON.md @@ -19,10 +19,21 @@ Testo focuses on extensibility through plugins and stays a thin layer over usual Other frameworks may be a better fit if you need BDD scenarios or some unique features "out of the box". Testo can support BDD-style tests through plugins. -[^1]: Ginkgo runs parallel tests in [separate processes](https://onsi.github.io/ginkgo/#mental-model-how-ginkgo-runs-parallel-specs) with its own runner (not available through `go test`). This is _less performant_ than native `go test` parallelization based on goroutines. +## Why we moved off Allure-Go + +Testo was created at Ozon as the successor to [Allure-Go]. +Allure-Go couples the test framework with Allure report generation: +reporting cannot be swapped out, and the framework cannot be extended +beyond what it ships with. Testo decouples the two - the framework +itself is a dependency-free layer over `testing.T`, and Allure +reporting is [one plugin among many](https://github.com/ozontech/testo-allure). +If you are on Allure-Go today, see the +[migration guide](./docs/migration.md#migrating-from-allure-go). + +[^1]: Ginkgo runs parallel tests in [separate processes](https://onsi.github.io/ginkgo/#mental-model-how-ginkgo-runs-parallel-specs) with its own runner (not available through `go test`). This is heavier-weight than native `go test` parallelization based on goroutines, trading startup and memory cost for process isolation. [^2]: [Not supported](https://github.com/smartystreets/goconvey/issues/360). [^3]: See [issue #934](https://github.com/stretchr/testify/issues/934). -[^4]: DSL — Domain-Specific Language. Requires describing tests in a specific way, different from usual Go tests. Not necessarily a bad thing, but has a learning curve and is less flexible. +[^4]: DSL: Domain-Specific Language. Requires describing tests in a specific way, different from usual Go tests. Not necessarily a bad thing, but has a learning curve and is less flexible. [^5]: Whether it has any dependencies. Not necessarily a bad thing, but fewer dependencies mean a smaller footprint, faster build times and avoiding potential vulnerabilities. [^6]: Any report format is achievable through plugins, but none is baked into Testo by default. See [testo-allure](https://github.com/ozontech/testo-allure). [^7]: "Native `go test`" means _all features_ are supported using only the `go test` command without any other CLIs. diff --git a/README.md b/README.md index d461db8..ad8cdaa 100644 --- a/README.md +++ b/README.md @@ -24,9 +24,9 @@ Add some flavor to your tests with - [Lifecycle hooks](./examples/02_hooks/main_test.go) - before and after any suite, test & sub-test. - [Test annotations](./examples/07_annotations/main_test.go) - attach static options to any test. - [Informative errors and traces](./examples/06_errors/main_test.go) - no need to guess what went wrong. -- Sub-tests & sub-suites - support for nested tests and nested suites. -- Test reflection - deeply inspect test's meta-information. -- Caching - key-value storage persistent between test runs. +- [Sub-tests & sub-suites](./examples/08_subsuites/main_test.go) - support for nested tests and nested suites. +- [Test reflection](https://pkg.go.dev/github.com/ozontech/testo/testoreflect) - deeply inspect test's meta-information. +- [Caching](./docs/how-to.md#how-to-use-persistent-cache) - key-value storage persistent between test runs. - [Zero dependencies](./go.mod). ## Why Testo @@ -80,6 +80,7 @@ See also [VS Code extension for Testo](#vs-code-extension). - Take [a guided tour of Testo](./docs/tutorial.md) by making simple plugins and running the tests using various features. - See [test examples](./examples). - Learn [how to use various Testo features](./docs/how-to.md). +- Migrating from testify or allure-go? See the [migration guide](./docs/migration.md). - Read a [brief description and technical overview](./docs/technical-overview.md) of Testo. - View [API documentation](https://pkg.go.dev/github.com/ozontech/testo). @@ -89,7 +90,7 @@ Testo features a powerful plugin system. Plugins can: -- Provide `BeforeAll`/`AfterAll`, `BeforeEach`/`AfterEach` & `BeforeSubEach`/`AfterSubEach` hooks. +- Provide `BeforeAll`/`AfterAll`, `BeforeEach`/`AfterEach` & `BeforeEachSub`/`AfterEachSub` hooks. - Plan tests for execution - filter, duplicate & reorder. - Override built-in `T` methods, such as `Log`, `Error` and _etc._ - Extend `T` by adding new methods. @@ -97,6 +98,10 @@ Plugins can: - Communicate with other plugins. - Add command line flags for `go test` command. +The plugin API (`testoplugin` package) follows the same +[SemVer](https://semver.org) compatibility promise as the rest of the module: +no breaking changes within a major version. New `Spec` fields may be added in minor releases. + Examples: - [Testo Allure Plugin](https://github.com/ozontech/testo-allure) - enhance your tests with automatically generated [Allure Reports](https://allurereport.org/). diff --git a/docs/README.md b/docs/README.md index d9f022a..3521776 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,7 +1,10 @@ # Documentation -- [Tutorial](./tutorial.md) -- [How to use Testo features](./how-to.md) -- [Persistent cache](./how-to.md#how-to-use-persistent-cache) -- [Technical Overview](./technical-overview.md) +Suggested reading order: + +- [Tutorial](./tutorial.md) - a guided tour from a plain Go test to a full Testo suite. +- [Examples](../examples) - small runnable projects, one per feature. +- [How to use Testo features](./how-to.md) - goal-oriented recipes, including the [persistent cache](./how-to.md#how-to-use-persistent-cache). +- [Migration guide](./migration.md) - moving from testify or allure-go. +- [Technical overview](./technical-overview.md) - lifecycle, panics, plugin internals. - [API documentation](https://pkg.go.dev/github.com/ozontech/testo). diff --git a/docs/how-to.md b/docs/how-to.md index 44977cb..5a515c5 100644 --- a/docs/how-to.md +++ b/docs/how-to.md @@ -2,6 +2,20 @@ Learn how to use the features of Testo. +Snippets on this page are fragments: they assume a project set up as +in the [tutorial](./tutorial.md), with a suite and a `T` type already +defined. + +- [How to write parametrized tests](#how-to-write-parametrized-tests) +- [How to write parallel tests](#how-to-write-parallel-tests) +- [How to use plugin options](#how-to-use-plugin-options) +- [How to use persistent cache](#how-to-use-persistent-cache) +- [How to structure tests](#how-to-structure-tests) +- [How to run and skip specific tests](#how-to-run-and-skip-specific-tests) +- [How to annotate tests](#how-to-annotate-tests) +- [How to integrate with CI](#how-to-integrate-with-ci) +- [How to run sub-suites](#how-to-run-sub-suites) + ## How to write parametrized tests Parametrized tests are defined as regular tests with a second argument: @@ -31,15 +45,48 @@ Field names used in a `struct{ Name string; Age int}` must be equal to existing Given that, test `TestFoo` will be invoked with all possible combinations of names and ages: -```python -TestFoo(name=John, age=18) -TestFoo(name=John, age=60) -TestFoo(name=John, age=6) -TestFoo(name=Joe, age=18) -TestFoo(name=Joe, age=60) -TestFoo(name=Joe, age=6) +```txt +TestFoo with p = {Name: "John", Age: 18} +TestFoo with p = {Name: "John", Age: 60} +TestFoo with p = {Name: "John", Age: 6} +TestFoo with p = {Name: "Joe", Age: 18} +TestFoo with p = {Name: "Joe", Age: 60} +TestFoo with p = {Name: "Joe", Age: 6} ``` +Note that parameter values do not appear in test names. +In `go test -v` output the runs are named `TestFoo`, `TestFoo#01`, `TestFoo#02` and so on, +following the standard `go test` convention for repeated names. + +### Table tests (correlated parameters) + +Separate parameters always produce the Cartesian product of their values. +When values are correlated - a classic table test, where each case is one +row with its own input and expected output - use a single struct-typed +parameter instead: + +```go +type Case struct { + Input string + Want int +} + +func (*Suite) CasesCase() []Case { + return []Case{ + {Input: "one", Want: 1}, + {Input: "two", Want: 2}, + } +} + +func (*Suite) TestParse(t *testo.T, p struct{ Case Case }) { + if got := Parse(p.Case.Input); got != p.Case.Want { + t.Errorf("Parse(%q) = %d, want %d", p.Case.Input, got, p.Case.Want) + } +} +``` + +The test runs once per element of the slice, with no cross-combination. + If for at least one required parameter function `CasesXxx` returns zero values Testo will log a warning with similar message: @@ -72,6 +119,9 @@ func (*Suite) TestFoo(t *testo.T) { } ``` +Standard `go test` flags such as `-parallel`, `-count` and `-timeout` +apply unchanged - Testo tests are regular Go tests underneath. + You can expect all `AfterEach` and `AfterAll` hooks to execute at the end of each test properly. Please note, that `AfterEach` (only applies to it) hook is deferred to run at the end of the test. @@ -99,12 +149,16 @@ func (*Suite) Test(t T) { } ``` -Unless you need to run sub-tests during this hook, -it is recommended to use t.Cleanup during BeforeEach. +If your teardown must run after all parallel sub-tests finish, +register it with `t.Cleanup` inside `BeforeEach` instead of using the `AfterEach` hook - +cleanups run after all parallel sub-tests of the test are done: ```go func (*Suite) BeforeEach(t T) { - t.Cleanup(t.afterEach) + t.Cleanup(func() { + // Teardown logic here. + // Runs after the test AND all its parallel sub-tests finish. + }) } ``` @@ -179,10 +233,17 @@ packages that do not import `testocache` do not receive an unknown test flag: TESTO_CACHE_DIR=/tmp/my-testo-cache go test ./... ``` +To disable the cache entirely, set `TESTO_CACHE_DISABLE`: + ```bash TESTO_CACHE_DISABLE=true go test ./... ``` +> [!NOTE] +> `go test` caches successful test results. A cached pass skips execution +> entirely, so `testocache` state will not refresh on such runs. +> Pass `-count=1` to force execution when that matters. + For plugin state, prefer a namespace: ```go @@ -207,8 +268,7 @@ processes sharing the same cache directory. Testo does not enforce any particular file structure to work. However, some patterns are proved to be useful. - -### Standalone suites +Here is one with standalone suite packages: ```txt go.mod @@ -244,7 +304,7 @@ type PluginCommon struct { // This method implements Plugin interface. func (*PluginCommon) Plugin(testoplugin.Plugin, ...testoplugin.Option) testoplugin.Spec { - return testolpugin.Spec{} + return testoplugin.Spec{} } ``` @@ -253,7 +313,11 @@ Contents of `tests/suite/suitefoo/t.go`: ```go package suitefoo -import "example/tests/testcommon" +import ( + "example/tests/testcommon" + + "github.com/ozontech/testo" +) // Global (common) T used by all suites. type T struct { @@ -311,7 +375,24 @@ Testo works with default go test flags, such as `-run` and `-skip`. > See `go help testflag` for detailed flags description. -Testo also provides its own flag: `-testo.m regexp` to run specific suite tests. +To make hooks work correctly with parallel tests, Testo inserts a hidden +`testo!` level into every suite. The full name of a suite test is: + +```txt +TestFunc/SuiteName/testo!/TestMethod[/sub-test...] +``` + +> [!WARNING] +> A `-run` pattern that omits the `testo!` segment, such as +> `-run 'Test/MySuite/TestFoo'`, matches **zero tests** - and `go test` +> still reports `PASS`. This is also the pattern most IDE "run test" buttons +> generate. Either include the segment (`-run 'Test/MySuite/testo!/TestFoo'`) +> or - better - use the `-testo.m` flag below. +> In VS Code, the [Testo extension](../vscode-extension) generates +> correct run/debug commands for you. + +Testo provides its own flag `-testo.m regexp` to select suite tests by +method name, without worrying about the `testo!` segment. For example, given the following suite: @@ -330,15 +411,84 @@ func (MySuite) TestBar(t T) { We can run only `TestFoo` like that: ```shell -go test . -run ./MySuite -testo.m TestFoo +go test . -run 'Test/MySuite' -testo.m TestFoo ``` +Here `-run 'Test/MySuite'` selects the suite and `-testo.m TestFoo` +selects the method inside it. + +Note that if `-testo.m` matches no tests, the suite runs empty (its +`BeforeAll`/`AfterAll` hooks still execute) and `go test` reports `PASS`. +Guard against "0 tests ran" in CI if that is a concern. + +> [!NOTE] +> `t.Name()` on `testo.T` returns the logical name without the `testo!` +> segment (e.g. `Test/MySuite/TestFoo`), while the underlying `testing.T` +> name - the one shown in `go test -v` output and required by `-run` - +> includes it. Keep this in mind when mapping names back to `-run` patterns +> or parsing test output. + > [!TIP] -> See also [Visual Studio Code extension](../vscode-extension) which does just that for you. +> See also [Visual Studio Code extension](../vscode-extension) which +> generates correct run/debug commands for you. + +## How to annotate tests + +Annotations attach static plugin options to a specific test, +so plugins can see them before the test runs (useful for planning, +reporting, retries and similar features). + +Use `testo.For` for regular tests and `testo.ForEach` for parametrized ones: + +```go +var _ = testo.For(MySuite.TestFoo, myplugin.WithRetry()) + +func (MySuite) TestFoo(t T) { + // ... +} + +var _ = testo.ForEach(MySuite.TestBar, myplugin.WithRetry()) + +func (MySuite) TestBar(t T, p struct{ N int }) { + // ... +} +``` + +Multiple annotation calls for the same test append options. + +An option is just a `testoplugin.Option` value wrapping any type your +plugin knows how to interpret: + +```go +func WithRetry() testoplugin.Option { + return testoplugin.Option{Value: retryOption{}} +} +``` + +The plugin receives options in its `Plugin` method and type-asserts +the `Value` field. + +See [annotations example](../examples/07_annotations/main_test.go) +(its `plugin.go` defines the options) +and [API documentation](https://pkg.go.dev/github.com/ozontech/testo#For). + +## How to integrate with CI + +Testo output is standard `go test` output. `go test -json`, `test2json`, +[gotestsum](https://github.com/gotestyourself/gotestsum) and similar tools +work unchanged. + +Notes: + +- The hidden `testo!` node appears in reports as an extra nesting level + (e.g. JUnit converters render it as an empty intermediate node). +- For Allure reports, use the [testo-allure plugin](https://github.com/ozontech/testo-allure). +- For rerunning only failed tests (flaky-test handling), see the + [rerun plugin](https://github.com/ozontech/testo-toppings/tree/main/rerun). ## How to run sub-suites -There a `testo.RunSubSuite` function for that: +There is a `testo.RunSubSuite` function for that: ```go type OuterSuite struct{ testo.Suite[T] } diff --git a/docs/migration.md b/docs/migration.md new file mode 100644 index 0000000..9a9dfd5 --- /dev/null +++ b/docs/migration.md @@ -0,0 +1,144 @@ +# Migration guide + +How to move an existing test base to Testo - from +[testify](#migrating-from-testify) or from +[allure-go](#migrating-from-allure-go). + +Both migrations can be done incrementally: Testo tests are regular +`go test` tests, so old and new suites run side by side in the same +package with no interference. Migrate one suite at a time. + +## Migrating from testify + +### Suites and hooks + +| testify | Testo | +| :------ | :---- | +| `suite.Suite` (embedded) | `testo.Suite[T]` (embedded) | +| `suite.Run(t, new(S))` | `testo.RunSuite(t, new(S))` | +| `func (s *S) SetupSuite()` | `func (*S) BeforeAll(t T)` | +| `func (s *S) SetupTest()` | `func (*S) BeforeEach(t T)` | +| `func (s *S) TearDownTest()` | `func (*S) AfterEach(t T)` | +| `func (s *S) TearDownSuite()` | `func (*S) AfterAll(t T)` | +| `func (s *S) TestFoo()` | `func (*S) TestFoo(t T)` | +| `s.T()` | `t` (passed to every method) | +| `s.Run(name, func())` | `testo.Run(t, name, func(t T))` | +| `func (s *S) BeforeTest(suiteName, testName string)` | `BeforeEach(t T)` + `t.Name()` (same for `AfterTest`) | +| `SetupSubTest()` / `TearDownSubTest()` | `BeforeEachSub`/`AfterEachSub` plugin hooks (no suite-level equivalent) | +| `go test -run TestSuite -testify.m TestFoo` | `go test -run TestSuite -testo.m TestFoo` | + +One flag caveat: testify method filters like `-run 'TestMySuite/TestFoo'` +stop matching after migration, because Testo inserts a hidden `testo!` +level into test names. Update CI filters to use `-testo.m` - see +[how to run and skip specific tests](./how-to.md#how-to-run-and-skip-specific-tests). + +`T` here is your own type built on `*testo.T` - in the simplest case +just an alias: + +```go +type T = *testo.T +``` + +See the [tutorial](./tutorial.md) for defining a `T` with plugins. + +### Assertions + +testify's `require` and `assert` packages work with Testo unchanged, +because `testo.T` implements `testing.TB`: + +```go +// testify/suite: +func (s *MySuite) TestFoo() { + s.Require().NoError(err) + s.Equal(want, got) // shorthand from the embedded suite.Suite +} + +// Testo: +func (*MySuite) TestFoo(t T) { + require.NoError(t, err) + assert.Equal(t, want, got) +} +``` + +The `s.Require().X(...)` and shorthand `s.X(...)` forms both become +plain `require.X(t, ...)` / `assert.X(t, ...)` calls. + +There is no need to drop testify to adopt Testo - it replaces +`suite`, not `require`/`assert`. + +### Suite state + +testify suites keep per-test state in suite struct fields. +A Testo suite is also a single instance, so fields shared by all +tests still work - but for per-test state prefer methods on your `T` +(fixtures) or plugin fields, which get a fresh instance per test and +are safe with parallel tests. See +[fixtures in the tutorial](./tutorial.md#fixtures). + +### Parallelism + +testify's `suite` does not support `t.Parallel()` +([issue #934](https://github.com/stretchr/testify/issues/934)). +With Testo, call `t.Parallel()` in any test - hooks handle parallel +tests correctly. See +[how to write parallel tests](./how-to.md#how-to-write-parallel-tests). + +## Migrating from allure-go + +Testo grew out of [allure-go](https://github.com/ozontech/allure-go) +at Ozon: allure-go couples the test framework with Allure reporting, +while Testo splits them - the framework is dependency-free, and +Allure reporting is one plugin among many. Keep in mind you need +**two** pieces: `github.com/ozontech/testo` and the +[testo-allure](https://github.com/ozontech/testo-allure) plugin. + +```go +import ( + "github.com/ozontech/testo" + + allure "github.com/ozontech/testo-allure" +) + +type T struct { + *testo.T + *allure.PluginAllure +} +``` + +### Concept mapping + +| allure-go | Testo (+ testo-allure) | +| :-------- | :--------------------- | +| `suite.Suite` | `testo.Suite[T]` | +| `suite.RunSuite(t, new(S))` (or `runner.NewSuiteRunner(...)`) | `testo.RunSuite(t, new(S))` | +| `BeforeAll(t provider.T)` | `BeforeAll(t T)` (same for the other hooks) | +| `func (s *S) TestFoo(t provider.T)` | `func (*S) TestFoo(t T)` | +| `t.WithNewStep("name", func(ctx provider.StepCtx) {...})` | `allure.Step(t, "name", func(t T) {...})` | +| `t.NewStep("name")` | `allure.Step(t, "name", func(T) {})` | +| `t.Title`, `t.Epic`, `t.Feature`, `t.Story`, `t.Tags`, `t.Severity`, `t.Owner`, `t.ID` | same-named methods added to `T` by `PluginAllure` | +| `t.Link(...)`, `t.TmsLink(...)`, `t.TmsLinks(...)` | `t.Links(...)` with the `allure.NewLink`/`allure.TMS`/`allure.Issue` constructors | +| `t.WithNewAttachment(name, mimeType, content)`, `t.WithAttachments(...)` | `t.Attach(name, allure.Bytes(...))` or `t.Attach(name, allure.File(...))` | +| table tests / `ParametrizedRunner` | single struct-typed parameter, see [table tests](./how-to.md#table-tests-correlated-parameters) | +| `t.XSkip()` | `t.XFail()` via the [xfail plugin](https://github.com/ozontech/testo-toppings/tree/main/xfail) from testo-toppings | + +### Steps + +Steps are sub-tests. Two ways to create them: + +- `testo.Run(t, "name", func(t T) {...})` - the framework primitive. + `t.Fatal` inside stops only the sub-test, not the outer test. +- `allure.Step(t, "name", func(t T) {...})` - the testo-allure + helper. Same as `testo.Run`, but a fatal failure propagates to the + parent test and stops it - matching `WithNewStep` semantics from + allure-go. + +When migrating `WithNewStep`-heavy code, `allure.Step` is the +drop-in choice. Nested steps started from a `StepCtx` +(`sCtx.WithNewStep`, `sCtx.NewStep`) become plain nested +`allure.Step(t, ...)` calls, and step-level assertions +(`sCtx.Assert()`, `sCtx.Require()`) become the `t.Assert()` / +`t.Require()` methods that `PluginAllure` adds to `T`. + +For the full testo-allure API (output directory, labels, attachments +and so on), see the +[testo-allure documentation](https://github.com/ozontech/testo-allure). diff --git a/docs/technical-overview.md b/docs/technical-overview.md index 836aff3..69aeef2 100644 --- a/docs/technical-overview.md +++ b/docs/technical-overview.md @@ -2,6 +2,25 @@ A technical overview of Testo. +## Mechanism + +Testo works through runtime reflection over suite method sets and `T` +types - there is no code generation, `go:generate` step or separate +CLI. This puts a few constraints on your types, checked before any +test runs: + +- Plugins (and all exported fields of `T` and plugin structs) must be + embedded as pointers. +- Recursive plugin type references are detected and rejected. + +Signature and `CasesXxx` mismatches are reported as test errors; +violations of the type constraints above panic with a descriptive +message, aborting the test binary. + +Plugins are re-instantiated for every test and sub-test. This is what +makes writing to plugin fields safe without synchronization, at the +cost of a constructor call per (sub-)test. + ## Lifecycle When you call `testo.RunSuite` the following happens: @@ -56,6 +75,9 @@ Other tests will run even if some tests are panicking. Testo **will** catch panics from `BeforeAll` & `AfterAll`. Panic in these hooks will result in suite tests not running. +See also [suite hooks in the tutorial](./tutorial.md#suite-hooks) and +[hook behavior with parallel tests](./how-to.md#how-to-write-parallel-tests). + ## Plugins Testo uses dependency-injection-like mechanism to enable cross-plugin communication. diff --git a/docs/tutorial.md b/docs/tutorial.md index d145dd4..c2aa34d 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -1,363 +1,666 @@ # Tutorial -## Create a New Go Project +In this tutorial we take a plain Go test and grow it, step by step, +into a full Testo suite with a plugin, parametrization, hooks, +fixtures and steps. + +Each section changes only a small piece of code, and the file compiles +and passes after every change. The complete final file is listed +[at the end](#the-finished-test-file). + +## Setting Up + +Create a new project and add Testo to it: ```bash -mkdir testo-tutorial -cd testo-tutorial -go mod init testo-tutorial +mkdir testo-bakery +cd testo-bakery +go mod init testo-bakery +go get github.com/ozontech/testo ``` -Create `main.go`: +We are going to test a tiny bakery. Create `main.go`: ```go package main -// Add returns sum of the given integers. -func Add(a, b int) int { return a + b } +// Pastry is something a bakery bakes. +type Pastry struct { + Name string + Tasty bool + Ingredients string +} + +var recipes = map[string]string{ + "honey cake": "honey, flour, eggs, sour cream", + "tiramisu": "mascarpone, coffee, ladyfingers", +} + +// Bake bakes a pastry, if the bakery knows the recipe. +func Bake(name string) (Pastry, bool) { + ingredients, ok := recipes[name] + if !ok { + return Pastry{}, false + } + + return Pastry{Name: name, Tasty: true, Ingredients: ingredients}, true +} + +// Eat disposes of a pastry in the most pleasant way possible. +func Eat(Pastry) {} + +// Oven turns the bakery oven on or off. +func Oven(on bool) error { return nil } func main() {} ``` -## Writing Tests +## A Plain Go Test + +We start without Testo. Create `main_test.go`: + +```go +package main + +import "testing" -Testo uses `testo.T` - a wrapper around `testing.T` with extended features. -All methods available in `testing.T` are also available in `testo.T` (except `t.Run`, more on that later in [Sub-tests chapter](#sub-tests)). -Also, `testo.T` implements the `testing.TB` interface. +func TestBake(t *testing.T) { + t.Run("HoneyCake", func(t *testing.T) { + pastry, ok := Bake("honey cake") + if !ok { + t.Fatal("the bakery must know how to bake a honey cake") + } -**Important**: We'll create an alias to avoid repetition. + // no reason to waste it after the test + t.Cleanup(func() { Eat(pastry) }) -Create file `main_test.go`: + if !pastry.Tasty { + t.Error("honey cake must be tasty") + } + }) +} +``` + +```bash +go test . -v +``` + +```txt +=== RUN TestBake +=== RUN TestBake/HoneyCake +--- PASS: TestBake (0.00s) + --- PASS: TestBake/HoneyCake (0.00s) +PASS +``` + +So far, so ordinary. Let's bring in Testo. + +## Switching to Testo + +Replace the `t.Run` call with `testo.RunTest`: ```go package main import ( - "testing" + "testing" - "github.com/ozontech/testo" + "github.com/ozontech/testo" ) -type T = *testo.T +func TestBake(t *testing.T) { + testo.RunTest(t, func(t *testo.T) { + pastry, ok := Bake("honey cake") + if !ok { + t.Fatal("the bakery must know how to bake a honey cake") + } + + // no reason to waste it after the test + t.Cleanup(func() { Eat(pastry) }) + + if !pastry.Tasty { + t.Error("honey cake must be tasty") + } + }) +} ``` -Now we need a Suite. A Suite must "inherit" `testo.Suite[T]` by embedding it. +The body of the test did not change - only the type of `t`. -> [!NOTE] -> It's possible to run tests without suites, more on that in [later](#running-tests-without-suites). +`testo.T` is a wrapper around `testing.T`. +All methods available on `testing.T` are also available on `testo.T` +(except `t.Run` - sub-tests are started with `testo.Run`, covered +[later](#steps-sub-tests)), and `testo.T` implements the `testing.TB` +interface, so libraries built for standard tests (assertion helpers, +mocks and so on) keep working. + +Run the test again: + +```txt +=== RUN TestBake +=== RUN TestBake/#00 +=== RUN TestBake/#00/testo! +=== RUN TestBake/#00/testo!/TestBake +--- PASS: TestBake (0.00s) + --- PASS: TestBake/#00 (0.00s) + --- PASS: TestBake/#00/testo! (0.00s) + --- PASS: TestBake/#00/testo!/TestBake (0.00s) +PASS +``` + +Two technical levels appeared in the names: + +- `#00` is the index of the test inside this `RunTest` call. +- `testo!` is a special test Testo inserts to guarantee correct work + of hooks with parallel tests. It does not affect your tests, but it + does affect `-run` patterns - see + [how to run specific tests](./how-to.md#how-to-run-and-skip-specific-tests). + +So far Testo gave us nothing new. Its value starts with plugins. + +## Plugins + +A plugin is a struct that describes what it does through the +`testoplugin.Plugin` interface: ```go -type Suite struct{ testo.Suite[T] } +type Plugin interface { + Plugin(parent Plugin, options ...Option) Spec +} + +type Spec struct { + // Plan tests for execution: filter, duplicate, reorder. + Plan Plan -// Tests in Suites are regular methods, following the same -// naming rules as regular tests in Go. -// That means they must have the "Test" prefix. -// They also must use the same type T as specified in `testo.Suite[T]`. + // Hooks around suites, tests and sub-tests. + Hooks Hooks -func (*Suite) TestAdd(t T) { - if Add(2, 2) != 4 { - t.Fatal("2 + 2 must equal 4") - } + // Middleware for built-in T methods, + // such as Fail, Log, Skip and others. + Overrides Overrides } ``` -## Running Tests +Let's write a plugin that measures how long each test takes. +In `main_test.go`, update the import block as shown below and add +the plugin after it: + +```go +import ( + "testing" + "time" + + "github.com/ozontech/testo" + "github.com/ozontech/testo/testoplugin" +) + +type PluginTimer struct { + *testo.T + + start time.Time +} + +func (p *PluginTimer) Plugin(testoplugin.Plugin, ...testoplugin.Option) (spec testoplugin.Spec) { + spec.Hooks.BeforeEach.Func = func() { + p.start = time.Now() + } + + spec.Hooks.AfterEach.Func = func() { + p.Logf("test %q took %s", p.Name(), time.Since(p.start)) + } + + return spec +} +``` -To connect Testo with `go test` we must run it from a regular test: +The plugin embeds `*testo.T` - Testo initializes it with the same `T` +as the current test, so the plugin sees everything the test sees. +The `Plugin` method is called for each test, so writing to plugin +fields is safe: every test gets its own instance. + +To use the plugin, define your own `T` type that embeds `*testo.T` +together with the plugins you want: ```go -func Test(t *testing.T) { - testo.RunSuite(t, new(Suite)) +type T struct { + *testo.T + *PluginTimer } ``` -Now we can run tests: +And change the test function to take this `T`: -```bash -go test . -v +```go +func TestBake(t *testing.T) { + testo.RunTest(t, func(t T) { + // ... body unchanged ... + }) +} ``` +That is the whole installation. Testo looks at the `T` type a test +accepts, then collects and initializes the plugins listed in it: + ```txt -=== RUN Test -=== RUN Test/Suite -=== RUN Test/Suite/testo! -=== RUN Test/Suite/testo!/TestAdd ---- PASS: Test (0.00s) - --- PASS: Test/Suite (0.00s) - --- PASS: Test/Suite/testo! (0.00s) - --- PASS: Test/Suite/testo!/TestAdd (0.00s) +=== RUN TestBake +=== RUN TestBake/#00 + main_test.go:35: testo: plugins collected: 1: main.PluginTimer +=== RUN TestBake/#00/testo! +=== RUN TestBake/#00/testo!/TestBake + main_test.go:23: test "TestBake/#00/TestBake" took 98.125µs +--- PASS: TestBake (0.00s) +... PASS ``` -You may notice a special test `testo!` - Testo creates it "under the hood" -to guarantee correct work of hooks with parallel tests. +> [!NOTE] +> Plugins must be embedded as pointers. Pointers let plugins share +> state with each other by pointing to the same memory. +> +> Also note that `t.Name()` on `testo.T` returns the logical test name +> without the `testo!` segment. -Don't worry too much about this, as its existence doesn't affect your tests. +Plugins can do much more than hooks: reorder or filter the test plan, +wrap built-in methods like `t.Log`, add new methods to `T`, accept +options and command line flags. See the +[plugins example](../examples/04_plugins/main_test.go) for a tour of +`Plan` and `Overrides`, and the ready-made plugins: +[testo-allure](https://github.com/ozontech/testo-allure) for Allure +reports and [testo-toppings](https://github.com/ozontech/testo-toppings) +for a collection of small plugins. -## Suite Hooks +For instance, here is a complete plugin that makes every test parallel: + +```go +type PluginParallel struct{ *testo.T } + +func (p *PluginParallel) Plugin(testoplugin.Plugin, ...testoplugin.Option) (spec testoplugin.Spec) { + spec.Hooks.BeforeEach.Func = func() { + p.Parallel() + } + + return spec +} +``` + +This one is for illustration only - don't add it to your file, or the +outputs (and their line numbers) below will differ. A more complete +version of it is available as the +[parallel plugin](https://github.com/ozontech/testo-toppings/tree/main/parallel) +from testo-toppings. -Suites can have the following hooks: +Plugins can also accept options passed to `testo.RunSuite` or +`testo.Run` - see +[how to use plugin options](./how-to.md#how-to-use-plugin-options) +and the [annotations example](../examples/07_annotations/plugin.go) +for a plugin that defines its own options. -- `BeforeAll(T)` - called _once before_ all tests. The passed `T` refers to the top test, i.e. `Test/Suite`. -- `BeforeEach(T)` - called _before each_ test. The passed `T` refers to the same test that the hook runs before. -- `AfterEach(T)` - called _after each_ test, but before `t.Cleanup`. The passed `T` refers to the same test that the hook runs after. -- `AfterAll(T)` - called _once after_ all tests. The hook waits for all (parallel) tests to finish. The passed `T` refers to the top test, i.e. `Test/Suite`. +## Suites -Hooks are defined as Suite methods: +When tests share setup and plugins, it is convenient to group them +into a suite. A suite embeds `testo.Suite[T]`, where `T` is the type +we defined above - i.e. the set of plugins every test in the suite +gets: ```go -func (*Suite) BeforeEach(t T) { - t.Logf("Starting: %s", t.Name()) -} +type Bakery struct{ testo.Suite[T] } +``` + +Tests become methods of the suite. They follow the usual Go naming +rules (the `Test` prefix) and must accept the same `T` as specified in +`testo.Suite[T]`. Replace the `TestBake` function with: + +```go +func (Bakery) TestBake(t T) { + pastry, ok := Bake("honey cake") + if !ok { + t.Fatal("the bakery must know how to bake a honey cake") + } -func (*Suite) AfterEach(t T) { - t.Logf("Finished: %s", t.Name()) + t.Cleanup(func() { Eat(pastry) }) + + if !pastry.Tasty { + t.Error("honey cake must be tasty") + } } ``` -
-Output: +Go has no native notion of suites, so we launch the suite from one +regular test function: + +```go +func Test(t *testing.T) { + testo.RunSuite(t, new(Bakery)) +} +``` ```txt === RUN Test -=== RUN Test/Suite -=== RUN Test/Suite/testo! -=== RUN Test/Suite/testo!/Add - main_test.go:20: Starting: Test/Suite/TestAdd - main_test.go:28: Test/Suite/TestAdd - main_test.go:24: Finished: Test/Suite/TestAdd +=== RUN Test/Bakery + main_test.go:50: testo: plugins collected: 1: main.PluginTimer +=== RUN Test/Bakery/testo! +=== RUN Test/Bakery/testo!/TestBake + main_test.go:23: test "Test/Bakery/TestBake" took 66.583µs --- PASS: Test (0.00s) - --- PASS: Test/Suite (0.00s) - --- PASS: Test/Suite/testo! (0.00s) - --- PASS: Test/Suite/testo!/TestAdd (0.00s) + --- PASS: Test/Bakery (0.00s) + --- PASS: Test/Bakery/testo! (0.00s) + --- PASS: Test/Bakery/testo!/TestBake (0.00s) PASS ``` -
- ## Parametrized Tests -Parametrized tests let you run the same tests with different input parameters. +Our bakery bakes more than honey cake, and the test scenario is the +same for every dessert. Instead of copying the test, we make the +dessert a parameter. -Parametrized tests follow the same naming rules as regular tests, but take -a second argument after `T`. -This argument must be a struct containing the required parameters. +A parametrized test takes a second argument after `T`: a struct whose +fields are the parameters. Replace `TestBake` with: ```go -func (*Suite) TestAddButParametrized(t T, p struct{ A, B int }) { - if Add(p.A, p.B) != Add(p.B, p.A) { - t.Errorf("%[1]d + %[2]d != %[2]d + %[1]d", p.A, p.B) - } +func (Bakery) TestBake(t T, p struct{ Dessert string }) { + pastry, ok := Bake(p.Dessert) + if !ok { + t.Fatalf("the bakery must know how to bake %s", p.Dessert) + } + + t.Cleanup(func() { Eat(pastry) }) + + if !pastry.Tasty { + t.Errorf("%s must be tasty", p.Dessert) + } } ``` -We also need to declare the parameter values with which the test will run. -This is done by declaring special methods with names like `CasesXXX`, where -`XXX` is the parameter name. +Parameter values are declared with `CasesXxx` methods, where `Xxx` +matches the field name: ```go -func (*Suite) CasesA() []int { - return []int{1, 2, 3, 4, 5} +func (Bakery) CasesDessert() []string { + return []string{"honey cake", "tiramisu"} } +``` -func (*Suite) CasesB() []int { - return []int{11, 1000, 13} -} +```txt +=== RUN Test +=== RUN Test/Bakery + main_test.go:54: testo: plugins collected: 1: main.PluginTimer +=== RUN Test/Bakery/testo! +=== RUN Test/Bakery/testo!/TestBake + main_test.go:23: test "Test/Bakery/TestBake" took 108.791µs +=== RUN Test/Bakery/testo!/TestBake#01 + main_test.go:23: test "Test/Bakery/TestBake#01" took 15.042µs +--- PASS: Test (0.00s) + --- PASS: Test/Bakery (0.00s) + --- PASS: Test/Bakery/testo! (0.00s) + --- PASS: Test/Bakery/testo!/TestBake (0.00s) + --- PASS: Test/Bakery/testo!/TestBake#01 (0.00s) +PASS ``` -Parametrized tests are called with the [Cartesian product](https://en.wikipedia.org/wiki/Cartesian_product) -of all values obtained from `CasesXXX` functions. -For the example above, this is 15 different pairs of values. +The test ran once per dessert. With several parameters, the test runs +with the [Cartesian product](https://en.wikipedia.org/wiki/Cartesian_product) +of all their values. For correlated values (classic table tests), use +a single struct-typed parameter - see +[how to write parametrized tests](./how-to.md#how-to-write-parametrized-tests). -If a test specifies a parameter for which no corresponding `Cases` function is found, -Testo will print an informative error before running any tests. -The same applies to type mismatches. +If a test declares a parameter with no matching `CasesXxx` method, or +the types mismatch, Testo reports an informative error before running +any tests. The [errors example](../examples/06_errors/main_test.go) +demonstrates these messages (that example fails on purpose). -See [error examples](../../examples/07_errors/main_test.go). +## Suite Hooks + +One day the whole bakery stopped producing pastry. It turned out +somebody had switched off the oven. The suite should manage the oven +itself. -## Sub-tests +Suites can define these hooks as methods: -Running sub-tests is a bit different from the usual `t.Run` call. +- `BeforeAll(T)` - called _once before_ all tests. Its `T` refers to + the top-level test, i.e. `Test/Bakery`. +- `BeforeEach(T)` - called _before each_ test, with that test's `T`. +- `AfterEach(T)` - called _after each_ test, before `t.Cleanup` + callbacks, with that test's `T`. +- `AfterAll(T)` - called _once after_ all tests, including parallel + ones. Its `T` refers to the top-level test. -Sub-tests must be started using the `testo.Run` function: +We need the oven on once before all tests and off once after them: ```go -func (*Suite) CasesC() []int { - return []int{-4, -99, 9} +func (Bakery) BeforeAll(t T) { + if err := Oven(true); err != nil { + t.Fatalf("failed to turn the oven on: %v", err) + } } -func (*Suite) TestAddButParametrized(t T, p struct{ A, B, C int }) { - testo.Run(t, "commutative", func(t T) { - if Add(p.A, p.B) != Add(p.B, p.A) { - t.Errorf("%[1]d + %[2]d != %[2]d + %[1]d", p.A, p.B) - } - }) - - testo.Run(t, "associative", func(t T) { - if Add(Add(p.A, p.B), p.C) != Add(p.A, Add(p.B, p.C)) { - t.Errorf("(%[1]d + %[2]d) + %[3]d != %[1]d + (%[2]d + %[3]d)", p.A, p.B, p.C) - } - }) +func (Bakery) AfterAll(t T) { + if err := Oven(false); err != nil { + t.Errorf("failed to turn the oven off: %v", err) + } } ``` -## Plugins - -One of the main features of Testo is the plugin system. +If `BeforeAll` fails, the suite's tests do not run at all. +For hook behavior with parallel sub-tests and panics, see +[how to write parallel tests](./how-to.md#how-to-write-parallel-tests) +and the [technical overview](./technical-overview.md#panics). -### Writing Plugins +## Fixtures -Plugin examples: +The test currently creates a pastry and remembers to clean it up. +With more resources this gets repetitive, so we move the +create-and-cleanup logic out of the test. -1. A plugin that runs tests in reverse order. -2. A plugin that extends the `t.Log` method. -3. A plugin that adds new methods for `T` (fixtures). -4. A plugin that measures the time of each test. +`T` is our own type, which means we can add methods to it. A method +that creates a resource and schedules its cleanup is a fixture: ```go -import ( - "github.com/ozontech/testo" - "github.com/ozontech/testo/plugin" -) +// Bake is a fixture: it bakes a pastry and +// schedules the cleanup for the end of the test. +func (t T) Bake(name string) (Pastry, bool) { + pastry, ok := Bake(name) + + if ok { + t.Cleanup(func() { Eat(pastry) }) + } -// We can embed testo.T in plugins - it will point to the same T -// as in the current test. -type ReverseTestsOrder struct{ *testo.T } - -// Plugins must implement this function -// so that testo understands it's a plugin and can handle it correctly. -func (*ReverseTestsOrder) Plugin(testoplugin.Plugin, ...testoplugin.Option) testoplugin.Spec { - return testoplugin.Spec{ - Plan: testoplugin.Plan{ - Modify: func(tests *[]testoplugin.PlannedTest) { - slices.Reverse(*tests) - }, - }, - } -} - -type OverrideLog struct { *testo.T } - -func (*OverrideLog) Plugin(testoplugin.Plugin, ...testoplugin.Option) testoplugin.Spec { - return testoplugin.Spec{ - Overrides: testoplugin.Overrides{ - Log: func(f testoplugin.FuncLog) testoplugin.FuncLog { - return func(args ...any) { - // will be called each time t.Log is called - fmt.Println("Inside log override") - f(args...) - } - }, - }, - } -} - -type AddNewMethods struct{ *testo.T } - -func (*AddNewMethods) Plugin(testoplugin.Plugin, ...testoplugin.Option) testoplugin.Spec { - // We have nothing to use here, so we return an empty specification - return testoplugin.Spec{} -} - -// Later we will see how we can call this method from tests. -func (a *AddNewMethods) Explode() { a.Fatal("BOOM") } - -type Timer struct { - *testo.T - start time.Time -} - -func (t *Timer) Plugin(testoplugin.Plugin, ...testoplugin.Option) testoplugin.Spec { - return testoplugin.Spec{ - Hooks: testoplugin.Hooks{ - BeforeEach: testoplugin.Hook{ - Priority: testoplugin.TryLast, - Func: func() { - // the .Plugin method is called for each (sub-)test, - // so we can modify fields without risk of synchronization errors. - t.start = time.Now() - }, - }, - AfterEach: testoplugin.Hook{ - Priority: testoplugin.TryFirst, - Func: func() { - elapsed := time.Since(t.start) - - fmt.Printf("Test %q took %s\n", t.Name(), elapsed) - }, - }, - }, - } + return pastry, ok } ``` -### Using Plugins - -Recall the alias for `T` defined earlier: +The test no longer deals with cleanup: ```go -type T = *testo.T +func (Bakery) TestBake(t T, p struct{ Dessert string }) { + pastry, ok := t.Bake(p.Dessert) + if !ok { + t.Fatalf("the bakery must know how to bake %s", p.Dessert) + } + + if !pastry.Tasty { + t.Errorf("%s must be tasty", p.Dessert) + } +} ``` -Now we can add (install) plugins to it: +Fixtures in Testo are ordinary methods, without any registration +step, and they are reusable across all tests of the suite. -```go -type T struct{ - *testo.T - *ReverseTestsOrder - *OverrideLog - *AddNewMethods - *Timer -} -``` +## Steps (Sub-tests) -That's the only change. -Testo will take care of the rest. +Right now a failure tells us *which* test failed, but not *which +check* inside it. We can structure the test into named steps using +sub-tests. -And since `AddNewMethods` is now embedded in `T`, we can use its methods without magic: +Sub-tests are started with the `testo.Run` function (this is the one +place where Testo differs from `t.Run`). Replace `TestBake` with: ```go -func (*Suite) TestBoom(t T) { - t.Explode() +func (Bakery) TestBake(t T, p struct{ Dessert string }) { + pastry, ok := t.Bake(p.Dessert) + + testo.Run(t, "check the bakery can bake it", func(t T) { + if !ok { + t.Fatalf("the bakery must know how to bake %s", p.Dessert) + } + }) + + testo.Run(t, "check it is tasty", func(t T) { + if !pastry.Tasty { + t.Errorf("%s must be tasty", p.Dessert) + } + }) } ``` -> [!NOTE] -> Plugins must be installed as pointers. -> -> Pointers allow plugins to share their state with other plugins, -> by pointing to the same memory location through pointers. +```txt +=== RUN Test/Bakery/testo!/TestBake +=== RUN Test/Bakery/testo!/TestBake/check_the_bakery_can_bake_it +=== RUN Test/Bakery/testo!/TestBake/check_it_is_tasty +... +--- PASS: Test/Bakery/testo!/TestBake (0.00s) + --- PASS: Test/Bakery/testo!/TestBake/check_the_bakery_can_bake_it (0.00s) + --- PASS: Test/Bakery/testo!/TestBake/check_it_is_tasty (0.00s) +``` + +> [!WARNING] +> `t.Fatal` inside a sub-test stops only that sub-test, not the outer +> test - standard `go test` behavior. If a failed step must stop the +> whole test, check the sub-test result in the outer test, or use a +> plugin that propagates the failure, such as `allure.Step` from +> [testo-allure](https://github.com/ozontech/testo-allure). + +Plugins get their own hooks around every sub-test: +`BeforeEachSub` and `AfterEachSub`. +Reporting plugins use sub-tests as steps - with +[testo-allure](https://github.com/ozontech/testo-allure) installed, +each `testo.Run` above becomes a step in the Allure report. -## Running tests without suites +## Running Tests Without Suites -It's possible: +Suites are optional. You have already seen `testo.RunTest`: ```go -type T struct{ - *testo.T - *ReverseTestsOrder - *OverrideLog - *AddNewMethods - *Timer +func TestFoo(t *testing.T) { + testo.RunTest(t, func(t T) { + t.Log("Hello from Testo!") + }) } +``` + +And to run several tests from a single test function, there is the +`testo.Test` adapter for `t.Run`: +```go func TestFoo(t *testing.T) { - testo.RunSuite(t, func(t T) { - t.Log("Hello from testo!") - }) + t.Run("FirstTest", testo.Test(func(t T) { + t.Log("1!") + })) + + t.Run("SecondTest", testo.Test(func(t T) { + t.Log("2!") + })) } ``` -Or, if you need to run several tests from a single "real" test: +## The Finished Test File + +The complete `main_test.go` we built in this tutorial: ```go -func TestFoo(t *testing.T) { - t.Run("FirstTest", testo.Test(func(t T) { - t.Log("1!") - })) +package main + +import ( + "testing" + "time" + + "github.com/ozontech/testo" + "github.com/ozontech/testo/testoplugin" +) + +type PluginTimer struct { + *testo.T + + start time.Time +} + +func (p *PluginTimer) Plugin(testoplugin.Plugin, ...testoplugin.Option) (spec testoplugin.Spec) { + spec.Hooks.BeforeEach.Func = func() { + p.start = time.Now() + } + + spec.Hooks.AfterEach.Func = func() { + p.Logf("test %q took %s", p.Name(), time.Since(p.start)) + } + + return spec +} - t.Run("SecondTest", testo.Test(func(t T) { - t.Log("2!") - })) +type T struct { + *testo.T + *PluginTimer +} + +// Bake is a fixture: it bakes a pastry and +// schedules the cleanup for the end of the test. +func (t T) Bake(name string) (Pastry, bool) { + pastry, ok := Bake(name) + + if ok { + t.Cleanup(func() { Eat(pastry) }) + } + + return pastry, ok +} + +type Bakery struct{ testo.Suite[T] } + +func (Bakery) BeforeAll(t T) { + if err := Oven(true); err != nil { + t.Fatalf("failed to turn the oven on: %v", err) + } +} + +func (Bakery) AfterAll(t T) { + if err := Oven(false); err != nil { + t.Errorf("failed to turn the oven off: %v", err) + } +} + +func (Bakery) CasesDessert() []string { + return []string{"honey cake", "tiramisu"} +} + +func (Bakery) TestBake(t T, p struct{ Dessert string }) { + pastry, ok := t.Bake(p.Dessert) + + testo.Run(t, "check the bakery can bake it", func(t T) { + if !ok { + t.Fatalf("the bakery must know how to bake %s", p.Dessert) + } + }) + + testo.Run(t, "check it is tasty", func(t T) { + if !pastry.Tasty { + t.Errorf("%s must be tasty", p.Dessert) + } + }) +} + +func Test(t *testing.T) { + testo.RunSuite(t, new(Bakery)) } ``` + +## Next Steps + +- Learn [how to use various Testo features](./how-to.md) - filtering + tests, plugin options, persistent cache, project structure and more. +- Browse the [examples](../examples) - each is a small runnable + project with its expected output. +- Read the [technical overview](./technical-overview.md) for the full + lifecycle of a suite run. +- Migrating from testify or allure-go? See the + [migration guide](./migration.md). +- View the [API documentation](https://pkg.go.dev/github.com/ozontech/testo). diff --git a/examples/07_annotations/main_test.go b/examples/07_annotations/main_test.go index 96b2096..6fd0fc2 100644 --- a/examples/07_annotations/main_test.go +++ b/examples/07_annotations/main_test.go @@ -11,7 +11,9 @@ import ( type T struct { *testo.T - // This is an example plugin that supports annotations + // This is an example plugin that supports annotations. + // It (and the WithOrder & WithRuns options used below) + // is defined in plugin.go next to this file. *PluginUtils } diff --git a/examples/07_annotations/output.golden b/examples/07_annotations/output.golden index 18b6088..fd4bb8a 100644 --- a/examples/07_annotations/output.golden +++ b/examples/07_annotations/output.golden @@ -1,6 +1,6 @@ === RUN Test === RUN Test/Suite - main_test.go:32: testo: plugins collected: 1: main.PluginUtils + main_test.go:34: testo: plugins collected: 1: main.PluginUtils === RUN Test/Suite/testo! === RUN Test/Suite/testo!/TestFizz === RUN Test/Suite/testo!/TestBar @@ -16,4 +16,4 @@ --- PASS: Test/Suite/testo!/TestFoo#01 (0.00s) --- PASS: Test/Suite/testo!/TestFoo#02 (0.00s) PASS -ok github.com/ozontech/testo/examples/07_annotations 0.319s +ok github.com/ozontech/testo/examples/07_annotations 0.272s diff --git a/examples/README.md b/examples/README.md index 8d0a055..5ee250b 100644 --- a/examples/README.md +++ b/examples/README.md @@ -1,17 +1,32 @@ # Examples -Showcase of various Testo features. +Small runnable projects, one per Testo feature. -Examples are sorted by their simplicity in ascending order from basic to advanced. -You may want to start from the simplest -examples such as [01_suite](./01_suite/main_test.go) or [01_suiteless](./01_suiteless/main_test.go). +| Example | What it shows | +| :------ | :------------ | +| [01_suite](./01_suite/main_test.go) | A minimal suite with a few tests. | +| [01_suiteless](./01_suiteless/main_test.go) | Running tests without a suite. | +| [02_hooks](./02_hooks/main_test.go) | Suite hooks: `BeforeAll`, `BeforeEach`, `AfterEach`, `AfterAll`. | +| [03_parametrized](./03_parametrized/main_test.go) | Parametrized tests with `CasesXxx` methods. | +| [04_plugins](./04_plugins/main_test.go) | Writing plugins: hooks, method overrides, test planning, new `T` methods. | +| [05_parallel](./05_parallel/main_test.go) | Parallel tests and how hooks behave around them. | +| [06_errors](./06_errors/main_test.go) | Error messages for common mistakes. **Fails on purpose.** | +| [07_annotations](./07_annotations/main_test.go) | Attaching static options to tests with `testo.For` & `testo.ForEach`. | +| [08_subsuites](./08_subsuites/main_test.go) | Nesting suites with `testo.RunSubSuite`. | -To run each test execute the following command in the example directory: +Examples are sorted from basic to advanced. +Start with [01_suite](./01_suite/main_test.go) or [01_suiteless](./01_suiteless/main_test.go). + +To run an example, execute the following command in its directory: ```bash go test . -v -tags example -count=1 ``` -Or, if you can run `make` to do the same. +Or run `make` to do the same. + +> [!NOTE] +> `06_errors` demonstrates how Testo reports mistakes, +> so its `go test` run is expected to fail. -Each example includes its output in `output.golden` file. +Each example includes its expected output in an `output.golden` file. diff --git a/examples_test.go b/examples_test.go index a31d540..a6ee931 100644 --- a/examples_test.go +++ b/examples_test.go @@ -17,7 +17,7 @@ import ( var ignoreLineOrder = []string{ "02_hooks", - "06_parallel", + "05_parallel", } func TestExamples(t *testing.T) { From 7e1f1c2d1943fe167090bf0c993d0b602af2bd94 Mon Sep 17 00:00:00 2001 From: metafates Date: Sat, 25 Jul 2026 16:52:41 +0300 Subject: [PATCH 2/8] update docs --- COMPARISON.md | 2 +- README.md | 19 ++------ docs/how-to.md | 113 ++++++++++++++++++++------------------------- docs/migration.md | 17 +++---- docs/tutorial.md | 43 ++++++++--------- examples/README.md | 12 +---- 6 files changed, 81 insertions(+), 125 deletions(-) diff --git a/COMPARISON.md b/COMPARISON.md index d1fdd76..e91382f 100644 --- a/COMPARISON.md +++ b/COMPARISON.md @@ -34,7 +34,7 @@ If you are on Allure-Go today, see the [^2]: [Not supported](https://github.com/smartystreets/goconvey/issues/360). [^3]: See [issue #934](https://github.com/stretchr/testify/issues/934). [^4]: DSL: Domain-Specific Language. Requires describing tests in a specific way, different from usual Go tests. Not necessarily a bad thing, but has a learning curve and is less flexible. -[^5]: Whether it has any dependencies. Not necessarily a bad thing, but fewer dependencies mean a smaller footprint, faster build times and avoiding potential vulnerabilities. +[^5]: Whether it has any dependencies. Fewer dependencies mean a smaller footprint, faster builds and fewer potential vulnerabilities. [^6]: Any report format is achievable through plugins, but none is baked into Testo by default. See [testo-allure](https://github.com/ozontech/testo-allure). [^7]: "Native `go test`" means _all features_ are supported using only the `go test` command without any other CLIs. diff --git a/README.md b/README.md index ad8cdaa..f83d6b9 100644 --- a/README.md +++ b/README.md @@ -13,12 +13,12 @@ with an extensive plugin system. > Testo (/tɛstɒ/) is a play on words "test" and "тесто", meaning "dough". > Just like you can cook anything from dough, you can test anything with Testo! -Add some flavor to your tests with -[toppings - a collection of small, miscellaneous plugins for Testo framework.](https://github.com/ozontech/testo-toppings) +See also [toppings](https://github.com/ozontech/testo-toppings) - +a collection of small optional plugins for Testo. ## Features -- [Plugins](./examples/04_plugins/main_test.go) - adapt your tests to any scenario with features you need. +- [Plugins](./examples/04_plugins/main_test.go) - hook, filter and extend `T` without forking the framework. - [Parametrized tests](./examples/03_parametrized/main_test.go) - describe a test once, repeat it with different parameters. - [Parallel tests](./examples/05_parallel/main_test.go) - make your tests faster by running them all at once. - [Lifecycle hooks](./examples/02_hooks/main_test.go) - before and after any suite, test & sub-test. @@ -36,8 +36,6 @@ At Ozon, Testo powers thousands of end-to-end tests daily in production. With plugins, it is flexible enough to adapt to diverse requirements, without leaving the Go ecosystem - just a layer over `testing.T`. -If your needs are outgrowing standard `testing` package, Testo is a great choice. - See [comparison with other frameworks](./COMPARISON.md). ## Quick Start @@ -70,10 +68,7 @@ And run it with `go test` as usual: go test . ``` -But there is more! -Testo supports suites, parametrized tests & plugins, see [Next steps](#next-steps). - -See also [VS Code extension for Testo](#vs-code-extension). +Testo also supports suites, parametrized tests & plugins. ### Next steps @@ -132,9 +127,3 @@ See [contributing guidelines](./CONTRIBUTING.md). ## License This project is released under the [Apache-2.0 license](./LICENSE). - ---- - -> [!TIP] -> If you find Testo useful, [consider giving this repository a star](https://github.com/ozontech/testo) to help it reach more people. -> Thank you! diff --git a/docs/how-to.md b/docs/how-to.md index 5a515c5..79c5111 100644 --- a/docs/how-to.md +++ b/docs/how-to.md @@ -87,15 +87,14 @@ func (*Suite) TestParse(t *testo.T, p struct{ Case Case }) { The test runs once per element of the slice, with no cross-combination. -If for at least one required parameter function `CasesXxx` -returns zero values Testo will log a warning with similar message: +If a `CasesXxx` method returns an empty slice, Testo logs a warning +like this: ```txt main_test.go:15: testo: (*main.Suite).CasesName returned empty slice, (*main.Suite).TestFoo will not run ``` -To turn this log into fatal error and do not proceed with further execution -pass flag `-testo.strict` to the `go test` command invocation: +To make this a fatal error that stops the run, pass the `-testo.strict` flag: ```bash go test ./... -testo.strict @@ -122,11 +121,9 @@ func (*Suite) TestFoo(t *testo.T) { Standard `go test` flags such as `-parallel`, `-count` and `-timeout` apply unchanged - Testo tests are regular Go tests underneath. -You can expect all `AfterEach` and `AfterAll` hooks to execute at the end of each test properly. - -Please note, that `AfterEach` (only applies to it) hook is deferred to run at the end of the test. -If that test has sub-tests marked as parallel, -this hook will run BEFORE those sub-tests are finished. +`AfterEach` and `AfterAll` still run for parallel tests, but +`AfterEach` is deferred to the end of the test body - if the test has +parallel sub-tests, the hook runs BEFORE they finish: ```go func (*Suite) Test(t T) { @@ -210,65 +207,63 @@ func (s *Suite) TestFoo(t *testo.T) { } ``` -> Options can be propagated to inner sub-tests, but it's up to a plugin author -> to decide whether an option should be propagated or not. -> -> If needed, you can change propagation for certain options -> by setting `.Propagate` field to `false` (disable) or `true` (enable). -> However, it is highly discouraged to do so. +> Whether an option propagates to inner sub-tests is the plugin +> author's call. The `.Propagate` field overrides that, but you +> rarely should. ## How to use persistent cache -`testocache` stores key-value data between `go test` runs. -By default it uses `.testo_cache` in the test working directory. +`testocache` stores key-value data between `go test` runs: -```bash -go test ./path/to/package -cache.dir /tmp/my-testo-cache +```go +import "github.com/ozontech/testo/testocache" + +err := testocache.Set("token", []byte("abc")) +value, err := testocache.Get("token") ``` -When testing multiple packages with `./...`, use the environment variable so -packages that do not import `testocache` do not receive an unknown test flag: +By default the cache lives in `.testo_cache` in the test working +directory: ```bash -TESTO_CACHE_DIR=/tmp/my-testo-cache go test ./... -``` +# change the directory for one package +go test ./path/to/package -cache.dir /tmp/my-testo-cache -To disable the cache entirely, set `TESTO_CACHE_DISABLE`: +# for ./... use the env var, so packages that don't import +# testocache don't fail on an unknown flag +TESTO_CACHE_DIR=/tmp/my-testo-cache go test ./... -```bash +# disable the cache entirely TESTO_CACHE_DISABLE=true go test ./... ``` -> [!NOTE] -> `go test` caches successful test results. A cached pass skips execution -> entirely, so `testocache` state will not refresh on such runs. -> Pass `-count=1` to force execution when that matters. - -For plugin state, prefer a namespace: +For plugin state, prefer a namespace - isolated from other namespaces +and from the package-level functions, with the same `Get`, `Set`, +`Keys` and `Remove` methods: ```go var cache = testocache.Namespace("myplugin") ``` -Namespaces are isolated from each other and from package-level -`testocache.Get`, `Set`, `Keys`, and `Remove`. -The scoped cache provides the same `Get`, `Set`, `Keys`, and `Remove` methods. -`Keys` accepts the same glob syntax as `path.Match`. +Behavior details: -If cache is disabled, operations return `testocache.ErrDisabled`. -If a key is missing, `Get` and `Remove` return `testocache.ErrNotFound`. -Cache writes use a temporary file followed by `os.Rename`; atomic replacement -follows the guarantees of `os.Rename` on the host platform. -Malformed entries and entries whose stored key does not match the requested key -are treated as missing. Concurrent operations are synchronized inside one test -process; Testo does not provide cross-process locking for multiple `go test` -processes sharing the same cache directory. +- `Get` and `Remove` return `testocache.ErrNotFound` for missing keys. +- All operations return `testocache.ErrDisabled` when the cache is off. +- `Keys` accepts the same glob syntax as `path.Match`. +- Writes are atomic (temporary file + `os.Rename`); malformed entries + are treated as missing. +- Operations are synchronized within one process. There is no locking + between separate `go test` processes sharing a cache directory. + +> [!NOTE] +> `go test` caches successful test results, and a cached pass skips +> execution - so `testocache` state will not refresh. Pass `-count=1` +> to force a run. ## How to structure tests -Testo does not enforce any particular file structure to work. -However, some patterns are proved to be useful. -Here is one with standalone suite packages: +Testo does not enforce any particular file structure. +Here is one pattern we find useful - standalone suite packages: ```txt go.mod @@ -293,11 +288,9 @@ import ( "github.com/ozontech/testo/testoplugin" ) -// Here we define a global plugin common for all tests. -// -// It is highly advised to make it even if you don't use any plugins yet, -// as in the future it won't require any further changes in other files if you -// decide to add plugins. +// A plugin shared by all tests. +// Define it even with no plugins yet: adding one later +// then touches no other file. type PluginCommon struct { *testo.T } @@ -357,12 +350,9 @@ import ( "github.com/ozontech/testo" ) -// In this file we actually run our suites. -// The reason for starting tests here is that -// single file makes it easier to see what suites will we run or skip. -// -// Moreover, if you add some build tag, like we do here ("integration" on the top), -// specifying it only once makes it less typo-prone. +// All suites are started here. +// One file shows at a glance which suites run, +// and the build tag ("integration") is declared once. func TestSuiteFoo(t *testing.T) { testo.RunSuite(t, new(suitefoo.Suite)) @@ -417,9 +407,8 @@ go test . -run 'Test/MySuite' -testo.m TestFoo Here `-run 'Test/MySuite'` selects the suite and `-testo.m TestFoo` selects the method inside it. -Note that if `-testo.m` matches no tests, the suite runs empty (its -`BeforeAll`/`AfterAll` hooks still execute) and `go test` reports `PASS`. -Guard against "0 tests ran" in CI if that is a concern. +If `-testo.m` matches nothing, the suite runs empty (hooks still +fire) and `go test` reports `PASS` - guard against "0 tests ran" in CI. > [!NOTE] > `t.Name()` on `testo.T` returns the logical name without the `testo!` @@ -428,10 +417,6 @@ Guard against "0 tests ran" in CI if that is a concern. > includes it. Keep this in mind when mapping names back to `-run` patterns > or parsing test output. -> [!TIP] -> See also [Visual Studio Code extension](../vscode-extension) which -> generates correct run/debug commands for you. - ## How to annotate tests Annotations attach static plugin options to a specific test, diff --git a/docs/migration.md b/docs/migration.md index 9a9dfd5..80bfdd6 100644 --- a/docs/migration.md +++ b/docs/migration.md @@ -85,10 +85,8 @@ tests correctly. See ## Migrating from allure-go -Testo grew out of [allure-go](https://github.com/ozontech/allure-go) -at Ozon: allure-go couples the test framework with Allure reporting, -while Testo splits them - the framework is dependency-free, and -Allure reporting is one plugin among many. Keep in mind you need +Testo is [allure-go's successor at Ozon](../COMPARISON.md#why-we-moved-off-allure-go), +with reporting split out of the framework. That means you need **two** pieces: `github.com/ozontech/testo` and the [testo-allure](https://github.com/ozontech/testo-allure) plugin. @@ -132,12 +130,11 @@ Steps are sub-tests. Two ways to create them: parent test and stops it - matching `WithNewStep` semantics from allure-go. -When migrating `WithNewStep`-heavy code, `allure.Step` is the -drop-in choice. Nested steps started from a `StepCtx` -(`sCtx.WithNewStep`, `sCtx.NewStep`) become plain nested -`allure.Step(t, ...)` calls, and step-level assertions -(`sCtx.Assert()`, `sCtx.Require()`) become the `t.Assert()` / -`t.Require()` methods that `PluginAllure` adds to `T`. +For code full of `WithNewStep`, use `allure.Step` - same semantics. +Nested steps (`sCtx.WithNewStep`, `sCtx.NewStep`) become nested +`allure.Step(t, ...)` calls. Step assertions (`sCtx.Assert()`, +`sCtx.Require()`) become `t.Assert()` / `t.Require()`, which +`PluginAllure` adds to `T`. For the full testo-allure API (output directory, labels, attachments and so on), see the diff --git a/docs/tutorial.md b/docs/tutorial.md index c2aa34d..3b852b1 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -93,7 +93,7 @@ go test . -v PASS ``` -So far, so ordinary. Let's bring in Testo. +So far this is a plain Go test. Let's bring in Testo. ## Switching to Testo @@ -127,12 +127,11 @@ func TestBake(t *testing.T) { The body of the test did not change - only the type of `t`. -`testo.T` is a wrapper around `testing.T`. -All methods available on `testing.T` are also available on `testo.T` -(except `t.Run` - sub-tests are started with `testo.Run`, covered -[later](#steps-sub-tests)), and `testo.T` implements the `testing.TB` -interface, so libraries built for standard tests (assertion helpers, -mocks and so on) keep working. +`testo.T` wraps `testing.T` and keeps its interface, with one +exception: sub-tests are started with `testo.Run` instead of `t.Run` +(covered [later](#steps-sub-tests)). It also implements `testing.TB`, +so assertion libraries and mocks built for standard tests keep +working. Run the test again: @@ -253,21 +252,22 @@ accepts, then collects and initializes the plugins listed in it: PASS ``` +Note the logged name has no `testo!` in it: `t.Name()` returns the +logical test name +([details](./how-to.md#how-to-run-and-skip-specific-tests)). + > [!NOTE] > Plugins must be embedded as pointers. Pointers let plugins share > state with each other by pointing to the same memory. -> -> Also note that `t.Name()` on `testo.T` returns the logical test name -> without the `testo!` segment. Plugins can do much more than hooks: reorder or filter the test plan, wrap built-in methods like `t.Log`, add new methods to `T`, accept -options and command line flags. See the -[plugins example](../examples/04_plugins/main_test.go) for a tour of -`Plan` and `Overrides`, and the ready-made plugins: +[options](./how-to.md#how-to-use-plugin-options) and command line +flags. The [plugins example](../examples/04_plugins/main_test.go) +shows `Plan` and `Overrides` in action. Ready-made plugins: [testo-allure](https://github.com/ozontech/testo-allure) for Allure -reports and [testo-toppings](https://github.com/ozontech/testo-toppings) -for a collection of small plugins. +reports, [testo-toppings](https://github.com/ozontech/testo-toppings) +for small everyday ones. For instance, here is a complete plugin that makes every test parallel: @@ -283,17 +283,10 @@ func (p *PluginParallel) Plugin(testoplugin.Plugin, ...testoplugin.Option) (spec } ``` -This one is for illustration only - don't add it to your file, or the -outputs (and their line numbers) below will differ. A more complete -version of it is available as the +Don't add this one to your file (it would change the outputs below) - +a fuller version ships as the [parallel plugin](https://github.com/ozontech/testo-toppings/tree/main/parallel) -from testo-toppings. - -Plugins can also accept options passed to `testo.RunSuite` or -`testo.Run` - see -[how to use plugin options](./how-to.md#how-to-use-plugin-options) -and the [annotations example](../examples/07_annotations/plugin.go) -for a plugin that defines its own options. +in testo-toppings. ## Suites diff --git a/examples/README.md b/examples/README.md index 5ee250b..ad4ff61 100644 --- a/examples/README.md +++ b/examples/README.md @@ -14,19 +14,11 @@ Small runnable projects, one per Testo feature. | [07_annotations](./07_annotations/main_test.go) | Attaching static options to tests with `testo.For` & `testo.ForEach`. | | [08_subsuites](./08_subsuites/main_test.go) | Nesting suites with `testo.RunSubSuite`. | -Examples are sorted from basic to advanced. -Start with [01_suite](./01_suite/main_test.go) or [01_suiteless](./01_suiteless/main_test.go). - -To run an example, execute the following command in its directory: +To run an example, execute the following command in its directory +(or run `make`, which does the same): ```bash go test . -v -tags example -count=1 ``` -Or run `make` to do the same. - -> [!NOTE] -> `06_errors` demonstrates how Testo reports mistakes, -> so its `go test` run is expected to fail. - Each example includes its expected output in an `output.golden` file. From c74436fe04b4791fd8c2c027062b143eb1907a14 Mon Sep 17 00:00:00 2001 From: metafates Date: Sat, 25 Jul 2026 17:04:49 +0300 Subject: [PATCH 3/8] update docs --- COMPARISON.md | 2 +- README.md | 9 +++---- docs/how-to.md | 52 +++++++++++++++++++++----------------- docs/migration.md | 17 ++++++------- docs/technical-overview.md | 4 +-- docs/tutorial.md | 30 ++++++++++++---------- 6 files changed, 61 insertions(+), 53 deletions(-) diff --git a/COMPARISON.md b/COMPARISON.md index e91382f..c3a1e79 100644 --- a/COMPARISON.md +++ b/COMPARISON.md @@ -34,7 +34,7 @@ If you are on Allure-Go today, see the [^2]: [Not supported](https://github.com/smartystreets/goconvey/issues/360). [^3]: See [issue #934](https://github.com/stretchr/testify/issues/934). [^4]: DSL: Domain-Specific Language. Requires describing tests in a specific way, different from usual Go tests. Not necessarily a bad thing, but has a learning curve and is less flexible. -[^5]: Whether it has any dependencies. Fewer dependencies mean a smaller footprint, faster builds and fewer potential vulnerabilities. +[^5]: Whether it has any dependencies. [^6]: Any report format is achievable through plugins, but none is baked into Testo by default. See [testo-allure](https://github.com/ozontech/testo-allure). [^7]: "Native `go test`" means _all features_ are supported using only the `go test` command without any other CLIs. diff --git a/README.md b/README.md index f83d6b9..79b1bff 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,8 @@ a collection of small optional plugins for Testo. - [Parallel tests](./examples/05_parallel/main_test.go) - make your tests faster by running them all at once. - [Lifecycle hooks](./examples/02_hooks/main_test.go) - before and after any suite, test & sub-test. - [Test annotations](./examples/07_annotations/main_test.go) - attach static options to any test. -- [Informative errors and traces](./examples/06_errors/main_test.go) - no need to guess what went wrong. +- [Test filtering](./docs/how-to.md#how-to-run-and-skip-specific-tests) - `-run`, `-testo.m`, and the one trap to know before writing CI filters. +- [Informative errors and traces](./examples/06_errors/main_test.go) - error messages name the exact method and type that caused them. - [Sub-tests & sub-suites](./examples/08_subsuites/main_test.go) - support for nested tests and nested suites. - [Test reflection](https://pkg.go.dev/github.com/ozontech/testo/testoreflect) - deeply inspect test's meta-information. - [Caching](./docs/how-to.md#how-to-use-persistent-cache) - key-value storage persistent between test runs. @@ -33,8 +34,8 @@ a collection of small optional plugins for Testo. At Ozon, Testo powers thousands of end-to-end tests daily in production. -With plugins, it is flexible enough to adapt to diverse requirements, -without leaving the Go ecosystem - just a layer over `testing.T`. +Plugins let teams add what they need: reporting, retries, custom `T` +methods. Tests stay plain `go test` tests. See [comparison with other frameworks](./COMPARISON.md). @@ -81,8 +82,6 @@ Testo also supports suites, parametrized tests & plugins. ## Plugins -Testo features a powerful plugin system. - Plugins can: - Provide `BeforeAll`/`AfterAll`, `BeforeEach`/`AfterEach` & `BeforeEachSub`/`AfterEachSub` hooks. diff --git a/docs/how-to.md b/docs/how-to.md index 79c5111..3731b18 100644 --- a/docs/how-to.md +++ b/docs/how-to.md @@ -54,7 +54,7 @@ TestFoo with p = {Name: "Joe", Age: 60} TestFoo with p = {Name: "Joe", Age: 6} ``` -Note that parameter values do not appear in test names. +Parameter values do not appear in test names. In `go test -v` output the runs are named `TestFoo`, `TestFoo#01`, `TestFoo#02` and so on, following the standard `go test` convention for repeated names. @@ -121,8 +121,10 @@ func (*Suite) TestFoo(t *testo.T) { Standard `go test` flags such as `-parallel`, `-count` and `-timeout` apply unchanged - Testo tests are regular Go tests underneath. +### Hooks and parallel sub-tests + `AfterEach` and `AfterAll` still run for parallel tests, but -`AfterEach` is deferred to the end of the test body - if the test has +`AfterEach` is deferred to the end of the test body. If the test has parallel sub-tests, the hook runs BEFORE they finish: ```go @@ -207,9 +209,9 @@ func (s *Suite) TestFoo(t *testo.T) { } ``` -> Whether an option propagates to inner sub-tests is the plugin -> author's call. The `.Propagate` field overrides that, but you -> rarely should. +> The plugin author decides if an option is passed to inner +> sub-tests. You can force this with the `.Propagate` field, +> but usually you should not. ## How to use persistent cache @@ -237,9 +239,9 @@ TESTO_CACHE_DIR=/tmp/my-testo-cache go test ./... TESTO_CACHE_DISABLE=true go test ./... ``` -For plugin state, prefer a namespace - isolated from other namespaces -and from the package-level functions, with the same `Get`, `Set`, -`Keys` and `Remove` methods: +For plugin state, prefer a namespace. It is isolated from other +namespaces and from the package-level functions, and has the same +`Get`, `Set`, `Keys` and `Remove` methods: ```go var cache = testocache.Namespace("myplugin") @@ -289,8 +291,8 @@ import ( ) // A plugin shared by all tests. -// Define it even with no plugins yet: adding one later -// then touches no other file. +// Define it even if you have no plugins yet. When you add +// a plugin later, you will change only this file. type PluginCommon struct { *testo.T } @@ -373,13 +375,14 @@ TestFunc/SuiteName/testo!/TestMethod[/sub-test...] ``` > [!WARNING] -> A `-run` pattern that omits the `testo!` segment, such as +> A `-run` pattern without the `testo!` segment, such as > `-run 'Test/MySuite/TestFoo'`, matches **zero tests** - and `go test` -> still reports `PASS`. This is also the pattern most IDE "run test" buttons -> generate. Either include the segment (`-run 'Test/MySuite/testo!/TestFoo'`) -> or - better - use the `-testo.m` flag below. -> In VS Code, the [Testo extension](../vscode-extension) generates -> correct run/debug commands for you. +> still reports `PASS`. Most IDE "run test" buttons generate exactly +> this broken pattern. +> +> Either include the segment (`-run 'Test/MySuite/testo!/TestFoo'`) +> or, better, use the `-testo.m` flag below. In VS Code, the +> [Testo extension](../vscode-extension) generates correct commands. Testo provides its own flag `-testo.m regexp` to select suite tests by method name, without worrying about the `testo!` segment. @@ -407,15 +410,15 @@ go test . -run 'Test/MySuite' -testo.m TestFoo Here `-run 'Test/MySuite'` selects the suite and `-testo.m TestFoo` selects the method inside it. -If `-testo.m` matches nothing, the suite runs empty (hooks still -fire) and `go test` reports `PASS` - guard against "0 tests ran" in CI. +If `-testo.m` matches no tests, the suite runs with zero tests +(hooks still run) and `go test` shows `PASS`. Add a CI check that +at least one test ran. > [!NOTE] -> `t.Name()` on `testo.T` returns the logical name without the `testo!` -> segment (e.g. `Test/MySuite/TestFoo`), while the underlying `testing.T` -> name - the one shown in `go test -v` output and required by `-run` - -> includes it. Keep this in mind when mapping names back to `-run` patterns -> or parsing test output. +> `t.Name()` returns the name without `testo!`, e.g. `Test/MySuite/TestFoo`. +> The real `testing.T` name (the one in `go test -v` output and in `-run` +> patterns) still contains `testo!`. Remember this when you parse test +> output or build `-run` patterns. ## How to annotate tests @@ -465,6 +468,9 @@ work unchanged. Notes: +- Before writing CI test filters, read + [how to run and skip specific tests](#how-to-run-and-skip-specific-tests) - + a `-run` pattern without `testo!` silently runs zero tests. - The hidden `testo!` node appears in reports as an extra nesting level (e.g. JUnit converters render it as an empty intermediate node). - For Allure reports, use the [testo-allure plugin](https://github.com/ozontech/testo-allure). diff --git a/docs/migration.md b/docs/migration.md index 80bfdd6..cfeb5af 100644 --- a/docs/migration.md +++ b/docs/migration.md @@ -29,7 +29,7 @@ package with no interference. Migrate one suite at a time. One flag caveat: testify method filters like `-run 'TestMySuite/TestFoo'` stop matching after migration, because Testo inserts a hidden `testo!` -level into test names. Update CI filters to use `-testo.m` - see +level into test names. Update CI filters to use `-testo.m`. See [how to run and skip specific tests](./how-to.md#how-to-run-and-skip-specific-tests). `T` here is your own type built on `*testo.T` - in the simplest case @@ -69,11 +69,10 @@ There is no need to drop testify to adopt Testo - it replaces ### Suite state testify suites keep per-test state in suite struct fields. -A Testo suite is also a single instance, so fields shared by all -tests still work - but for per-test state prefer methods on your `T` -(fixtures) or plugin fields, which get a fresh instance per test and -are safe with parallel tests. See -[fixtures in the tutorial](./tutorial.md#fixtures). +A Testo suite is also a single instance, so shared fields still work. +For per-test state, prefer [fixtures](./tutorial.md#fixtures) +(methods on your `T`) or plugin fields. Both get a fresh instance +for each test and are safe with parallel tests. ### Parallelism @@ -85,9 +84,9 @@ tests correctly. See ## Migrating from allure-go -Testo is [allure-go's successor at Ozon](../COMPARISON.md#why-we-moved-off-allure-go), -with reporting split out of the framework. That means you need -**two** pieces: `github.com/ozontech/testo` and the +Testo [replaces allure-go at Ozon](../COMPARISON.md#why-we-moved-off-allure-go). +Reporting is now a separate plugin, not part of the framework. +So you need **two** pieces: `github.com/ozontech/testo` and the [testo-allure](https://github.com/ozontech/testo-allure) plugin. ```go diff --git a/docs/technical-overview.md b/docs/technical-overview.md index 69aeef2..5db5842 100644 --- a/docs/technical-overview.md +++ b/docs/technical-overview.md @@ -5,8 +5,8 @@ A technical overview of Testo. ## Mechanism Testo works through runtime reflection over suite method sets and `T` -types - there is no code generation, `go:generate` step or separate -CLI. This puts a few constraints on your types, checked before any +types. There is no code generation and no separate CLI. +This puts a few constraints on your types, checked before any test runs: - Plugins (and all exported fields of `T` and plugin structs) must be diff --git a/docs/tutorial.md b/docs/tutorial.md index 3b852b1..d94643e 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -155,7 +155,8 @@ Two technical levels appeared in the names: does affect `-run` patterns - see [how to run specific tests](./how-to.md#how-to-run-and-skip-specific-tests). -So far Testo gave us nothing new. Its value starts with plugins. +So far Testo has given us nothing new. Plugins are where it gets +interesting. ## Plugins @@ -212,10 +213,11 @@ func (p *PluginTimer) Plugin(testoplugin.Plugin, ...testoplugin.Option) (spec te } ``` -The plugin embeds `*testo.T` - Testo initializes it with the same `T` -as the current test, so the plugin sees everything the test sees. -The `Plugin` method is called for each test, so writing to plugin -fields is safe: every test gets its own instance. +The plugin embeds `*testo.T`. Testo fills it with the same `T` as +the current test, so the plugin sees everything the test sees. + +The `Plugin` method is called for each test, and every test gets its +own plugin instance. That is why writing to plugin fields is safe. To use the plugin, define your own `T` type that embeds `*testo.T` together with the plugins you want: @@ -264,10 +266,12 @@ Plugins can do much more than hooks: reorder or filter the test plan, wrap built-in methods like `t.Log`, add new methods to `T`, accept [options](./how-to.md#how-to-use-plugin-options) and command line flags. The [plugins example](../examples/04_plugins/main_test.go) -shows `Plan` and `Overrides` in action. Ready-made plugins: +shows `Plan` and `Overrides` in action. + +Ready-made plugins: [testo-allure](https://github.com/ozontech/testo-allure) for Allure -reports, [testo-toppings](https://github.com/ozontech/testo-toppings) -for small everyday ones. +reports, and [testo-toppings](https://github.com/ozontech/testo-toppings) - +a collection of small utility plugins. For instance, here is a complete plugin that makes every test parallel: @@ -283,8 +287,8 @@ func (p *PluginParallel) Plugin(testoplugin.Plugin, ...testoplugin.Option) (spec } ``` -Don't add this one to your file (it would change the outputs below) - -a fuller version ships as the +Don't add this one to your file, or it would change the outputs +below. A more complete version is available as the [parallel plugin](https://github.com/ozontech/testo-toppings/tree/main/parallel) in testo-toppings. @@ -477,8 +481,8 @@ func (Bakery) TestBake(t T, p struct{ Dessert string }) { } ``` -Fixtures in Testo are ordinary methods, without any registration -step, and they are reusable across all tests of the suite. +A fixture in Testo is an ordinary method. There is nothing to +register, and every test of the suite can use it. ## Steps (Sub-tests) @@ -519,7 +523,7 @@ func (Bakery) TestBake(t T, p struct{ Dessert string }) { > [!WARNING] > `t.Fatal` inside a sub-test stops only that sub-test, not the outer -> test - standard `go test` behavior. If a failed step must stop the +> test. That is standard `go test` behavior. If a failed step must stop the > whole test, check the sub-test result in the outer test, or use a > plugin that propagates the failure, such as `allure.Step` from > [testo-allure](https://github.com/ozontech/testo-allure). From 8c6a4c64abea7a38d5d7cdb3ba71459fa57cf6f9 Mon Sep 17 00:00:00 2001 From: metafates Date: Sat, 25 Jul 2026 17:16:17 +0300 Subject: [PATCH 4/8] update docs --- COMPARISON.md | 12 ++--- README.md | 8 ++-- docs/how-to.md | 71 +++++++++++++++++------------- docs/migration.md | 28 ++++++++++-- docs/technical-overview.md | 14 +++--- docs/tutorial.md | 10 ++++- examples/05_parallel/main_test.go | 2 + examples/05_parallel/output.golden | 38 ++++++++-------- 8 files changed, 113 insertions(+), 70 deletions(-) diff --git a/COMPARISON.md b/COMPARISON.md index c3a1e79..3f8a6f2 100644 --- a/COMPARISON.md +++ b/COMPARISON.md @@ -22,15 +22,15 @@ Testo can support BDD-style tests through plugins. ## Why we moved off Allure-Go Testo was created at Ozon as the successor to [Allure-Go]. -Allure-Go couples the test framework with Allure report generation: -reporting cannot be swapped out, and the framework cannot be extended -beyond what it ships with. Testo decouples the two - the framework -itself is a dependency-free layer over `testing.T`, and Allure -reporting is [one plugin among many](https://github.com/ozontech/testo-allure). +In Allure-Go, the test framework and Allure report generation are one +piece: you can't swap the reporting or extend the framework. Testo +splits them. The framework is a dependency-free layer over +`testing.T`, and Allure reporting is +[one plugin among many](https://github.com/ozontech/testo-allure). If you are on Allure-Go today, see the [migration guide](./docs/migration.md#migrating-from-allure-go). -[^1]: Ginkgo runs parallel tests in [separate processes](https://onsi.github.io/ginkgo/#mental-model-how-ginkgo-runs-parallel-specs) with its own runner (not available through `go test`). This is heavier-weight than native `go test` parallelization based on goroutines, trading startup and memory cost for process isolation. +[^1]: Ginkgo runs parallel tests in [separate processes](https://onsi.github.io/ginkgo/#mental-model-how-ginkgo-runs-parallel-specs) with its own runner (not available through `go test`). It costs more startup time and memory than goroutine-based `go test` parallelization, but you get process isolation in return. [^2]: [Not supported](https://github.com/smartystreets/goconvey/issues/360). [^3]: See [issue #934](https://github.com/stretchr/testify/issues/934). [^4]: DSL: Domain-Specific Language. Requires describing tests in a specific way, different from usual Go tests. Not necessarily a bad thing, but has a learning curve and is less flexible. diff --git a/README.md b/README.md index 79b1bff..b029044 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,7 @@ a collection of small optional plugins for Testo. - [Plugins](./examples/04_plugins/main_test.go) - hook, filter and extend `T` without forking the framework. - [Parametrized tests](./examples/03_parametrized/main_test.go) - describe a test once, repeat it with different parameters. - [Parallel tests](./examples/05_parallel/main_test.go) - make your tests faster by running them all at once. -- [Lifecycle hooks](./examples/02_hooks/main_test.go) - before and after any suite, test & sub-test. +- [Lifecycle hooks](./examples/02_hooks/main_test.go) - before and after any suite, test & sub-test ([hooks and parallel tests](./docs/how-to.md#how-to-write-parallel-tests)). - [Test annotations](./examples/07_annotations/main_test.go) - attach static options to any test. - [Test filtering](./docs/how-to.md#how-to-run-and-skip-specific-tests) - `-run`, `-testo.m`, and the one trap to know before writing CI filters. - [Informative errors and traces](./examples/06_errors/main_test.go) - error messages name the exact method and type that caused them. @@ -32,7 +32,7 @@ a collection of small optional plugins for Testo. ## Why Testo -At Ozon, Testo powers thousands of end-to-end tests daily in production. +Ozon runs thousands of end-to-end tests on Testo every day. Plugins let teams add what they need: reporting, retries, custom `T` methods. Tests stay plain `go test` tests. @@ -77,7 +77,7 @@ Testo also supports suites, parametrized tests & plugins. - See [test examples](./examples). - Learn [how to use various Testo features](./docs/how-to.md). - Migrating from testify or allure-go? See the [migration guide](./docs/migration.md). -- Read a [brief description and technical overview](./docs/technical-overview.md) of Testo. +- Read the [technical overview](./docs/technical-overview.md) - lifecycle, panics, plugin internals. - View [API documentation](https://pkg.go.dev/github.com/ozontech/testo). ## Plugins @@ -86,7 +86,7 @@ Plugins can: - Provide `BeforeAll`/`AfterAll`, `BeforeEach`/`AfterEach` & `BeforeEachSub`/`AfterEachSub` hooks. - Plan tests for execution - filter, duplicate & reorder. -- Override built-in `T` methods, such as `Log`, `Error` and _etc._ +- Override built-in `T` methods, such as `Log` and `Error`. - Extend `T` by adding new methods. - Allow users to configure their behavior through options. - Communicate with other plugins. diff --git a/docs/how-to.md b/docs/how-to.md index 3731b18..95a56ec 100644 --- a/docs/how-to.md +++ b/docs/how-to.md @@ -8,6 +8,7 @@ defined. - [How to write parametrized tests](#how-to-write-parametrized-tests) - [How to write parallel tests](#how-to-write-parallel-tests) + - [Hooks and parallel sub-tests](#hooks-and-parallel-sub-tests) - [How to use plugin options](#how-to-use-plugin-options) - [How to use persistent cache](#how-to-use-persistent-cache) - [How to structure tests](#how-to-structure-tests) @@ -41,7 +42,7 @@ func (*Suite) CasesAge() []int { > [!TIP] > `CasesXxx` are invoked *after* `BeforeAll` hook. -Field names used in a `struct{ Name string; Age int}` must be equal to existing `CasesXxx` functions. +Field names used in a `struct{ Name string; Age int }` must be equal to existing `CasesXxx` functions. Given that, test `TestFoo` will be invoked with all possible combinations of names and ages: @@ -58,6 +59,27 @@ Parameter values do not appear in test names. In `go test -v` output the runs are named `TestFoo`, `TestFoo#01`, `TestFoo#02` and so on, following the standard `go test` convention for repeated names. +If a `CasesXxx` method returns an empty slice, Testo logs a warning +like this: + +```txt +main_test.go:15: testo: warning: (*main.Suite).CasesName returned empty slice, (*main.Suite).TestFoo will not run +``` + +To make this a fatal error that stops the run, use strict mode: + +```bash +# one package +go test ./path/to/package -testo.strict + +# for ./... use the env var - packages that don't import +# Testo would fail on the unknown flag +TESTO_STRICT=true go test ./... +``` + +> [!TIP] +> Flags have higher priority than environment variables. + ### Table tests (correlated parameters) Separate parameters always produce the Cartesian product of their values. @@ -87,25 +109,6 @@ func (*Suite) TestParse(t *testo.T, p struct{ Case Case }) { The test runs once per element of the slice, with no cross-combination. -If a `CasesXxx` method returns an empty slice, Testo logs a warning -like this: - -```txt -main_test.go:15: testo: (*main.Suite).CasesName returned empty slice, (*main.Suite).TestFoo will not run -``` - -To make this a fatal error that stops the run, pass the `-testo.strict` flag: - -```bash -go test ./... -testo.strict -``` - -> [!TIP] -> You can also set `TESTO_STRICT` environment variable to `true` -> for the same effect. -> -> Flags have higher priority than environment variables. - ## How to write parallel tests You can use your regular `t.Parallel` method to mark a test as parallel. @@ -120,6 +123,7 @@ func (*Suite) TestFoo(t *testo.T) { Standard `go test` flags such as `-parallel`, `-count` and `-timeout` apply unchanged - Testo tests are regular Go tests underneath. +With `-count=N`, `BeforeAll` and `AfterAll` run once per iteration. ### Hooks and parallel sub-tests @@ -229,6 +233,7 @@ directory: ```bash # change the directory for one package +# (cache flags use the -cache. prefix, not -testo.) go test ./path/to/package -cache.dir /tmp/my-testo-cache # for ./... use the env var, so packages that don't import @@ -328,7 +333,7 @@ package suitefoo import "github.com/ozontech/testo" -// An actual test logic for this suite goes here. +// The test logic for this suite goes here. type Suite struct{ testo.Suite[T] } @@ -377,8 +382,9 @@ TestFunc/SuiteName/testo!/TestMethod[/sub-test...] > [!WARNING] > A `-run` pattern without the `testo!` segment, such as > `-run 'Test/MySuite/TestFoo'`, matches **zero tests** - and `go test` -> still reports `PASS`. Most IDE "run test" buttons generate exactly -> this broken pattern. +> still reports `PASS`. Suite hooks (`BeforeAll`/`AfterAll`) still run +> even when zero tests match. Most IDE "run test" buttons generate +> exactly this broken pattern. > > Either include the segment (`-run 'Test/MySuite/testo!/TestFoo'`) > or, better, use the `-testo.m` flag below. In VS Code, the @@ -401,7 +407,7 @@ func (MySuite) TestBar(t T) { } ``` -We can run only `TestFoo` like that: +To run only `TestFoo`: ```shell go test . -run 'Test/MySuite' -testo.m TestFoo @@ -422,9 +428,9 @@ at least one test ran. ## How to annotate tests -Annotations attach static plugin options to a specific test, -so plugins can see them before the test runs (useful for planning, -reporting, retries and similar features). +Annotations attach static plugin options to a specific test, so +plugins can see them before the test runs - for example to plan +retries or add report labels. Use `testo.For` for regular tests and `testo.ForEach` for parametrized ones: @@ -453,8 +459,10 @@ func WithRetry() testoplugin.Option { } ``` -The plugin receives options in its `Plugin` method and type-asserts -the `Value` field. +Plugins see annotations in two places: during planning, via +`PlannedTest.Annotations()` in `Plan.Prepare`, and per test, merged +into the `options` argument of the `Plugin` method. In both cases the +plugin type-asserts the `Value` field. See [annotations example](../examples/07_annotations/main_test.go) (its `plugin.go` defines the options) @@ -495,5 +503,8 @@ func (InnerSuite) Test(t T) { } ``` +See the [sub-suites example](../examples/08_subsuites/main_test.go). + > [!WARNING] -> Running the same suite as sub-suite may cause infinite loop. +> Running a suite inside itself causes an infinite loop. +> Keep sub-suite nesting acyclic. diff --git a/docs/migration.md b/docs/migration.md index cfeb5af..51142e2 100644 --- a/docs/migration.md +++ b/docs/migration.md @@ -24,6 +24,12 @@ package with no interference. Migrate one suite at a time. | `s.T()` | `t` (passed to every method) | | `s.Run(name, func())` | `testo.Run(t, name, func(t T))` | | `func (s *S) BeforeTest(suiteName, testName string)` | `BeforeEach(t T)` + `t.Name()` (same for `AfterTest`) | + +Two notes on the hook mapping. `t.Name()` returns the full path +(`Test/MySuite/TestFoo`), not the bare method name - compare with +`strings.HasSuffix`, not `==`. And if a suite uses both `SetupTest` +and `BeforeTest`, merge them into one `BeforeEach` (testify runs +`SetupTest` first; keep that order). | `SetupSubTest()` / `TearDownSubTest()` | `BeforeEachSub`/`AfterEachSub` plugin hooks (no suite-level equivalent) | | `go test -run TestSuite -testify.m TestFoo` | `go test -run TestSuite -testo.m TestFoo` | @@ -63,13 +69,15 @@ func (*MySuite) TestFoo(t T) { The `s.Require().X(...)` and shorthand `s.X(...)` forms both become plain `require.X(t, ...)` / `assert.X(t, ...)` calls. -There is no need to drop testify to adopt Testo - it replaces -`suite`, not `require`/`assert`. +Testo only replaces testify's `suite` package. Keep using `require` +and `assert`. ### Suite state testify suites keep per-test state in suite struct fields. A Testo suite is also a single instance, so shared fields still work. +But writes to shared fields from parallel tests are a data race - +run migrated packages with `-race` before enabling `t.Parallel()`. For per-test state, prefer [fixtures](./tutorial.md#fixtures) (methods on your `T`) or plugin fields. Both get a fresh instance for each test and are safe with parallel tests. @@ -115,8 +123,20 @@ type T struct { | `t.Title`, `t.Epic`, `t.Feature`, `t.Story`, `t.Tags`, `t.Severity`, `t.Owner`, `t.ID` | same-named methods added to `T` by `PluginAllure` | | `t.Link(...)`, `t.TmsLink(...)`, `t.TmsLinks(...)` | `t.Links(...)` with the `allure.NewLink`/`allure.TMS`/`allure.Issue` constructors | | `t.WithNewAttachment(name, mimeType, content)`, `t.WithAttachments(...)` | `t.Attach(name, allure.Bytes(...))` or `t.Attach(name, allure.File(...))` | -| table tests / `ParametrizedRunner` | single struct-typed parameter, see [table tests](./how-to.md#table-tests-correlated-parameters) | -| `t.XSkip()` | `t.XFail()` via the [xfail plugin](https://github.com/ozontech/testo-toppings/tree/main/xfail) from testo-toppings | +| table tests (`ParametrizedSuite` / `TableTestXxx` methods) | single struct-typed parameter, see [table tests](./how-to.md#table-tests-correlated-parameters) | +| `t.XSkip()` | `t.XFail()` via the [xfail plugin](https://github.com/ozontech/testo-toppings/tree/main/xfail) from testo-toppings (semantics differ slightly - check its README) | + +Other labels (`Description`, `Labels`, `Layer`, `Lead`, `Stage`, +`WithParameters`) map to the `Description`, `Labels` and `Parameters` +methods of `PluginAllure`. + +A minimal setup to start a proof of concept: + +```go +func Test(t *testing.T) { + testo.RunSuite(t, new(Bakery), allure.WithOutputDir("allure-results")) +} +``` ### Steps diff --git a/docs/technical-overview.md b/docs/technical-overview.md index 5db5842..3184cff 100644 --- a/docs/technical-overview.md +++ b/docs/technical-overview.md @@ -1,6 +1,7 @@ # Technical overview -A technical overview of Testo. +How Testo works under the hood. Read this if you write plugins or +debug hook ordering. ## Mechanism @@ -13,9 +14,9 @@ test runs: embedded as pointers. - Recursive plugin type references are detected and rejected. -Signature and `CasesXxx` mismatches are reported as test errors; -violations of the type constraints above panic with a descriptive -message, aborting the test binary. +Signature and `CasesXxx` mismatches are reported as test errors. +Violations of the type constraints above panic before any test runs, +aborting the test binary. Plugins are re-instantiated for every test and sub-test. This is what makes writing to plugin fields safe without synchronization, at the @@ -75,12 +76,15 @@ Other tests will run even if some tests are panicking. Testo **will** catch panics from `BeforeAll` & `AfterAll`. Panic in these hooks will result in suite tests not running. +If the suite-level `T` is skipped (for example, `t.Skip` inside +`BeforeAll`), the `AfterAll` hooks are skipped too. + See also [suite hooks in the tutorial](./tutorial.md#suite-hooks) and [hook behavior with parallel tests](./how-to.md#how-to-write-parallel-tests). ## Plugins -Testo uses dependency-injection-like mechanism to enable cross-plugin communication. +Testo uses a dependency-injection-like mechanism for cross-plugin communication. For example, assume we have plugin `X` and plugin `Y`. Plugin `X` needs to interact with plugin `Y`. To make it possible, `X` needs to embed `Y`: diff --git a/docs/tutorial.md b/docs/tutorial.md index d94643e..e42f15f 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -215,6 +215,9 @@ func (p *PluginTimer) Plugin(testoplugin.Plugin, ...testoplugin.Option) (spec te The plugin embeds `*testo.T`. Testo fills it with the same `T` as the current test, so the plugin sees everything the test sees. +The unused first argument of the `Plugin` method is the parent plugin +instance - see the +[technical overview](./technical-overview.md#lifecycle) if you need it. The `Plugin` method is called for each test, and every test gets its own plugin instance. That is why writing to plugin fields is safe. @@ -266,7 +269,9 @@ Plugins can do much more than hooks: reorder or filter the test plan, wrap built-in methods like `t.Log`, add new methods to `T`, accept [options](./how-to.md#how-to-use-plugin-options) and command line flags. The [plugins example](../examples/04_plugins/main_test.go) -shows `Plan` and `Overrides` in action. +shows `Plan` and `Overrides` in action, and the +[technical overview](./technical-overview.md#plugins) explains how +plugins talk to each other. Ready-made plugins: [testo-allure](https://github.com/ozontech/testo-allure) for Allure @@ -305,7 +310,8 @@ type Bakery struct{ testo.Suite[T] } Tests become methods of the suite. They follow the usual Go naming rules (the `Test` prefix) and must accept the same `T` as specified in -`testo.Suite[T]`. Replace the `TestBake` function with: +`testo.Suite[T]`. Value and pointer receivers both work. +Replace the `TestBake` function with: ```go func (Bakery) TestBake(t T) { diff --git a/examples/05_parallel/main_test.go b/examples/05_parallel/main_test.go index 27de900..a253914 100644 --- a/examples/05_parallel/main_test.go +++ b/examples/05_parallel/main_test.go @@ -26,6 +26,8 @@ func (Suite) AfterAll(t T) { } func (Suite) BeforeEach(t T) { + // AfterEach fires before parallel sub-tests finish; + // Cleanup runs after them. That is why teardown goes here. t.Cleanup(func() { t.Logf("cleanup %q", t.Name()) }) diff --git a/examples/05_parallel/output.golden b/examples/05_parallel/output.golden index 89cb061..f9ab823 100644 --- a/examples/05_parallel/output.golden +++ b/examples/05_parallel/output.golden @@ -3,21 +3,21 @@ main_test.go:21: starting suite === RUN Test/Suite/testo! === RUN Test/Suite/testo!/TestBar - main_test.go:33: before "Test/Suite/TestBar" + main_test.go:35: before "Test/Suite/TestBar" === PAUSE Test/Suite/testo!/TestBar === RUN Test/Suite/testo!/TestFoo - main_test.go:33: before "Test/Suite/TestFoo" + main_test.go:35: before "Test/Suite/TestFoo" === PAUSE Test/Suite/testo!/TestFoo === CONT Test/Suite/testo!/TestBar - main_test.go:62: inside test bar - main_test.go:37: after "Test/Suite/TestBar" - main_test.go:30: cleanup "Test/Suite/TestBar" + main_test.go:64: inside test bar + main_test.go:39: after "Test/Suite/TestBar" + main_test.go:32: cleanup "Test/Suite/TestBar" === CONT Test/Suite/testo!/TestFoo - main_test.go:44: inside test foo + main_test.go:46: inside test foo === RUN Test/Suite/testo!/TestFoo/sub-test === PAUSE Test/Suite/testo!/TestFoo/sub-test === NAME Test/Suite/testo!/TestFoo - main_test.go:37: after "Test/Suite/TestFoo" + main_test.go:39: after "Test/Suite/TestFoo" === CONT Test/Suite/testo!/TestFoo/sub-test === RUN Test/Suite/testo!/TestFoo/sub-test/inner_0 === PAUSE Test/Suite/testo!/TestFoo/sub-test/inner_0 @@ -40,27 +40,27 @@ === RUN Test/Suite/testo!/TestFoo/sub-test/inner_9 === PAUSE Test/Suite/testo!/TestFoo/sub-test/inner_9 === CONT Test/Suite/testo!/TestFoo/sub-test/inner_0 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_0" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_0" === CONT Test/Suite/testo!/TestFoo/sub-test/inner_5 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_5" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_5" === CONT Test/Suite/testo!/TestFoo/sub-test/inner_9 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_9" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_9" === CONT Test/Suite/testo!/TestFoo/sub-test/inner_8 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_8" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_8" === CONT Test/Suite/testo!/TestFoo/sub-test/inner_7 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_7" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_7" === CONT Test/Suite/testo!/TestFoo/sub-test/inner_6 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_6" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_6" === CONT Test/Suite/testo!/TestFoo/sub-test/inner_3 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_3" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_3" === CONT Test/Suite/testo!/TestFoo/sub-test/inner_4 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_4" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_4" === CONT Test/Suite/testo!/TestFoo/sub-test/inner_2 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_2" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_2" === CONT Test/Suite/testo!/TestFoo/sub-test/inner_1 - main_test.go:53: inside "Test/Suite/TestFoo/sub-test/inner_1" + main_test.go:55: inside "Test/Suite/TestFoo/sub-test/inner_1" === NAME Test/Suite/testo!/TestFoo - main_test.go:30: cleanup "Test/Suite/TestFoo" + main_test.go:32: cleanup "Test/Suite/TestFoo" === NAME Test/Suite main_test.go:25: finishing suite --- PASS: Test (0.00s) @@ -80,4 +80,4 @@ --- PASS: Test/Suite/testo!/TestFoo/sub-test/inner_2 (0.00s) --- PASS: Test/Suite/testo!/TestFoo/sub-test/inner_1 (0.00s) PASS -ok github.com/ozontech/testo/examples/05_parallel 0.314s +ok github.com/ozontech/testo/examples/05_parallel 0.364s From b1d5d1684e86353432326cc802215a3461c20adb Mon Sep 17 00:00:00 2001 From: metafates Date: Sat, 25 Jul 2026 18:57:15 +0300 Subject: [PATCH 5/8] update docs --- COMPARISON.md | 10 +++--- README.md | 8 ++--- docs/how-to.md | 68 ++++++++++++++++++++++---------------- docs/migration.md | 6 ++-- docs/technical-overview.md | 30 ++++++++++------- docs/tutorial.md | 15 ++++++--- examples/README.md | 3 ++ 7 files changed, 83 insertions(+), 57 deletions(-) diff --git a/COMPARISON.md b/COMPARISON.md index 3f8a6f2..d416719 100644 --- a/COMPARISON.md +++ b/COMPARISON.md @@ -19,22 +19,22 @@ Testo focuses on extensibility through plugins and stays a thin layer over usual Other frameworks may be a better fit if you need BDD scenarios or some unique features "out of the box". Testo can support BDD-style tests through plugins. -## Why we moved off Allure-Go +## Why we moved off allure-go -Testo was created at Ozon as the successor to [Allure-Go]. -In Allure-Go, the test framework and Allure report generation are one +Testo was created at Ozon as the successor to [allure-go][Allure-Go]. +In allure-go, the test framework and Allure report generation are one piece: you can't swap the reporting or extend the framework. Testo splits them. The framework is a dependency-free layer over `testing.T`, and Allure reporting is [one plugin among many](https://github.com/ozontech/testo-allure). -If you are on Allure-Go today, see the +If you are on allure-go today, see the [migration guide](./docs/migration.md#migrating-from-allure-go). [^1]: Ginkgo runs parallel tests in [separate processes](https://onsi.github.io/ginkgo/#mental-model-how-ginkgo-runs-parallel-specs) with its own runner (not available through `go test`). It costs more startup time and memory than goroutine-based `go test` parallelization, but you get process isolation in return. [^2]: [Not supported](https://github.com/smartystreets/goconvey/issues/360). [^3]: See [issue #934](https://github.com/stretchr/testify/issues/934). [^4]: DSL: Domain-Specific Language. Requires describing tests in a specific way, different from usual Go tests. Not necessarily a bad thing, but has a learning curve and is less flexible. -[^5]: Whether it has any dependencies. +[^5]: "Yes" means no third-party dependencies: only the standard library is used. [^6]: Any report format is achievable through plugins, but none is baked into Testo by default. See [testo-allure](https://github.com/ozontech/testo-allure). [^7]: "Native `go test`" means _all features_ are supported using only the `go test` command without any other CLIs. diff --git a/README.md b/README.md index b029044..2631bc9 100644 --- a/README.md +++ b/README.md @@ -20,7 +20,7 @@ a collection of small optional plugins for Testo. - [Plugins](./examples/04_plugins/main_test.go) - hook, filter and extend `T` without forking the framework. - [Parametrized tests](./examples/03_parametrized/main_test.go) - describe a test once, repeat it with different parameters. -- [Parallel tests](./examples/05_parallel/main_test.go) - make your tests faster by running them all at once. +- [Parallel tests](./examples/05_parallel/main_test.go) - run independent tests concurrently. - [Lifecycle hooks](./examples/02_hooks/main_test.go) - before and after any suite, test & sub-test ([hooks and parallel tests](./docs/how-to.md#how-to-write-parallel-tests)). - [Test annotations](./examples/07_annotations/main_test.go) - attach static options to any test. - [Test filtering](./docs/how-to.md#how-to-run-and-skip-specific-tests) - `-run`, `-testo.m`, and the one trap to know before writing CI filters. @@ -75,7 +75,7 @@ Testo also supports suites, parametrized tests & plugins. - Take [a guided tour of Testo](./docs/tutorial.md) by making simple plugins and running the tests using various features. - See [test examples](./examples). -- Learn [how to use various Testo features](./docs/how-to.md). +- Learn [how to use Testo features](./docs/how-to.md). - Migrating from testify or allure-go? See the [migration guide](./docs/migration.md). - Read the [technical overview](./docs/technical-overview.md) - lifecycle, panics, plugin internals. - View [API documentation](https://pkg.go.dev/github.com/ozontech/testo). @@ -107,7 +107,7 @@ Examples: Testo has its own [VS Code extension](./vscode-extension). -Makes it easier to run and debug individual suite tests and adds helpful snippets. +The extension adds run/debug buttons for individual suite tests, plus snippets. ![VSCode extension screenshot showing codelens buttons for running and debugging a test](./vscode-extension/example.png) @@ -115,7 +115,7 @@ Makes it easier to run and debug individual suite tests and adds helpful snippet Testo guarantees to support at least **3 latest major** [Go releases](https://go.dev/doc/devel/release). -Currently, minimum supported Go version is **1.24** +Currently, the minimum supported Go version is **1.24**. ## Contributing diff --git a/docs/how-to.md b/docs/how-to.md index 95a56ec..860a7fa 100644 --- a/docs/how-to.md +++ b/docs/how-to.md @@ -1,12 +1,12 @@ -# How to - -Learn how to use the features of Testo. +# How to use Testo features Snippets on this page are fragments: they assume a project set up as in the [tutorial](./tutorial.md), with a suite and a `T` type already defined. - [How to write parametrized tests](#how-to-write-parametrized-tests) + - [Table tests](#table-tests-correlated-parameters) +- [How to enable strict mode](#how-to-enable-strict-mode) - [How to write parallel tests](#how-to-write-parallel-tests) - [Hooks and parallel sub-tests](#hooks-and-parallel-sub-tests) - [How to use plugin options](#how-to-use-plugin-options) @@ -22,7 +22,7 @@ defined. Parametrized tests are defined as regular tests with a second argument: ```go -func (*Suite) TestFoo(t *testo.T, p struct{ Name string; Age int }) { +func (*Suite) TestFoo(t T, p struct{ Name string; Age int }) { t.Logf("Using name=%q and age=%d", p.Name, p.Age) } ``` @@ -42,7 +42,7 @@ func (*Suite) CasesAge() []int { > [!TIP] > `CasesXxx` are invoked *after* `BeforeAll` hook. -Field names used in a `struct{ Name string; Age int }` must be equal to existing `CasesXxx` functions. +Each parameter field must have a matching `CasesXxx` method. Given that, test `TestFoo` will be invoked with all possible combinations of names and ages: @@ -58,27 +58,19 @@ TestFoo with p = {Name: "Joe", Age: 6} Parameter values do not appear in test names. In `go test -v` output the runs are named `TestFoo`, `TestFoo#01`, `TestFoo#02` and so on, following the standard `go test` convention for repeated names. +Log the parameters at the start of the test, so a failing `TestFoo#03` +identifies its case - see the +[parametrized example](../examples/03_parametrized/main_test.go). If a `CasesXxx` method returns an empty slice, Testo logs a warning -like this: +(visible with `go test -v`): ```txt main_test.go:15: testo: warning: (*main.Suite).CasesName returned empty slice, (*main.Suite).TestFoo will not run ``` -To make this a fatal error that stops the run, use strict mode: - -```bash -# one package -go test ./path/to/package -testo.strict - -# for ./... use the env var - packages that don't import -# Testo would fail on the unknown flag -TESTO_STRICT=true go test ./... -``` - -> [!TIP] -> Flags have higher priority than environment variables. +To make this warning fatal, enable +[strict mode](#how-to-enable-strict-mode). ### Table tests (correlated parameters) @@ -100,7 +92,7 @@ func (*Suite) CasesCase() []Case { } } -func (*Suite) TestParse(t *testo.T, p struct{ Case Case }) { +func (*Suite) TestParse(t T, p struct{ Case Case }) { if got := Parse(p.Case.Input); got != p.Case.Want { t.Errorf("Parse(%q) = %d, want %d", p.Case.Input, got, p.Case.Want) } @@ -109,12 +101,30 @@ func (*Suite) TestParse(t *testo.T, p struct{ Case Case }) { The test runs once per element of the slice, with no cross-combination. +## How to enable strict mode + +Strict mode turns every Testo warning into a fatal error. +Today there are two warnings: a suite with no tests, and a +`CasesXxx` method returning an empty slice (shown above): + +```bash +# one package +go test ./path/to/package -testo.strict + +# for ./... use the env var - packages that don't import +# Testo would fail on the unknown flag +TESTO_STRICT=true go test ./... +``` + +> [!TIP] +> Flags have higher priority than environment variables. + ## How to write parallel tests -You can use your regular `t.Parallel` method to mark a test as parallel. +Mark a test as parallel with the standard `t.Parallel`: ```go -func (*Suite) TestFoo(t *testo.T) { +func (*Suite) TestFoo(t T) { t.Parallel() // your test here @@ -129,7 +139,7 @@ With `-count=N`, `BeforeAll` and `AfterAll` run once per iteration. `AfterEach` and `AfterAll` still run for parallel tests, but `AfterEach` is deferred to the end of the test body. If the test has -parallel sub-tests, the hook runs BEFORE they finish: +parallel sub-tests, the hook runs **before** they finish: ```go func (*Suite) Test(t T) { @@ -206,13 +216,14 @@ func Test(t *testing.T) { Passing to `testo.Run`: ```go -func (s *Suite) TestFoo(t *testo.T) { - testo.Run(t, "my sub-test", func(t *testo.T) { +func (s *Suite) TestFoo(t T) { + testo.Run(t, "my sub-test", func(t T) { // ... }, myplugin.SomeOption()) } ``` +> [!NOTE] > The plugin author decides if an option is passed to inner > sub-tests. You can force this with the `.Propagate` field, > but usually you should not. @@ -368,9 +379,9 @@ func TestSuiteFoo(t *testing.T) { ## How to run and skip specific tests -Testo works with default go test flags, such as `-run` and `-skip`. +Testo supports the standard `go test` flags, such as `-run` and `-skip`. -> See `go help testflag` for detailed flags description. +See `go help testflag` for the full flag reference. To make hooks work correctly with parallel tests, Testo inserts a hidden `testo!` level into every suite. The full name of a suite test is: @@ -418,7 +429,8 @@ selects the method inside it. If `-testo.m` matches no tests, the suite runs with zero tests (hooks still run) and `go test` shows `PASS`. Add a CI check that -at least one test ran. +at least one test ran - for example, grep `go test -json` output +for `testo!/` test events. > [!NOTE] > `t.Name()` returns the name without `testo!`, e.g. `Test/MySuite/TestFoo`. diff --git a/docs/migration.md b/docs/migration.md index 51142e2..3367b4f 100644 --- a/docs/migration.md +++ b/docs/migration.md @@ -1,6 +1,6 @@ # Migration guide -How to move an existing test base to Testo - from +How to move existing tests to Testo - from [testify](#migrating-from-testify) or from [allure-go](#migrating-from-allure-go). @@ -24,14 +24,14 @@ package with no interference. Migrate one suite at a time. | `s.T()` | `t` (passed to every method) | | `s.Run(name, func())` | `testo.Run(t, name, func(t T))` | | `func (s *S) BeforeTest(suiteName, testName string)` | `BeforeEach(t T)` + `t.Name()` (same for `AfterTest`) | +| `SetupSubTest()` / `TearDownSubTest()` | `BeforeEachSub`/`AfterEachSub` plugin hooks (no suite-level equivalent) | +| `go test -run TestSuite -testify.m TestFoo` | `go test -run TestSuite -testo.m TestFoo` | Two notes on the hook mapping. `t.Name()` returns the full path (`Test/MySuite/TestFoo`), not the bare method name - compare with `strings.HasSuffix`, not `==`. And if a suite uses both `SetupTest` and `BeforeTest`, merge them into one `BeforeEach` (testify runs `SetupTest` first; keep that order). -| `SetupSubTest()` / `TearDownSubTest()` | `BeforeEachSub`/`AfterEachSub` plugin hooks (no suite-level equivalent) | -| `go test -run TestSuite -testify.m TestFoo` | `go test -run TestSuite -testo.m TestFoo` | One flag caveat: testify method filters like `-run 'TestMySuite/TestFoo'` stop matching after migration, because Testo inserts a hidden `testo!` diff --git a/docs/technical-overview.md b/docs/technical-overview.md index 3184cff..fab36a5 100644 --- a/docs/technical-overview.md +++ b/docs/technical-overview.md @@ -10,8 +10,8 @@ types. There is no code generation and no separate CLI. This puts a few constraints on your types, checked before any test runs: -- Plugins (and all exported fields of `T` and plugin structs) must be - embedded as pointers. +- Plugins must be embedded as pointers. More generally, every + exported field of `T` and of plugin structs must be a pointer type. - Recursive plugin type references are detected and rejected. Signature and `CasesXxx` mismatches are reported as test errors. @@ -30,25 +30,24 @@ When you call `testo.RunSuite` the following happens: A root test named the same as a suite is run { Suite tests are collected and verified. - Plugins are collected and initialized with ".Plugin(parent, options)" method call, if implemented. Innermost plugins are initialized first. - At this top level, parent is a typed-nil instance of the plugin's own type: the interface is non-nil, but the concrete pointer inside is nil. + Plugins are initialized (innermost first; see note below). "BeforeAll" plugin hooks are called. "BeforeAll" suite hook is called. - "CasesXXX" functions are called and parametrized tests are collected. + "CasesXxx" functions are called and parametrized tests are collected. Test plan from plugins is applied to the final test collection. A test named "testo!" is run { For each test in collection { - Plugins are collected and initialized with ".Plugin(parent: parent, options)" method call, if implemented. Innermost plugins are initialized first. + Plugins are initialized (innermost first). "BeforeEach" plugin hooks are called. "BeforeEach" suite hook is called. Actual test is run { - For each sub-test in test (ran through "testo.Run") { - Plugins are collected and initialized with ".Plugin(parent: parent, options)" method call, if implemented. Innermost plugins are initialized first. + For each sub-test in test (run via "testo.Run") { + Plugins are initialized (innermost first). "BeforeEachSub" plugin hooks are called. @@ -68,13 +67,20 @@ A root test named the same as a suite is run { } ``` +Plugin initialization calls the `.Plugin(parent, options)` method, +if implemented, innermost plugins first. At the top level, `parent` +is a typed-nil instance of the plugin's own type: the interface is +non-nil, but the concrete pointer inside is nil. + ## Panics -Testo **will** catch panics from tests, including `BeforeEach`, `BeforeEachSub`, `AfterEachSub` & `AfterEach` hooks. -Other tests will run even if some tests are panicking. +Unlike plain `go test`, a panicking test does not abort the whole +binary: Testo catches panics from tests and from all hooks. +Other tests keep running even if some panic. -Testo **will** catch panics from `BeforeAll` & `AfterAll`. -Panic in these hooks will result in suite tests not running. +A panic in `BeforeAll` prevents the suite's tests from running. +A panic in `AfterAll` is caught like any other hook panic and fails +the suite. If the suite-level `T` is skipped (for example, `t.Skip` inside `BeforeAll`), the `AfterAll` hooks are skipped too. diff --git a/docs/tutorial.md b/docs/tutorial.md index e42f15f..dbb2e04 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -126,6 +126,8 @@ func TestBake(t *testing.T) { ``` The body of the test did not change - only the type of `t`. +The `HoneyCake` name is gone, though: `RunTest` numbers its tests +instead. Suites (below) give tests proper names again. `testo.T` wraps `testing.T` and keeps its interface, with one exception: sub-tests are started with `testo.Run` instead of `t.Run` @@ -219,8 +221,9 @@ The unused first argument of the `Plugin` method is the parent plugin instance - see the [technical overview](./technical-overview.md#lifecycle) if you need it. -The `Plugin` method is called for each test, and every test gets its -own plugin instance. That is why writing to plugin fields is safe. +The `Plugin` method is called for each test and sub-test, and each +gets its own plugin instance. That is why writing to plugin fields +is safe. To use the plugin, define your own `T` type that embeds `*testo.T` together with the plugins you want: @@ -257,7 +260,7 @@ accepts, then collects and initializes the plugins listed in it: PASS ``` -Note the logged name has no `testo!` in it: `t.Name()` returns the +The logged name has no `testo!` in it: `t.Name()` returns the logical test name ([details](./how-to.md#how-to-run-and-skip-specific-tests)). @@ -292,7 +295,7 @@ func (p *PluginParallel) Plugin(testoplugin.Plugin, ...testoplugin.Option) (spec } ``` -Don't add this one to your file, or it would change the outputs +Don't add this one to your file, or it will change the outputs below. A more complete version is available as the [parallel plugin](https://github.com/ozontech/testo-toppings/tree/main/parallel) in testo-toppings. @@ -401,7 +404,9 @@ func (Bakery) CasesDessert() []string { PASS ``` -The test ran once per dessert. With several parameters, the test runs +The test ran once per dessert. The name `TestBake#01` does not say +which dessert it was, so log `p.Dessert` at the top of the test if +that matters to you. With several parameters, the test runs with the [Cartesian product](https://en.wikipedia.org/wiki/Cartesian_product) of all their values. For correlated values (classic table tests), use a single struct-typed parameter - see diff --git a/examples/README.md b/examples/README.md index ad4ff61..fbf8f56 100644 --- a/examples/README.md +++ b/examples/README.md @@ -14,6 +14,9 @@ Small runnable projects, one per Testo feature. | [07_annotations](./07_annotations/main_test.go) | Attaching static options to tests with `testo.For` & `testo.ForEach`. | | [08_subsuites](./08_subsuites/main_test.go) | Nesting suites with `testo.RunSubSuite`. | +`01_` has two variants on purpose: the same starting point, with and +without a suite. + To run an example, execute the following command in its directory (or run `make`, which does the same): From 9dc6cd2e6356b8b7f25a21528e71069d33b54449 Mon Sep 17 00:00:00 2001 From: metafates Date: Sat, 25 Jul 2026 19:51:34 +0300 Subject: [PATCH 6/8] add plugin docs --- README.md | 2 + docs/README.md | 1 + docs/how-to.md | 58 +++++---- docs/migration.md | 3 - docs/plugins.md | 253 +++++++++++++++++++++++++++++++++++++ docs/technical-overview.md | 8 +- docs/tutorial.md | 14 +- testoplugin/plugin.go | 14 +- 8 files changed, 307 insertions(+), 46 deletions(-) create mode 100644 docs/plugins.md diff --git a/README.md b/README.md index 2631bc9..e1d822d 100644 --- a/README.md +++ b/README.md @@ -92,6 +92,8 @@ Plugins can: - Communicate with other plugins. - Add command line flags for `go test` command. +See [the guide on writing plugins](./docs/plugins.md). + The plugin API (`testoplugin` package) follows the same [SemVer](https://semver.org) compatibility promise as the rest of the module: no breaking changes within a major version. New `Spec` fields may be added in minor releases. diff --git a/docs/README.md b/docs/README.md index 3521776..a7fafe4 100644 --- a/docs/README.md +++ b/docs/README.md @@ -5,6 +5,7 @@ Suggested reading order: - [Tutorial](./tutorial.md) - a guided tour from a plain Go test to a full Testo suite. - [Examples](../examples) - small runnable projects, one per feature. - [How to use Testo features](./how-to.md) - goal-oriented recipes, including the [persistent cache](./how-to.md#how-to-use-persistent-cache). +- [Writing plugins](./plugins.md) - the full plugin API: hooks, overrides, planning, options, flags. - [Migration guide](./migration.md) - moving from testify or allure-go. - [Technical overview](./technical-overview.md) - lifecycle, panics, plugin internals. - [API documentation](https://pkg.go.dev/github.com/ozontech/testo). diff --git a/docs/how-to.md b/docs/how-to.md index 860a7fa..b353f46 100644 --- a/docs/how-to.md +++ b/docs/how-to.md @@ -6,13 +6,13 @@ defined. - [How to write parametrized tests](#how-to-write-parametrized-tests) - [Table tests](#table-tests-correlated-parameters) -- [How to enable strict mode](#how-to-enable-strict-mode) - [How to write parallel tests](#how-to-write-parallel-tests) - [Hooks and parallel sub-tests](#hooks-and-parallel-sub-tests) - [How to use plugin options](#how-to-use-plugin-options) - [How to use persistent cache](#how-to-use-persistent-cache) - [How to structure tests](#how-to-structure-tests) - [How to run and skip specific tests](#how-to-run-and-skip-specific-tests) +- [How to enable strict mode](#how-to-enable-strict-mode) - [How to annotate tests](#how-to-annotate-tests) - [How to integrate with CI](#how-to-integrate-with-ci) - [How to run sub-suites](#how-to-run-sub-suites) @@ -59,8 +59,9 @@ Parameter values do not appear in test names. In `go test -v` output the runs are named `TestFoo`, `TestFoo#01`, `TestFoo#02` and so on, following the standard `go test` convention for repeated names. Log the parameters at the start of the test, so a failing `TestFoo#03` -identifies its case - see the -[parametrized example](../examples/03_parametrized/main_test.go). +identifies its case - or let a +[ten-line plugin](./plugins.md#reading-test-metadata) do it for every +test automatically. If a `CasesXxx` method returns an empty slice, Testo logs a warning (visible with `go test -v`): @@ -101,24 +102,6 @@ func (*Suite) TestParse(t T, p struct{ Case Case }) { The test runs once per element of the slice, with no cross-combination. -## How to enable strict mode - -Strict mode turns every Testo warning into a fatal error. -Today there are two warnings: a suite with no tests, and a -`CasesXxx` method returning an empty slice (shown above): - -```bash -# one package -go test ./path/to/package -testo.strict - -# for ./... use the env var - packages that don't import -# Testo would fail on the unknown flag -TESTO_STRICT=true go test ./... -``` - -> [!TIP] -> Flags have higher priority than environment variables. - ## How to write parallel tests Mark a test as parallel with the standard `t.Parallel`: @@ -330,7 +313,8 @@ import ( "github.com/ozontech/testo" ) -// Global (common) T used by all suites. +// T for this suite. With several suites, +// hoist it into a shared package. type T struct { *testo.T *testcommon.PluginCommon @@ -393,9 +377,10 @@ TestFunc/SuiteName/testo!/TestMethod[/sub-test...] > [!WARNING] > A `-run` pattern without the `testo!` segment, such as > `-run 'Test/MySuite/TestFoo'`, matches **zero tests** - and `go test` -> still reports `PASS`. Suite hooks (`BeforeAll`/`AfterAll`) still run -> even when zero tests match. Most IDE "run test" buttons generate -> exactly this broken pattern. +> still exits 0 (at best you get an easy-to-miss `[no tests to run]` +> note). Suite hooks (`BeforeAll`/`AfterAll`) still run even when zero +> tests match. Most IDE "run test" buttons generate exactly this +> broken pattern. > > Either include the segment (`-run 'Test/MySuite/testo!/TestFoo'`) > or, better, use the `-testo.m` flag below. In VS Code, the @@ -438,6 +423,26 @@ for `testo!/` test events. > patterns) still contains `testo!`. Remember this when you parse test > output or build `-run` patterns. +## How to enable strict mode + +Strict mode turns every Testo warning into a fatal error. +Currently two warnings exist - a suite with no tests, and a +`CasesXxx` method returning an +[empty slice](#how-to-write-parametrized-tests) - and the list +may grow: + +```bash +# one package +go test ./path/to/package -testo.strict + +# for ./... use the env var - packages that don't import +# Testo would fail on the unknown flag +TESTO_STRICT=true go test ./... +``` + +> [!TIP] +> Flags have higher priority than environment variables. + ## How to annotate tests Annotations attach static plugin options to a specific test, so @@ -476,7 +481,8 @@ Plugins see annotations in two places: during planning, via into the `options` argument of the `Plugin` method. In both cases the plugin type-asserts the `Value` field. -See [annotations example](../examples/07_annotations/main_test.go) +See [options in the plugins guide](./plugins.md#options), +the [annotations example](../examples/07_annotations/main_test.go) (its `plugin.go` defines the options) and [API documentation](https://pkg.go.dev/github.com/ozontech/testo#For). diff --git a/docs/migration.md b/docs/migration.md index 3367b4f..a205542 100644 --- a/docs/migration.md +++ b/docs/migration.md @@ -66,9 +66,6 @@ func (*MySuite) TestFoo(t T) { } ``` -The `s.Require().X(...)` and shorthand `s.X(...)` forms both become -plain `require.X(t, ...)` / `assert.X(t, ...)` calls. - Testo only replaces testify's `suite` package. Keep using `require` and `assert`. diff --git a/docs/plugins.md b/docs/plugins.md new file mode 100644 index 0000000..5e46afa --- /dev/null +++ b/docs/plugins.md @@ -0,0 +1,253 @@ +# Writing plugins + +Everything a plugin can do, with a compiling example for each part. +This page assumes you finished the [tutorial](./tutorial.md), where a +first plugin (the timer) is built step by step. + +A plugin is a struct that users embed into their `T`: + +```go +type T struct { + *testo.T + *PluginShout +} +``` + +Testo collects the plugins from the `T` type, creates a fresh +instance of each for every test and sub-test, and asks each one what +it wants to do by calling its `Plugin` method: + +```go +func (p *PluginShout) Plugin(parent testoplugin.Plugin, options ...testoplugin.Option) testoplugin.Spec +``` + +The returned `testoplugin.Spec` has three parts, all optional: + +| Part | What it does | +| :--- | :----------- | +| `Hooks` | run code around suites, tests and sub-tests | +| `Overrides` | wrap built-in `T` methods (`Log`, `Fail`, ...) | +| `Plan` | filter, reorder or duplicate the tests before the run | + +A plugin usually embeds `*testo.T`. Testo fills it with the same `T` +as the current test, so the plugin can log, fail, or inspect the test +it runs in. Pin the interface at compile time: + +```go +var _ testoplugin.Plugin = (*PluginShout)(nil) +``` + +The `parent` argument is the plugin instance of the enclosing scope: +the suite root for a test, the test for a sub-test. Only at the suite +root itself is it a typed-nil pointer, so assert and nil-check the +pointer, not the interface: + +```go +prev, _ := parent.(*PluginShout) +if prev != nil { + // not the suite root: prev belongs to the enclosing test or suite +} +``` + +## Hooks + +Six hooks: the four suite hooks plus `BeforeEachSub`/`AfterEachSub`, +which have no suite-level counterpart: + +```go +spec.Hooks.BeforeEach = testoplugin.Hook{ + Priority: testoplugin.TryFirst, + Func: func() { + p.Logf("starting %s", p.Name()) + }, +} +``` + +`BeforeAll`/`AfterAll` run once per suite, `BeforeEach`/`AfterEach` +around each test, `BeforeEachSub`/`AfterEachSub` around each +sub-test. `Priority` is an `int` that orders hooks across plugins: +lower values run earlier, any value works, `TryFirst` and `TryLast` +are the extremes, zero keeps declaration order. + +`AfterEach` and `AfterEachSub` are deferred, so with parallel +sub-tests they run before those sub-tests finish - same caveat as +[suite hooks](./how-to.md#hooks-and-parallel-sub-tests). + +## Overrides + +Overrides wrap built-in `T` methods in middleware style: your +function receives the next implementation and returns a replacement. + +```go +spec.Overrides.Log = func(next testoplugin.FuncLog) testoplugin.FuncLog { + return func(args ...any) { + p.Helper() // keep the caller's file:line in the output + next(append([]any{"[quiet]"}, args...)...) + } +} +``` + +Call `t.Helper()` inside the wrapper, or every log line will point +at the wrapper's own file and line. + +Overridable methods: `Log`, `Error`, `Fatal`, `Fail`, `FailNow`, +`Failed`, `Skip`, `SkipNow`, `Skipped`, `Parallel`, `Cleanup`, +`Context`, `Deadline`, `TempDir`, `Setenv`, `Chdir`. +Each method `X` has a matching `testoplugin.FuncX` signature type. + +Methods call each other underneath - `Error` is `Log` + `Fail`, +`Fatal` is `Log` + `FailNow` - so overriding `Log` also affects +`Error` and `Fatal`. `Overrides.Priority` orders stacks across +plugins: the lowest priority becomes the outermost wrapper, so its +code runs first. For a real example, the +[xfail plugin](https://github.com/ozontech/testo-toppings/tree/main/xfail) +overrides `Fail` and `FailNow` to turn expected failures into skips. + +## Planning tests + +`Plan.Prepare` receives the collected tests before the run and may +filter, reorder or duplicate them in place: + +```go +import "github.com/ozontech/testo/testoreflect" + +spec.Plan.Prepare = func(suite testoreflect.SuiteInfo, tests *[]testoplugin.PlannedTest) { + slices.Reverse(*tests) +} +``` + +Each `PlannedTest` exposes `Annotations()` - the options attached to +that test with [`testo.For`](./how-to.md#how-to-annotate-tests) - and +`Info()`, which returns a `testoreflect.TestInfo` interface: + +```go +name := t.Info().GetName() +``` + +`GetName()` returns the full run path without the `testo!` segment, +e.g. `Test/Suite/TestFoo#01` - match with `strings.HasSuffix`, not `==`. + +Parametrized tests arrive already expanded, one `PlannedTest` per +case. The initial order is regular tests first (alphabetical), then +parametrized cases. One display quirk: after reordering, `go test` +reassigns the `#01` suffixes by run order, while `t.Name()` keeps the +original case index. + +The [rerun plugin](https://github.com/ozontech/testo-toppings/tree/main/rerun) +uses `Prepare` to drop every test that passed in the previous run. + +## Adding methods to T + +Any method on the plugin becomes a method on `T` through embedding. +If methods are all your plugin does, you don't even need a `Plugin` +method: + +```go +type PluginGreet struct{ *testo.T } + +func (g *PluginGreet) Greet() { g.Log("hello") } +``` + +```go +func (Suite) TestFoo(t T) { + t.Greet() +} +``` + +## Options + +An option is a `testoplugin.Option` wrapping any value your plugin +recognizes. Users pass options to `testo.RunSuite`, `testo.Run` or +`testo.Options` - see +[how to use plugin options](./how-to.md#how-to-use-plugin-options). +The common pattern is an unexported function type: + +```go +type shoutOption func(*PluginShout) + +func WithPrefix(prefix string) testoplugin.Option { + return testoplugin.Option{ + Value: shoutOption(func(p *PluginShout) { p.prefix = prefix }), + Propagate: true, // pass to sub-tests too + } +} +``` + +Consume options at the top of the `Plugin` method. All user-supplied +options come through the variadic argument (including per-test +[annotations](./how-to.md#how-to-annotate-tests)); ignore values that +are not yours: + +```go +func (p *PluginShout) Plugin(_ testoplugin.Plugin, options ...testoplugin.Option) (spec testoplugin.Spec) { + for _, o := range options { + if o, ok := o.Value.(shoutOption); ok { + o(p) + } + } + // ... + return spec +} +``` + +## Command line flags + +Plugins register flags with the standard `flag` package at package +level. The `go test` machinery parses them like any other test flag: + +```go +var flagShout = flag.Bool("shout.enabled", false, "print a banner before each test") +``` + +```bash +go test . -shout.enabled +``` + +Read the value inside `Plugin` or a hook, after flags are parsed. +Prefix flag names with your plugin name to avoid collisions. Real +examples: `-rerun.failed` in the rerun plugin, `-allure.dir` in +testo-allure. + +Flags only exist in packages that import the plugin, so `go test +./... -shout.enabled` fails in packages that don't. Pair each flag +with an environment variable if your plugin must be configurable +across a whole repo (Testo does this with `TESTO_STRICT` and +`TESTO_CACHE_DIR`). + +## Reading test metadata + +`testo.Reflect` works on any `T` - including the plugin itself, via +its embedded `*testo.T`: + +```go +spec.Hooks.BeforeEach.Func = func() { + info, ok := testo.Reflect(p).Test.(testoreflect.ParametrizedTestInfo) + if ok { + p.Logf("params: %v", info.Params) + } +} +``` + +That snippet answers "which parameters did `TestFoo#01` run with": +the plugin logs them for every parametrized test. The `Reflection` struct also carries the suite info, failure +kind and source, and panic details - which is how +[testo-allure](https://github.com/ozontech/testo-allure) fills its +reports. See the +[testoreflect reference](https://pkg.go.dev/github.com/ozontech/testo/testoreflect). + +## Talking to other plugins + +Plugins reference each other by embedding; Testo reuses one instance +per test across all references. See +[cross-plugin communication](./technical-overview.md#plugins). + +## Plugins worth reading + +- [examples/04_plugins](../examples/04_plugins/main_test.go) - hooks, + overrides and planning in one small file. +- [examples/07_annotations](../examples/07_annotations/plugin.go) - a + plugin that defines options and consumes annotations in `Plan`. +- [testo-toppings](https://github.com/ozontech/testo-toppings) - four + small production plugins (rerun, xfail, parallel, async). +- [testo-allure](https://github.com/ozontech/testo-allure) - a large + reporting plugin using every part of the `Spec`. diff --git a/docs/technical-overview.md b/docs/technical-overview.md index fab36a5..eece09f 100644 --- a/docs/technical-overview.md +++ b/docs/technical-overview.md @@ -68,9 +68,11 @@ A root test named the same as a suite is run { ``` Plugin initialization calls the `.Plugin(parent, options)` method, -if implemented, innermost plugins first. At the top level, `parent` -is a typed-nil instance of the plugin's own type: the interface is -non-nil, but the concrete pointer inside is nil. +if implemented, innermost plugins first: a plugin embedded by another +plugin initializes before the plugin embedding it. At the suite root, +`parent` is a typed-nil instance of the plugin's own type: the +interface is non-nil, but the concrete pointer inside is nil. Below +the root, `parent` is the plugin instance of the enclosing scope. ## Panics diff --git a/docs/tutorial.md b/docs/tutorial.md index dbb2e04..3e4677f 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -152,8 +152,8 @@ PASS Two technical levels appeared in the names: - `#00` is the index of the test inside this `RunTest` call. -- `testo!` is a special test Testo inserts to guarantee correct work - of hooks with parallel tests. It does not affect your tests, but it +- `testo!` is a special test Testo inserts so hooks run correctly + around parallel tests. It does not affect your tests, but it does affect `-run` patterns - see [how to run specific tests](./how-to.md#how-to-run-and-skip-specific-tests). @@ -260,8 +260,8 @@ accepts, then collects and initializes the plugins listed in it: PASS ``` -The logged name has no `testo!` in it: `t.Name()` returns the -logical test name +Your timings will differ. The logged name has no `testo!` in it: +`t.Name()` returns the logical test name ([details](./how-to.md#how-to-run-and-skip-specific-tests)). > [!NOTE] @@ -271,10 +271,8 @@ logical test name Plugins can do much more than hooks: reorder or filter the test plan, wrap built-in methods like `t.Log`, add new methods to `T`, accept [options](./how-to.md#how-to-use-plugin-options) and command line -flags. The [plugins example](../examples/04_plugins/main_test.go) -shows `Plan` and `Overrides` in action, and the -[technical overview](./technical-overview.md#plugins) explains how -plugins talk to each other. +flags. The [writing plugins guide](./plugins.md) covers the full +API with an example for each part. Ready-made plugins: [testo-allure](https://github.com/ozontech/testo-allure) for Allure diff --git a/testoplugin/plugin.go b/testoplugin/plugin.go index c9b15b0..552d09c 100644 --- a/testoplugin/plugin.go +++ b/testoplugin/plugin.go @@ -5,15 +5,17 @@ // Plugins can implement [Plugin] interface to be registered as such. // // Method "Plugin" will be called for each plugin before running a suite. -// For sub-tests, parent refers to the plugin instance of the parent test. -// For top-level tests, parent is a typed-nil instance of the plugin's own -// type: the interface itself is non-nil, but the concrete pointer is nil, -// so an unconditional type assertion like parent.(*PluginFoo) is safe. -// To detect the top level, nil-check the asserted pointer, not the interface: +// The parent is the plugin instance of the enclosing scope: +// the suite root for a test, the test for a sub-test. +// Only at the suite root itself is parent a typed-nil instance of the +// plugin's own type: the interface itself is non-nil, but the concrete +// pointer is nil, so an unconditional type assertion like +// parent.(*PluginFoo) is safe. +// To detect the suite root, nil-check the asserted pointer, not the interface: // // prev, _ := parent.(*PluginFoo) // if prev != nil { -// // sub-test: use prev +// // not the suite root: prev belongs to the enclosing test or suite // } // // It is encouraged to ensure a plugin implements [Plugin] interface with the following line: From 4649c490defa8e3fc3208fa82d003cb18830acca Mon Sep 17 00:00:00 2001 From: metafates Date: Sat, 25 Jul 2026 20:09:28 +0300 Subject: [PATCH 7/8] better plugin docs --- README.md | 32 +++++++++++++++++++++++++++----- docs/how-to.md | 28 ++++++++++++---------------- docs/plugins.md | 38 +++++++++++++++++++++++++------------- docs/tutorial.md | 31 +++++++++++++++++++++++++++++-- 4 files changed, 93 insertions(+), 36 deletions(-) diff --git a/README.md b/README.md index e1d822d..f2940fa 100644 --- a/README.md +++ b/README.md @@ -18,11 +18,11 @@ a collection of small optional plugins for Testo. ## Features -- [Plugins](./examples/04_plugins/main_test.go) - hook, filter and extend `T` without forking the framework. -- [Parametrized tests](./examples/03_parametrized/main_test.go) - describe a test once, repeat it with different parameters. +- [Plugins](./docs/plugins.md) - hook, filter and extend `T` without forking the framework ([example](./examples/04_plugins/main_test.go)). +- [Parametrized tests](./docs/how-to.md#how-to-write-parametrized-tests) - describe a test once, repeat it with different parameters ([example](./examples/03_parametrized/main_test.go)). - [Parallel tests](./examples/05_parallel/main_test.go) - run independent tests concurrently. -- [Lifecycle hooks](./examples/02_hooks/main_test.go) - before and after any suite, test & sub-test ([hooks and parallel tests](./docs/how-to.md#how-to-write-parallel-tests)). -- [Test annotations](./examples/07_annotations/main_test.go) - attach static options to any test. +- [Lifecycle hooks](./docs/tutorial.md#suite-hooks) - before and after any suite, test & sub-test ([example](./examples/02_hooks/main_test.go), [parallel caveat](./docs/how-to.md#hooks-and-parallel-sub-tests)). +- [Test annotations](./docs/how-to.md#how-to-annotate-tests) - attach static options to any test ([example](./examples/07_annotations/main_test.go)). - [Test filtering](./docs/how-to.md#how-to-run-and-skip-specific-tests) - `-run`, `-testo.m`, and the one trap to know before writing CI filters. - [Informative errors and traces](./examples/06_errors/main_test.go) - error messages name the exact method and type that caused them. - [Sub-tests & sub-suites](./examples/08_subsuites/main_test.go) - support for nested tests and nested suites. @@ -69,7 +69,29 @@ And run it with `go test` as usual: go test . ``` -Testo also supports suites, parametrized tests & plugins. +The part Testo exists for is suites with parametrized tests. +In the same file: + +```go +type Suite struct{ testo.Suite[*testo.T] } + +func (Suite) CasesWord() []string { + return []string{"dough", "bread"} +} + +func (Suite) TestLen(t *testo.T, p struct{ Word string }) { + if len(p.Word) == 0 { + t.Error("word must not be empty") + } +} + +func TestSuite(t *testing.T) { + testo.RunSuite(t, new(Suite)) +} +``` + +`TestLen` runs once per word from `CasesWord`. Plugins install the +same way - as fields of a struct. ### Next steps diff --git a/docs/how-to.md b/docs/how-to.md index b353f46..9aa277c 100644 --- a/docs/how-to.md +++ b/docs/how-to.md @@ -4,19 +4,6 @@ Snippets on this page are fragments: they assume a project set up as in the [tutorial](./tutorial.md), with a suite and a `T` type already defined. -- [How to write parametrized tests](#how-to-write-parametrized-tests) - - [Table tests](#table-tests-correlated-parameters) -- [How to write parallel tests](#how-to-write-parallel-tests) - - [Hooks and parallel sub-tests](#hooks-and-parallel-sub-tests) -- [How to use plugin options](#how-to-use-plugin-options) -- [How to use persistent cache](#how-to-use-persistent-cache) -- [How to structure tests](#how-to-structure-tests) -- [How to run and skip specific tests](#how-to-run-and-skip-specific-tests) -- [How to enable strict mode](#how-to-enable-strict-mode) -- [How to annotate tests](#how-to-annotate-tests) -- [How to integrate with CI](#how-to-integrate-with-ci) -- [How to run sub-suites](#how-to-run-sub-suites) - ## How to write parametrized tests Parametrized tests are defined as regular tests with a second argument: @@ -227,7 +214,6 @@ directory: ```bash # change the directory for one package -# (cache flags use the -cache. prefix, not -testo.) go test ./path/to/package -cache.dir /tmp/my-testo-cache # for ./... use the env var, so packages that don't import @@ -238,6 +224,10 @@ TESTO_CACHE_DIR=/tmp/my-testo-cache go test ./... TESTO_CACHE_DISABLE=true go test ./... ``` +> [!NOTE] +> Cache flags use the `-cache.` prefix, not `-testo.` - +> there is no `-testo.cache.dir`. + For plugin state, prefer a namespace. It is isolated from other namespaces and from the package-level functions, and has the same `Get`, `Set`, `Keys` and `Remove` methods: @@ -414,8 +404,14 @@ selects the method inside it. If `-testo.m` matches no tests, the suite runs with zero tests (hooks still run) and `go test` shows `PASS`. Add a CI check that -at least one test ran - for example, grep `go test -json` output -for `testo!/` test events. +at least one test ran: + +```bash +go test -json ./... | grep -q '"Test":".*testo!/' || { echo "no tests ran"; exit 1; } +``` + +Unlike `-testo.strict` and the cache flags, `-testo.m` has no +env-var form - pass it only to packages that import Testo. > [!NOTE] > `t.Name()` returns the name without `testo!`, e.g. `Test/MySuite/TestFoo`. diff --git a/docs/plugins.md b/docs/plugins.md index 5e46afa..f0ee011 100644 --- a/docs/plugins.md +++ b/docs/plugins.md @@ -39,8 +39,9 @@ var _ testoplugin.Plugin = (*PluginShout)(nil) The `parent` argument is the plugin instance of the enclosing scope: the suite root for a test, the test for a sub-test. Only at the suite -root itself is it a typed-nil pointer, so assert and nil-check the -pointer, not the interface: +root itself is it a typed-nil pointer. A non-nil interface can hold a +nil pointer, so `parent != nil` would lie - assert and nil-check the +pointer instead: ```go prev, _ := parent.(*PluginShout) @@ -49,10 +50,17 @@ if prev != nil { } ``` +Snippets in the Hooks, Overrides and Planning sections live inside +a `Plugin` method with a named return, like the tutorial's timer: +Go creates `spec` empty, the snippets fill its fields. + ## Hooks -Six hooks: the four suite hooks plus `BeforeEachSub`/`AfterEachSub`, -which have no suite-level counterpart: +`BeforeAll`/`AfterAll` run once per suite, `BeforeEach`/`AfterEach` +around each test, `BeforeEachSub`/`AfterEachSub` around each +sub-test. The first four mirror the suite hook methods but are a +separate mechanism - both run, side by side. The `Sub` pair exists +only for plugins. ```go spec.Hooks.BeforeEach = testoplugin.Hook{ @@ -63,11 +71,10 @@ spec.Hooks.BeforeEach = testoplugin.Hook{ } ``` -`BeforeAll`/`AfterAll` run once per suite, `BeforeEach`/`AfterEach` -around each test, `BeforeEachSub`/`AfterEachSub` around each -sub-test. `Priority` is an `int` that orders hooks across plugins: -lower values run earlier, any value works, `TryFirst` and `TryLast` -are the extremes, zero keeps declaration order. +`Priority` is an `int` that orders hooks across plugins: lower +values run earlier, any value works, `TryFirst` and `TryLast` are +the extremes, zero keeps declaration order. Logs from hooks point at +the plugin's own file, which is expected. `AfterEach` and `AfterEachSub` are deferred, so with parallel sub-tests they run before those sub-tests finish - same caveat as @@ -75,8 +82,11 @@ sub-tests they run before those sub-tests finish - same caveat as ## Overrides -Overrides wrap built-in `T` methods in middleware style: your -function receives the next implementation and returns a replacement. +Overrides wrap built-in `T` methods. Testo hands your function the +current implementation (`next`) and installs whatever you return in +its place. When the test calls `t.Log`, your replacement runs and +decides if and when to call `next`. With several plugins the +replacements nest, like HTTP middleware. ```go spec.Overrides.Log = func(next testoplugin.FuncLog) testoplugin.FuncLog { @@ -167,12 +177,14 @@ type shoutOption func(*PluginShout) func WithPrefix(prefix string) testoplugin.Option { return testoplugin.Option{ - Value: shoutOption(func(p *PluginShout) { p.prefix = prefix }), - Propagate: true, // pass to sub-tests too + Value: shoutOption(func(p *PluginShout) { p.prefix = prefix }), } } ``` +Set `Propagate: true` only when sub-tests need the option too; +leave it unset otherwise. + Consume options at the top of the `Plugin` method. All user-supplied options come through the variadic argument (including per-test [annotations](./how-to.md#how-to-annotate-tests)); ignore values that diff --git a/docs/tutorial.md b/docs/tutorial.md index 3e4677f..60d0145 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -126,6 +126,8 @@ func TestBake(t *testing.T) { ``` The body of the test did not change - only the type of `t`. +This swap lets everything below (plugins, suites, parametrization) +attach to a plain test. The `HoneyCake` name is gone, though: `RunTest` numbers its tests instead. Suites (below) give tests proper names again. @@ -215,6 +217,11 @@ func (p *PluginTimer) Plugin(testoplugin.Plugin, ...testoplugin.Option) (spec te } ``` +Two Go details in the signature above. The parameters are unnamed +because this plugin ignores them. And `(spec testoplugin.Spec)` is a +named return: Go creates `spec` as an empty value, the method fills +its fields and returns it. + The plugin embeds `*testo.T`. Testo fills it with the same `T` as the current test, so the plugin sees everything the test sees. The unused first argument of the `Plugin` method is the parent plugin @@ -235,7 +242,11 @@ type T struct { } ``` -And change the test function to take this `T`: +(Both fields contain a `*testo.T`, but the direct one is shallower, +so calls like `t.Log` stay unambiguous.) + +And change the test function to take this `T` - no `*` this time, +since our `T` is a struct that already contains the pointers: ```go func TestBake(t *testing.T) { @@ -447,6 +458,9 @@ func (Bakery) AfterAll(t T) { } ``` +The run output looks the same as before: these hooks are silent. +Add a `t.Log` inside one if you want to see it fire. + If `BeforeAll` fails, the suite's tests do not run at all. For hook behavior with parallel sub-tests and panics, see [how to write parallel tests](./how-to.md#how-to-write-parallel-tests) @@ -475,7 +489,9 @@ func (t T) Bake(name string) (Pastry, bool) { } ``` -The test no longer deals with cleanup: +The `Bake(name)` inside the method is the package function from +`main.go` - methods don't shadow package names, so this is not +recursion. The test no longer deals with cleanup: ```go func (Bakery) TestBake(t T, p struct{ Dessert string }) { @@ -570,6 +586,17 @@ func TestFoo(t *testing.T) { } ``` +Each `testo.Test` call gets its own technical levels, so the names +run deep - that is normal: + +```txt +=== RUN TestFoo/FirstTest +=== RUN TestFoo/FirstTest/#00 +=== RUN TestFoo/FirstTest/#00/testo! +=== RUN TestFoo/FirstTest/#00/testo!/FirstTest +... +``` + ## The Finished Test File The complete `main_test.go` we built in this tutorial: From cc9b84ced0679496a95df88488262dea0db6ede0 Mon Sep 17 00:00:00 2001 From: metafates Date: Sat, 25 Jul 2026 20:34:21 +0300 Subject: [PATCH 8/8] remove not needed docs --- COMPARISON.md | 8 ++-- README.md | 12 ++---- docs/README.md | 2 +- docs/how-to.md | 22 +++------- docs/migration.md | 106 ++++++++++++++++++++-------------------------- docs/plugins.md | 11 +++-- docs/tutorial.md | 1 + 7 files changed, 66 insertions(+), 96 deletions(-) diff --git a/COMPARISON.md b/COMPARISON.md index d416719..24aa7ed 100644 --- a/COMPARISON.md +++ b/COMPARISON.md @@ -19,15 +19,15 @@ Testo focuses on extensibility through plugins and stays a thin layer over usual Other frameworks may be a better fit if you need BDD scenarios or some unique features "out of the box". Testo can support BDD-style tests through plugins. -## Why we moved off allure-go +## Why we moved off [Allure-Go] -Testo was created at Ozon as the successor to [allure-go][Allure-Go]. -In allure-go, the test framework and Allure report generation are one +Testo was created at Ozon as the successor to [Allure-Go]. +In Allure-Go, the test framework and Allure report generation are one piece: you can't swap the reporting or extend the framework. Testo splits them. The framework is a dependency-free layer over `testing.T`, and Allure reporting is [one plugin among many](https://github.com/ozontech/testo-allure). -If you are on allure-go today, see the +If you are on Allure-Go today, see the [migration guide](./docs/migration.md#migrating-from-allure-go). [^1]: Ginkgo runs parallel tests in [separate processes](https://onsi.github.io/ginkgo/#mental-model-how-ginkgo-runs-parallel-specs) with its own runner (not available through `go test`). It costs more startup time and memory than goroutine-based `go test` parallelization, but you get process isolation in return. diff --git a/README.md b/README.md index f2940fa..8f16f91 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ a collection of small optional plugins for Testo. - [Parallel tests](./examples/05_parallel/main_test.go) - run independent tests concurrently. - [Lifecycle hooks](./docs/tutorial.md#suite-hooks) - before and after any suite, test & sub-test ([example](./examples/02_hooks/main_test.go), [parallel caveat](./docs/how-to.md#hooks-and-parallel-sub-tests)). - [Test annotations](./docs/how-to.md#how-to-annotate-tests) - attach static options to any test ([example](./examples/07_annotations/main_test.go)). -- [Test filtering](./docs/how-to.md#how-to-run-and-skip-specific-tests) - `-run`, `-testo.m`, and the one trap to know before writing CI filters. +- [Test filtering](./docs/how-to.md#how-to-run-and-skip-specific-tests) - `-run`, `-testo.m`. - [Informative errors and traces](./examples/06_errors/main_test.go) - error messages name the exact method and type that caused them. - [Sub-tests & sub-suites](./examples/08_subsuites/main_test.go) - support for nested tests and nested suites. - [Test reflection](https://pkg.go.dev/github.com/ozontech/testo/testoreflect) - deeply inspect test's meta-information. @@ -69,8 +69,7 @@ And run it with `go test` as usual: go test . ``` -The part Testo exists for is suites with parametrized tests. -In the same file: +Testo can do more. For example, run parametrized tests in suite: ```go type Suite struct{ testo.Suite[*testo.T] } @@ -90,8 +89,7 @@ func TestSuite(t *testing.T) { } ``` -`TestLen` runs once per word from `CasesWord`. Plugins install the -same way - as fields of a struct. +`TestLen` runs once per word from `CasesWord`. ### Next steps @@ -116,10 +114,6 @@ Plugins can: See [the guide on writing plugins](./docs/plugins.md). -The plugin API (`testoplugin` package) follows the same -[SemVer](https://semver.org) compatibility promise as the rest of the module: -no breaking changes within a major version. New `Spec` fields may be added in minor releases. - Examples: - [Testo Allure Plugin](https://github.com/ozontech/testo-allure) - enhance your tests with automatically generated [Allure Reports](https://allurereport.org/). diff --git a/docs/README.md b/docs/README.md index a7fafe4..0034790 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,7 +4,7 @@ Suggested reading order: - [Tutorial](./tutorial.md) - a guided tour from a plain Go test to a full Testo suite. - [Examples](../examples) - small runnable projects, one per feature. -- [How to use Testo features](./how-to.md) - goal-oriented recipes, including the [persistent cache](./how-to.md#how-to-use-persistent-cache). +- [How to use Testo features](./how-to.md) - goal-oriented recipes. - [Writing plugins](./plugins.md) - the full plugin API: hooks, overrides, planning, options, flags. - [Migration guide](./migration.md) - moving from testify or allure-go. - [Technical overview](./technical-overview.md) - lifecycle, panics, plugin internals. diff --git a/docs/how-to.md b/docs/how-to.md index 9aa277c..a44ef20 100644 --- a/docs/how-to.md +++ b/docs/how-to.md @@ -369,8 +369,7 @@ TestFunc/SuiteName/testo!/TestMethod[/sub-test...] > `-run 'Test/MySuite/TestFoo'`, matches **zero tests** - and `go test` > still exits 0 (at best you get an easy-to-miss `[no tests to run]` > note). Suite hooks (`BeforeAll`/`AfterAll`) still run even when zero -> tests match. Most IDE "run test" buttons generate exactly this -> broken pattern. +> tests match. > > Either include the segment (`-run 'Test/MySuite/testo!/TestFoo'`) > or, better, use the `-testo.m` flag below. In VS Code, the @@ -391,6 +390,10 @@ func (MySuite) TestFoo(t T) { func (MySuite) TestBar(t T) { // ... } + +func Test(t *testing.T) { + testo.RunSuite(t, new(MySuite)) +} ``` To run only `TestFoo`: @@ -402,17 +405,6 @@ go test . -run 'Test/MySuite' -testo.m TestFoo Here `-run 'Test/MySuite'` selects the suite and `-testo.m TestFoo` selects the method inside it. -If `-testo.m` matches no tests, the suite runs with zero tests -(hooks still run) and `go test` shows `PASS`. Add a CI check that -at least one test ran: - -```bash -go test -json ./... | grep -q '"Test":".*testo!/' || { echo "no tests ran"; exit 1; } -``` - -Unlike `-testo.strict` and the cache flags, `-testo.m` has no -env-var form - pass it only to packages that import Testo. - > [!NOTE] > `t.Name()` returns the name without `testo!`, e.g. `Test/MySuite/TestFoo`. > The real `testing.T` name (the one in `go test -v` output and in `-run` @@ -490,9 +482,7 @@ work unchanged. Notes: -- Before writing CI test filters, read - [how to run and skip specific tests](#how-to-run-and-skip-specific-tests) - - a `-run` pattern without `testo!` silently runs zero tests. +- Read [how to run and skip specific tests](#how-to-run-and-skip-specific-tests). - The hidden `testo!` node appears in reports as an extra nesting level (e.g. JUnit converters render it as an empty intermediate node). - For Allure reports, use the [testo-allure plugin](https://github.com/ozontech/testo-allure). diff --git a/docs/migration.md b/docs/migration.md index a205542..ed12cbd 100644 --- a/docs/migration.md +++ b/docs/migration.md @@ -8,32 +8,32 @@ Both migrations can be done incrementally: Testo tests are regular `go test` tests, so old and new suites run side by side in the same package with no interference. Migrate one suite at a time. -## Migrating from testify +## Migrating from Testify ### Suites and hooks -| testify | Testo | -| :------ | :---- | -| `suite.Suite` (embedded) | `testo.Suite[T]` (embedded) | -| `suite.Run(t, new(S))` | `testo.RunSuite(t, new(S))` | -| `func (s *S) SetupSuite()` | `func (*S) BeforeAll(t T)` | -| `func (s *S) SetupTest()` | `func (*S) BeforeEach(t T)` | -| `func (s *S) TearDownTest()` | `func (*S) AfterEach(t T)` | -| `func (s *S) TearDownSuite()` | `func (*S) AfterAll(t T)` | -| `func (s *S) TestFoo()` | `func (*S) TestFoo(t T)` | -| `s.T()` | `t` (passed to every method) | -| `s.Run(name, func())` | `testo.Run(t, name, func(t T))` | -| `func (s *S) BeforeTest(suiteName, testName string)` | `BeforeEach(t T)` + `t.Name()` (same for `AfterTest`) | -| `SetupSubTest()` / `TearDownSubTest()` | `BeforeEachSub`/`AfterEachSub` plugin hooks (no suite-level equivalent) | -| `go test -run TestSuite -testify.m TestFoo` | `go test -run TestSuite -testo.m TestFoo` | - -Two notes on the hook mapping. `t.Name()` returns the full path -(`Test/MySuite/TestFoo`), not the bare method name - compare with -`strings.HasSuffix`, not `==`. And if a suite uses both `SetupTest` -and `BeforeTest`, merge them into one `BeforeEach` (testify runs -`SetupTest` first; keep that order). - -One flag caveat: testify method filters like `-run 'TestMySuite/TestFoo'` +| Testify | Testo | +| :--------------------------------------------------- | :---------------------------------------------------------------------- | +| `suite.Suite` (embedded) | `testo.Suite[T]` (embedded) | +| `suite.Run(t, new(S))` | `testo.RunSuite(t, new(S))` | +| `func (s *S) SetupSuite()` | `func (*S) BeforeAll(t T)` | +| `func (s *S) SetupTest()` | `func (*S) BeforeEach(t T)` | +| `func (s *S) TearDownTest()` | `func (*S) AfterEach(t T)` | +| `func (s *S) TearDownSuite()` | `func (*S) AfterAll(t T)` | +| `func (s *S) TestFoo()` | `func (*S) TestFoo(t T)` | +| `s.T()` | `t` (passed to every method) | +| `s.Run(name, func())` | `testo.Run(t, name, func(t T))` | +| `func (s *S) BeforeTest(suiteName, testName string)` | `BeforeEach(t T)` + `t.Name()` (same for `AfterTest`) | +| `SetupSubTest()` / `TearDownSubTest()` | `BeforeEachSub`/`AfterEachSub` plugin hooks (no suite-level equivalent) | +| `go test -run TestSuite -testify.m TestFoo` | `go test -run TestSuite -testo.m TestFoo` | + +> [!NOTE] +> `t.Name()` returns the full path (`Test/MySuite/TestFoo`), +> not the bare method name - compare with `strings.HasSuffix`, not `==`. +> And if a suite uses both `SetupTest` and `BeforeTest`, +> merge them into one `BeforeEach` (testify runs `SetupTest` first; keep that order). + +One caveat: testify method filters like `-run 'TestMySuite/TestFoo'` stop matching after migration, because Testo inserts a hidden `testo!` level into test names. Update CI filters to use `-testo.m`. See [how to run and skip specific tests](./how-to.md#how-to-run-and-skip-specific-tests). @@ -71,7 +71,7 @@ and `assert`. ### Suite state -testify suites keep per-test state in suite struct fields. +Testify suites keep per-test state in suite struct fields. A Testo suite is also a single instance, so shared fields still work. But writes to shared fields from parallel tests are a data race - run migrated packages with `-race` before enabling `t.Parallel()`. @@ -81,15 +81,13 @@ for each test and are safe with parallel tests. ### Parallelism -testify's `suite` does not support `t.Parallel()` -([issue #934](https://github.com/stretchr/testify/issues/934)). -With Testo, call `t.Parallel()` in any test - hooks handle parallel -tests correctly. See -[how to write parallel tests](./how-to.md#how-to-write-parallel-tests). +Testify's `suite` does not support `t.Parallel()` ([issue #934](https://github.com/stretchr/testify/issues/934)). +With Testo, call `t.Parallel()` in any test - hooks handle parallel tests correctly. +See [how to write parallel tests](./how-to.md#how-to-write-parallel-tests). -## Migrating from allure-go +## Migrating from Allure-Go -Testo [replaces allure-go at Ozon](../COMPARISON.md#why-we-moved-off-allure-go). +Testo [replaces Allure-Go at Ozon](../COMPARISON.md#why-we-moved-off-allure-go). Reporting is now a separate plugin, not part of the framework. So you need **two** pieces: `github.com/ozontech/testo` and the [testo-allure](https://github.com/ozontech/testo-allure) plugin. @@ -109,31 +107,19 @@ type T struct { ### Concept mapping -| allure-go | Testo (+ testo-allure) | -| :-------- | :--------------------- | -| `suite.Suite` | `testo.Suite[T]` | -| `suite.RunSuite(t, new(S))` (or `runner.NewSuiteRunner(...)`) | `testo.RunSuite(t, new(S))` | -| `BeforeAll(t provider.T)` | `BeforeAll(t T)` (same for the other hooks) | -| `func (s *S) TestFoo(t provider.T)` | `func (*S) TestFoo(t T)` | -| `t.WithNewStep("name", func(ctx provider.StepCtx) {...})` | `allure.Step(t, "name", func(t T) {...})` | -| `t.NewStep("name")` | `allure.Step(t, "name", func(T) {})` | -| `t.Title`, `t.Epic`, `t.Feature`, `t.Story`, `t.Tags`, `t.Severity`, `t.Owner`, `t.ID` | same-named methods added to `T` by `PluginAllure` | -| `t.Link(...)`, `t.TmsLink(...)`, `t.TmsLinks(...)` | `t.Links(...)` with the `allure.NewLink`/`allure.TMS`/`allure.Issue` constructors | -| `t.WithNewAttachment(name, mimeType, content)`, `t.WithAttachments(...)` | `t.Attach(name, allure.Bytes(...))` or `t.Attach(name, allure.File(...))` | -| table tests (`ParametrizedSuite` / `TableTestXxx` methods) | single struct-typed parameter, see [table tests](./how-to.md#table-tests-correlated-parameters) | -| `t.XSkip()` | `t.XFail()` via the [xfail plugin](https://github.com/ozontech/testo-toppings/tree/main/xfail) from testo-toppings (semantics differ slightly - check its README) | - -Other labels (`Description`, `Labels`, `Layer`, `Lead`, `Stage`, -`WithParameters`) map to the `Description`, `Labels` and `Parameters` -methods of `PluginAllure`. - -A minimal setup to start a proof of concept: - -```go -func Test(t *testing.T) { - testo.RunSuite(t, new(Bakery), allure.WithOutputDir("allure-results")) -} -``` +| allure-go | Testo (+ testo-allure) | +| :------------------------------------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `suite.Suite` | `testo.Suite[T]` | +| `suite.RunSuite(t, new(S))` (or `runner.NewSuiteRunner(...)`) | `testo.RunSuite(t, new(S))` | +| `BeforeAll(t provider.T)` | `BeforeAll(t T)` (same for the other hooks) | +| `func (s *S) TestFoo(t provider.T)` | `func (*S) TestFoo(t T)` | +| `t.WithNewStep("name", func(ctx provider.StepCtx) {...})` | `allure.Step(t, "name", func(t T) {...})` | +| `t.NewStep("name")` | `allure.Step(t, "name", func(T) {})` | +| `t.Title`, `t.Epic`, `t.Feature`, `t.Story`, `t.Tags`, `t.Severity`, `t.Owner`, `t.ID` | same-named methods added to `T` by `PluginAllure` | +| `t.Link(...)`, `t.TmsLink(...)`, `t.TmsLinks(...)` | `t.Links(...)` with the `allure.NewLink`/`allure.TMS`/`allure.Issue` constructors | +| `t.WithNewAttachment(name, mimeType, content)`, `t.WithAttachments(...)` | `t.Attach(name, allure.Bytes(...))` or `t.Attach(name, allure.File(...))` | +| table tests (`ParametrizedSuite` / `TableTestXxx` methods) | single struct-typed parameter, see [table tests](./how-to.md#table-tests-correlated-parameters) | +| `t.XSkip()` | `t.XFail()` via the [xfail plugin](https://github.com/ozontech/testo-toppings/tree/main/xfail) from testo-toppings (semantics differ slightly - check its README) | ### Steps @@ -146,11 +132,11 @@ Steps are sub-tests. Two ways to create them: parent test and stops it - matching `WithNewStep` semantics from allure-go. -For code full of `WithNewStep`, use `allure.Step` - same semantics. +Use `allure.Step` instead of `WithNewStep`. Nested steps (`sCtx.WithNewStep`, `sCtx.NewStep`) become nested -`allure.Step(t, ...)` calls. Step assertions (`sCtx.Assert()`, -`sCtx.Require()`) become `t.Assert()` / `t.Require()`, which -`PluginAllure` adds to `T`. +`allure.Step(t, ...)` calls. + +Assertions `sCtx.Assert()` and `sCtx.Require()` become `t.Assert()` & `t.Require()`, which `PluginAllure` adds to `T`. For the full testo-allure API (output directory, labels, attachments and so on), see the diff --git a/docs/plugins.md b/docs/plugins.md index f0ee011..a2be206 100644 --- a/docs/plugins.md +++ b/docs/plugins.md @@ -1,6 +1,5 @@ # Writing plugins -Everything a plugin can do, with a compiling example for each part. This page assumes you finished the [tutorial](./tutorial.md), where a first plugin (the timer) is built step by step. @@ -23,11 +22,11 @@ func (p *PluginShout) Plugin(parent testoplugin.Plugin, options ...testoplugin.O The returned `testoplugin.Spec` has three parts, all optional: -| Part | What it does | -| :--- | :----------- | -| `Hooks` | run code around suites, tests and sub-tests | -| `Overrides` | wrap built-in `T` methods (`Log`, `Fail`, ...) | -| `Plan` | filter, reorder or duplicate the tests before the run | +| Part | What it does | +| :---------- | :---------------------------------------------------- | +| `Hooks` | run code around suites, tests and sub-tests | +| `Overrides` | wrap built-in `T` methods (`Log`, `Fail`, ...) | +| `Plan` | filter, reorder or duplicate the tests before the run | A plugin usually embeds `*testo.T`. Testo fills it with the same `T` as the current test, so the plugin can log, fail, or inspect the test diff --git a/docs/tutorial.md b/docs/tutorial.md index 60d0145..3be4124 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -323,6 +323,7 @@ type Bakery struct{ testo.Suite[T] } Tests become methods of the suite. They follow the usual Go naming rules (the `Test` prefix) and must accept the same `T` as specified in `testo.Suite[T]`. Value and pointer receivers both work. + Replace the `TestBake` function with: ```go