feat: implement test suites on computed metrics by GregoryAlbouy · Pull Request #16 · benchttp/engine

GregoryAlbouy · 2022-07-27T18:21:29Z

Description

Some exploration regarding specs and implementation of formal test suite runs, without the use of template.

Specs

Features

Error handling

Error messages for each situation

CLI run

Config

Output

request:
  url: https://example.com
runner:
  requests: 5
  concurrency: 1
tests:
  - name: minimum response time
    field: MIN
    predicate: GT
    target: 80ms
  - name: maximum response time
    field: MAX
    predicate: LTE
    target: 120ms
  - name: 100% availability
    field: FAILURE_COUNT
    predicate: EQ
    target: 0

Server run

Config

Output

{
  "request": {
    "url": "https://example.com"
  },
  "runner": {
    "requests": 5,
    "concurrency": 1
  },
  "tests": [
    {
      "name": "minimum response time",
      "field": "MIN",
      "predicate": "GT",
      "target": "80ms"
    },
    {
      "name": "maximum response time",
      "field": "MAX",
      "predicate": "LTE",
      "target": "120ms"
    },
    {
      "name": "100% availability",
      "field": "FAILURE_COUNT",
      "predicate": "EQ",
      "target": "0"
    },
  ]
}

{
  "Metrics": {},
  "Metadata": {},
  "Tests": {
    "Pass": false,
    "Results": [
      {
        "Input": {
          "Name": "minimum response time",
          "Field": "MIN",
          "Predicate": "GT",
          "Target": 80000000
        },
        "Pass": true,
        "Summary": "want MIN > 80ms, got 83.392463ms"
      },
      {
        "Input": {
          "Name": "maximum response time",
          "Field": "MAX",
          "Predicate": "LTE",
          "Target": 120000000
        },
        "Pass": false,
        "Summary": "want MAX <= 120ms, got 433.415742ms"
      },
      {
        "Input": {
          "Name": "100% availability",
          "Field": "FAILURE_COUNT",
          "Predicate": "EQ",
          "Target": 0
        },
        "Pass": true,
        "Summary": "want FAILURE_COUNT == 0, got 0"
      }
    ]
  }
}

Demo

Changes

Linked issues

Closes Implement test suites #4
Closes Investigate: test suites #19

Notes

moreirathomas

Really nice! It offers a great solution to the feature request 🙂

Question: do we continue supporting template outputs?

If it was used to answer the need for testing capabilities, this probably is a better replacement?

runner/internal/tests/tests_test.go

runner/internal/tests/predicate.go

moreirathomas · 2022-07-31T16:02:25Z

runner/internal/report/report.go

 	"strings"
 	"time"

+	"github.com/benchttp/engine/internal/cli/ansi"


Nitpick: I'm not sure about this dependency.

runner/internal/... depends on internal/cli/....

Yet internal/cli depends on runner.

It is ok because it is different packages. Still I find it strange because as ansi is nested in cli.

We may argue that it is strange that a core package (report) has io/visualization concerns.

Although, not deal breaker at all for the pr.

We may argue that it is strange that a core package (report) has io/visualization concerns.

I agree, this is the root of the problem to me. It's ok having a Report.String but it should not be specific to a CLI with ansi codes.

Probably a good candidate for refactoring in #26

runner/internal/metrics/metrics.go

internal/configparse/parse.go

moreirathomas

Great refactoring since last viewed 👍

- create package runner/internal/tests

- add result to server reponse (auto) - display summary in CLI

- delegate metrics comparison logics to package metrics - remove usage of metric getter - use tests.Case as config.Global.Tests input - adapt TestPredicate to new tests API - several renamings: - metrics.Metric -> metrics.Source - tests.Input -> tests.Case - tests.SingleResult -> tests.CaseResult

- e.g. durations: "85675992" -> "85.675992ms"

- PassValues and FailValues were inverted for comparison cases

Before: `a.Compare(b) == compare(a, b)` `0.Compare(1) == compare(0, 1) == SUP` Now: `a.Compare(b) == compare(a, b)` `0.Compare(1) == compare(1, 0) == INF` The new order makes more sense because `a` remains the main interest of the operation: when calling `a.Compare(b)` we want to get information about `a` (when compared to `b`)

- remove helper metricGetter - some renamings - add documenting comments

- implement errorutil.WithDetails - replace local implementations

- for field tests[n].target, JSON did not accept int values (but YAML did as they were read as string)

GregoryAlbouy added the poc label Jul 27, 2022

GregoryAlbouy added this to the Next milestone Jul 27, 2022

GregoryAlbouy force-pushed the poc/test-suite branch from ee57770 to c7c1e24 Compare July 27, 2022 18:36

GregoryAlbouy added must have Blocking feature for the targeted release scope:core scope:server scope:cli labels Jul 29, 2022

GregoryAlbouy marked this pull request as draft July 29, 2022 22:01

GregoryAlbouy force-pushed the poc/test-suite branch from 42ed54b to de3be90 Compare July 31, 2022 14:22

GregoryAlbouy marked this pull request as ready for review July 31, 2022 14:25

GregoryAlbouy changed the title ~~poc: test suite~~ feat: implement test suites on computed metrics Jul 31, 2022

GregoryAlbouy removed the poc label Jul 31, 2022

GregoryAlbouy force-pushed the poc/test-suite branch from de3be90 to b00f830 Compare July 31, 2022 14:54

GregoryAlbouy requested review from c1-ra and moreirathomas July 31, 2022 14:54

moreirathomas reviewed Jul 31, 2022

View reviewed changes

GregoryAlbouy force-pushed the poc/test-suite branch 2 times, most recently from 0f37ce8 to fe3f93c Compare August 1, 2022 00:51

GregoryAlbouy mentioned this pull request Aug 1, 2022

chore: remove templating logics #28

Merged

moreirathomas previously approved these changes Aug 1, 2022

View reviewed changes

GregoryAlbouy dismissed moreirathomas’s stale review via 29b0c26 August 2, 2022 18:30

GregoryAlbouy force-pushed the poc/test-suite branch from 63f6014 to 29b0c26 Compare August 2, 2022 18:30

GregoryAlbouy added 8 commits August 2, 2022 21:15

feat: implement test runner

76387d2

- create package runner/internal/tests

feat: parse input test configuration

cfba02e

feat: run test suite from config

2dbe0ad

feat: report test results

1587db7

- add result to server reponse (auto) - display summary in CLI

test(configparse): update unit tests

2568b59

test(report): update unit test for Report.String

aa639e6

feat(metrics,configparse): allow any metric types for tests

cdae0dc

GregoryAlbouy added 17 commits August 2, 2022 21:16

fix(tests): use accurate format for value display

3391b4e

- e.g. durations: "85675992" -> "85.675992ms"

fix(metrics): set TypeDuration to last reflect.Kind + 1

8538c1a

test(tests): implement TestRun

6ce2447

test(tests): fix and refactor TestPredicate

4fe0e37

- PassValues and FailValues were inverted for comparison cases

docs(metrics): add doc comments

6df8217

test(metrics): implement TestMetric_Compare

cd471c6

refactor(metrics): rename Source -> Field

e6196c0

refactor(metrics): use fieldRecord as single source of truth

05a2514

refactor(metrics): reorganize files

fb5147f

refactor(metrics): improve field understandability

3f5a104

- remove helper metricGetter - some renamings - add documenting comments

refactor(configparse): Ultimate Once And For All Solution

b0402d0

feat(configparse): error handling

360fa20

refactor: create package errorutil

24b2b4c

- implement errorutil.WithDetails - replace local implementations

feat(cli): exit 1 if the test suite fails

76c397d

ui(cli): improve test suite results display

7aa80df

fix(configparse): accept any value when unmarshaling tests.target

54791be

- for field tests[n].target, JSON did not accept int values (but YAML did as they were read as string)

GregoryAlbouy force-pushed the poc/test-suite branch from 29b0c26 to 54791be Compare August 2, 2022 19:17

refactor(configparse): remove unnecessary helper

7c9c4ce

GregoryAlbouy requested a review from moreirathomas August 2, 2022 19:29

GregoryAlbouy mentioned this pull request Aug 3, 2022

Remove dependencies to cli from runner #30

Closed

moreirathomas approved these changes Aug 3, 2022

View reviewed changes

GregoryAlbouy merged commit bb6d64f into main Aug 3, 2022

moreirathomas deleted the poc/test-suite branch August 19, 2022 17:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement test suites on computed metrics#16

feat: implement test suites on computed metrics#16
GregoryAlbouy merged 26 commits intomainfrom
poc/test-suite

GregoryAlbouy commented Jul 27, 2022 •

edited

Loading

Uh oh!

moreirathomas left a comment

Uh oh!

Uh oh!

Uh oh!

moreirathomas Jul 31, 2022

Uh oh!

GregoryAlbouy Jul 31, 2022

Uh oh!

Uh oh!

Uh oh!

moreirathomas left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

GregoryAlbouy commented Jul 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Specs

Features

Error handling

CLI run

Server run

Demo

Changes

Linked issues

Notes

Uh oh!

moreirathomas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

moreirathomas Jul 31, 2022

Choose a reason for hiding this comment

Uh oh!

GregoryAlbouy Jul 31, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

moreirathomas left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

GregoryAlbouy commented Jul 27, 2022 •

edited

Loading