From 6053b6c80b063fd01d4b82f0fac1355f08a19a72 Mon Sep 17 00:00:00 2001 From: James Naftel Date: Thu, 22 Apr 2021 15:33:55 -0400 Subject: [PATCH 1/9] First draft of ruby instructions --- docs/yourbase-ruby.md | 59 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 docs/yourbase-ruby.md diff --git a/docs/yourbase-ruby.md b/docs/yourbase-ruby.md new file mode 100644 index 0000000..9fe125b --- /dev/null +++ b/docs/yourbase-ruby.md @@ -0,0 +1,59 @@ +# YourBase Ruby Acceleration + +Tests are important. For large monoliths, they're also a major source of drag on velocity. + +YourBase is a tool that traces your tests to determine which functions each test depends on. It can later use this information to determine which tests do not need to run because their code paths have not changed. These tests are skipped automatically. + +YourBase works with Ruby versions >= 2.3 + +## Getting Started +No configuration, setup, or babysitting required. To get started, you need the YourBase gem and a download token. To request a token, please visit [YourBase.io](https://yourbase.io/download) + +Once you have a token, simply follow the steps below: +```sh +YOURBASE_DOWNLOAD_TOKEN= +bundle add yourbase-rspec --git "https://$YOURBASE_DOWNLOAD_TOKEN:x-oauth-basic@github.com/yourbase/yourbase-rspec-skipper-engine.git" + +bundle install +``` + +## First run +After installing yourbase-rspec, if you are not using Rails you must add +"require 'yourbase-rspec'" in your spec folder. + +```sh +# Add require for non Rails projects +echo "require 'yourbase-rspec'" >> spec/yourbase_spec.rb +``` + +Run your tests with the same command you typically use. You should see a rocket ship at the beginning the RSpec test section. + +``` +πŸš€ +``` + +The first run will be cold, so if you just want to see YourBase in action and your tests are going to take a while, you can run a subset of tests. Tracing data for the subset will be used correctly even if you later run all tests. + +After the run finishes, running again will skip all tests. Modifying a dependency will run only tests whose code paths touched the changed code. You're YourBased! πŸš€ + +## RSpec Output + +YourBase enhances the output so that you can clearly see the results of the Gem. + +For an accelerated run, you will see the number of skipped tested added to your +RSpec summary line: +```sh +1 examples, 0 failures, 1 skipped with YourBaseπŸš€ +``` + +To get additional details, please use -f, --formatter [p]rogress | [d]ocumentation] + +## Cohorting / Sharding +YourBase supports sharding your tests without negatively affecting tracing or acceleration. It uses consistent hashing to split tests into cohorts that stay the same between runs, even as the test pool grows or shrinks. + +1) Set YB_COHORT_COUNT to the number of cohorts your tests should be split into. This should be the same among all shards. +1) Set YB_TEST_COHORT to the cohort ID this run should identify as, starting with 1. This should be different among all shards. +Without these set, YourBase assumes a value of "1" for both, meaning one shard will run one cohort; that cohort will contain all tests. + +Note that tests are only guaranteed to remain in the same cohort as long as +YB_COHORT_COUNT doesn't change. From da78f3dad029550fccdc3d1e03ed99730a8eef4e Mon Sep 17 00:00:00 2001 From: James Naftel Date: Thu, 22 Apr 2021 15:41:22 -0400 Subject: [PATCH 2/9] link from main README.md to Ruby --- docs/README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/README.md b/docs/README.md index eaeed1a..e974fd8 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,3 +8,7 @@ - [Package Configuration Reference](configuration.md) - [CI Acceleration and Caching](ci-caching.md) - [Design Philosophy](design-philosophy.md) + +# Quick Start Acceleration + +- [YourBase Ruby Acceleration](yourbase-ruby.md) From 4f8a1a2cae2899bbda68137081e375216d06a7a1 Mon Sep 17 00:00:00 2001 From: James Naftel Date: Thu, 29 Apr 2021 09:22:11 -0400 Subject: [PATCH 3/9] updated based on Ross' comments and added observation mode --- docs/README.md | 8 +++++--- docs/yourbase-ruby.md | 28 +++++++++++++++++++++------- 2 files changed, 26 insertions(+), 10 deletions(-) diff --git a/docs/README.md b/docs/README.md index e974fd8..28eb4d8 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,5 +1,10 @@ YourBase +# Quick Start Acceleration + +- [YourBase Ruby Acceleration](yourbase-ruby.md) + + # CI and CLI Documentation - [Installation](installation.md) @@ -9,6 +14,3 @@ - [CI Acceleration and Caching](ci-caching.md) - [Design Philosophy](design-philosophy.md) -# Quick Start Acceleration - -- [YourBase Ruby Acceleration](yourbase-ruby.md) diff --git a/docs/yourbase-ruby.md b/docs/yourbase-ruby.md index 9fe125b..9c56a9d 100644 --- a/docs/yourbase-ruby.md +++ b/docs/yourbase-ruby.md @@ -11,10 +11,7 @@ No configuration, setup, or babysitting required. To get started, you need the Y Once you have a token, simply follow the steps below: ```sh -YOURBASE_DOWNLOAD_TOKEN= -bundle add yourbase-rspec --git "https://$YOURBASE_DOWNLOAD_TOKEN:x-oauth-basic@github.com/yourbase/yourbase-rspec-skipper-engine.git" - -bundle install +bundle add yourbase-rspec --git "https://${YOURBASE_DOWNLOAD_TOKEN?}:x-oauth-basic@github.com/yourbase/yourbase-rspec-skipper-engine.git" && bundle install ``` ## First run @@ -28,7 +25,7 @@ echo "require 'yourbase-rspec'" >> spec/yourbase_spec.rb Run your tests with the same command you typically use. You should see a rocket ship at the beginning the RSpec test section. -``` +```plain πŸš€ ``` @@ -42,11 +39,12 @@ YourBase enhances the output so that you can clearly see the results of the Gem. For an accelerated run, you will see the number of skipped tested added to your RSpec summary line: -```sh +```plain 1 examples, 0 failures, 1 skipped with YourBaseπŸš€ ``` -To get additional details, please use -f, --formatter [p]rogress | [d]ocumentation] +YourBase adds additional details when using the RSpec formatter option with the +progress and documentation formatters. ## Cohorting / Sharding YourBase supports sharding your tests without negatively affecting tracing or acceleration. It uses consistent hashing to split tests into cohorts that stay the same between runs, even as the test pool grows or shrinks. @@ -57,3 +55,19 @@ Without these set, YourBase assumes a value of "1" for both, meaning one shard w Note that tests are only guaranteed to remain in the same cohort as long as YB_COHORT_COUNT doesn't change. + +## Observation Mode +The yourbase-rspec gem includes an "observation mode" which will run all [command-line specified] examples. "Observation mode" allows you to test drive the plugin without actually skipping any tests. + +To enable observation mode, set YOURBASE_OBSERVATION_MODE to true in the environment. The documentation formatter isn't strictly required, but it will print the reasons why YourBase would run or skip a given example group. + +```sh +YOURBASE_OBSERVATION_MODE=true bundle exec rspec --format documentation +``` + +Instead of a single rocketship, you'll see the following at the top of the rspec output for observation mode: + +```plain +πŸš€ YourBase test selection is in observation mode. All example groups will be run. πŸš€ +``` + From 0fd4892f66083353823caeb98066e4cb8cb6936f Mon Sep 17 00:00:00 2001 From: James Naftel Date: Mon, 3 May 2021 11:50:40 -0400 Subject: [PATCH 4/9] add link to pypi docs --- docs/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/README.md b/docs/README.md index 28eb4d8..872a8ca 100644 --- a/docs/README.md +++ b/docs/README.md @@ -3,6 +3,7 @@ # Quick Start Acceleration - [YourBase Ruby Acceleration](yourbase-ruby.md) +- [YourBase Python Acceleration](https://pypi.org/project/yourbase/) # CI and CLI Documentation From c67d7626b0bf7a766625845a869c882ac214c986 Mon Sep 17 00:00:00 2001 From: James Naftel Date: Wed, 12 May 2021 09:57:39 -0400 Subject: [PATCH 5/9] Added Christina's PR feedback. --- docs/yourbase-ruby.md | 65 ++++++++++++++++++++++++------------------- 1 file changed, 36 insertions(+), 29 deletions(-) diff --git a/docs/yourbase-ruby.md b/docs/yourbase-ruby.md index 9c56a9d..c5d5736 100644 --- a/docs/yourbase-ruby.md +++ b/docs/yourbase-ruby.md @@ -2,26 +2,30 @@ Tests are important. For large monoliths, they're also a major source of drag on velocity. -YourBase is a tool that traces your tests to determine which functions each test depends on. It can later use this information to determine which tests do not need to run because their code paths have not changed. These tests are skipped automatically. +YourBase is a tool that traces your tests to determine which files they depend on. It uses this information to determine which tests must run, because they are new or have changed dependencies. Tests without changed dependencies are skipped. -YourBase works with Ruby versions >= 2.3 +YourBase works with Ruby versions >= 2.3 and RSpec 3+. ## Getting Started -No configuration, setup, or babysitting required. To get started, you need the YourBase gem and a download token. To request a token, please visit [YourBase.io](https://yourbase.io/download) +1. Add `gem 'yourbase-rspec', '~> 0.5.6',` to your Gemfile. If you have a `:test` group, add it there. +2. `bundle install` from the command line. +3. If you are not in a Rails project, you will also need to `require 'yourbase-rspec` your spec_helper (or at the top of the spec file you want to run). Once you have a token, simply follow the steps below: ```sh bundle add yourbase-rspec --git "https://${YOURBASE_DOWNLOAD_TOKEN?}:x-oauth-basic@github.com/yourbase/yourbase-rspec-skipper-engine.git" && bundle install ``` -## First run -After installing yourbase-rspec, if you are not using Rails you must add -"require 'yourbase-rspec'" in your spec folder. +> After installing yourbase-rspec, if you are not using Rails you must add +"require 'yourbase-rspec'" to in your spec folder. ```sh -# Add require for non Rails projects +# Add require 'yourbase-rspec' for non Rails projects. echo "require 'yourbase-rspec'" >> spec/yourbase_spec.rb ``` +## First run + +> Note: If you are using Spring, please run `spring stop` before starting your tests. Run your tests with the same command you typically use. You should see a rocket ship at the beginning the RSpec test section. @@ -29,45 +33,48 @@ Run your tests with the same command you typically use. You should see a rocket πŸš€ ``` -The first run will be cold, so if you just want to see YourBase in action and your tests are going to take a while, you can run a subset of tests. Tracing data for the subset will be used correctly even if you later run all tests. +The first time you run your tests with `yourbase-rspec` will take the typical amount of time as it records tracing data to map dependencies (a "cold build"). If you run the same test again without changing any code, you should see everything skipped! Subsequent runs will only run examples that are new or depend on changed files. After the run finishes, running again will skip all tests. Modifying a dependency will run only tests whose code paths touched the changed code. You're YourBased! πŸš€ ## RSpec Output -YourBase enhances the output so that you can clearly see the results of the Gem. +YourBase adds to the RSpec output to give you information while about if examples are being run or skipped, and why. -For an accelerated run, you will see the number of skipped tested added to your -RSpec summary line: +The default RSpec output from the `ProgressFormatter` (`.....*..... F........`) will print a `.` in yellow (or the color you have set for `:pending`) when an example is skipped. + +The `DocumentationFormatter` will add the reason an example group is being run in green (or the color your have set for `:success`), and the reason an example group is being skipped in yellow (or the color you have set for `:pending`). + +The summary line will show how show how many examples were skipped! πŸš€ ```plain 1 examples, 0 failures, 1 skipped with YourBaseπŸš€ ``` -YourBase adds additional details when using the RSpec formatter option with the -progress and documentation formatters. +## Parallelization and Sharding +The `yourbase-rspec` gem supports your workflows for both parallelization (running tests in more than one process at a time on the same machine) and sharding (running tests on more than one machine). Dependency histories are keyed off of the code state (git hash), and all tracing data derived from an identical code state is grouped for future use. + +The environment variables `YOURBASE_ACTIVE_COHORT` and `YOURBASE_COHORT COUNT` control which tests **might** run. Tests that are in the active cohort will be checked against dependency changes, and _tests that are not in the active cohort will be automatically skipped_. -## Cohorting / Sharding -YourBase supports sharding your tests without negatively affecting tracing or acceleration. It uses consistent hashing to split tests into cohorts that stay the same between runs, even as the test pool grows or shrinks. +YourBase cohorts are assigned based on consistent hashing of the example group name AND the number of cohorts. An example group that is in cohort `1` will always be in cohort `1` unless either the `YOURBASE_COHORT_COUNT` OR example group name are changed. -1) Set YB_COHORT_COUNT to the number of cohorts your tests should be split into. This should be the same among all shards. -1) Set YB_TEST_COHORT to the cohort ID this run should identify as, starting with 1. This should be different among all shards. -Without these set, YourBase assumes a value of "1" for both, meaning one shard will run one cohort; that cohort will contain all tests. +The `YOURBASE_ACTIVE_COHORT` is 1-indexed (it starts at 1, not 0). If you are sharding with YourBase cohorts, and you set `YOURBASE_COHORT_COUNT=2`, then one of your shard should have `YOURBASE_ACTIVE_COHORT=1` and the other should have `YOURBASE_ACTIVE_COHORT=2`. -Note that tests are only guaranteed to remain in the same cohort as long as -YB_COHORT_COUNT doesn't change. +Unless the value of `YOURBASE_COHORT_COUNT` is set and is greater than 1, cohorts are turned off. ## Observation Mode -The yourbase-rspec gem includes an "observation mode" which will run all [command-line specified] examples. "Observation mode" allows you to test drive the plugin without actually skipping any tests. +The yourbase-rspec gem includes an "observation mode" which allows you to test drive the gem without actually skipping any tests. -To enable observation mode, set YOURBASE_OBSERVATION_MODE to true in the environment. The documentation formatter isn't strictly required, but it will print the reasons why YourBase would run or skip a given example group. +In β€œobservation mode” all [command-line specified] examples will be run, but `yourbase-rspec` will monitor if our test selection would have skipped any examples that ultimately failed. At the end, it will print out the names of any example group that would have been incorrectly skipped, or it will confirm that none were. -```sh -YOURBASE_OBSERVATION_MODE=true bundle exec rspec --format documentation -``` +To access observation mode, set `YOURBASE_OBSERVATION_MODE=true` in the environment, and run your specs. The documentation formatter isn’t required, but it will print the reasons why YourBase would select to run or skip a given example group. -Instead of a single rocketship, you'll see the following at the top of the rspec output for observation mode: +`$ YOURBASE_OBSERVATION_MODE=true bundle exec rspec --format documentation` -```plain -πŸš€ YourBase test selection is in observation mode. All example groups will be run. πŸš€ -``` +Instead of a single rocketship, you’ll see the following at the top of the rspec output for observation mode: +`:rocket: YourBase test selection is in observation mode. All example groups will be run. :rocket:` + +And then at the bottom, below the RSpec summary, you should see this: +`πŸš€ YourBase observation mode: all "skipped" example groups passed successfully! πŸš€` + +If you instead see: `πŸš€ YourBase observation mode: some \"skipped\" example groups contained failures πŸ›Έ` followed by one or more example group names, it means that yourbase-spec would have skipped at least one test that failed when it was actually run. We hope you'll never see this, and we hope that you'll email us if you do, at: From 6320aeef72154b3a27347b2a847f68bc7bedcd83 Mon Sep 17 00:00:00 2001 From: James Naftel Date: Wed, 12 May 2021 10:11:04 -0400 Subject: [PATCH 6/9] fixed formatting and added support email --- docs/yourbase-ruby.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/yourbase-ruby.md b/docs/yourbase-ruby.md index c5d5736..f379876 100644 --- a/docs/yourbase-ruby.md +++ b/docs/yourbase-ruby.md @@ -16,7 +16,7 @@ Once you have a token, simply follow the steps below: bundle add yourbase-rspec --git "https://${YOURBASE_DOWNLOAD_TOKEN?}:x-oauth-basic@github.com/yourbase/yourbase-rspec-skipper-engine.git" && bundle install ``` -> After installing yourbase-rspec, if you are not using Rails you must add +> Note: After installing yourbase-rspec, if you are not using Rails you must add "require 'yourbase-rspec'" to in your spec folder. ```sh @@ -35,7 +35,9 @@ Run your tests with the same command you typically use. You should see a rocket The first time you run your tests with `yourbase-rspec` will take the typical amount of time as it records tracing data to map dependencies (a "cold build"). If you run the same test again without changing any code, you should see everything skipped! Subsequent runs will only run examples that are new or depend on changed files. -After the run finishes, running again will skip all tests. Modifying a dependency will run only tests whose code paths touched the changed code. You're YourBased! πŸš€ +After the run finishes, running again will skip all tests. Modifying a dependency will run only tests whose code paths touched the changed code. + +You're YourBased! πŸš€ ## RSpec Output @@ -76,5 +78,5 @@ Instead of a single rocketship, you’ll see the following at the top of the rsp And then at the bottom, below the RSpec summary, you should see this: `πŸš€ YourBase observation mode: all "skipped" example groups passed successfully! πŸš€` -If you instead see: `πŸš€ YourBase observation mode: some \"skipped\" example groups contained failures πŸ›Έ` followed by one or more example group names, it means that yourbase-spec would have skipped at least one test that failed when it was actually run. We hope you'll never see this, and we hope that you'll email us if you do, at: +If you instead see: `πŸš€ YourBase observation mode: some "skipped" example groups contained failures πŸ›Έ` followed by one or more example group names, it means that yourbase-spec would have skipped at least one test that failed when it was actually run. We hope you'll never see this, and we hope that you'll email us if you do, at: From 45426239c6462fcc36928302e96f3a273e958b43 Mon Sep 17 00:00:00 2001 From: James Naftel Date: Thu, 20 May 2021 08:35:58 -0400 Subject: [PATCH 7/9] Update docs/yourbase-ruby.md Co-authored-by: Christina Austin <4240737+clausti@users.noreply.github.com> --- docs/yourbase-ruby.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/yourbase-ruby.md b/docs/yourbase-ruby.md index f379876..7597808 100644 --- a/docs/yourbase-ruby.md +++ b/docs/yourbase-ruby.md @@ -21,7 +21,7 @@ bundle add yourbase-rspec --git "https://${YOURBASE_DOWNLOAD_TOKEN?}:x-oauth-bas ```sh # Add require 'yourbase-rspec' for non Rails projects. -echo "require 'yourbase-rspec'" >> spec/yourbase_spec.rb +If you are not in a Rails project, you will also need to add `require 'yourbase-rspec'` to your `spec_helper.rb` (or another project file which will be loaded, such as the file containing the specs you are running). ``` ## First run @@ -79,4 +79,3 @@ And then at the bottom, below the RSpec summary, you should see this: `πŸš€ YourBase observation mode: all "skipped" example groups passed successfully! πŸš€` If you instead see: `πŸš€ YourBase observation mode: some "skipped" example groups contained failures πŸ›Έ` followed by one or more example group names, it means that yourbase-spec would have skipped at least one test that failed when it was actually run. We hope you'll never see this, and we hope that you'll email us if you do, at: - From 0174eacdf8aa27ed43d9bcc372bf864408716ee2 Mon Sep 17 00:00:00 2001 From: James Naftel Date: Thu, 20 May 2021 09:05:28 -0400 Subject: [PATCH 8/9] updated based on PR comments --- docs/yourbase-ruby.md | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) diff --git a/docs/yourbase-ruby.md b/docs/yourbase-ruby.md index 7597808..793448f 100644 --- a/docs/yourbase-ruby.md +++ b/docs/yourbase-ruby.md @@ -7,25 +7,12 @@ YourBase is a tool that traces your tests to determine which files they depend o YourBase works with Ruby versions >= 2.3 and RSpec 3+. ## Getting Started -1. Add `gem 'yourbase-rspec', '~> 0.5.6',` to your Gemfile. If you have a `:test` group, add it there. +1. Add `gem 'yourbase-rspec', '~> x.x.x',` to your Gemfile. If you have a `:test` group, add it there. 2. `bundle install` from the command line. 3. If you are not in a Rails project, you will also need to `require 'yourbase-rspec` your spec_helper (or at the top of the spec file you want to run). -Once you have a token, simply follow the steps below: -```sh -bundle add yourbase-rspec --git "https://${YOURBASE_DOWNLOAD_TOKEN?}:x-oauth-basic@github.com/yourbase/yourbase-rspec-skipper-engine.git" && bundle install -``` - -> Note: After installing yourbase-rspec, if you are not using Rails you must add -"require 'yourbase-rspec'" to in your spec folder. - -```sh -# Add require 'yourbase-rspec' for non Rails projects. -If you are not in a Rails project, you will also need to add `require 'yourbase-rspec'` to your `spec_helper.rb` (or another project file which will be loaded, such as the file containing the specs you are running). -``` ## First run -> Note: If you are using Spring, please run `spring stop` before starting your tests. Run your tests with the same command you typically use. You should see a rocket ship at the beginning the RSpec test section. @@ -33,6 +20,8 @@ Run your tests with the same command you typically use. You should see a rocket πŸš€ ``` +> Note: If you are using Spring and do not see the rocket ship, please run `spring stop` before running your tests. + The first time you run your tests with `yourbase-rspec` will take the typical amount of time as it records tracing data to map dependencies (a "cold build"). If you run the same test again without changing any code, you should see everything skipped! Subsequent runs will only run examples that are new or depend on changed files. After the run finishes, running again will skip all tests. Modifying a dependency will run only tests whose code paths touched the changed code. From 10dd8635bcbc3be35b972c52a094f005b833b4b1 Mon Sep 17 00:00:00 2001 From: James Naftel Date: Thu, 20 May 2021 09:35:03 -0400 Subject: [PATCH 9/9] clarity and naming improvements --- docs/yourbase-ruby.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/docs/yourbase-ruby.md b/docs/yourbase-ruby.md index 793448f..c145199 100644 --- a/docs/yourbase-ruby.md +++ b/docs/yourbase-ruby.md @@ -1,4 +1,4 @@ -# YourBase Ruby Acceleration +# YourBase Ruby Test Selection Tests are important. For large monoliths, they're also a major source of drag on velocity. @@ -14,11 +14,7 @@ YourBase works with Ruby versions >= 2.3 and RSpec 3+. ## First run -Run your tests with the same command you typically use. You should see a rocket ship at the beginning the RSpec test section. - -```plain -πŸš€ -``` +Run your tests with the same command you typically use. You should see a rocket ship πŸš€ at the beginning the RSpec test section. The rocket ship signifies that the YourBase Test Selection is properly loaded and tracing your tests. > Note: If you are using Spring and do not see the rocket ship, please run `spring stop` before running your tests.