review of the InSpec profile process

src/README.md

@@ -18,7 +18,7 @@ actions:
   - text: Guidance Development
     type: primary
     link: /courses/guidance/
-  - text: InSpec Profile Updating & Development
+  - text: Developing And Testing InSpec Profiles
     type: primary
     link: /courses/profile-dev-test
 

src/courses/profile-dev-test/02.md

@@ -5,7 +5,7 @@ title: Repository Organization
 author: Aaron Lippold
 ---
 
-The repository and profile are organized into two primary branches: `main` and `TBD`. The repository has a set of `tags` representing iterative releases of the STIG from one Benchmark major version to the next. It also has a set of releases for fixes and updates to the profile between STIG Benchmark Releases.
+Each InSpec profile repository has a set of `tags` representing iterative releases of the underlying benchmark from one major version to the next. It also has a set of releases for fixes and updates to the profile between benchmark releases.
 
 # Branches
 
@@ -38,3 +38,9 @@ Major tags point to the latest patch release of the benchmark. For example, `v1.
 The latest patch release always points to the major release for the profile.
 
 For example, after releasing `v1.12.0`, we will point `v1.12` to that patch release: `v1.12.0`. When an issue is found, we will fix, tag, and release `v1.12.1`. We will then 'move' the `v1.12` tag so that it points to tag `v1.12.1`. This way, your pipelines can choose if they want to pin on a specific release of the InSpec profile or always run 'current'.
+
+::: tip
+It helps to recall that all we're really doing when we write an InSpec profile for a benchmark is _transforming the benchmark document into code._ It's still the "same" document.
+
+For example, the RHEL8 STIG profile is published by DISA as an XCCDF XML document describing requirements and instructions on meeting them. The MITRE SAF InSpec profile for that STIG _includes all of the original XML tags as metadata in the control files,_ and adds some InSpec tests alongside them. Thus, the InSpec profile _is_ the STIG document, and therefore needs to reflect the version and release of the last published STIG.
+:::

src/courses/profile-dev-test/03.md

@@ -7,14 +7,71 @@ author: Aaron Lippold
 
 # Required Software
 
-- RVM, or another Ruby Management Tool
+- [RVM](https://rvm.io/), or another Ruby Management Tool
 - Ruby v3 or higher
 - Git
+- [Test Kitchen](https://docs.chef.io/workstation/kitchen/)
 - VS Code or another IDE
 - Docker (if you want to test hardened and non-hardened containers)
 - AWS CLI
 - AWS Account
 
+## Bundler
+
+[Bundler](https://bundler.io/) is Ruby's dependency manager (think `pip` for Python, or `npm` for NodeJS). Since InSpec is ultimately Ruby code, Bundler is how we're going to keep our development environment squared up.
+
+### The Gemfile
+
+You tell Bundler what Ruby dependencies ("gems") you want available by listing them out in a file called (appropriately) `Gemfile` at the root or your project.
+
+InSpec profiles occasionally require you to download gems to support individual tests. If this is the case, they should be listed in the Gemfile. This is also how we install the Test Kitchen suite (`test-kitchen`).
+
+Here's an example from the [RHEL 8 STIG profile](https://github.com/mitre/redhat-enterprise-linux-8-stig-baseline/blob/main/Gemfile):
+``` ruby
+source 'https://rubygems.org'
+
+gem 'cookstyle'
+gem 'highline'
+gem 'inspec', '>= 6.6.0'
+gem 'inspec-bin'
+gem 'inspec-core'
+gem 'kitchen-ansible'
+gem 'kitchen-docker'
+gem 'kitchen-dokken'
+gem 'kitchen-ec2'
+gem 'kitchen-inspec'
+gem 'kitchen-sync'
+gem 'kitchen-vagrant'
+gem 'pry-byebug'
+gem 'rake'
+gem 'rubocop'
+gem 'rubocop-rake'
+gem 'test-kitchen'
+gem 'train-awsssm'
+```
+
+Running the command `bundle install` inside your profile repo will cause Bundler to read your Gemfile and install every listed gem (which the docs call your 'gem bundle'). See the Bundler docs linked above for more details.
+
+::: Note Wait, does that gem say `inspec`?
+Correct. For convenience, we on the SAF team tend to install InSpec itself as a gem (since it is, after all, just Ruby code under the hood). Note that installing InSpec this way _can_ cause confusion if you install it alongside one of the regular executable releases (i.e. from the Chef downloads page, or via `brew install inspec` or similar), which actually include their own little Ruby environment internally. This is why we need to use `bundle exec` to _force_ commands to use the locally configured Ruby environment, including the "local" InSpec exectuable, as discussed in the next section.
+:::
+
+### bundle exec
+
+Note that running `bundle install` does not mean that every subsequent time you run InSpec it will correctly use your Gemfile's gem bundle. If you want to _force_ a command to run in the context of the gem bundle, you append it with `bundle exec`.
+
+This is why most of the commands in the rest of this guide have a `bundle exec` in front of them.
+
+## RVM
+
+We use the Ruby Version Manager to segregate our development environment for Ruby from the rest of our system. Think `venv` for Python. Make sure you [install RVM](https://rvm.io/rvm/install) and use a [gemset](https://rvm.io/gemsets/basics) for each profile you want to run.
+
+::: note Doesn't that seem like overkill for just writing a few tests?
+It isn't overkill.
+
+Trust us; if you don't take the time to configure your environment to do this stuff, you will eventually wish you had.
+:::
+
 # Required Accounts
 
 1. [AWS Console Account](https://aws.amazon.com/console/ "AWS Console Account")

src/courses/profile-dev-test/04.md

@@ -44,4 +44,109 @@ You can also isolate which of the 'target suites' - either `vanilla` or `hardene
 # Useful Test Kitchen Commands
 
 - `login`: Allows you to easily log in using the credentials created when you ran `bundle exec kitchen create`.
-- `test`: Runs all the Test Kitchen stages starting with create through destroy to easily allow you to go through a full clean test run.
+- `test`: Runs all the Test Kitchen stages starting with create through destroy to easily allow you to go through a full clean test run.
+
+# Converge
+
+For RHEL8, we use Ansible as our `provisioner`, which handles configuring the test target during the `kitchen converge` step. We use the `suites` attribute alongside `provisioner` to define what test targets we want to create and how to configure them:
+
+Let's take a quick look at the `provisioner` section:
+``` yaml
+provisioner:
+  name: ansible_playbook
+  hosts: all
+  require_chef_for_busser: false
+  require_ruby_for_busser: false
+  ansible_binary_path: /usr/local/bin
+  # require_pip3: true
+  ansible_verbose: true
+  roles_path: spec/ansible/roles
+  galaxy_ignore_certs: true
+  requirements_path: spec/ansible/roles/requirements.yml
+  requirements_collection_path: spec/ansible/roles/requirements.yml
+  ansible_extra_flags: <%= ENV['ANSIBLE_EXTRA_FLAGS'] %>
+
+suites:
+  - name: vanilla
+    provisioner:
+      playbook: spec/ansible/roles/ansible-role-rhel-vanilla.yml
+    driver:
+      tags:
+        Name: Vanilla-<%= ENV['USER'] %>
+        CreatedBy: test-kitchen
+
+  - name: hardened
+    provisioner:
+      playbook: spec/ansible/roles/ansible-role-rhel-hardened.yml
+    driver:
+      tags:
+        Name: Hardened-<%= ENV['USER'] %>
+        CreatedBy: test-kitchen
+```
+
+There are quite a few variables set here to make Ansible run correctly, but the really important ones are:
+- `name: ansible_playbook`, which informs Kitchen that we are using Ansible in the first place, and
+- `roles_path: spec/ansible/roles`, which tells Ansible where the roles we want to use on our target will live.
+
+Meanwhile, in `suites`, we invoke the `provisioner` attribute again to add one more line to the `provisioner` data for each individual item in the suite. We tell Kitchen that the playbook for the `hardened` test target lives in `spec/ansible/roles/ansible-role-rhel-hardened.yml`, and the `vanilla` playbook lives in `spec/ansible/roles/ansible-role-rhel-vanilla.yml`.
+
+## The `spec` folder
+
+By convention, we drop off all of our configuration management artifacts, including our Ansible content, in the `spec` directory inside the profile (this folder is for defining the 'specification' we want our test suite to meet).
+
+If we look inside `spec/ansible/roles`, we see that we have a role for both of the hardened and vanilla test targets we want. We also see a `requirements.yml` file, which defines extra content outside of the `spec` directory we will be referencing in these roles (we also told Kitchen about these dependencies using `requirements_path: spec/ansible/roles/requirements.yml`).
+
+That `requirements.yml` file includes a dependency for a piece of code in the MITRE SAF [Hardening Library](https://saf.mitre.org/libs/harden):
+
+``` yaml
+roles:
+  - name: rhel8STIG
+    src: https://github.com/ansible-lockdown/RHEL8-STIG
+```
+
+### The Playbooks
+
+The `ansible-role-rhel-vanilla.yml` file in the `spec/ansible/roles` directory describes what role we want run against our vanilla target:
+
+``` yaml
+---
+- hosts:
+    - localhost
+  roles:
+    - roles/ansible-role-rhel-vanilla
+  serial: 50
+```
+
+It's a pretty short playbook. All we really want to do is run a single role that we have written specifically for the vanilla test target. If you're curious, all that particular role really does is run a quick `yum update` to ensure that our test target is nice and up to date.
+
+::: tip
+Remember that the whole point of a vanilla test target is to see what happens when we run our profile against a target that is fresh out of the box. That's why we aren't doing anything to it besides a quick package update.
+:::
+
+The `ansible-role-rhel-hardened.yml` file, meanwhile, references that dependency from the `requirements.yml` file:
+
+``` yaml
+---
+- hosts:
+    - localhost
+  roles:
+    - roles/ansible-role-rhel-vanilla
+    - roles/rhel8STIG
+  serial: 50
+  become: yes
+  vars:
+    rhel8stig_bootloader_password_hash: "changethispassword"
+    rhel_08_040123: false # RHEL 8 must mount /tmp with the nodev option.
+    rhel_08_040124: false # RHEL 8 must mount /tmp with the nosuid option.
+    rhel_08_040125: false # RHEL 8 must mount /tmp with the noexec option.
+    rhel_08_010380: false # RHEL 8 must require users to provide a password for privilege escalation.
+```
+
+Note that actually run two roles here: the local `vanilla` role, which runs a quick `yum update`, and then the STIG-hardening code from the MITRE SAF Hardening Library.
+
+So, ultimately, we have:
+- told Kitchen to invoke Ansible
+- told Ansible to look in the `spec` folder for a playbook on how to configure each test target
+- told the playbook to go grab already-written hardening content from the Hardening Library for the `hardened` target
+
+Kitchen and Ansible can thus work together to converge both test targets to their desired state.

src/courses/profile-dev-test/05.md

@@ -25,6 +25,10 @@ author: Aaron Lippold
   hardened-rhel-8  Ec2     AnsiblePlaybook  Inspec    Ssh        Verified     None
 ```
 
+::: warning
+If you instead see `Dummy` under the `Driver` column, it means you forgot to set the `KITCHEN_LOCAL_YAML` envar as described in the previous page.
+:::
+
 7. Create a kitchen instance: `bundle exec kitchen create vanilla`.
 
 ```shell
@@ -36,6 +40,8 @@ author: Aaron Lippold
 -----> Test Kitchen is finished. (0m1.21s)
 ```
 
+Kitchen has now done the legwork to create a brand-new, unconfigured EC2 instance for us.
+
 8. Converge the kitchen instance: `bundle exec kitchen converge`.
 
 ```shell
@@ -54,6 +60,8 @@ author: Aaron Lippold
 -----> Test Kitchen is finished. (1m13.52s)
 ```
 
+We now have an EC2 that has been configured to match the spec we gave to Kitchen.
+
 9. Run InSpec on the kitchen instance: `bundle exec kitchen verify`.
 
 ```shell

src/courses/profile-dev-test/06.md

@@ -7,6 +7,12 @@ author: Aaron Lippold
 
 # Docker Suite Setup
 
+Let's take a look at one of our other test suites, the Docker suite.
+
+::: Note Why Run Both?
+We define multiple test suites because we want to know that our profile works **regardless of what kind of deployment we run it against.** That's why the pipeline runs _both_ a Docker and an EC2 full VM test suite.
+:::
+
 1. Make sure Docker or Podman is running
 2. Login to your docker registry
 3. Clone the repository
@@ -28,6 +34,12 @@ vanilla-ubi8   Dokken  Dummy        Inspec    Dokken     <Not Created>  <None>
 hardened-ubi8  Dokken  Dummy        Inspec    Dokken     <Not Created>  <None>
 ```
 
+::: note Why is 'Provisioner' set to Dummy for these targets?
+Because these are containers. We assume that we are using immutable container images -- one 'vanilla,' one 'hardened' -- for testing.
+
+You don't really do "configuration management" on a live container -- you just tear it down, update the container image, and redeploy it. Therefore, we don't really need to know how hardening functions on the container.
+:::
+
 11. Create the kitchen instance: `bundle exec kitchen create vanilla`
 
 ```shell
@@ -73,7 +85,7 @@ Test Summary: 0 successful, 4 failures, 0 skipped
 
 ## This error is just fine
 
-The error below is just Test Kitchen telling you that not all of the Contrls in the profile passed.
+The error below is just Test Kitchen telling you that not all of the Controls in the profile passed.
 
 ```shell
 >>>>>> ------Exception-------
@@ -85,6 +97,14 @@ The error below is just Test Kitchen telling you that not all of the Contrls in
 >>>>>> Also try running `kitchen diagnose --all` for configuration
 ```
 
+::: Warning Wait, isn't that bad?!
+For most use cases for Test Kitchen, yes. For us, no.
+
+Remember that our goal here is to make sure our InSpec profile _tests accurately_. As long as the failed tests are true failures, then our profile is working as expected.
+
+In fact, failed tests are good in this context -- it means we correctly caught a misconfiguration in our container!
+:::
+
 14. For steps that apply to making updates, patches, and updates to the profile, see the next section, [Updating the Profile](#updating-the-profile).
 15. Your InSpec scan results are located in the `./spec/results/` directory, named `./spec/results/ubi-8_*.`
 16. Use Heimdall Lite to load both the `hardened` and `vanilla` results to ensure your changes and updates, "failed as expected and passed as expected and covered your corner cases."

src/courses/profile-dev-test/21.md

@@ -91,6 +91,8 @@ provisioner:
 
 This section configures the provisioner, which is the tool that brings your system to the desired state. Here, it's using Ansible playbooks. The various options configure how Ansible is run, such as the path to the Ansible binary (`ansible_binary_path: /usr/local/bin`), whether to require pip3 (`require_pip3: true`), and the path to the roles and requirements files.
 
+For more detail on how Kitchen and Ansible are configured in this section, see the section for [Test your Test Environment](04.md).
+
 ```yaml
 lifecycle:
   pre_converge:

src/courses/profile-dev-test/README.md

@@ -1,17 +1,56 @@
 ---
 order: 1
 next: 02.md
-title: Development & Testing InSpec Profile
+title: Developing And Testing InSpec Profiles
 author: Aaron Lippold
 ---
 
+Prior courses have been focused on the process of writing InSpec profile _code_, i.e. individual tests and resources. The following content is concerned with development best practices for your workflow for producing or updating an entire InSpec profile. The MITRE SAF team follows this process whenever we write a new profile or conduct a major overhaul of one to match an update to the underlying benchmark. 
+
+You may note that much of this content is really just describing DevSecOps best practices. This is not a coincidence.
+
 # Overview
 
-The development and testing of profiles is accomplished by a  variety of tools including: Ruby, the Test Kitchen suite, InSpec compliance language, Ansible, Docker, and shell scripting (bash/zsh). To contribute with Pull Requests and fixes, you'll need to set up your local test suite following the instructions provided below.
+Developing and testing profiles requires a variety of software tools, including (but not limited to): 
+
+- Ruby
+- Progress Chef's [Test Kitchen](https://docs.chef.io/workstation/kitchen/) suite
+- [InSpec](https://inspec.io) itself
+- [Ansible](https://www.ansible.com) (or your configuration management orchestration tool of choice)
+- [Docker](https://www.docker.com)
+- Good old fashioned shell scripting (bash/zsh)
+
+Our profiles, and any supporting automation content to support them, are hosted by [MITRE's GitHub](https://github.com/mitre) organization. To contribute to these repos with Pull Requests and fixes, you'll need to set up your local test suite following the instructions provided below.
 
 Our development and testing workflow is managed by Test Kitchen. This tool is integral to our GitHub Actions CI/CD Pipelines and is also used for local development, testing, and releasing updates, patches, and full releases of the profile.
 
-Test Kitchen uses Docker (or Podman, if preferred) and AWS (using free-tier resources) for testing. We provide example files for testing on a local Vagrant Red Hat (or other RHEL variant) box in the repository.
+Test Kitchen uses Docker (or Podman, if preferred) and AWS (using free-tier resources) for testing. For convenience, we provide example files for testing on a local Vagrant Red Hat (or other RHEL variant) box in the repository.
 
 Additionally, Test Kitchen uses the Red Hat hardened `ubi8 base container` from Platform One for testing. To test the hardened container portion of the testing suite, you'll need to set up and log in to your P1 Free account, then obtain a CLI token to pull the Platform One Iron Bank Red Hat Enterprise Linux 8 Universal Base Image (P1 IB UBI8) image into the test suite.
 
+## Running The Pipeline Locally
+
+The pipeline will, of course, run when code is pushed to the remote GitHub repository where it "lives." However, note that you can (and should!) use the pipeline code to construct your own test resources from your local development environment to play around with.
+
+The Test Kitchen files we discuss in this guide, for example, can be executed from your laptop just as well as on a GitHub action runner node. 
+
+## Examples in this guide
+
+This guide will be describing the workflow that is in place around MITRE SAF's [Red Hat 8 STIG InSpec profile](https://github.com/mitre/redhat-enterprise-linux-8-stig-baseline) as a convenient example. We encourage anyone interested in contributing to one of our profiles, or writing one of their own, to poke around that repo and examine our test suite files in that repo. This guide will explain how they all work together to support the overall profile.
+
+## Why Bother?
+
+> Quis custodiet ipsos custodes?
+
+        - Juvenal, Satire VI, lines 347–348, 1st-2nd century CE
+
+> Who InSpecs the InSpec-ers?
+
+        - MITRE SAF developer, five seconds after running test with bugs on lines 347-348, 2024 CE
+
+
+You may be wondering why we develop all this automation content to support InSpec profiles. The reason that we set up our development environment for an InSpec profile this way is because setting up a full DevSecOps CI/CD pipeline makes for better code and better tests. The reason that we build the CI/CD content directly into the profile repo, and publish the pipeline artifacts on GitHub where anybody can see them, is because we want to be able to _prove that they work._
+
+Remember that your users downstream who run your profile have no idea if the tests are necessarily accurate. We want to be able to point to evidence that InSpec produces expected results when run against a benchmarked configuration. That's why we want a full CI/CD pipeline set up for each profile; one that runs the profile against an off-the-shelf, completely fresh component (the "vanilla" test) and one that runs against a component that we have run hardening content against (the "hardened" test). Note also that we get the added benefit of testing whatever hardening content we use to produce the hardened test target.
+
+This all hopefully reinforces the general theme of the MITRE SAF trainings on this site - it's not enough to write some slapdash test code to check if your system matches the configuration you think it should have and call it a day. Automation code needs to be _well-documented_, _easy to maintain_ and above all _proven to work_ for it to be valuable long-term.