Heimdall Result doesn't match OpenSCAP HTML

[gmisura]

The Heimdall result says Compliance Level: 51.31%, Passed: 175, Failed: 127 (High: 1, Medium: 118, Low:8), N/A: 34, Not Reviewed: 36, Profile Error: 13

• using the https://github.com/mitre/amazon-linux-2023-stig-ready-baseline

The OpenSCAP HTML says: 75.05%, Passed: 183, Failed: 122 (High:2, Medium: 105, Low:12, Other:3), Other: 29

• using the "CIS Amazon Linux 2023 Benchmark for Level 2 - Server"

Clearly the percentages are being calculated differently because the pass/failed numbers are not that different.

Which number is "correct"? Which will a DoD Auditor accept?

[Amndeep7]

Hello @gmisura.

1. Your comparison is between a CIS benchmark and some reference/example InSpec for a WIP attempt at developing STIG-ready content for Amazon Linux 2023. We'd begun the collaboration process with Amazon to help them develop a STIG, but it hasn't been completed yet. As such, you can see that the InSpec and Readme content in that repo still refers to RHEL 8. Please keep that in mind w/r to the exact numbers from running that InSpec scan.
2. Talking about numbers, please note that CIS benchmarks and STIGs are independent security guidance documents created by separate entities so comparing them is an apples/oranges situation w/r to the total number of controls, assigned severity to a given control, etc.
3. The calculations are different as you can see on Heimdall itself: our calculation is [Passed/(Passed + Failed + Not Reviewed + Profile Error) * 100]. Additionally, we've often found that SCAP content often does not include every requirement required by the security guidance whereas our InSpec content does even if we don't have an automated means of assessment for that control. Consequently, the SCAP scores are often artificially higher than InSpec's/Heimdall's since it's a percentage score for the content that it assesses, not all the content that has to be assessed. We believe our number ends up more "correct" than SCAP's for that reason.
4. This is a discussion that you will need to have with your ISSO/ISSM/AO to determine what security guidance is applicable and will be used to assess your software system. The particular score that you get from your assessment tool of choice is merely a quick means of understanding your security posture. Your assessor will (or should) be vastly more interested in the actual findings of your tools.

If you are interested in having a longer conversation, please reach out to amann@mitre.org and I can arrange a meeting to go more in-depth and answer any other questions you might have.

[gmisura]

@Amndeep7

Thanks for the info!

Any date for AL2023?

As the person in the details, I can see the differences. I'm having a hard time explaining this to management. They want "all issues fixed." So they want to focus on OpenSCAP results because of the higher compliance percentage. I want to focus on being more secure.

Our 3PAO says CIS is "fine." We'll see.

[Amndeep7]

@gmisura

We are not actively working on AL2023 at this moment so I cannot give you a timeline for its completion. If you are aware of a government agency that is utilizing AL2023 or has a contractor that is utilizing it, please let us know and we could start having discussions to accelerate and complete this work.

CIS vs STIG (or STIG-ready content) is not a quality determination - just an originating source and underlying security guidance determination. DOD agencies are required to use STIGs where STIGs exist, but often use CIS benchmarks where STIGs are not available.

We'd be more than willing to have a conversation with you and your management chain to help explain that "all issues fixed" means that you have to actually fix all the issues, not just the issues that SCAP gives you a failing assessment on (which are not an equivalent set of issues).

In fact, we have several InSpec profiles that are built off of CIS' security guidance. You're more than welcome to compare the results you get from one of those InSpec profiles and compare them to your SCAP results. This might be the best way to showcase that your SCAP results are possibly not as complete as what you would get from InSpec.

Another benefit of using InSpec is that we can write overlay profiles that can explain why certain findings are mitigated or not applicable to your particular usecase or deployment environment. Inputs to an InSpec profile can similarly be tailored so as to do more accurate testing.