BinSkim checks identical files every time, needs to deduplicate by file hash

[KirillOsenkov]

If I have 300 identical .dlls in the inputs, they will be scanned 300 times.

BinSkim (or sarif-sdk) should deduplicate by file hash such as SHA256, scan each file only once, and reuse results.

[danmoseley]

looking at dotnet/dotnet outputs

• 63,856 total artifacts (all scanned, all hashed)
• 11,983 unique SHA-256 hashes
• 51,873 duplicates (81.2%) — wasted work
• Top offender: System.Runtime.CompilerServices.Unsafe.dll at 189 copies

[danmoseley]

Looks like SARIF does calculate hashes (for downstream pipeline correlation, reporting etc I guess) but no longer de dupes

https://github.com/microsoft/sarif-sdk/pull/2624/changes#diff-dc29abf67f9853673774977ab7e753c3d292701f022b9a5503e1d916643849a1R343-R354
microsoft/sarif-sdk#2620

[danmoseley]

On dotnet/dotnet this would only save 2-3 mins because even 64K files don't take that long. And the build takes hours overall...