How UTopia generate code coverage?

[AxelHowe]

Hello,

I encountered an issue while trying to reproduce evaluation

I want to check the code coverage and I'm not sure if my approach is correct.

I followed the steps below, using assimp as an example:

python3 -m helper.make assimp
python3 -m helper.build assimp

I executed the fuzz driver VTAdjacencyTest_unreferencedVerticesSet_Test to obtain default.profraw

cd exp/assimp/output/profiles/
./VTAdjacencyTest_unreferencedVerticesSet_Test

llvm-profdata-10 merge -sparse default.profraw -o test.profdata
llvm-cov-10 report ./VTAdjacencyTest_unreferencedVerticesSet_Test -instr-profile=test.profdata

The report shows that the code coverage is 0%.

I tested other projects in the same way and got the same result.

Filename                                                              Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
code/AssetLib/3DS/3DSConverter.cpp                                        281               281     0.00%          10                10     0.00%         681               681     0.00%
code/AssetLib/3DS/3DSExporter.cpp                                         146               146     0.00%          19                19     0.00%         448               448     0.00%
code/AssetLib/3DS/3DSHelper.h                                              21                21     0.00%          18                18     0.00%          89                89     0.00%
code/AssetLib/3DS/3DSLoader.cpp                                           572               572     0.00%          27                27     0.00%        1171              1171     0.00%
code/AssetLib/3MF/D3MFExporter.cpp                                        114               114     0.00%          20                20     0.00%         311               311     0.00%

... 

include/assimp/commonMetaData.h                                             0                 0         -           0                 0         -           0                 0         -
include/assimp/version.h                                                    0                 0         -           0                 0         -           0                 0         -
test/unit/ImportExport/MDL/MDLHL1TestFiles.h                                0                 0         -           0                 0         -           0                 0         -
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                   96166             96166     0.00%        8372              8372     0.00%      150380            150380     0.00%

I would like to know how UTopia calculates code coverage.

Thank you for your response.

[AxelHowe]

Additionally, I have another question.

Why do some fuzz driver executions result in outputs like this:

root@b2e10585dfcb:/home/exp/libvpx/output/profiles# ./ByteAlignmentTest_SwitchByteAlignment_Test | less
../test/webm_video_source.h:43: Failure
Value of: vpx_ctx_->file != NULL
  Actual: false
Expected: true
Input file open failed. Filename: vp90-2-02-size-lf-1920x1080.webm
../test/decode_test_driver.h:74: Failure
Expected equality of these values:
  expected_value
    Which is: 0
  res
    Which is: 8
Invalid parameter
../test/webm_video_source.h:57: Failure
Value of: vpx_ctx_->file != NULL
  Actual: false
Expected: true
../test/decode_test_driver.h:74: Failure
Expected equality of these values:
  expected_value
    Which is: 0
  res
    Which is: 8
Invalid parameter
../test/webm_video_source.h:57: Failure
Value of: vpx_ctx_->file != NULL
  Actual: false
Expected: true
../test/decode_test_driver.h:74: Failure
Expected equality of these values:
  expected_value
    Which is: 0
  res
    Which is: 8
Invalid parameter
../test/webm_video_source.h:57: Failure
Value of: vpx_ctx_->file != NULL
  Actual: false
Expected: true

Shouldn't the correct execution look like the libFuzzer output?

root@b2e10585dfcb:/home/exp/libvpx/output/profiles# ./VP9_TestBitIO_Test
INFO: Seed: 2734834166
INFO: Loaded 1 modules   (128229 inline 8-bit counters): 128229 [0x1a12cc0, 0x1a321a5),
INFO: Loaded 1 PC tables (128229 PCs): 128229 [0x15e6570,0x17db3c0),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2      INITED cov: 158 ft: 159 corp: 1/1b exec/s: 0 rss: 50Mb
        NEW_FUNC[1/34]: 0xcd1e50 in std::vector<unsigned char, std::allocator<unsigned char> >::size() const /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_vector.h:916
        NEW_FUNC[2/34]: 0x129de20 in AutoFuzz::FuzzArgsProfile::~FuzzArgsProfile() /home/exp/libvpx/test/FuzzArgsProfile.pb.cc:116
#3      NEW    cov: 213 ft: 232 corp: 2/220b lim: 4096 exec/s: 0 rss: 52Mb L: 219/219 MS: 2 InsertRepeatedBytes-Custom-
#7      NEW    cov: 214 ft: 245 corp: 3/366b lim: 4096 exec/s: 0 rss: 52Mb L: 146/219 MS: 6 InsertRepeatedBytes-Custom-CustomCrossOver-CustomCrossOver-InsertByte-Custom-
#9      NEW    cov: 214 ft: 247 corp: 4/609b lim: 4096 exec/s: 0 rss: 52Mb L: 243/243 MS: 4 ChangeBit-Custom-CopyPart-Custom-
#11     NEW    cov: 214 ft: 248 corp: 5/628b lim: 4096 exec/s: 0 rss: 52Mb L: 19/243 MS: 2 CustomCrossOver-CustomCrossOver-
#12     NEW    cov: 214 ft: 249 corp: 6/666b lim: 4096 exec/s: 0 rss: 52Mb L: 38/243 MS: 1 CustomCrossOver-
#14     NEW    cov: 214 ft: 251 corp: 7/942b lim: 4096 exec/s: 0 rss: 52Mb L: 276/276 MS: 3 CustomCrossOver-CrossOver-Custom-
#15     NEW    cov: 214 ft: 252 corp: 8/1095b lim: 4096 exec/s: 0 rss: 52Mb L: 153/276 MS: 2 InsertRepeatedBytes-Custom-
#19     REDUCE cov: 214 ft: 252 corp: 8/1011b lim: 4096 exec/s: 0 rss: 52Mb L: 192/243 MS: 5 CustomCrossOver-ChangeBinInt-Custom-CustomCrossOver-CustomCrossOver-
#21     REDUCE cov: 214 ft: 252 corp: 8/992b lim: 4096 exec/s: 0 rss: 52Mb L: 173/243 MS: 4 ShuffleBytes-CrossOver-Custom-Custom-
#24     NEW    cov: 214 ft: 271 corp: 9/1025b lim: 4096 exec/s: 0 rss: 52Mb L: 33/243 MS: 7 ChangeBit-Custom-ChangeBit-Custom-CopyPart-ChangeBinInt-Custom-

I'm not sure if it's because I'm unfamiliar with libFuzzer. Apologies if this is a naive question, I'm new to fuzz testing.