background：AddressSanitizer (ASan) can effectively detect a variety of memory errors at runtime. However, studies have shown that ASan incurs extremely high performance overhead: its CPU overhead is approximately 2×, code size overhead is around 20×, and memory overhead is also significant, at roughly 2.5×. In recent years, many state-of-the-art techniques have emerged in the performance optimization of ASan, aiming to mitigate the performance overhead introduced by ASan. Most research focuses on reducing ASan’s CPU overhead, while some work targets lowering its memory overhead. This project aims to conduct horizontal (across different techniques and tools) and vertical (evolution of similar techniques) analyses of the current state-of-the-art research, and theoretically analyze the pros, cons, and applicability of various approaches. Furthermore, we experimentally compare and evaluate state-of-the-art runtime overhead optimization techniques for ASan, and fairly assess their actual performance on unified benchmark datasets and large-scale real-world applications.
This warehouse primarily stores relevant tools used in empirical research, as well as related experimental datasets.
Workloads and Benchmarks
- SPEC CPU2006
- CVEs
- Juliet Test Case
- MSET
Tools：
- Asan
- Asan--
- GiantSan
- Sanrazor
- LFP
- ASAP
- MEDs
- RSan
Metrics and Measurement：
- Runtime overhead
- Memory overhead
- Binary Code-Size Growth
- Vulnerability Detection
We experimentally evaluate the tools using the SPEC CPU2006 benchmark suite. All tools are executed under their default configurations to ensure a fair comparison.
The overall performance comparison across different tools is summarized in the figure below.
As shown in the figure, different optimization techniques lead to varying levels of performance improvement. Some approaches significantly reduce the runtime overhead introduced by ASan, while others focus on reducing memory consumption or code size growth.
To better understand the behavior of each tool, we further present a detailed breakdown of the experimental results in the following sections.
Runtime overhead The runtime overhead of different tools on the SPEC CPU2006 benchmark suite is shown below.
| Programs | Native | ASan | R | ASan-- | R | LFP | R | ASAP | R | MEDs | R | SanRazor_L0 | R | SanRazor_L1 | R | SanRazor_L2 | R | GiantSan | R | RSan | R |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 433.milc | 176 | 283 | 160.80% | 222 | 126.14% | - | - | 215 | 122.16% | 229 | 130.11% | 202 | 114.77% | 197 | 111.93% | 190 | 107.95% | 259 | 147.16% | 172 | 97.73% |
| 444.namd | 169 | 335 | 198.22% | 305 | 180.47% | - | - | 180 | 106.51% | 259 | 153.25% | 234 | 138.46% | 216 | 127.81% | 198 | 117.16% | 235 | 139.05% | 242 | 143.20% |
| 450.soplex | 110 | 197 | 179.09% | 185 | 168.18% | - | - | 136 | 123.64% | - | - | - | - | - | - | - | - | 159 | 144.55% | 150 | 136.36% |
| 453.povray | 62 | 208 | 335.48% | 161 | 259.68% | - | - | 113 | 182.26% | - | - | 163 | 262.90% | 151 | 243.55% | 139 | 224.19% | 183 | 295.16% | 101 | 162.90% |
| 470.lbm | 104 | 165 | 158.65% | 142 | 136.54% | 127 | 122.12% | 103 | 99.04% | 118 | 113.46% | 119 | 114.42% | 105 | 100.96% | 104 | 100.00% | 117 | 112.50% | 105 | 100.96% |
| 482.sphinx3 | 233 | 395 | 169.53% | 368 | 157.94% | 389 | 166.95% | 250 | 107.30% | - | - | 383 | 164.38% | 289 | 124.03% | 287 | 123.18% | 282 | 121.03% | 288 | 123.61% |
| 400.perlbench | 147 | 637 | 433.33% | 549 | 373.47% | - | - | - | - | - | - | - | - | - | - | 550 | 374.15% | - | - | 294 | 200.00% |
| 401.bzip2 | 271 | 440 | 162.36% | 400 | 147.60% | 414 | 152.77% | 295 | 108.86% | 423 | 156.09% | 417 | 153.87% | 389 | 143.54% | 388 | 143.17% | 348 | 128.41% | 416 | 153.51% |
| 403.gcc | 118 | 372 | 315.25% | 353 | 299.15% | - | - | 302 | 255.93% | - | - | - | - | - | - | - | - | 317 | 268.64% | 302 | 255.93% |
| 429.mcf | 149 | 205 | 137.58% | 189 | 126.85% | 185 | 124.16% | 128 | 85.91% | 196 | 131.54% | 181 | 121.48% | 138 | 92.62% | 152 | 102.01% | 166 | 111.41% | 186 | 124.83% |
| 445.gobmk | 252 | 445 | 176.59% | 369 | 146.43% | 352 | 139.68% | 297 | 117.86% | 404 | 160.32% | 412 | 163.49% | 380 | 150.79% | 375 | 148.81% | 410 | 162.70% | 363 | 144.05% |
| 456.hmmer | 151 | 415 | 274.83% | 340 | 225.17% | 409 | 270.86% | 186 | 123.18% | - | - | 296 | 196.03% | 190 | 125.83% | 190 | 125.83% | 224 | 148.34% | 197 | 130.46% |
| 458.sjeng | 271 | 479 | 176.75% | 400 | 147.60% | 389 | 143.54% | - | - | 457 | 168.63% | 513 | 189.30% | 468 | 172.69% | 469 | 173.06% | 434 | 160.15% | 419 | 154.61% |
| 462.libquantum | 154 | 223 | 144.81% | 209 | 135.71% | 177 | 114.94% | 154 | 100.00% | - | - | 1215 | 139.61% | 193 | 125.32% | 192 | 124.68% | 155 | 100.65% | 149 | 96.75% |
| 464.h264ref | 206 | 482 | 233.98% | 433 | 210.19% | - | - | - | - | - | - | - | - | - | - | - | - | 359 | 174.27% | 319 | 154.85% |
| 473.astar | 222 | 335 | 150.90% | 289 | 130.18% | 334 | 150.45% | 228 | 102.70% | - | - | - | - | - | - | - | - | 230 | 103.60% | 296 | 133.33% |
| 483.xalancbmk | 86.5 | 251 | 290.17% | 227 | 262.43% | - | - | - | - | - | - | - | - | - | - | - | - | 170 | 196.53% | - | - |
| geomean | - | - | 217.55% | - | 190.22% | - | 153.94% | - | 125.79% | - | 144.77% | - | 159.88% | - | 138.10% | - | 135.46% | - | 169.90% | - | 144.57% |
Memory overhead The memory overhead of different tools on the SPEC CPU2006 benchmark suite is shown below.
| Programs | Native | ASan | R | ASan-- | R | LFP | R | ASAP | R | MEDs | R | SanRazor_L0 | R | SanRazor_L1 | R | SanRazor_L2 | R | GiantSan | R | RSan | R |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 433.milc | 679 | 959 | 141.24% | 958 | 141.09% | - | - | 959 | 141.24% | 699 | 102.95% | 958 | 141.09% | 957 | 140.94% | 958 | 141.09% | 1045 | 153.90% | 1519.51 | 223.79% |
| 444.namd | 148 | 183 | 123.65% | 174 | 117.57% | - | - | 63 | 42.57% | 132 | 89.19% | 122 | 82.43% | 122 | 82.43% | 121 | 81.76% | 163 | 110.14% | 121.25 | 81.93% |
| 450.soplex | 422 | 827 | 195.97% | 825 | 195.50% | - | - | 829 | 196.45% | - | - | - | - | - | - | - | - | 895 | 212.09% | 1417.25 | 335.84% |
| 453.povray | 112 | 231 | 206.25% | 231 | 206.25% | - | - | 227 | 202.68% | - | - | 232 | 207.14% | 232 | 207.14% | 233 | 208.04% | 232 | 207.14% | 236.75 | 211.38% |
| 470.lbm | 410 | 465 | 113.41% | 465 | 113.41% | 410 | 100.00% | 464 | 113.17% | 416 | 101.46% | 463 | 112.93% | 464 | 113.17% | 463 | 112.93% | 467 | 113.90% | 442.5 | 107.93% |
| 482.sphinx3 | 83 | 420 | 506.02% | 421 | 507.23% | 60 | 72.29% | 423 | 509.64% | - | - | 402 | 484.34% | 402 | 484.34% | 402 | 484.34% | 406 | 489.16% | 568.43 | 684.86% |
| 400.perlbench | 662 | 2090 | 315.71% | 2089 | 315.56% | - | - | - | - | - | - | - | - | - | - | 1996 | 301.51% | - | - | 5096.71 | 769.90% |
| 401.bzip2 | 849 | 872 | 102.71% | 872 | 102.71% | 848 | 99.88% | 872 | 102.71% | 854 | 100.59% | 872 | 102.71% | 871 | 102.59% | 871 | 102.59% | 2133 | 251.24% | 3765.75 | 443.55% |
| 403.gcc | 886 | 2794 | 315.35% | 2793 | 315.24% | - | - | 2786 | 314.45% | - | - | - | - | - | - | - | - | 1687 | 190.41% | 13517.34 | 1525.66% |
| 429.mcf | 1677 | 1889 | 112.64% | 1889 | 112.64% | 1678 | 100.06% | 1889 | 112.64% | 1683 | 100.36% | 1889 | 112.64% | 1889 | 112.64% | 1888 | 112.58% | 1892 | 112.82% | 1718 | 102.44% |
| 445.gobmk | 150 | 366 | 244.00% | 369 | 246.00% | 119 | 79.33% | 362 | 241.33% | 125 | 83.33% | 371 | 247.33% | 370 | 246.67% | 370 | 246.67% | 374 | 249.33% | 1575.25 | 1050.17% |
| 456.hmmer | 87 | 458 | 526.44% | 465 | 534.48% | 70 | 80.46% | 457 | 525.29% | - | - | 413 | 474.71% | 423 | 486.21% | 423 | 486.21% | 417 | 479.31% | 938 | 1078.16% |
| 458.sjeng | 175 | 179 | 102.29% | 179 | 102.29% | 177 | 101.14% | - | - | 182 | 104.00% | 180 | 102.86% | 179 | 102.29% | 180 | 102.86% | 205 | 117.14% | 211.5 | 120.86% |
| 462.libquantum | 97 | 358 | 369.07% | 357 | 368.04% | 97 | 100.00% | 358 | 369.07% | - | - | 357 | 368.04% | 356 | 367.01% | 357 | 368.04% | 368 | 379.38% | 348 | 358.76% |
| 464.h264ref | 109 | 330 | 302.75% | 324 | 297.25% | - | - | - | - | - | - | - | - | - | - | - | - | 326 | 299.08% | 794 | 728.44% |
| 473.astar | 326 | 976 | 299.39% | 976 | 299.39% | 338 | 103.68% | 971 | 297.85% | - | - | - | - | - | - | - | - | 833 | 255.52% | 1927.82 | 591.35% |
| 483.xalancbmk | 418 | 1589 | 380.14% | 1587 | 379.67% | - | - | - | - | - | - | - | - | - | - | - | - | 1405 | 336.12% | - | - |
| geomean | - | - | 256.30% | - | 256.14% | - | 92.98% | - | 243.78% | - | 97.41% | - | 221.47% | - | 222.31% | - | 222.46% | - | 250.48% | - | 525.94% |
Binary code-size growth
The binary size growth of different tools on the SPEC CPU2006 benchmark suite is shown below.
| Programs | Native | ASan | R | ASan-- | R | LFP | R | ASAP | R | MEDs | R | SanRazor_L0 | R | SanRazor_L1 | R | SanRazor_L2 | R | GiantSan | R | RSan | R |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 433.milc | 0.12 | 1.48 | 1233.33% | 1.34 | 1116.67% | - | - | 1.33 | 1108.33% | 1.48 | 1233.33% | 1.42 | 1183.33% | 1.4 | 1166.67% | 1.4 | 1166.67% | 1.29 | 1075.00% | 0.22 | 183.33% |
| 444.namd | 0.28 | 1.94 | 692.86% | 1.82 | 650.00% | - | - | 1.66 | 592.86% | 1.78 | 635.71% | 1.77 | 632.14% | 1.73 | 617.86% | 1.72 | 614.29% | 1.88 | 671.43% | 0.45 | 160.71% |
| 450.soplex | 0.37 | 2.52 | 681.08% | 2.29 | 618.92% | - | - | 2.19 | 591.89% | - | - | - | - | - | - | - | - | 1.9 | 513.51% | 0.94 | 254.05% |
| 453.povray | 1.02 | 4.44 | 435.29% | 3.45 | 338.24% | - | - | 3.64 | 356.86% | - | - | 3.7 | 362.75% | 3.61 | 353.92% | 3.58 | 350.98% | 3.11 | 304.90% | 2.64 | 258.82% |
| 470.lbm | 0.01 | 1.12 | 11200.00% | 1.12 | 11200.00% | 0.05 | 500.00% | 1.03 | 10300.00% | 1.15 | 11500.00% | 1.12 | 11200.00% | 1.12 | 11200.00% | 1.12 | 11200.00% | 1.1 | 11000.00% | 0.03 | 300.00% |
| 482.sphinx3 | 0.18 | 1.78 | 988.89% | 1.64 | 911.11% | 0.44 | 244.44% | 1.57 | 872.22% | - | - | 1.65 | 916.67% | 1.6 | 888.89% | 1.58 | 877.78% | 1.49 | 827.78% | 0.38 | 211.11% |
| 400.perlbench | 1.03 | 4.92 | 477.67% | 3.78 | 366.99% | - | - | - | - | - | - | - | - | - | - | - | - | 3.36 | 326.21% | 3.04 | 295.15% |
| 401.bzip2 | 0.08 | 1.49 | 1862.50% | 1.32 | 1650.00% | 0.22 | 275.00% | 1.26 | 1575.00% | 1.38 | 1725.00% | 1.32 | 1650.00% | 1.25 | 1562.50% | 1.25 | 1562.50% | 1.74 | 2175.00% | 0.19 | 237.50% |
| 403.gcc | 3.34 | 11.25 | 336.83% | 8.53 | 255.39% | - | - | 9.05 | 270.96% | - | - | - | - | - | - | - | - | 7.3 | 218.56% | 9.14 | 273.65% |
| 429.mcf | 0.01 | 1.13 | 11300.00% | 1.12 | 11200.00% | 0.06 | 600.00% | 1.04 | 10400.00% | 1.16 | 11600.00% | 1.13 | 11300.00% | 1.12 | 11200.00% | 1.12 | 11200.00% | 1.11 | 11100.00% | 0.03 | 300.00% |
| 445.gobmk | 1.37 | 5.3 | 386.86% | 4.69 | 342.34% | 2.21 | 161.31% | 4.34 | 316.79% | 4.88 | 356.20% | 4.51 | 329.20% | 4.31 | 314.60% | 4.29 | 313.14% | 4.31 | 314.60% | 4.83 | 352.55% |
| 456.hmmer | 0.29 | 2.22 | 765.52% | 1.97 | 679.31% | 0.78 | 268.97% | 1.94 | 668.97% | - | - | 2.05 | 706.90% | 2.03 | 700.00% | 2.03 | 700.00% | 1.68 | 579.31% | 0.41 | 141.38% |
| 458.sjeng | 0.12 | 1.61 | 1341.67% | 1.4 | 1166.67% | 0.28 | 233.33% | - | - | 1.54 | 1283.33% | 1.49 | 1241.67% | 1.46 | 1216.67% | 1.46 | 1216.67% | 1.33 | 1108.33% | 0.26 | 216.67% |
| 462.libquantum | 0.04 | 1.19 | 2975.00% | 1.17 | 2925.00% | 0.1 | 250.00% | 1.09 | 2725.00% | - | - | 1.18 | 2950.00% | 1.18 | 2950.00% | 1.17 | 2925.00% | 1.14 | 2850.00% | 0.04 | 100.00% |
| 464.h264ref | 0.6 | 4.46 | 743.33% | 3.54 | 590.00% | - | - | - | - | - | - | - | - | - | - | - | - | 2.24 | 373.33% | 1.67 | 278.33% |
| 473.astar | 0.04 | 1.27 | 3175.00% | 1.22 | 3050.00% | 0.13 | 325.00% | 1.15 | 2875.00% | - | - | - | - | - | - | - | - | 1.16 | 2900.00% | 0.07 | 175.00% |
| 483.xalancbmk | 4.02 | 11.82 | 294.03% | 10.01 | 249.00% | - | - | - | - | - | - | - | - | - | - | - | - | 7.53 | 187.31% | - | - |
| geomean | - | - | 2287.64% | - | 2194.68% | - | 317.56% | - | 2332.42% | - | 4047.66% | - | 2952.06% | - | 2924.65% | - | 2920.64% | - | 2148.55% | - | 233.64% |
Vulnerability detection capability
These analyses provide a comprehensive comparison of state-of-the-art ASan optimization techniques.