VOOZH about

URL: https://www.fuzzbench.com/reports/2020-12-19-bug/index.html

⇱ FuzzBench: 2020-12-19-bug report (running)

FuzzBench: 2020-12-19-bug report

(experiment incomplete/still running...)
warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median bug-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached bug-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
aflplusplus_optimal 68.02
aflplusplus 67.54
eclipser 66.31
entropic 58.06
afl 56.43
aflsmart 54.17
mopt 51.51
libfuzzer 50.95
aflfast 50.56
fairfuzz 41.63
honggfuzz 39.25
By avg. rank
average rank
fuzzer
afl 4.94
eclipser 5.22
aflsmart 5.44
mopt 5.83
entropic 5.89
aflfast 6.00
aflplusplus_optimal 6.00
aflplusplus 6.11
libfuzzer 6.78
fairfuzz 6.89
honggfuzz 6.89
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median bug-coverages on each benchmark
    fuzzer afl aflfast aflplusplus aflplusplus_optimal aflsmart eclipser entropic fairfuzz honggfuzz libfuzzer mopt
    benchmark
    arrow_parquet-arrow-fuzz 38.0 36.0 18.0 18.0 38.0 37.0 11.0 29.0 1.0 32.0 40.0
    harfbuzz_hb-subset-fuzzer 30.0 30.0 15.0 16.0 29.0 29.0 22.0 19.0 25.0 18.0 29.0
    libhevc_hevc_dec_fuzzer 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    matio_matio_fuzzer 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    openexr_openexr_exrenvmap_fuzzer 0.0 0.0 1.0 1.0 0.0 1.0 1.0 0.0 0.0 1.0 0.0
    openh264_decoder_fuzzer 3.0 3.0 0.0 0.0 3.0 3.0 5.0 0.0 6.0 5.0 3.0
    php_php-fuzz-execute 0.0 0.0 2.0 2.0 0.0 0.0 1.0 1.0 0.0 0.0 0.0
    php_php-fuzz-parser-2020-07-25 2.0 1.0 4.0 4.0 1.5 1.0 2.0 2.0 0.0 0.0 1.0
    stb_stbi_read_fuzzer 9.0 8.0 7.0 7.0 9.0 9.0 2.0 5.0 8.0 3.0 8.0

arrow_parquet-arrow-fuzz summary

Discovered bug coverage distribution
👁 Image
Reached code coverage distribution
👁 Image
Mean code coverage growth over time
👁 Image
Mean code coverage growth over time
👁 Image
Mean bug coverage growth over time
👁 Image
Mean bug coverage growth over time
👁 Image
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    mopt 82800 19.0 39.631579 2.165654 36.0 38.00 40.0 41.00 44.0
    afl 82800 16.0 37.625000 2.473190 33.0 35.75 38.0 39.25 41.0
    aflsmart 82800 16.0 38.562500 3.244868 34.0 37.00 38.0 40.00 48.0
    eclipser 82800 18.0 36.888889 2.398256 33.0 35.00 37.0 38.00 42.0
    aflfast 82800 16.0 37.500000 3.076795 33.0 35.00 36.0 40.00 43.0
    libfuzzer 82800 17.0 32.117647 6.881882 13.0 28.00 32.0 37.00 41.0
    fairfuzz 82800 18.0 29.055556 3.588744 22.0 26.50 29.0 31.00 37.0
    aflplusplus 82800 17.0 17.470588 1.374666 15.0 17.00 18.0 19.00 19.0
    aflplusplus_optimal 82800 12.0 17.916667 2.108784 13.0 17.50 18.0 20.00 20.0
    entropic 82800 16.0 9.937500 3.623419 2.0 7.75 11.0 12.00 15.0
    honggfuzz 82800 18.0 1.277778 0.460889 1.0 1.00 1.0 1.75 2.0

    Mann-Whitney U test
    👁 Image
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    mopt 82800 19.0 5214.947368 72.952400 5073.0 5161.00 5211.0 5268.50 5367.0
    afl 82800 16.0 5164.625000 86.198124 5028.0 5091.50 5154.5 5243.00 5278.0
    aflfast 82800 16.0 5149.250000 52.533799 5051.0 5130.75 5152.5 5180.00 5259.0
    aflsmart 82800 16.0 5124.562500 54.329818 5047.0 5085.50 5114.5 5138.00 5229.0
    eclipser 82800 18.0 5112.555556 80.908243 4975.0 5058.00 5096.0 5168.75 5255.0
    fairfuzz 82800 18.0 4534.055556 113.648826 4302.0 4473.50 4557.0 4600.50 4772.0
    libfuzzer 82800 17.0 4296.000000 71.333723 4189.0 4272.00 4296.0 4333.00 4488.0
    entropic 82800 16.0 4187.625000 64.362385 4090.0 4146.50 4174.5 4211.25 4315.0
    aflplusplus 82800 17.0 4111.470588 61.715190 3984.0 4066.00 4112.0 4162.00 4229.0
    aflplusplus_optimal 82800 12.0 4074.333333 48.612257 3993.0 4054.50 4067.0 4119.00 4158.0
    honggfuzz 82800 18.0 3833.666667 76.266560 3723.0 3778.00 3826.5 3870.75 3974.0

    Mann-Whitney U test
    👁 Image
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

harfbuzz_hb-subset-fuzzer summary

Discovered bug coverage distribution
👁 Image
Reached code coverage distribution
👁 Image
Mean code coverage growth over time
👁 Image
Mean code coverage growth over time
👁 Image
Mean bug coverage growth over time
👁 Image
Mean bug coverage growth over time
👁 Image
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 14.0 29.714286 2.301457 24.0 29.0 30.0 30.75 33.0
    aflfast 82800 17.0 30.058824 1.919329 27.0 29.0 30.0 32.00 34.0
    aflsmart 82800 17.0 29.235294 2.385619 26.0 27.0 29.0 31.00 33.0
    eclipser 82800 18.0 29.222222 2.184317 26.0 27.0 29.0 31.00 33.0
    mopt 82800 19.0 28.631579 1.977949 25.0 27.0 29.0 30.00 32.0
    honggfuzz 82800 19.0 24.789474 1.512134 22.0 24.0 25.0 25.50 28.0
    entropic 82800 18.0 22.888889 2.054805 20.0 22.0 22.0 24.75 27.0
    fairfuzz 82800 20.0 19.200000 4.111601 7.0 18.0 19.0 21.25 26.0
    libfuzzer 82800 17.0 17.882353 2.368606 12.0 17.0 18.0 19.00 21.0
    aflplusplus_optimal 82800 18.0 15.388889 2.703786 8.0 15.0 16.0 17.00 19.0
    aflplusplus 82800 18.0 15.166667 2.854305 10.0 13.0 15.0 16.00 21.0

    Mann-Whitney U test
    👁 Image
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    mopt 82800 19.0 10375.157895 55.904942 10280.0 10342.00 10377.0 10402.50 10504.0
    eclipser 82800 18.0 10326.555556 80.402826 10151.0 10299.25 10346.5 10375.75 10453.0
    afl 82800 14.0 10308.642857 93.880945 10124.0 10261.25 10308.0 10383.50 10436.0
    aflsmart 82800 17.0 10302.411765 92.281539 10101.0 10253.00 10298.0 10345.00 10464.0
    aflfast 82800 17.0 10255.352941 79.770719 10094.0 10195.00 10263.0 10316.00 10397.0
    honggfuzz 82800 19.0 10134.789474 79.412831 10042.0 10073.50 10116.0 10197.50 10306.0
    entropic 82800 18.0 9856.333333 88.362888 9677.0 9798.50 9840.5 9893.00 10012.0
    fairfuzz 82800 20.0 9738.800000 297.370688 8755.0 9632.25 9741.0 9833.25 10278.0
    aflplusplus 82800 18.0 9507.944444 157.534113 9162.0 9420.00 9483.5 9624.75 9806.0
    aflplusplus_optimal 82800 18.0 9372.444444 239.210029 8935.0 9178.00 9419.5 9538.25 9788.0
    libfuzzer 82800 17.0 9203.411765 135.570581 8905.0 9134.00 9208.0 9264.00 9478.0

    Mann-Whitney U test
    👁 Image
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libhevc_hevc_dec_fuzzer summary

Discovered bug coverage distribution
👁 Image
Reached code coverage distribution
👁 Image
Mean code coverage growth over time
👁 Image
Mean code coverage growth over time
👁 Image
Mean bug coverage growth over time
👁 Image
Mean bug coverage growth over time
👁 Image
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 19.0 0.315789 0.477567 0.0 0.0 0.0 1.0 1.0
    aflfast 82800 18.0 0.277778 0.574513 0.0 0.0 0.0 0.0 2.0
    aflplusplus 82800 16.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    aflplusplus_optimal 82800 16.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    aflsmart 82800 18.0 0.222222 0.427793 0.0 0.0 0.0 0.0 1.0
    eclipser 82800 13.0 0.230769 0.438529 0.0 0.0 0.0 0.0 1.0
    entropic 82800 19.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    fairfuzz 82800 17.0 0.235294 0.437237 0.0 0.0 0.0 0.0 1.0
    honggfuzz 82800 19.0 0.052632 0.229416 0.0 0.0 0.0 0.0 1.0
    libfuzzer 82800 18.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    mopt 82800 18.0 0.555556 0.704792 0.0 0.0 0.0 1.0 2.0

    Mann-Whitney U test
    👁 Image
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 19.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflfast 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflplusplus 82800 16.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflplusplus_optimal 82800 16.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflsmart 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    eclipser 82800 13.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    entropic 82800 19.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    fairfuzz 82800 17.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    honggfuzz 82800 19.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    libfuzzer 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    mopt 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Mann-Whitney U test
    👁 Image
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

matio_matio_fuzzer summary

Discovered bug coverage distribution
👁 Image
Reached code coverage distribution
👁 Image
Mean code coverage growth over time
👁 Image
Mean code coverage growth over time
👁 Image
Mean bug coverage growth over time
👁 Image
Mean bug coverage growth over time
👁 Image
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 15.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    aflfast 82800 15.0 0.066667 0.258199 0.0 0.0 0.0 0.0 1.0
    aflplusplus 82800 17.0 0.058824 0.242536 0.0 0.0 0.0 0.0 1.0
    aflplusplus_optimal 82800 15.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    aflsmart 82800 17.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    eclipser 82800 15.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    entropic 82800 16.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    fairfuzz 82800 16.0 0.187500 0.403113 0.0 0.0 0.0 0.0 1.0
    honggfuzz 82800 15.0 0.066667 0.258199 0.0 0.0 0.0 0.0 1.0
    libfuzzer 82800 17.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    mopt 82800 18.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0

    Mann-Whitney U test
    👁 Image
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 15.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflfast 82800 15.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflplusplus 82800 17.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflplusplus_optimal 82800 15.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflsmart 82800 17.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    eclipser 82800 15.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    entropic 82800 16.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    fairfuzz 82800 16.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    honggfuzz 82800 15.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    libfuzzer 82800 17.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    mopt 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Mann-Whitney U test
    👁 Image
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openexr_openexr_exrenvmap_fuzzer summary

Discovered bug coverage distribution
👁 Image
Reached code coverage distribution
👁 Image
Mean code coverage growth over time
👁 Image
Mean code coverage growth over time
👁 Image
Mean bug coverage growth over time
👁 Image
Mean bug coverage growth over time
👁 Image
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 16.0 1.062500 0.250000 1.0 1.0 1.0 1.0 2.0
    aflplusplus_optimal 82800 16.0 1.062500 0.250000 1.0 1.0 1.0 1.0 2.0
    eclipser 82800 16.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    entropic 82800 17.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    libfuzzer 82800 17.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    afl 82800 14.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    aflfast 82800 16.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    aflsmart 82800 19.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    fairfuzz 82800 17.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    honggfuzz 82800 17.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    mopt 82800 15.0 0.066667 0.258199 0.0 0.0 0.0 0.0 1.0

    Mann-Whitney U test
    👁 Image
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 17.0 1015.411765 25.828905 973.0 1001.0 1017.0 1029.0 1051.0
    libfuzzer 82800 17.0 990.411765 33.075782 943.0 966.0 981.0 1019.0 1060.0
    aflplusplus 82800 16.0 1031.375000 167.675828 920.0 922.5 972.0 1050.0 1494.0
    aflplusplus_optimal 82800 16.0 1003.812500 93.072888 921.0 931.5 967.5 1044.5 1228.0
    eclipser 82800 16.0 897.000000 0.000000 897.0 897.0 897.0 897.0 897.0
    afl 82800 14.0 252.000000 0.000000 252.0 252.0 252.0 252.0 252.0
    aflfast 82800 16.0 252.000000 0.000000 252.0 252.0 252.0 252.0 252.0
    aflsmart 82800 19.0 252.000000 0.000000 252.0 252.0 252.0 252.0 252.0
    fairfuzz 82800 17.0 252.000000 0.000000 252.0 252.0 252.0 252.0 252.0
    honggfuzz 82800 17.0 248.705882 5.632103 239.0 248.0 252.0 252.0 252.0
    mopt 82800 15.0 295.000000 166.538284 252.0 252.0 252.0 252.0 897.0

    Mann-Whitney U test
    👁 Image
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openh264_decoder_fuzzer summary

Discovered bug coverage distribution
👁 Image
Reached code coverage distribution
👁 Image
Mean code coverage growth over time
👁 Image
Mean code coverage growth over time
👁 Image
Mean bug coverage growth over time
👁 Image
Mean bug coverage growth over time
👁 Image
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 17.0 6.117647 1.363926 4.0 5.0 6.0 7.00 8.0
    entropic 82800 17.0 5.176471 1.424574 3.0 4.0 5.0 6.00 8.0
    libfuzzer 82800 15.0 4.333333 1.447494 1.0 3.5 5.0 5.00 6.0
    afl 82800 17.0 3.058824 1.144038 1.0 3.0 3.0 4.00 5.0
    aflfast 82800 17.0 2.941176 1.248529 1.0 2.0 3.0 4.00 5.0
    aflsmart 82800 17.0 3.000000 1.274755 1.0 2.0 3.0 4.00 5.0
    eclipser 82800 19.0 2.947368 0.911268 1.0 2.0 3.0 4.00 4.0
    mopt 82800 18.0 2.833333 1.098127 1.0 2.0 3.0 3.75 5.0
    aflplusplus 82800 16.0 0.437500 0.963933 0.0 0.0 0.0 0.00 3.0
    aflplusplus_optimal 82800 16.0 0.312500 0.793200 0.0 0.0 0.0 0.00 3.0
    fairfuzz 82800 18.0 0.555556 0.783823 0.0 0.0 0.0 1.00 2.0

    Mann-Whitney U test
    👁 Image
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 17.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflfast 82800 17.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflplusplus 82800 16.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflplusplus_optimal 82800 16.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflsmart 82800 17.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    eclipser 82800 19.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    entropic 82800 17.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    fairfuzz 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    honggfuzz 82800 17.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    libfuzzer 82800 15.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    mopt 82800 18.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Mann-Whitney U test
    👁 Image
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-execute summary

Discovered bug coverage distribution
👁 Image
Reached code coverage distribution
👁 Image
Mean code coverage growth over time
👁 Image
Mean code coverage growth over time
👁 Image
Mean bug coverage growth over time
👁 Image
Mean bug coverage growth over time
👁 Image
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 17.0 2.176471 1.590412 1.0 1.0 2.0 2.0 6.0
    aflplusplus_optimal 82800 15.0 2.333333 1.290994 1.0 1.0 2.0 3.0 5.0
    entropic 82800 20.0 0.750000 0.786398 0.0 0.0 1.0 1.0 2.0
    fairfuzz 82800 19.0 0.736842 0.733493 0.0 0.0 1.0 1.0 2.0
    afl 82800 18.0 0.111111 0.323381 0.0 0.0 0.0 0.0 1.0
    aflfast 82800 17.0 0.529412 0.717430 0.0 0.0 0.0 1.0 2.0
    aflsmart 82800 17.0 0.176471 0.392953 0.0 0.0 0.0 0.0 1.0
    eclipser 82800 17.0 0.235294 0.437237 0.0 0.0 0.0 0.0 1.0
    honggfuzz 82800 15.0 0.200000 0.414039 0.0 0.0 0.0 0.0 1.0
    libfuzzer 82800 15.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    mopt 82800 16.0 0.437500 0.629153 0.0 0.0 0.0 1.0 2.0

    Mann-Whitney U test
    👁 Image
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 82800 15.0 201474.066667 2102.890102 198520.0 199530.50 201906.0 202538.50 205136.0
    aflplusplus 82800 17.0 200266.470588 1054.051951 198473.0 199662.00 200312.0 201049.00 202044.0
    aflfast 82800 17.0 178143.529412 4001.388636 172700.0 175397.00 177274.0 180888.00 186097.0
    fairfuzz 82800 19.0 172900.894737 8828.807783 154388.0 171479.00 175206.0 177689.00 185100.0
    afl 82800 18.0 176655.722222 5152.855631 170593.0 173035.00 175097.5 178397.50 187130.0
    eclipser 82800 17.0 176280.000000 3416.748217 172702.0 174057.00 174823.0 177484.00 184934.0
    aflsmart 82800 17.0 173724.470588 3128.592078 168663.0 171974.00 173738.0 175924.00 179102.0
    mopt 82800 16.0 174076.875000 2876.917700 169958.0 172387.75 173373.5 175304.25 182699.0
    entropic 82800 20.0 165539.950000 7930.712701 154324.0 157886.00 170020.5 172715.75 174954.0
    honggfuzz 82800 15.0 155064.266667 2488.391634 146308.0 155231.50 155707.0 156168.00 156563.0
    libfuzzer 82800 15.0 142544.466667 4338.972243 138970.0 140219.50 140756.0 142549.00 154432.0

    Mann-Whitney U test
    👁 Image
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-parser-2020-07-25 summary

Discovered bug coverage distribution
👁 Image
Reached code coverage distribution
👁 Image
Mean code coverage growth over time
👁 Image
Mean code coverage growth over time
👁 Image
Mean bug coverage growth over time
👁 Image
Mean bug coverage growth over time
👁 Image
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 18.0 4.111111 1.131833 3.0 3.00 4.0 5.0 6.0
    aflplusplus_optimal 82800 16.0 4.625000 1.746425 1.0 4.00 4.0 6.0 7.0
    afl 82800 15.0 1.600000 1.298351 0.0 0.50 2.0 2.5 4.0
    entropic 82800 14.0 2.428571 1.283881 1.0 1.25 2.0 3.0 5.0
    fairfuzz 82800 17.0 2.941176 1.983387 0.0 2.00 2.0 4.0 7.0
    aflsmart 82800 18.0 1.722222 1.447332 0.0 0.25 1.5 3.0 4.0
    aflfast 82800 18.0 1.555556 1.381484 0.0 1.00 1.0 2.0 6.0
    eclipser 82800 15.0 1.533333 1.407463 0.0 0.50 1.0 2.0 5.0
    mopt 82800 17.0 0.823529 0.951006 0.0 0.00 1.0 1.0 3.0
    honggfuzz 82800 17.0 0.352941 0.492592 0.0 0.00 0.0 1.0 1.0
    libfuzzer 82800 19.0 0.157895 0.374634 0.0 0.00 0.0 0.0 1.0

    Mann-Whitney U test
    👁 Image
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 18.0 45467.666667 53.106995 45379.0 45418.00 45461.5 45516.00 45565.0
    aflplusplus_optimal 82800 16.0 45474.687500 57.880012 45396.0 45434.50 45460.5 45499.50 45600.0
    entropic 82800 14.0 45100.285714 200.584695 44839.0 44916.00 45163.5 45266.50 45446.0
    eclipser 82800 15.0 44991.400000 231.415458 44556.0 44835.00 45080.0 45162.50 45225.0
    aflsmart 82800 18.0 44988.055556 195.849123 44660.0 44811.75 45025.5 45139.50 45254.0
    afl 82800 15.0 44824.400000 334.573033 43997.0 44665.50 44828.0 45083.50 45302.0
    aflfast 82800 18.0 44791.333333 155.554795 44481.0 44706.00 44802.0 44868.25 45132.0
    fairfuzz 82800 17.0 44697.529412 693.622657 43623.0 44124.00 44766.0 45378.00 45458.0
    mopt 82800 17.0 44722.176471 333.558322 43881.0 44567.00 44737.0 44991.00 45167.0
    libfuzzer 82800 19.0 43509.578947 413.771450 42733.0 43313.50 43542.0 43754.50 44128.0
    honggfuzz 82800 17.0 43474.705882 418.333116 42985.0 43182.00 43306.0 43521.00 44319.0

    Mann-Whitney U test
    👁 Image
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

stb_stbi_read_fuzzer summary

Discovered bug coverage distribution
👁 Image
Reached code coverage distribution
👁 Image
Mean code coverage growth over time
👁 Image
Mean code coverage growth over time
👁 Image
Mean bug coverage growth over time
👁 Image
Mean bug coverage growth over time
👁 Image
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 14.0 8.571429 1.157868 6.0 8.00 9.0 9.00 10.0
    aflsmart 82800 19.0 9.052632 1.177270 6.0 9.00 9.0 9.50 11.0
    eclipser 82800 19.0 9.157895 1.213954 7.0 8.50 9.0 10.00 12.0
    aflfast 82800 17.0 7.941176 0.899346 6.0 7.00 8.0 9.00 9.0
    honggfuzz 82800 16.0 8.312500 1.078193 6.0 8.00 8.0 9.00 10.0
    mopt 82800 16.0 8.062500 0.997914 6.0 7.75 8.0 9.00 10.0
    aflplusplus 82800 14.0 6.714286 0.913874 5.0 6.00 7.0 7.00 8.0
    aflplusplus_optimal 82800 18.0 7.333333 1.084652 6.0 6.25 7.0 8.00 9.0
    fairfuzz 82800 14.0 4.785714 1.311404 2.0 4.00 5.0 5.75 7.0
    libfuzzer 82800 19.0 3.421053 0.768533 2.0 3.00 3.0 4.00 5.0
    entropic 82800 19.0 2.421053 0.606977 2.0 2.00 2.0 3.00 4.0

    Mann-Whitney U test
    👁 Image
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 16.0 2902.750000 94.349351 2682.0 2878.25 2934.5 2958.25 2996.0
    eclipser 82800 19.0 2869.473684 65.862962 2730.0 2820.00 2866.0 2931.00 2958.0
    aflplusplus 82800 14.0 2623.142857 116.904864 2411.0 2545.25 2655.5 2720.25 2758.0
    aflplusplus_optimal 82800 18.0 2627.277778 119.425490 2409.0 2549.00 2627.5 2676.00 2928.0
    fairfuzz 82800 14.0 2479.857143 247.245085 1979.0 2296.25 2567.0 2580.00 2782.0
    afl 82800 14.0 2549.857143 143.717595 2401.0 2422.75 2538.0 2622.00 2815.0
    aflfast 82800 17.0 2541.352941 141.944065 2404.0 2434.00 2459.0 2629.00 2847.0
    aflsmart 82800 19.0 2526.894737 151.709479 2402.0 2420.50 2429.0 2624.50 2815.0
    mopt 82800 16.0 2463.437500 82.944937 2392.0 2413.00 2422.5 2481.75 2616.0
    entropic 82800 19.0 2135.526316 99.035554 1848.0 2088.50 2176.0 2191.00 2271.0
    libfuzzer 82800 19.0 2083.052632 94.222239 1831.0 2026.00 2123.0 2153.00 2180.0

    Mann-Whitney U test
    👁 Image
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

The experiment was conducted using this FuzzBench commit: 8ed23f235e9086571f306ad29109b687b39a3074

Experiment Description:

('Official bug benchmarking experiment.',)