I have come to realise---or rather, I have become more and more convinced that---fuzzing research has stalled not because we have no further contributions to make, but because the contributions that we are making are either incremental and merely sound impressive or presented in ways that obscure their utility. To be more concrete: we are spending time trying to "improve" fuzzing generally rather than identifying what can be improved; everyone is trying to be "the best" rather than trying to identify what is actually happening . This is not the first time that I have felt this, but perhaps my understanding of this problem has improved in the last two years. It's time for a revisit!
Last year, I was involved in a paper which tried to standardise fuzzer evaluation . While I still think that this paper is incredibly important in providing baseline evaluation requirements, something that I've only realised in the last year or so is that it asks the wrong questions.
Statistical significance is the golden standard for scientific advancement. This shows that there is indeed a difference between two experimental configurations. The only problem is: it is incredibly trivial to have statistical significance in fuzzing.
Last time , I mentioned that I was to run the fuzzing competition for SBFT'25. Despite having only two contestants, I think this competition truly highlighted the problem of fuzzer evaluation. The first contestant ensembled AFL++ and LibAFL and used fixed-interval corpus minimisation during execution. The second ensembled AFL++, LibAFL, FOX , and ZTaint-Havoc . This submission is an engineering marvel, utilising program-specific knowledge to give far greater knowledge to the fuzzer search pattern.
The kicker? The first contestant, Kraken, won---though this is in part due to a bug that caused the second contestant, HFuzz, to crash on one of the targets. In raw scores, Kraken beat HFuzz on 3 of the targets (+1 additionally, if we include the crashing target), and 4 vice versa. The improvements shown by these fuzzers are statistically significant, by classical evaluation metrics and tests.
Yet, I suspect that if you handed these tools to a bunch of reviewers, they would reject Kraken and accept HFuzz. Why? The fuzzers involved in HFuzz are much more technically interesting ; indeed, FOX was accepted at CCS'24 and ZTaint-Havoc was accepted at ISSTA'25 , an A* and an A venue, respectively.