Editing Software bug (section)

=== Benchmark ===

To facilitate reproducible research on testing and debugging, researchers use curated benchmarks of bugs:
* the Siemens benchmark
* ManyBugs<ref name="Le GouesHoltschulte2015">{{cite journal|last1=Le Goues|first1=Claire|author1-link=Claire Le Goues|last2=Holtschulte|first2=Neal|last3=Smith|first3=Edward K.|last4=Brun|first4=Yuriy|last5=Devanbu|first5=Premkumar|last6=Forrest|first6=Stephanie|last7=Weimer|first7=Westley|title=The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs|journal=IEEE Transactions on Software Engineering|volume=41|issue=12|year=2015|pages=1236–1256|issn=0098-5589|doi=10.1109/TSE.2015.2454513|doi-access=free}}</ref> is a benchmark of 185 C bugs in nine open-source programs.
* Defects4J<ref name="JustJalali2014">{{cite book|last1=Just|first1=René|title=Proceedings of the 2014 International Symposium on Software Testing and Analysis – ISSTA 2014|pages=437–440|last2=Jalali|first2=Darioush|last3=Ernst|first3=Michael D.|s2cid=12796895|chapter=Defects4J: a database of existing faults to enable controlled testing studies for Java programs|year=2014|doi=10.1145/2610384.2628055|isbn=9781450326452|citeseerx=10.1.1.646.3086}}</ref> is a benchmark of 341 Java bugs from 5 open-source projects. It contains the corresponding patches, which cover a variety of patch type.