Editing Chess engine (section)

===Ratings===
Chess engine rating lists aim to provide statistically significant measures of relative engine strength. These lists play multiple games between engines. Some also standardize the opening books, the [[time control]]s, and the [[computer hardware]] the engines use, in an attempt to measure the strength differences of the engines only. These lists provide not only a ranking, but also margins of error on the given ratings.

The ratings on the rating lists, although calculated by using the [[Elo rating system|Elo system]] (or similar rating methods), have no direct relation to FIDE Elo ratings or to other chess federation ratings of human players. Except for some man versus machine games which the SSDF had organized many years ago (when engines were far from today's strength), there is ''no calibration'' between any of these rating lists and player pools. Hence, the results which matter are the ''ranks'' and the ''differences'' between the ratings, and not the absolute values.

Missing from many rating lists are [[IPPOLIT]] and its derivatives. Although very strong and [[open source]], there are allegations from commercial software interests that they were derived from a disassembled binary of [[Rybka]].<ref>{{Cite web |url=http://www.chessvibes.com/reports/chess-engine-controversy/ |title=Chess engine controversy at chessvibes.com, retrieved 28/May/2010 |access-date=2010-05-28 |archive-url=https://web.archive.org/web/20100528153718/http://www.chessvibes.com/reports/chess-engine-controversy/ |archive-date=2010-05-28 |url-status=dead }}</ref> Due to the [[Rybka#IPPOLIT controversy|controversy]], all these engines have been blacklisted from many tournaments and rating lists. Rybka in turn was accused of being based on [[Fruit (chess)|Fruit]],<ref>{{Cite web |url=https://webspace.utexas.edu/zzw57/rtc/eval/eval.html |title=Evaluation<!-- Bot generated title --> |access-date=2010-12-18 |archive-url=https://web.archive.org/web/20110811141639/https://webspace.utexas.edu/zzw57/rtc/eval/eval.html |archive-date=2011-08-11 |url-status=dead }}</ref> and in June 2011, the ICGA formally claimed Rybka was derived from Fruit and Crafty and banned Rybka from the International Computer Games Association World Computer Chess Championship, and revoked its previous victories (2007, 2008, 2009, and 2010).<ref>{{Cite web |url=http://www.chessvibes.com/reports/rybka-disqualified-and-banned-from-world-computer-chess-championships/ |title=Rybka disqualified and banned from World Computer Chess Championships {{!}} ChessVibes<!-- Bot generated title --> |access-date=2011-06-30 |archive-url=https://web.archive.org/web/20120302233915/http://www.chessvibes.com/reports/rybka-disqualified-and-banned-from-world-computer-chess-championships/ |archive-date=2012-03-02 |url-status=dead }}</ref> The ICGA received some criticism for this decision.<ref name=AGMoJiCCpo>{{cite news|last=Riis|first=Dr. Søren|title=A Gross Miscarriage of Justice in Computer Chess (part one)|url=http://www.chessbase.com/newsdetail.asp?newsid=7791|access-date=19 February 2012|newspaper=Chessbase News|date=January 2, 2012|archive-date=7 January 2012|archive-url=https://web.archive.org/web/20120107191340/http://www.chessbase.com/newsdetail.asp?newsid=7791|url-status=live}}</ref> Despite all this, Rybka is still included on many rating lists, such as CCRL and CEGT, in addition to [[Houdini]], a derivative of the IPPOLIT derivative Robbolito,<ref name="chess vibes">{{cite web|title=Free Houdini beats commercial Rybka 23.5-16.5|url=http://www.chessvibes.com/reports/free-houdini-beats-commercial-rybka-23-5-16-5/|archive-url=https://web.archive.org/web/20140410195128/http://www.chessvibes.com/reports/free-houdini-beats-commercial-rybka-23-5-16-5/|archive-date=10 April 2014|publisher=Chessvibes|access-date=25 October 2013|date=10 February 2010}}</ref> and Fire, a derivative of Houdini. In addition, [[ChessBase#Criticism and legal issues|Fat Fritz 2]], a derivative of Stockfish,<ref>{{Cite web|last=The Stockfish Team|date=2021-07-21|title=Our lawsuit against ChessBase|url=https://stockfishchess.org/blog/2021/our-lawsuit-against-chessbase/|url-status=live|archive-url=https://web.archive.org/web/20210721190547/https://stockfishchess.org/blog/2021/our-lawsuit-against-chessbase/|archive-date=2021-07-21|access-date=2021-08-18|website=Stockfishchess.org}}</ref> is also included on most of the rating lists.

====Differences between rating lists====
There are a number of factors that vary among the chess engine rating lists:
* Number of games. [[Law of large numbers|More games when testing each engine]] result in higher [[statistical significance]]. 
* Formulae used to calculate the elo of each engine.
* [[Time control]]:
** Longer time controls are better suited for determining tournament play strength, but also either make testing more time-consuming or the results less statistically significant. 
** Increment time controls are better suited for determining tournament play strength since tournaments usually use increment time controls, but many rating lists use cyclic/repeating time controls instead. 
** Consistent time controls throughout the rating list vs different time controls for each test. The latter results in a smaller [[statistical significance]] than the former because different time controls is a potential [[confounder]]. This is particularly problematic for CCRL because CCRL uses both cyclic/repeating time controls (40/15) and increment time controls (15"+10') in its CCRL 40/15 list yet maintains both time controls on the same list.<ref>{{Cite web|url=https://talkchess.com/forum3/viewtopic.php?f=6&t=82754|title=CCRL 40/15, 2m1s and FRC 40/2 lists updated (21-10-2023)|website=talkchess.com|access-date=22 October 2023|archive-date=15 June 2024|archive-url=https://web.archive.org/web/20240615081446/https://talkchess.com/forum3/viewtopic.php?f=6&t=82754|url-status=live}}</ref>
* Opponents used in testing engines. 
** Some rating lists only test an engine against the most recent version of each opponent engine, while other rating lists test an engine against the version(s) of each opponent engine closest in elo to the engine being tested. 
** Most rating lists do not test every engine on the rating list vs every other engine on the rating list in a [[round-robin tournament]] format. This causes distortions in the rating lists, especially for CCRL and CEGT.<ref>{{Cite web|url=https://talkchess.com/viewtopic.php?t=83829|title=Experimental testruns of Stockfish / Torch|website=talkchess.com|access-date=29 May 2024|archive-date=15 June 2024|archive-url=https://web.archive.org/web/20240615081446/https://talkchess.com/viewtopic.php?t=83829|url-status=live}}</ref>
* Hardware used: 
** Faster hardware with more memory leads to stronger play.
** 64-bit (vs. 32-bit) hardware and operating systems favor [[bitboard]]-based programs
** Hardware using modern instruction sets such as [[AVX2]] or [[AVX512]] favor engines using vectors and vector intrinsics in their code, common in [[Artificial neural network|neural networks]]. 
** [[Graphics processing units]] favor programs with [[deep neural networks]].
** Multiprocessor vs. single processor hardware.
** Consistent hardware throughout the rating list vs different hardware for every test. The latter results in a smaller [[statistical significance]] than the former because different hardware is a potential [[confounder]]. This is particularly problematic for [[CEGT]] because multiple testers each with their own unique hardware are involved in testing each engine in CEGT.<ref>{{Cite web|url=http://www.cegt.net/testers/testers.html|title=CEGT Testers|website=Cegt.net|access-date=26 June 2022|archive-date=24 May 2022|archive-url=https://web.archive.org/web/20220524090535/http://www.cegt.net/testers/testers.html|url-status=live}}</ref> The same issue arises in [[CCRL]].<ref "CCRL issues">{{Cite web|url=https://talkchess.com/viewtopic.php?p=962945#p962945|title=ShashChess|author=Ray Banks (Modern Times)|website=talkchess.com|access-date=30 April 2024|archive-date=30 April 2024|archive-url=https://web.archive.org/web/20240430232313/https://talkchess.com/viewtopic.php?p=962945#p962945|url-status=live}}</ref>
* Ponder settings (speculative analysis while the opponent is thinking) aka Permanent Brain.
* Transposition table sizes.
* GUI settings. 
* Opening book settings. 

These differences affect the results, and make direct comparisons between rating lists difficult.

====List of rating lists====
Current rating lists and rating list organizations include:

* [https://www.computerchess.org.uk/ccrl/ Computer Chess Rating Lists] (CCRL)
** [https://www.computerchess.org.uk/ccrl/404/ CCRL Blitz 2+1]
** [https://www.computerchess.org.uk/ccrl/4040/ CCRL 40/15] 
** [https://www.computerchess.org.uk/ccrl/404FRC/ CCRL FRC]
* [[Chess Engines Grand Tournament]] (CEGT)
** [http://www.cegt.net/blitz.htm CEGT Blitz]
** [http://www.cegt.net/rating.htm CEGT 40/20]
** [http://www.cegt.net/rating120.htm CEGT 40/120]
** [http://www.cegt.net/rating3plus1pbon.htm CEGT 3"+1']
** [http://www.cegt.net/rating5plus3pbon.htm CEGT 5"+3']
** [http://www.cegt.net/rating25plus8.htm CEGT 25"+8']
* [https://www.sp-cc.de/ Stefan Pohl Computer Chess] (SPCC)
* [https://www.chessengeria.eu/mcerl Mac Chess Engine Rating List] (MCERL)
* [https://ipmanchess.yolasite.com/ Ipman Chess]
** [https://ipmanchess.yolasite.com/r9-7945hx.php Ipman Chess AMD Ryzen 9 7945HX]
** [https://ipmanchess.yolasite.com/i9-7980xe.php Ipman Chess Intel i9 7980XE]
** [https://ipmanchess.yolasite.com/i7-13700h.php Ipman Chess Intel i7 13700H]
* [https://ssdf.bosjo.net/list.htm SSDF]

Historic rating lists and rating list organizations include:
* [https://ccrl.chessdom.com/ccrl/ Computer Chess Rating Lists] (CCRL)
** [https://ccrl.chessdom.com/ccrl/402.archive/ CCRL 40/2 Archive]
* [http://fastgm.de FastGM's Rating List] (FGRL)
** [http://fastgm.de/240-2.40.html FGRL 4"+2.4']
** [http://fastgm.de/30min.html FGRL 30"]
* [https://ipmanchess.yolasite.com/ Ipman Chess]
** [https://ipmanchess.yolasite.com/list-i7-970.php Ipman Chess i7 970]
** [https://ipmanchess.yolasite.com/i7-11800h.php Ipman Chess Intel i7 11800H]
** [https://ipmanchess.yolasite.com/list-i7-2670qm.php Ipman Chess i7 2670QM]
** [https://ipmanchess.yolasite.com/list-i3-m380.php Ipman Chess List i3 M380]
** [https://ipmanchess.yolasite.com/i7-5960x.php Ipman Chess i7 5960x]
* [https://web.archive.org/web/20190125041233/http://inwoba.de/bayes.html IPON]
* [https://rebel13.nl/misc/gambit-rating-list.html Gambit Rating List] (GRL)
* [http://fastgm.de FastGM's Rating List] (FGRL)
** [http://fastgm.de/60-0.60.html FGRL 1"+0.6']
** [http://fastgm.de/10min.html FGRL 10"+6']
** [http://fastgm.de/60min.html FGRL 60"+15']
** [http://fastgm.de/16-60-0.6.html FGRL 16 Cores vs RTX 2070]
* [https://rebel7775.wixsite.com/rebel/ Anti Draw Rating List] (ADRL)
** [https://web.archive.org/web/20240331164628/https://rebel7775.wixsite.com/rebel/adrl-blitz ADRL Blitz]
** [https://web.archive.org/web/20240331164629/https://rebel7775.wixsite.com/rebel/kopie-van-adrl-blitz ADRL 40/120]