Editing Robots.txt (section)

===Search engines===
Some major search [[Facebook Paper|engines]] following this standard include Ask,<ref name="ask-webmasters">{{cite web |title=About Ask.com: Webmasters |url=http://about.ask.com/docs/about/webmasters.shtml |website=About.ask.com |access-date=16 February 2013 |archive-date=27 January 2013 |archive-url=https://web.archive.org/web/20130127134025/http://about.ask.com/docs/about/webmasters.shtml |url-status=live }}</ref> AOL,<ref name="about-aol-search">{{cite web |title=About AOL Search |url=http://search.aol.com/aol/about |website=Search.aol.com |access-date=16 February 2013 |archive-date=13 December 2012 |archive-url=https://web.archive.org/web/20121213134546/http://search.aol.com/aol/about |url-status=dead }}</ref> Baidu,<ref name="baidu-spider">{{cite web |title=Baiduspider |url=http://www.baidu.com/search/spider_english.html |website=Baidu.com |access-date=16 February 2013 |archive-date=6 August 2013 |archive-url=https://web.archive.org/web/20130806131031/http://www.baidu.com/search/spider_english.html |url-status=live }}</ref> Bing,<ref name="bing-blog-robots">{{cite web|url=https://blogs.bing.com/webmaster/2008/06/03/robots-exclusion-protocol-joining-together-to-provide-better-documentation/|title=Robots Exclusion Protocol: joining together to provide better documentation|website=Blogs.bing.com|date=3 June 2008 |archive-url=https://web.archive.org/web/20140818025412/http://blogs.bing.com/webmaster/2008/06/03/robots-exclusion-protocol-joining-together-to-provide-better-documentation/|archive-date=2014-08-18|url-status=live|access-date=16 February 2013}}</ref>  DuckDuckGo,<ref name="duckduckgo-bot">{{cite web|url=https://duckduckgo.com/duckduckbot|website=DuckDuckGo.com|title=DuckDuckGo Bot|access-date=25 April 2017|archive-date=16 February 2017|archive-url=https://web.archive.org/web/20170216043103/https://duckduckgo.com/duckduckbot|url-status=live}}</ref> Kagi,<ref name="KagiBot">{{cite web|url=https://kagi.com/bot|website=Kagi Search|title=Kagi Search KagiBot|access-date=20 November 2024|archive-date=12 April 2024|archive-url=https://web.archive.org/web/20240412192855/https://kagi.com/bot|url-status=live}}</ref> Google,<ref name="google-webmasters-spec">{{cite web |url=https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt |title=Webmasters: Robots.txt Specifications |work=Google Developers |access-date=16 February 2013 |archive-url=https://web.archive.org/web/20130115214137/https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt |archive-date=2013-01-15 |url-status=live }}</ref> Yahoo!,<ref name="yahoo-search-is-bing">{{cite web |url=http://help.yahoo.com/kb/index?page=content&y=PROD_SRCH&locale=en_US&id=SLN2217&impressions=true |title=Submitting your website to Yahoo! Search |access-date=16 February 2013 |archive-url=https://web.archive.org/web/20130121035801/http://help.yahoo.com/kb/index?page=content&y=PROD_SRCH&locale=en_US&id=SLN2217&impressions=true |archive-date=2013-01-21 |url-status=live }}</ref> and Yandex.<ref name="yandex-robots">{{cite web |url=http://help.yandex.com/webmaster/?id=1113851 |title=Using robots.txt |website=Help.yandex.com |access-date=16 February 2013 |archive-url=https://web.archive.org/web/20130125040017/http://help.yandex.com/webmaster/?id=1113851 |archive-date=2013-01-25 |url-status=live }}</ref>