Editing Search engine optimization (section)

=== Getting indexed ===
[[File:PageRanks-Example.svg|thumb|A simple illustration of the [[PageRank|Pagerank]] algorithm. Percentage shows the perceived importance.]]
The leading search engines, such as Google, Bing, [[Brave Search]] and Yahoo!, use [[Web crawler|crawlers]] to find pages for their algorithmic search results. Pages that are linked from other search engine-indexed pages do not need to be submitted because they are found automatically. The [[Yahoo! Directory]] and [[DMOZ]], two major directories which closed in 2014 and 2017 respectively, both required manual submission and human editorial review.<ref>{{cite web|url=http://searchenginewatch.com/showPage.html?page=2167881|title=Submitting To Directories: Yahoo & The Open Directory|date=March 12, 2007|access-date=May 15, 2007|publisher=[[Search Engine Watch]]|archive-url=https://web.archive.org/web/20070519052103/http://searchenginewatch.com/showPage.html?page=2167881|archive-date=May 19, 2007|df=mdy-all}}</ref> Google offers [[Google Webmaster Tools|Google Search Console]], for which an XML [[Sitemap]] feed can be created and submitted for free to ensure that all pages are found, especially pages that are not discoverable by automatically following links<ref>{{cite web|url=http://www.google.com/support/webmasters/bin/answer.py?answer=40318&topic=8514|title=What is a Sitemap file and why should I have one?|access-date=March 19, 2007|archive-date=July 1, 2007|archive-url=https://web.archive.org/web/20070701232719/http://www.google.com/support/webmasters/bin/answer.py?answer=40318&topic=8514|url-status=live}}</ref> in addition to their URL submission console.<ref>{{cite web |url=https://www.google.com/webmasters/tools/submit-url |title=Search Console - Crawl URL |access-date=2015-12-18 |archive-date=August 14, 2022 |archive-url=https://web.archive.org/web/20220814100500/https://accounts.google.com/_/bscframe |url-status=live }}</ref> Yahoo! formerly operated a paid submission service that guaranteed to crawl for a [[Pay per click|cost per click]];<ref>{{cite web|url=http://searchenginewatch.com/showPage.html?page=2167871|title=Submitting To Search Crawlers: Google, Yahoo, Ask & Microsoft's Live Search|date=March 12, 2007|access-date=May 15, 2007|publisher=[[Search Engine Watch]] |first1= Danny |last1=Sullivan |archive-url=https://web.archive.org/web/20070510090932/http://searchenginewatch.com/showPage.html?page=2167871#Teoma|archive-date=May 10, 2007|df=mdy-all}}</ref> however, this practice was discontinued in 2009.

[[Web search engine|Search engine]] crawlers may look at a number of different factors when [[Web crawler|crawling]] a site. Not every page is indexed by search engines. The distance of pages from the root directory of a site may also be a factor in whether or not pages get crawled.<ref name="cho">{{cite web|url=http://ilpubs.stanford.edu:8090/347/|title=Efficient crawling through URL ordering|author1=Cho, J. |author2=Garcia-Molina, H. |author3=Page, L. |year=1998|work=Seventh International World-Wide Web Conference |location=Brisbane, Australia |publisher=Stanford InfoLab Publication Server |access-date=May 9, 2007|archive-date=July 14, 2019|archive-url=https://web.archive.org/web/20190714141416/http://ilpubs.stanford.edu:8090/347/}}</ref>

Mobile devices are used for the majority of Google searches.<ref>{{cite web|url=https://webmasters.googleblog.com/2016/11/mobile-first-indexing.html|title=Mobile-first Index|access-date=March 19, 2018|archive-date=February 22, 2019|archive-url=https://web.archive.org/web/20190222000527/https://webmasters.googleblog.com/2016/11/mobile-first-indexing.html|url-status=live}}</ref> In November 2016, Google announced a major change to the way they are crawling websites and started to make their index mobile-first, which means the mobile version of a given website becomes the starting point for what Google includes in their index.<ref>{{cite web |last1=Phan |first1=Doantam |title=Mobile-first Indexing |url=https://webmasters.googleblog.com/2016/11/mobile-first-indexing.html |website=Official Google Webmaster Central Blog |access-date=16 January 2019 |date=4 November 2016 |archive-date=February 22, 2019 |archive-url=https://web.archive.org/web/20190222000527/https://webmasters.googleblog.com/2016/11/mobile-first-indexing.html |url-status=live }}</ref> In May 2019, Google updated the rendering engine of their crawler to be the latest version of Chromium (74 at the time of the announcement).  Google indicated that they would regularly update the [[Chromium (web browser)|Chromium]] rendering engine to the latest version.<ref>{{Cite web|url=https://webmasters.googleblog.com/2019/05/the-new-evergreen-googlebot.html|title=The new evergreen Googlebot|website=Official Google Webmaster Central Blog|language=en|access-date=2020-03-02|archive-date=November 6, 2020|archive-url=https://web.archive.org/web/20201106072307/https://webmasters.googleblog.com/2019/05/the-new-evergreen-googlebot.html|url-status=live}}</ref>  In December 2019, Google began updating the User-Agent string of their crawler to reflect the latest Chrome version used by their rendering service.  The delay was to allow webmasters time to update their code that responded to particular bot User-Agent strings.  Google ran evaluations and felt confident the impact would be minor.<ref>{{Cite web|url=https://webmasters.googleblog.com/2019/10/updating-user-agent-of-googlebot.html|title=Updating the user agent of Googlebot|website=Official Google Webmaster Central Blog|language=en|access-date=2020-03-02|archive-date=March 2, 2020|archive-url=https://web.archive.org/web/20200302132028/https://webmasters.googleblog.com/2019/10/updating-user-agent-of-googlebot.html|url-status=live}}</ref>