Editing Web crawler (section)

====Restricting followed links====
A crawler may only want to seek out HTML pages and avoid all other [[Internet media type|MIME types]]. In order to request only HTML resources, a crawler may make an HTTP HEAD request to determine a Web resource's MIME type before requesting the entire resource with a GET request. To avoid making numerous HEAD requests, a crawler may examine the URL and only request a resource if the URL ends with certain characters such as .html, .htm, .asp, .aspx, .php, .jsp, .jspx or a slash. This strategy may cause numerous HTML Web resources to be unintentionally skipped.

Some crawlers may also avoid requesting any resources that have a [[Query string|"?"]] in them (are dynamically produced) in order to avoid [[spider trap]]s that may cause the crawler to download an infinite number of URLs from a Web site. This strategy is unreliable if the site uses [[URL rewriting]] to simplify its URLs.