Editing Web crawler (section)

==Security==
While most of the website owners are keen to have their pages indexed as broadly as possible to have strong presence in [[Web search engine|search engines]], web crawling can also have [[unintended consequences]] and lead to a [[Web application security|compromise]] or [[data breach]] if a search engine indexes resources that should not be publicly available, or pages revealing potentially vulnerable versions of software.

{{main|Google hacking}}

Apart from standard [[web application security]] recommendations website owners can reduce their exposure to opportunistic hacking by only allowing search engines to index the public parts of their websites (with [[Robots exclusion standard|robots.txt]]) and explicitly blocking them from indexing transactional parts (login pages, private pages, etc.).