To avoid undesirable content from the look for indexes, webmasters can instruct spiders not to crawl sure documents or directories throughout the regular robots.txt file in the foundation directory of the area. On top of that, a web page may be explicitly excluded from the online search engine's database by using a meta tag precise to robots (ordin