Group items tagged

Robots.txt is a very important file for modern websites. Developers place it inside the root folder (the same place where the index file is). The reason behind it is that the file tells the search engine robots what pages to crawl and what pages to not crawl.