Robot Exclusion Protocol or Robots.txt file is believed to be an important aspect of a website. This file provides instructions to the search engine spiders on what the entire site comprises for them to crawl. Many designate the robots.txt file as an essential factor for effective SEO strategy, and if not used properly then it can affect visibility on search engine results page.
Robots.txt is a very important file for modern websites. Developers place it inside the root folder (the same place where the index file is). The reason behind it is that the file tells the search engine robots what pages to crawl and what pages to not crawl.