How to Create Robots.txt File For SEO - 0 views
-
techtoguru on 16 Mar 17The Robots Exclusion Standard was developed in 1994 so that website owners can advise search engines how to crawl your website. It works in a Similar way as the Robots meta tag. The main difference being that the robots.txt file stop search engines from seeing a page or directory, whereas the robots meta tag only controls whether it is indexed.