A robots.txt is a text file kept on your website’s home directory i.e., Public_HTML. The file is checked by all search engines spiders before they start crawling your website’s content to know which files, file types and directories are not allowed to be archived. So by using a Robot.Txt file you can exclude [...]




