What is a Robots.txt File?
A **robots.txt file** is a plain text document created in your website’s root directory that provides crawling directives to automated search indexing spiders (such as Googlebot and Bingbot). It instructs search engines which paths are safe to explore and index and which folders should be completely ignored.
By disallowing private directories (like administrator panels or temporary files), you optimize your site’s **Crawl Budget**—ensuring that crawlers focus exclusively on your highly-valuable content pages. All computation is handled entirely within the browser, meaning your directories lists never travel over the network.