What is the robots.txt File?

The robots.txt file is a special text file placed in the root directory of a website and is used to provide instructions to search engine bots (e.g., Googlebot) about which pages or directories should or should not be indexed.

Why is the robots.txt File Necessary?

1. To make certain parts of the website inaccessible to search engines (e.g., administrative pages).
2. To reduce server load by limiting access to certain bots.
3. To prevent the indexing of duplicate or irrelevant content.

Basic Syntax of the robots.txt File

The file format is simple and consists of the following directives:

    User-agent: *
    Disallow: /admin/
    Allow: /public/

1. User-agent: Specifies which bot the instructions are directed to. * means all bots.
2. Disallow: Defines the paths that are prohibited for bots.
3. Allow: Permits specific pages or directories to be indexed, even if there is a Disallow directive above.

Common Patterns

Block All Bots from Indexing the Entire Website

    User-agent: *
    Disallow: /

Allow All Bots to Fully Index the Website

User-agent: * Disallow:

Block Specific Directories

    User-agent: *
    Disallow: /private/
    Disallow: /temp/

Allow Specific Pages While Blocking the Rest

    User-agent: *
    Disallow: /
    Allow: /index.html

Specific Instructions for Googlebot Only

    User-agent: Googlebot
    Disallow: /no-google/

Including Sitemap in robots.txt

If you have an XML Sitemap, you can specify it in the robots.txt file to inform search engines:

    Sitemap: https://example.com/sitemap.xml

How to Check the robots.txt File

1. Google Search Console – Google's "Robots Testing Tool" allows you to check if your robots.txt file is working correctly.
2. Browser – Simply open your website's robots.txt file: https://example.com/robots.txt