Search engines use automated programs called robots or crawlers to go through websites and generate the site results found when visitors use their search engine. Artificial intelligence (AI) platforms are now crawling websites to use content for model training as well. Most of the time, websites want to be found on search engines. However, you may have particular parts of your site that you do not want to be crawled. Or you may want to limit how many requests crawlers are making on your site.

You can modify and disable search engine indexing in three methods:

• Edit the robots.txt file to block crawlers from indexing your entire website or certain pages;
• You can easily disable site indexing from the WordPress Toolkit, and;
• If you're using WooCommerce and Jetpack, you can also block crawlers by disabling search indexing from your WordPress.com privacy settings.

Note: The robots.txt and noindex meta tags are like "No Trespassing" signs. However, not all search engines honor this setting.  

Disable search engine indexing from the robots.txt file

You can block most search engine robots from indexing your website by doing the following:

1. Edit (or create) the robots.txt file in the public_html folder. To block all search engines from the entire site, the file should contain the following:

User-agent: *
Disallow: /

To block search engines from a specific folder, disallow lines should be added for each folder path to be blocked. For example, to block only the "images" folder and its subfolders, the file would contain:

User-agent: *
Disallow: /images/

To learn more about creating a robots.txt file, visit Google's help page on the topic.

2. Edit the index.html file (or all web pages) to include the following meta tag:

<meta name="robots"' content="noindex">

This method is recommended if you do not want any search engines indexing your site.

Note: It is not possible to edit the robots.txt file for a website created with the Store Editor.

Disable search engine indexing from the WordPress Toolkit

Follow the steps here to disable search engine indexing from the WordPress Toolkit.

Disable search engine indexing at WordPress.com

Follow the steps at WordPress.com to disable search engine indexing and third party sharing, including AI models.