What is a SEO robot meta tag?

A meta robot tag tells the search engine what to follow and what not to follow. It is a piece of code in the head section of our webpage. If you do not want any specific page to be indexed, you can let the search engine know by using a meta tag. Meta tags can have many values. 

The syntax of the meta robot tag is:

<meta name=”robots” content=”value”>

What are the different SEO robot meta tag values?

1. Index: This is the default value and it tells search engines to index the page.
2. Noindex: This value is used to inform the search engines not to index a specific page.
3. Noimageindex: This means you do not want the images on a page to be crawled. 
4. Note: This is another form of the noindex or nofollow value. By using this tag, you are saying to Google not to do anything with a particular page.
5. Follow: If you want search engines to follow all the links on your page, regardless of whether you want them to be indexed or not, then use the follow value meta tag.
6. Nofollow: To inform the search engines not to follow any links on a page, use the nofollow value.
7. noarchive: To stop the search engine from making a cached copy of the page, you can tell the crawler not to archive it. 
8. nosnippet: This prevents the search engines from showing a snippet of the page in search engine result pages. 
9. Noydir: this value is specifically used for yahoo search engines. This value blocks the Yahoo crawler so that it can not use the description for the page.

What are robot.txt files for SEO?

A robots.txt file tells the search engine what content crawl and what to avoid on your website. Its basic functionality is supported by all major search engines. The text file contains a strict syntax because it has to be computer-readable. Crawlers read this file in order to find which pages of the website it is allowed to access and which ones not to access.

What does the robot.txt file do for SEO?

Web pages are indexed by the spiders after crawling. Spiders follow the links from one page to another. The robots.txt file tells the search engines which URLs to index. 

Where should I put my robot.txt file for SEO?

The file robots.txt should be in the root of your domain. 

Example: https://www.websitetoon.com/robots.txt

If your domain responds without www, HTTP, or https, it should have the same robots.txt file. The name is case sensitive, so it should be named robots.txt file. 

What is the syntax of the robot.txt file for SEO?

A robots.txt file consists of one or more blocks of directives, each starting with a user-agent line. The user-agent is the name of a specific spider. 

For example:

User-agent: *

Disallow:/

User-agent: Googlebot

Disallow:

User-agent: bingbot

Disallow: /not-for-bing/

Directive like allow and disallow are not case sensitive, so you can write then either in capital letters or lower letters its your choice.

Best Practices:

• If you want to be found easily, place the robot.txt file in a website’s top-level directory. 
• As the name robots.txt is case sensitive so file name should be exactly “robots.txt”.
• The file robots.txt is publicly available. You just need to add robots.txt on the end of any root domain to see the website’s directives.