Search This Blog

Friday, 24 January 2020

How to add custom robots.txt file in Blogger | what is Robots.txt file

If you do not know what is Robots.txt file? And how to add Custom Robots.txt file to Blogger, then you read this article completely.
This is very important for the SEO of your blog, due to its incorrect use, your blog can be completely removed from the search engine. To know the benefits and use of Robots.txt File, follow this guide.

Robots.txt file is directly connected to indexing and crawl rate of your blog. Generally all search engines have their own robots or bots that crawl your blog and the links given in it and index it in the search engine. Because of which whatever you update in the blog, the entire update also shows in the search engine, and what crawl will happen in your blog, it controle robot.txt.

What Is Robot.txt


Robots.txt is a text file with some simple code, which gives instructions to web crawlers which pages to crawl in your blog and which do not.
This file is saved in the server of any website or blog so that search engine bots can be managed. It is mainly used to prevent any non-importance webpages (Demo page, Archive, Labels etc.) from crawling.

How to add custom robots.text file in Blogger,what is Robots.txt file, custom robots.txt blogger generator custom robots header tags, custom robots.txt generator, where does the robots txt file go,where to find robots txt, robots txt sitemap, robots txt not found, how to submit robots txt to google


In simple language, through robots.txt file, you have the authority of your blog, which page you want to show in the search engine and which page you want to hide.

Note- Search engine scans the robots.txt file before crawling any web page.

By default Blogger's robots.txt file looks like this, as shown in the screenshot.
To see the Custom Robots file, after slipping (/) after the homepage URL of your blog, type robots.txt and press Enter.

Now let us see what is the meaning of this Robots.txt code?

User-agent: Mediapartners-Google

This code is for Google AdSense robots, which manages what kind of Adsense ads will come on your blog and website.

User-agent: *

This code means that all search engine bots can crawl our blog. Asterisk (*) means all robot (googlebot, yahoo etc).

Disallow: / search

This means that wherever the first search keyword comes after your domain name, it will not crawl the search engine.

 Ex- https://www.stepstoblogging.com/search/label/Blogging

This link is the label name (Menu) of our blog. Similarly, all the labels in our blog will not be indexed.

Allow: /

This code helps to index your blog's Homepage by crawl.

Sitemap

Sitemap gives search engine information about all the existing and new posts of our blog. That is, whenever the web crawler scans our robots.txt file, sitemap will get information about all our published posts.

How to Add Robots.txt File Blogger

To add Custom Robots.txt file to Blogger, first of all login to Blogger and come to Dashboard.
Now you go to Settings and click on Search Preferences option.


  • In this page you will find a section Crawlers and indexing, where there will be an option of Custom robots.txt [Disabled]. To enable it, you have to click on Edit and tick Yes.
  • After this, copy the code given below and paste it in the box of Custom robots.txt and click on Save Changes.

Copy Robots.txt code
 User-agent: Mediapartners-Google
 Disallow:
 User-agent: *
 Disallow: / search
 Allow: /
 Sitemap: http://example.com/sitemap.xml
Use the sitemap of your blog instead of http://example.com/sitemap.xml inside the Note-Sitemap.

How to add custom robots.text file in Blogger,what is Robots.txt file, custom robots.txt blogger generator custom robots header tags, custom robots.txt generator, where does the robots txt file go,where to find robots txt, robots txt sitemap, robots txt not found, how to submit robots txt to google


I have explained the entire process of adding Custom Robots.txt file to your Blogger, you can also add robots.txt file to your blog by following it.

If you want to hide a post or page from here, then you should have proper knowledge of robots.txt because the misuse of its blog can completely damage the SEO of your blog.  Do not tamper with its settings at all without knowledge.

If you have any question related to custom robots txt ,then you can ask in the comment and you can like our Facebook page for similar blogging tips.

1 comment:

  1. Hey Naresh Yadav ,




    Excellent post with fab-work. I truly appreciate your hard-work.




    Glad to say that you have suggested helpful tips to add custom robotstext file in blogger. You have explained very nicely and cleared the concept too that is true enough to understand and educate the readers & learners as well. Your each of the suggested steps to add Robots.txt file in blogger ( including images ) are so clear and providing better understanding as well.



    Every time you comes up with an ideas that are truly engaging in every aspect and every time i get new ideas through your post.After going through this post i really got helpful ideas and have also boosted my knowledge.




    Eventually thanks for sharing your knowledge and such a helpful post.

    ReplyDelete