Robots.txt file is so much important to control what you want to share and what not on SE. So its very important to right the correct codes in to your robots.txt file or else you can down your blog SEO. So here we have a basic codes for newbies that will control his bold security and SEO too like pro.
Table of Contents
What Is Robots.txt?
Robots.txt is common name of a text file that is uploaded to a Web site’s root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private.
Where Is WordPress Robots.txt File?
Every site should have robots.txt that will help to secure and share his site. The robots.txt file URL should be same for every sites/blogs like at
How To Add Custom Robots.txt Tag In WordPress?
Step 1:) First of all, open
http://www.example.com/robots.txt and see that your blog contain a Robots.txt or not.
Step 2:) If yes then go to your FTP and open that folder where you have WordPress core files.
Step 3:) Now see there that is there any file named Robots.txt
Step 4:) If you will find it then edit it and if not find it then create a file with this name there.
Note: Some time, there will be no files with name Robots.txt but there is a file opening at
http://www.example.com/robots.txt so it means that you have a SEO plugin that is creating virtual Robots.txt. Dont worry, still follow all the steps.
Step 5:) After creating the file, add the below code there.
User-agent: * Allow: / Allow: /wp-content/uploads/ Disallow: /feed/ Disallow: /trackback/ Disallow: /wp-admin/ Disallow: /wp-content/ Disallow: /wp-includes/ Disallow: /xmlrpc.php Disallow: /wp- User-agent: Googlebot # Allow all files ending with these extensions Allow: /*.js$ Allow: /*.css$ Allow: /*.png$ Allow: /*.gif$ Allow: /*.jpg$ Allow: /*.woff$ Sitemap: http://www.example.com/sitemap.xml
Step 6:) Now save that file and you are done.
You just have to change
http://www.example.com/sitemap.xml with your sitemap URL. Add this line again and again if you have many sitemaps.
How To Check Your Robots.txt File?
You can check this file on your blog by adding
/robots.txt at last to your blog URL in the browser like
http://www.example.com/robots.txt. Once you visit the robots.txt file URL you will see the entire code which you are using in your custom robots.txt file so its confirmed that you did right.
Here we share complete basic knowledge on custom robots.txt file in WordPress. We tried to make this tutorial effective and simple. But still if you have and doubt or query then feel free to ask me. Think twice before edit it because a single mistake can lead your blog out of SE. Like it than share and comment here…