How To Create Robots.txt File Online For WordPress

Creating robots.txt file Online

Creating robots.txt file Online : To creating Robots.txt file Online for a website plays mandatory role if you are using blogger or WordPress its damn easy don’t worry. In order to do perfect SEO (Search engine optimization) and drive organic traffic you should do some basic setup at initial stage.

During that basic setup you should have created this robots.txt file.If you have not created because of some misconception don’t worry I’ll make you clear In this article you will learn  steps to create robots.txt file online for wordpress.

Before landing on this page you might have searched in google as :

How to create robots txt WordPress file?

Importance of robots txt file ?

What is robots txt in seo?

What is robots txt generator ?

Creating robots.txt file Online ?

If you have searched of above statements then you have landed at exact page were you are going to learn something new today.

What Is Robots.txt File?

Creating Robots.txt file online is a simple text file where you have to upload it to the root directory of your server. It tells search engines that which part of your site should get crawled (Scan) and which part of your site shouldn’t be crawled.

Simply it contains the syntax for bots where to go and where to not go on your site.If you don’t configure Robots.txt file correctly your site will not appear on search engines.

But creating robots.txt file online is not that much complicated you just have to follow small steps which we are going to discuss in this article.

So no Worries!!!

What If I Don’t Create Robots.txt File ?

As I said above you will be out of search engines(You site will not appear on search engines) and many issues raises.

  1. Private data could be seen
  2. Might get duplicate issues
  3. Hacking attack
  4. Search engines will be in Dailamo
  5. Negative SEO

Guide To Create Robots.txt File Online:

To create a robots.txt file there is some syntax to follow with three words and they are User-agent, Disallow, Allow.

You might be getting doubt what are these words Wait!!!!.

User agent : Search engine crawlers.

Disallow : The File or directory or URL that you want to block.

Allow : To give access File or Directory or URL to bots.

Syntax Of Robots.txt File :

User-agent : *

Disallow : /

Allow : /

Steps To Follow :

  1. Go to Your Hosting Cpanel 
  2. Select File Manager 
  3. Under Public_Html
  4. Create New File and name it as robots.txt
  5. Now Click Edit option and Paste the Following Code.

Note : Every site doesn’t have same robots.txt file it varies according to there requirements.

Different Types of Robots.txt File With Meaning :

1. If you want to allow the robots to crawl your whole website, then you should write :

User-agent: *

2. To restrict crawling all the bots from your server, then you should write :

User-agent: *
Disallow: /

3. If you don’t want any specific file or folder of the directory of your blog to get crawled, then you should write :

User-agent: *
Disallow: /cgi-bin/

If you don’t want to crawl multiple folder or directory then add another disallow statement as like below.

User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-plugins/

For More Detail Watch This video :

How To Edit Existing Robots.txt File In WordPress :

If you dont have knowledge about Google Webmaster Tools then you edit your robots.txt file using a plugin called Yoast By SEO.

I can say definatly no blogger without knowing this plugin every one uses it.


Steps To Follow :

  1. Go to Dashboard.
  2. Hover you mouse over Yost SEO plugin Under select Tools Option.wp
  3. You can see 3 options as like above image.
  4. Select File Editor Option.
  5. You can see Create Robots.txt file click on it and edit your file as like you want.

If your WordPress site is new born and till date you didn’t monetize you blog with any type of Monitization like Google Adsense, you can use the below code as your robots.txt file.


User-agent: *

Disallow: /cgi-bin/

Disallow: /page/

Disallow: /wp-admin/

Disallow: /wp-content/plugins

Disallow: /wp-content/themes

Disallow: /feed/

Disallow: /wp-includes/

Disallow: /recommended/

Disallow: /comments/feed/

Disallow: /trackback/

Disallow: /index.php

Disallow: /xmlrpc.php

If your WordPress site is not new and if you are using Adsense ads and your blog, Then you can go with the below robots.txt file format.

User-agent: *

Disallow: /cgi-bin/

Disallow: /wp-admin/

Disallow: /recommended/

Disallow: /comments/feed/

Disallow: /trackback/

Disallow: /index.php

Disallow: /xmlrpc.php

User-agent: NinjaBot

Allow: /

User-agent: Mediapartners-Google*

Allow: /

User-agent: Googlebot-Image

Allow: /wp-content/uploads/

User-agent: Adsbot-Google

Allow: /

User-agent: Googlebot-Mobile

Allow: /


Reference : WpGlossy

Bottom Line :

So this is the end of the article you just read “Complete Guide How To Create Robots.txt File and Edit In WordPress“.

I hope this article helped you to create robots.txt file in very easy method. Do share your views about this post through comment section it is specially designed for you.

karamath sheik

My Name is S M Karamath Founder of which helps you to guide about such things which I have marked as categories.Blogging is like a part which I was doing for fun but now I converted into as Passion.

Click Here to Leave a Comment Below 0 comments