J!Extensions Store™
Forum
Welcome, Guest
Please Login to access forum.
Re:Error in robots.txt file after upgrading to v. 3.5 (1 viewing) 
Go to bottom
TOPIC: Re:Error in robots.txt file after upgrading to v. 3.5
#2992
Better Web
Fresh Boarder
Posts: 5
User Offline
Re:Error in robots.txt file after upgrading to v. 3.5 4 Years ago Karma: 0  
The latest robots.txt included in Joomla goes in that direction :

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

The /media/ folder is removed but you can still find js/css files (not to mention images) in /cache/, /plugins/, /modules/ or /components/.

A quick and dirty way to solve this is to remove these directories.
Then you can add your lines to restrict access to js/css/images files. BUT, you still need to specify some Disallow, unless you don't care about restricting anything. But remember, what googlebot can access can be indexed, and you may not always want that.

Before I ran into your solution, I used to be very specific, so I have robots.txt looking like this:

User-agent: *
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/

User-agent: googlebot
Disallow: /administrator/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/
Allow: /plugins/system/jch_optimize/assets2/
Allow: /components/com_fabrik/views/tmpl/

I use the list of blocked ressources from the Google Search Console to customize the file. it's more work but gives you the most control.
 
Logged Logged  
  The administrator has disabled public write access.
      Topics Author Date
    thread link
Error in robots.txt file after upgrading to v. 3.5
Betterliving 2015/08/14 13:34
    thread link
thread linkthread link Re:Error in robots.txt file after upgrading to v. 3.5
John Dagelmore 2015/08/14 14:13
    thread link
thread linkthread linkthread link Re:Error in robots.txt file after upgrading to v. 3.5
Betterliving 2015/08/14 16:04
    thread link
thread linkthread linkthread linkthread link Re:Error in robots.txt file after upgrading to v. 3.5
John Dagelmore 2015/08/14 16:11
    thread link
thread linkthread linkthread linkthread linkthread link Re:Error in robots.txt file after upgrading to v. 3.5
Betterliving 2015/08/14 16:45
Go to top