J!Extensions Store™
Forum
Welcome, Guest
Please Login to access forum.
Google's new robots.txt warning (1 viewing) 
Go to bottom
TOPIC: Google's new robots.txt warning
#2821
Dave
Fresh Boarder
Posts: 10
User Offline
Google's new robots.txt warning 3 Years, 11 Months ago Karma: 0  
I started this thread in the Joomla Forum based on the Google messages about resources being blocked on my site. http://forum.joomla.org/viewtopic.php?f=708&t=892213&p=3319208#p3319208

Turns out Google wants to have the files in some of the default Joomla robots file that are blocked, allowed. So my question is how difficult would it be to make a extension or have your sitemap extension have some automated method to do this. Kind of like your add sitemap to robots file option.

The problem with doing it manually is figuring out all the files that need to be allowed and constantly updating the robots file. Just a thought.

Might be a nice paid extension for someone to present. Google is really off their rocker on this one. Threatening ranking penalties.
 
Logged Logged  
  The administrator has disabled public write access.
#2822
John Dagelmore
Admin
Posts: 3504
User Offline
Re:Google's new robots.txt warning 3 Years, 11 Months ago Karma: 75  
Hi Dave,

well this thread is out of topic here, has nothing to do with the sitemap extension.
I think that almost everyone has received the Google warning about JS/CSS files blocked.

The latest version of Joomla is ditributed already with the robots allowing access to common JS/CSS files thus the problem should not arise anymore.

In other cases normally it's enough remove the Disallow directive for:

Disallow: /images/
Disallow: /media/
Disallow: /templates/

Of course if some extensions uses JS/CSS files in its own folders this could not be enough, and other folders should be allowed in the robots.

Developing an extension that automates this could be an interesting idea.

Greetings

John
 
Logged Logged  
  The administrator has disabled public write access.
#2823
Dave
Fresh Boarder
Posts: 10
User Offline
Re:Google's new robots.txt warning 3 Years, 11 Months ago Karma: 0  
I checked into what Google wants and it's outside of the new robots.txt file shipping with Joomla 3.4.3 so my idea was a new way to manage this.

They are looking for files from 3rd party extensions in modules and plugins folder which is still blocked. Sorry to be off topic. I like your robots.txt editor and though of you.
 
Logged Logged  
  The administrator has disabled public write access.
#2824
John Dagelmore
Admin
Posts: 3504
User Offline
Re:Google's new robots.txt warning 3 Years, 11 Months ago Karma: 75  
I appreciate Dave, but i think that your post is only partially off topic.

I've an idea, developing a feature that parsing the page source code is able to detect the location of JS and CSS files configuring the robots.txt accordingly.

i will talk with some SEO expert and if this will result really important for Google it could be a new feature to include in JSitemap.

i will move this post in the Wishlist for the moment.

thanks,

John
 
Logged Logged  
  The administrator has disabled public write access.
#2827
Dave
Fresh Boarder
Posts: 10
User Offline
Re:Google's new robots.txt warning 3 Years, 11 Months ago Karma: 0  
Turns out there may be some easier methods. At least an easy way to make Google happy. May not be a good idea but it seems to work.
User-Agent: Googlebot
Allow: /*.js*
Allow: /*.css*
 
Logged Logged  
  The administrator has disabled public write access.
#2828
John Dagelmore
Admin
Posts: 3504
User Offline
Re:Google's new robots.txt warning 3 Years, 11 Months ago Karma: 75  
Thanks Dave, indeed i think that you are right! This will work
 
Logged Logged  
  The administrator has disabled public write access.
Go to top