I started this thread in the Joomla Forum based on the Google messages about resources being blocked on my site. http://forum.joomla.org/viewtopic.php?f=708&t=892213&p=3319208#p3319208

Turns out Google wants to have the files in some of the default Joomla robots file that are blocked, allowed. So my question is how difficult would it be to make a extension or have your sitemap extension have some automated method to do this. Kind of like your add sitemap to robots file option.

The problem with doing it manually is figuring out all the files that need to be allowed and constantly updating the robots file. Just a thought.

Might be a nice paid extension for someone to present. Google is really off their rocker on this one. Threatening ranking penalties.

Hi Dave,

well this thread is out of topic here, has nothing to do with the sitemap extension.
I think that almost everyone has received the Google warning about JS/CSS files blocked.

The latest version of Joomla is ditributed already with the robots allowing access to common JS/CSS files thus the problem should not arise anymore.

In other cases normally it's enough remove the Disallow directive for:

Disallow: /images/
Disallow: /media/
Disallow: /templates/

Of course if some extensions uses JS/CSS files in its own folders this could not be enough, and other folders should be allowed in the robots.

Developing an extension that automates this could be an interesting idea.

Greetings

John

I checked into what Google wants and it's outside of the new robots.txt file shipping with Joomla 3.4.3 so my idea was a new way to manage this.

They are looking for files from 3rd party extensions in modules and plugins folder which is still blocked. Sorry to be off topic. I like your robots.txt editor and though of you.

I appreciate Dave, but i think that your post is only partially off topic.

I've an idea, developing a feature that parsing the page source code is able to detect the location of JS and CSS files configuring the robots.txt accordingly.

i will talk with some SEO expert and if this will result really important for Google it could be a new feature to include in JSitemap.

i will move this post in the Wishlist for the moment.

thanks,

John

Turns out there may be some easier methods. At least an easy way to make Google happy. May not be a good idea but it seems to work.
User-Agent: Googlebot
Allow: /*.js*
Allow: /*.css*

Thanks Dave, indeed i think that you are right! This will work