Hi John

GWT is showing me that I have 1677 url's that are blocked by robots.txt. Here is an example of a url that is blocked from an extension I have: https://www.mywebsite.com/modules/mod_weather_gk4/icons/default/32/chance_of_storm.png

I think my robots.txt file is correct, here it is below:
----------------------------------------------
# If the Joomla site is installed within a folder such as at
# e.g. www.example.com/joomla/ the robots.txt file MUST be
# moved to the site root at e.g. www.example.com/robots.txt
# AND the joomla folder name MUST be prefixed to the disallowed
# path, e.g. the Disallow rule for the /administrator/ folder
# MUST be changed to read Disallow: /joomla/administrator/
#
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/orig.html
#
# For syntax checking, see:
# http://tool.motoricerca.info/robots-checker.phtml

User-agent: *
Disallow: /administrator/
Disallow: /bin/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/
----------------------------------------------

How do I stop my JSitemap pro including the 1677 links when submitting the sitemap? As you see from the url error link I use SSL on my website.

Many thanks

Hello Gary,

you need to remove the line: Disallow: /modules/
from the robots.txt in order to allow indexing of images such as: https://www.mywebsite.com/modules/mod_weather_gk4/icons/default/32/chance_of_storm.png

Keep in mind that after editing the robots.txt Google will need few days to refresh it, that's not instant. After the robots has refreshed you can resubmit the sitemap and warnings for blocked urls should not appear anymore.

Greetings,

John

John Dagelmore wrote:

QUOTE:
Hello Gary,

you need to remove the line: Disallow: /modules/
from the robots.txt in order to allow indexing of images such as:
https://www.mywebsite.com/modules/mod_weather_gk4/icons/default/32/chance_of_storm.png

I was thinking that the better solution rather than allowing the bot access would be if the module url was not submitted by the sitemap as it serves no purpose in being indexed by a search engine. Is this possible?

Thanks for your speedy help.

I think you are referring to the images sitemap, if so the answer is yes, if images are not needed to be indexed you can remove them from the sitemap.
You can configure exclusions filters for each data sources involved in the images sitemap to discard images with path containing 'modules', simply adding a comma separated filter.

If needed more info can be found in the pdf manual or at this forum thread: http://storejextensions.org/forum/view/10-jsitemap/1379-re-tips-for-beginners.html

John