Scritto il: mercoledì, 7 gennaio 2009 - 17:03:25

JobCrawlerBot

JobCrawlerBot

How often does JobCrawler check the linked pages?
Normally the website linking is updated every hour. Anyway this schedule is not followed in case of data updating or maintenance.  In this case the information that were not linked are retrieved with the following scanning . If you notice out-of-line data, please contact us at
infoJobCrawler.it .

How can I check if JobCrawlerBot accesses my website? What User-Agent do you use to link websites?
You can check the access log from your webserver and look for “JobCrawlerBot 1.0” as User-Agent, or more simply you can email us at
infoJobCrawler.it  and we will be glad to answer you.

How can I make JobCrawlerBot not link some parts of my website?
Through the standard document robots.txt, you can tell JobCrawler not to link some pages of your website. Robot.txt format is  showed in the Robot exclusion standards. When JobCrawlerBot links a page, it ensures if in robots.txt there is an  exclusion checking if its User-Agent is listed in this file. If it’s not listed, there is a following check by “*”.
Robots.txt specifics are available on http://www.robotstxt.org/wc/exclusion.html#robotstxt. To manage robots.txt files all you need is to have this file in your webserver and it will work as a filter for JobCrawlerBot and for every other search engine.

Why does JobCrawlerBot look for the file robots.txt, which I don’t have on my website?
Robots.txt is a standard that indicates to the search engines if they are allowed to link the site on their pages. To avoid the “not found file” error in your log, you just have to create an empty robots.txt file.

Why does JobCrawlerBot try to download non- existing pages on my website?

JobCrawler is programmed to link in specific times, some specific pages. If the structure of the website changes and the search engine doesn’t realize, there can be period when the old pages are still checked. In these cases as soon as the error is found, it’s immediately fixed. In case you can also email us at
infoJobCrawler.it

Why isn’t JobCrawlerBot following the guidelines of the file robots.txt?

JobCrawlerBot does follow the guidelines in robots.txt. Anyway it can happen that for software errors or configuration problems the indications are not followed in the correct way. If this is the case, first of all check if you file is correctly configured and if it accords to the standard http://www.robotstxt.org/wc/exclusion.html#robotstxt. If everything is correct, send an email to
infoJobCrawler.it  indicating the wrongly linked page.

I want JobCrawlerBot to link my job offers website. How can I do?
Write us at
infoJobCrawler.it, we will answer as soon as possible.