Whilst reviewing some error logfiles over the weekend, it became apparent that a rogue crawler called 192.comAgent was trawling through client websites. This crawler ignored the disallows in the robots.txt file.

To avoid the problems that this could cause, we have now added this to the list of banned user-agents. To complete this we detect the HTTP_USER_AGENT server variable and redirect undesirables to the custom error page.

 Problem over!