Microsoft's spam bot has been blocked

hurray! I was also getting a ton of these listings.

Thank you,

I only have a small site with a few visitors a day and almost as many Microsoft bots, so it will good to see just my visitors again.

Phil.

I'm glad you are excluding these from visit counts. But I don't think this is as evil as you imply. Basically, google does the same thing, just on a smaller scale. When spiders normally visit, they just grab the raw html and do not try to appear to the web site as a regular visitor with a browser. But I get this one occassional googlebot visit which DOES try to impersonate a browser and which is a googlebot visitors. Therefore, it shows up in my clicky. I think what google is doing is trying to check up on web sites to make sure they aren't giving the regular spider's different content from users. This is a sort of spotcheck against fraud where spiders might get more SEOed content in comparision to users.

Fast forward to Microsoft. They are probably just doing the same thing. Except, I'd bet that whoever programmed this did so in a rather clumsy way where they probably take a list of the top million (or so) keyword searches. Then their spider first spiders the top 10, or top 20 results for each of those searches on live.com Next, the spider then similates a click on that page to get to the actual site, where it then grabs that page as a part of what is really an "audit", just like the Google audit.

This really makes a lot of sense. They probably asked themselves, "If it is too much trouble and resources to audit ALL pages/sites in our database, which are the most important to audit?" Then they realized that the ones that are most important to audit are the ones which rank the highest for the most common searches.

And maybe similating a click was just the easiest way to program this. It ensures that they are getting the latest live.com data (not just a prototype or version, but an end-to-end FINAL product. No possible software bug could screw that up... and this ensures that the are only auditing the top X number of pages for each search)

They then compare their capture with their regular spider capture of that site and, if there are any differences, then he site owner is "caught" cheating and a penalty is imposed, such as a drop in ranking or complete removal from the index.

NOTE: This is totally an educated guess. But I'd bet money that I'm right (if I had any right now!)

If I'm right, i don't fault them much, if any for this... and Google does it too... just not as frequently.

That does make sense. But there's a problem. No one uses Live, which means no one is trying to scam it to get higher search results. So why would they even worry about it? :)

I don't think that live.com is checking for SEO cloaking etc. I see MSN "visitors" for search terms that I don't even rank for. It would be great if I did rank for "schools", but when I view the Live search results for "schools", my site is nowhere to be found. Yet, I my getclicky analytics showed that I got visitors for that search. I was wondering what that was all about. Thanks for clearing this up! For shame. -dan

That's what I hear too, that most of the searches people see don't even make sense for their site. The ones we see for Clicky typically do, as you can see from the screenshot, but I don't understand why they would be faking keywords for your site. It just all seems really slimy.

While it's good to see the Seattle bot ignored, it's gratifying to see an increase in my rankings on their site as a result of their crawl. On many of the sites I manage traffic a year ago from MSN was virtually nil, today it accounts for around 5% of traffic. Probably more if I look at US visitors.

Is there a Google bot, too, one from Germany? Is it from Google? Is it legit?

This issue with this bot isn't that it's a crawler that acts like a browser - it's that it spoofs referrer and keyword data, and does not provide any identification that allows users to filter its visits (or obey robots exclusions). MS claimed that it is part of "quality checks" which I find to be a far from satisfactory explanation - and invites speculation that the side effect of artificially inflating stats is potentially not unwanted.

Well done Clicky for excluding it.

Are you aware of something called robots.txt ?

You think a bot with this little respect for other people's sites is really going to respect robots.txt?

I was getting loads of visits from these guys, and suspected it was some kind of shabby Microsoft trick to big up their search service.

Happy to hear you're blocking MSN -- I've seen weird search terms showing up for keywords that I couldn't possibly rank for -- just assumed it was some lame MS bot. Good to hear you're on top of it. http://www.instarvalley.com

Are you aware of something called robots.txt ?
http://www.playerassist.com/maple-story

It's real, and Microsoft has admitted to doing it in the past, but they promised to fix it long ago. Looks like they "fixed it" temporarily and then once the storm died down, they came back at it again, hard. I've personally interviewed them in the past on seo-scoop.com so there's no doubt at all that it's real. It was wrong then, and it's wrong now. Glad you are taking a stand.

This fake IE6 bot MS is using can be blocked by checking the ACCEPT_ENCODING environment variable.

Normal IE6 will provide the value 'gzip, deflate'. This new bot will not.

This is just their way of trying to make the world think IE6 still has market share. Imagine how many sites won't even notice, and be deceived into thinking IE is still legitimate in today's market.

> This fake IE6 bot MS is using can be blocked by checking the ACCEPT_ENCODING environment variable.


Or you can just use getclicky.com for all your analytics needs and not worry about it :)