Google's new Ajax-powered search results breaks search keyword tracking for everyone

Do you run a web site? Do you use a service such as Clicky, Statcounter, AWStats, etc, to monitor the traffic to your web site? Do you find it useful how these services can tell you what search terms are leading visitors to your site? Have you ever wondered exactly how they are able to do that?

No, you probably haven't. But that's ok. Just know this: a major update that Google is testing has completely broken the ability for any external analytics service like Clicky to determine the search query used by a visitor arriving at your web site. Why would they do such a thing? Who knows. They aren't talking.

Knowing the search terms that are driving traffic to your web site is probably the #1 reason to use any kind of traffic monitoring service. And Google owns a huge chunk of the search market. If this update goes live for everyone, it effectively means that 2/3 of all searches leading to the average web site will be a complete mystery. This is huge.

So what is the problem exactly? Normally when do you a search on Google or any other search engine, the search term used become part of the URL. A search for Clicky, for example, gives you this:

http://www.google.com/search?q=clicky

When someone clicks a search result on that page, that URL above is sent as the "referrer" to the target site. An analytics app running on the target site can parse the referrer string and extract the word "clicky", and store that as a search that occured for that site. This is obviously very useful.

Here's what the new search result URLs look like with the new "Ajax" feature:

http://www.google.com/#q=clicky

See how there's a hash mark # in there now, and the "q=test" is after it? The problem is that web browsers don't send anything after the # in the referrer string. This means organic searches from Google will now show up as just "http://www.google.com/", with no search parameters. In other words, no analytics app can track these searches anymore. I started noticing lots of hits from just "http://www.google.com/" recently in our own search logs. I thought maybe it was just a bug with Clicky. But then one of our users contacted me about this article, and my jaw about broke from hitting the floor so hard.

This change isn't live for everyone yet so if you go to google.com and do a search, you may not see it in action. But it is happening for me when I go there and do a search.

So what can we do about it? If you run a blog, write about this. Submit this story or your own story to large tech blogs like TechCrunch, CenterNetworks, ReadWriteWeb, GigaOm, etc - no large site has written about this yet, and one of them needs to. Post in Google's Web Search forums (there's no way to directly contact them about web search unfortunately). Do anything you can to spread the word and let Google know that this is not acceptable.

Update: Google is watching