I'm in our data center today and tomorrow, doing maintenance and replacing some hardware on a bunch of our database servers to help prevent issues like we had last week from happening again. We only take down servers during the week if absolutely necessary, and this is one of those cases. This is because we will be in San Francisco for most of next week, and then I will be taking another small trip unrelated to work.

The last thing we want is to have a problem while we're on the road, because it may take much longer than normal to resolve depending on what we're doing at the time of such an incident (not to mention it's a much more of a PITA to do that type of thing while traveling).

Any given server may be down for as long as 3 hours. Data will not be lost during this time, however when a server does come back online, it will take it a while to catch back up with real time.

Our #1 priority is to make Clicky as reliable as possible for you. Thanks for your patience and understanding.

September 8th marked 1 year since we launched our iPhone web application. We've been meaning to add some new features for a while, and what better time to do that than on its first birthday? So we started adding some new features yesterday and just launched them tonight. It's still not meant as a full replacement for the desktop version, but these new features really add to the experience and make it that much more enjoyable to check your analytics on the go:

Visitor segmentation

You can now analyze any segment of visitors like you can in the desktop version. Just click on any item when viewing popular data, and you will see data about that specific segment of visitors:

 

Historical data / graphs

You can now view the historical data for any individual item. Just click the red/green percentage next to any item, and you will see its daily history. We're using the pretty amazing Google Charts API to generate these graphs, since Flash doesn't work on the iPhone.

 


You can view history in landscape mode as well, which will widen the graph to see better detail:

Organized dashboard

We've organized all of the menu options on the dashboard into groups, which makes it much quicker to find what you're looking for. As you can see here, we also added the new Short URLs data (from clicky.me) to the iPhone app:

More dates

You can now select dates going back as far as 6 months, including individual months:




Lastly, we wanted to let you know that the entire crew at Clicky (yeah, all 2 of us) will be at the TechCrunch 50 conference in San Francisco this coming Monday and Tuesday, so if you're going to be there, we'd love to meet up. Also, because of our traveling schedule (we'll actually be in San Fran for 5 days), please be patient if you send us an email. We'll of course still be checking them on a daily basis but unless it's an emergency, it may take a few days before we respond. Thanks for understanding. We look forward to seeing you there :)

At approximately 4am PST, two separate database servers (db1 and db16) had RAID failures that caused file system corruption. They kept trying to process traffic but Linux had switched part of the file system to "read only", so no traffic data was actually being written to the hard drives. This problem lasted from approximately 4am to 7am PST. Unfortunately, this traffic data is gone and unrecoverable.

We have alert systems setup so that when a significant event occurs, such as a server going offline or a RAID failure, we are alerted immediately. Unfortunately, the RAID notifications on a few servers were recently disabled while we were performing some maintenance, and wouldn't you know it, db1 and db16 were among those servers. Because of this, we weren't notified of the problem, and didn't discover it until we woke up to a flood of emails in our inbox this morning.

There were no problems on other servers that we could find, but if you have a site on a server other than db1 or db16 and it's experiencing issues, please leave a comment here explaining what's happening. Be sure to include the site ID.

We apologize for this issue, which we take very seriously. The RAID notifications are all back online, and we will be sure to always re-enable them immediately after this kind of maintenance in the future. Leaving them disabled was just an honest mistake.

One final note, these RAID failures occurred at the exact same time on two different servers. This happened once before as well, although it was three servers instead of two, and it didn't cause any corruption last time. This seems like very strange behavior to us, and we're not sure what could possibly cause such a thing to happen to separate servers (that don't talk to each other) at the exact same time. If any sysadmins out there have any ideas, please share.

The last thing the world needs is another damn URL shortener. There is no money to be made in this business, yet a new one launches pretty much every day. They're all mostly the same, offering some basic analytics like how many people clicked your link, and maybe the top referrers and geolocation of these clickers.

That's all well and good but there's something lacking: you don't get any data on your visitors after they arrive at your site. And that's the data that really matters. Who cares how many people clicked if you don't know what they did when they actually got to your web site?

That's why we're launching our own URL shortener, clicky.me. Here's what makes this product special:

The best analytics, period

Every visitor who arrives at your web site via a clicky.me link will be "tagged" with that link. This means you can filter/segment your visitors based on this URL, just like you already do with countries, referrers, etc. This gives you the full spectrum of segmentation data that Clicky offers, providing valuable insight into the actual impact of any link you share.

We made a brief announcement on Twitter about this, asking people to click a link to help beta test clicky.me. Here are the results, about 1 hour later:





If we had used another shortener such as the ever popular bit.ly, the only data we would have is the number of visitors, and the top referring domains and countries, as are shown in our screenshot above. That's great, but look how much more data we get from clicky.me!

We can get all this extra data because Clicky Analytics continues to track these visitors after they have arrived. We can see the bounce rate, the average pages per visit, average time per visit, and more. As you can see from the drop down menu that's open above the "countries" box, I can also view things like which goals these specific visitors completed, what web browsers they use, what cities they live - anything I want. Plus, since this is Clicky, you can actually see every individual visitor who arrived at your site from this link, so you can study them individually as desired. No other shortener gives you anything even remotely close to this.

The most accurate data

Just like Clicky only tracks human visitors, so does clicky.me. This means the numbers you get from us will be much more accurate and meaningful. Other shorteners log every hit, regardless if it's from a search engine, bot, or otherwise. clicky.me only tracks humans, so you'll know exactly how many people actually viewed your link.

No, we're not relying on Javascript for this :P We're doing it internally with a supar seakrat algorism.

Tons of ways to get your data

You can view the stats of your Short URLs by going to Links, then Short URLs. You can add the new Short URLs module to your dashboard (coming soon!). You can grab the data from our stats API, with type=shorturls.

Tons of ways to create short URLs

You can go right to clicky.me and paste in a URL. You can use our bookmarklet (available on the main clicky.me web site) to prefill both the URL field, and the site that the analytics will be attached to. You can create a short URL right from the Short URLs stats page (coming soon!). And we'll be launching an API in the near future to create them as well. We didn't consider the API a make or break launch feature, because we don't expect any of the major twitter application developers to integrate our service with them, since it requires an account. But we'll still add one eventually.

An actual business model (gasp!)

While clicky.me isn't going to make us money directly, it adds a lot of value to the service we already make our money from: Clicky Web Analytics. Any time we add value like that, more people upgrade to our paid services. As long as Clicky is around - and it's not going anywhere - clicky.me will be as well, so you can depend on us to keep redirecting those links well into the future.

(clicky.me is free for all users of Clicky Web Analytics. However, "free" accounts are limited to shortening 1 URL per day. Paid accounts have no limitations.)

Semantics, baby

Who can resist clicking a link that says "clicky me" right in the URL? Your click-throughs are going to skyrocket.


Well, that covers that. Enjoy!

One of the biggest complaints we receive is how long it can take to look up historical data that spans a range of more than a couple of days. I want to explain why this was and what steps we have taken to fix this.

Previously we had only been storing summary data on a per-day basis. So if you wanted to view an entire month's worth of searches for, say, July 2009, our system would have to do 31 queries (one for each day) to get all of the searches for the entire month, then sum all that data together before outputting anything. This is inefficient and could take quite a while.

To fix this, we have added systems that also summarize all of your data on both a weekly and monthly basis. So now if you do a query for July 2009 searches, it only has to do one database query, instead of 31. However, because there is more data in that single query output, the time it takes will be longer than just 1/31 of the original. For the average site it will be about 10-20% of the original size of data. (This is relevant because the more data it has to read off the disk, the longer it takes to get it). This means for the average site, large date range queries will take just 10-20% of the original time that they used to. That's a huge improvement!

The same thing applies to full week (Sunday - Saturday) queries. Instead of having to do seven queries per week, it only has to do one. And all of this works fine for spanning multiple weeks and months also.

(We've been running scripts in the background the past week or so that have been calculating summarizing this data into weekly and monthly chunks, all the way back to Jan 1. All servers are done with this, except db7, which should be done within 24 hours. At the end of each day going forward, it adds your daily totals onto the week and month that that date belongs to. This means that for data prior to Jan 1, these optimizations will not apply).

What if your date range doesn't fit exactly into full month or full week increments?

No problemo. We have designed this system to be very flexible, using the most optimized queries possible for whatever query you throw at it. Let's pretend instead of July 2009 (1-31), you queried for May 31 - July 15.



Here's how our new system deals with this query. This is a bit technical, but it's worth a read to learn how the system optimizes the queries, which will also allow you to create date ranges that are as optimized as possible.

First, it figures out all of the individual days in this date range. In this case, there are 46 full days. This is how many queries would have to be done under the old system.

Next, it checks if there are any full months in this date range. In this case there is one (June 2009), so it removes all of June from the daily queries, and adds that month to the monthly queries. We are now down to 17 queries, from 46.

It then checks if there are any full weeks (Sunday - Saturday) in the remaining single days. In this example, July 5-11 is the only full week. So it removes those 7 days from the daily queries, and adds that week to the weekly queries.

We are now at 11 queries - 1 monthly, 1 weekly, and 9 daily. This is down from 46 daily queries originally, which is ~75% less queries that need to be run, which will result in much faster response times. Hope you enjoy.