3 reasons the Technorati move was an absolute disaster
After “several months of preparation”, Dorion Carrol from Technorati announced late last month that they would be moving two colos and that they hoped to be finished by the end of May. I had no idea what a colo was (and still might not) but from what I gather it’s short for ‘co-location’ so presumably they are consolidating two locations into one.
With 600 servers being moved Dorion mentioned the possibility of outages, both planned and unplanned but what followed over the next few weeks (and is still going) has become an utter joke.
Now Technorati isn’t a googlesque giant with millions of dollars up their sleeve and I’m not expecting them to go out and buy 600 servers to ensure that there is absolutely no downtime for their users. Technorati, like my own site, is modestly supported by advertising and offers a free service.
What I do expect however is that in the year 2009 server moves should be efficient, fast and with minimal downtime. By my calculation after 27 days (and counting) you’d be moving close to 22 servers a day and then setting them up. This is painfully slow, are they moving their colos by bicycle or what?
In the interest of end-user sanity and hoping this never happens again, here are the three biggest mistakes Technorati have made over the last month:
The only thing more annoying then having a service disabled due to maintenance is having access to it but it in an unstable state.
With a sense of complete and utter randomness, over the past month Technorati services such as blog pinging, searching the blogosphere and headline updates on the main page have been relentlessly up and down.
The principle of limited unreliable access applies to anything, not just server moves. From a television that intermittently randomly changes channel on you to an internet connection that keeps breaking authentication requiring you to reconnect (pre auto login router days), the fact of the matter is having something half break down on you is just so much more annoying then having it gone for good.
For example recently my dvd player which I used to stream divx onto my television began randomly turning off and on everytime I tried to play something. I put up with this for weeks often sitting there hitting play on my remote over and over again until finally it worked.
It wasn’t until one day I ripped it from my tv stand and threw it as hard as I could against the floor completely destroying it that I realised how annoyed I’d become with the unit. As I swept up the shattered pieces a sense of peace drifted over me and I looked forward to replacing it with something that worked all of the time.
Don’t leave your users hanging with extended periods of unreliability when it comes to your services. Your users are far less likely to crack the shits if they don’t have access to various features altogether whilst you move your servers.
In the year 2009, one month of random downtime is simply unacceptable for a site that deals with cataloging the often up to the minute pace of current blogging entries worldwide.
I only use the site primarily to glance at the front page and I’m kind of over seeing blog posts from 8 days (and counting) ago on the front page. It’s outdated, unprofessional and a waste of uptime. Why have your site appearing to be functional if it isn’t?
If you cannot move some servers (even 600 of them) in less then a month then clearly you either hired the wrong people, or you’re in way over your head. It just shouldn’t happen, period. 2 weeks of downtime tops, which in internet land still is a massive time scale.
This one is simple, if you’re going to post updates about the progress give users detailed information. Hearing ‘we’re working on various issues’ for a month straight is a surefire way to just piss everyone off.
If you have a quick glance at the last month of posts on the Technorati blog (look at Dorion Carroll’s posts), it’s easy to feel that they’ve haven’t really fixed anything over the course of the past month.
With each new posts comes a whole host of new problems and very little on anything actually getting fixed. If you haven’t fixed stuff your users don’t want to hear about it. Posting problems upon problems just makes it look like the site is falling to pieces.
If even more stuff is broken and your site is live (Technorati has been painfully live during all these problems) then believe me people already know about it. Stop wasting time telling us about it and fix them.
“Hey guys we fixed problem A and then this, this and this happened” looks much more promising then “So we still have no idea what’s going on with problem A but in the meantime we’ve also found problems B through to H-oh great our cheeseburgers have arrived. Rest assured we’re working hard on the problems”.
Seriously guys, this has been such a shambles it’s not funny.
If a feature is down or not working, take it down while you fix it. Don’t let people use it until it is over 75% functional.
Plan your move properly. Create a timetable and stick to it. Plan for the worst, if you’re moving 600 servers assume that once you plug them all in nothing will work and you’ll need to spend some time setting it all up again. That way if things do go smoothly your users will appreciate the earlier uptime.
Don’t update unless you’ve got some good news. It’s no good posting an update on how the move is going and that you’re working on some things when 90% of your site is frustratingly useless. Work on something, fix it, post about it and then move on.
I know moving 600 servers can’t have been an easy task but surely there has got to be a better way then the absolute disaster that has been the Technorati move of May 2009. I don’t know who was responsible for planning the move but they’ve done a terrible job.
Twenty seven days after the move was announced the site is still hopelessly broken.
May 29th, 2009 at 4:46 am Van(Quote)
First – co-location (in the usage here) is where a provider has space, bandwidth options, power, back-up provided in a data center. The client then provides hardware that utilizes said services.
So, what was taking place was moving from one data center to another.
Second – HOLY GOD HAS THIS BEEN BAD.
There is no reason, NO REASON, this should have taken nearly a month to do. I was involved in a 900 server move that took less than a week. The majority of the time was spent prepping the new servers. All we did was flip the switch and we were done.
What is sounds like is they were physically moving each server from location a to location b. With 600 servers that is impractical. Machines are so cheap they could have easily purchased what they needed at minimal cost – especially for a company of their size.
Second, what this has done is make me question the need for Technorati. I mean to say, the world survived without them during this move, and bloggers turned to there services for aggregation needs during this time, is there any reason for the service to really exist any more?
Anyway… I’m on the same page with you here. Handled very poorly on their part and – I think – goes to show that Technorati may be irrelevant.
May 29th, 2009 at 4:46 am Van(Quote)
I guess that second “Second” should be a third. Just goes to show what happens when I start ranting.
May 29th, 2009 at 9:30 pm ozsoapbox(Quote)
Thanks for clearing up what colo meant, I had a feeling it was something like that.
At a month’s downtime forget about moving each server seperately, more like they’re moving the components from each of the 600 machines one a time.
I still think there’s a need for Technorati. I like browsing to the main page and seeing the headline posts of other blogs – even if it’s the same featured mega blogs every day.
May 30th, 2009 at 2:44 pm Van(Quote)
The mega blog aspect is why I really started to move away from Technorati. There was a time when I could find so much unique content, but it is long past for me. As a results, I moved over to http://popurls.com/
Someone should be fired because of that move, no reason for it!