The Cloud solves everything, right?

Subject tags: 

So, the much-vaunted launched today, purporting to show street-level crime figures for the UK. I’d provide a link to it, but there’s no point, because it doesn’t work. Instead, I’ll link to this BBC article which says, essentially, that it got Slashdotted.

Ah, who could have foreseen the demand? Well, I could have, for one, because every time there’s a major UK data-mining web site launch, it gets slashdotted. Remember the 1901 Census? I want to quote this BBC article from 2003. Sorry, I’ll just say that again: from 2003:

The National Archives must learn the lessons from the episode if it puts the 1911 census online, the report says. It says the level of interest in the website had not been expected by the Public Record Office (PRO), which had “developed a pre-launch strategy based on a low key launch”.

The website was launched on 2 January 2002 with the aim of catering for a maximum of 1.2 million users in a 24 hour period. However, just three hours after being launched, by noon 1.2 million users per hour were trying to access the site - with the same level of demand continuing during the next few days.

The report says this demand “overwhelmed the site” and led to the PRO and its contractor, QinetiQ to close it and launch an investigation.

That was 2003, and did “learn the lessons” of how to launch a public data site? Perhaps not.

The developers, Rock Kitchen Harris - whose fault this all is, frankly - placed the site onto the Amazon Electronic Compute Cloud. That’s a good start. After all, if you put your website “in the Cloud”, then it’s scalable, right?

And that, in a nutshell, is the problem. Because if anyone says that to you, they do not deserve to be anywhere near a major web application. (And they certainly don’t deserve to be anywhere near £300k of public money, but that’s for another time.)

No. No no no. “The Cloud” and “scalability” are really orthogonal concepts. “The Cloud” is not actually magic. It doesn’t solve all of your problems for you. In fact, it doesn’t solve any of your problems for you - it just gives you the tools you need to solve your problem, but you still have to use them.

EC2 and other cloud providers really are, at the end of the day, glorified vhost providers with more computer power than you can find elsewhere. You still need to design your application based on a scalable architecture. You still need to split front and back ends, replicate and/or shard your database (or better still, rewrite to use SimpleDB), use MapReduce, do whatever it takes. You still need to know what you’re doing. Scalability isn’t automatic.

If you don’t know what you’re doing, if you don’t use the tools provided for scalability, then sure, you can throw more instances at the problem - for as long as you’ve got the money - but that’s your only option, and all that achieves is turning EC2 into a Redundant Array of Slashdotted Webservers. Merely running your code on Amazon’s servers doesn’t make it faster.

One and a half years of testing (although obviously not load testing) and it crashes on day one; a perfect example of how not to design and deploy for the Cloud.