Amazon S3: For now at least, sometimes you have to reboot the cloud

commentary Amazon.com's Simple Storage Service, S3, spent a few hours Sunday in a big pothole on the road to the glorious cloud computing future, with an outage taking the storage system offline for several hours Sunday. Should we be surprised?

No. In short, the computing industry is making up what's called cloud computing as it goes along, often with a server and networking architecture that's one part improvisation to two parts proven best practice. Frankly, it's notable to me that some services are as reliable as they are.

Some Amazon Web Services were down for hours on July 20. (Credit: Amazon)

Computing practices tend to gravitate toward one of two poles. One is tight control, higher prices, and high reliability. The other is openness, lower cost, but some degree of flakiness. High-end mainframes and Unix servers can handle transaction loads that would crush most machines using Intel or AMD x86 processors, but they cost more and are less adaptable. Most of the cutting-edge, large-scale action in the Internet -- including various cloud computing efforts -- is happening with the more free-wheeling technology.

One company operating at colossal scale, Google, has concluded it's better to buy cheap x86 servers and write software that automatically paves over hardware failures. The bigger problem comes when a large system composed of many interacting components loses track of its self-conception, and rebooting a single system or swapping out a hard drive isn't sufficient.

Essentially, Amazon had to reboot S3. Here's how the company described its S3 problem in a statement:

"As a distributed system, the different components of S3 need to be aware of the state of each other. For example, this awareness makes it possible for the system to decide which redundant physical storage server to route a request to. We experienced a problem with those internal system communications, leaving the components unable to interact properly, and customers unable to successfully process requests. After exploring several alternatives, the team determined it had to take the service offline to restore proper communication and then bring service online again. These are sophisticated systems and it generally takes a while to get to root cause in such a situation," Amazon said. "We will be providing our customers with more information when we've fully investigated the incident."

Afterward, Om Malik called cloud computing frail: "The S3 outage points to a bigger (and a larger) issue: the cloud has many points of failure -- routers crashing, cable getting accidentally cut, load balancers getting misconfigured, or simply bad code. And he's right, to a degree, but there are three things that shouldn't be overlooked before writing cloud computing off as a failure.

• First, you should compare the problems of cloud computing to the alternatives, including running computing services in-house. Last I checked, corporate data centers also have crashing routers, bad code, and misconfigured load balancers.
• Second, you can expect reliability to increase as the companies providing cloud infrastructure and services figure out explore the terra igcognita.
• Third, don't confuse Web 2.0 with the foundational elements of cloud computing. A Web site that uses an online application at another site to mash up data from some other sites then present it using a service from yet another site is indeed susceptible to numerous points of failure. But a single-purpose infrastructure such as Amazon S3 is at least in theory a more tightly controlled, single-purpose utility that can offer higher reliability.

That's not to excuse Amazon's outage or gloss over the effect it had on business partners reliant on it. After all, S3 is the sole part of Amazon Web Services that comes with a service level agreement to promise customers reliability.

But a little silver lining to this particular cloud problem is that Amazon is setting expectations at the right level: They said in a statement, "Any downtime is unacceptable, and we won't be satisfied until it is perfect."