The Amazon S3 Outage - When the Internet's Hard Drive Fails …
March 06, 2017

Denis Goodwin
SmartBear

Share this

Last week, SmartBear observed a sudden and protracted 5X increase in web page timeout errors associated with the failure of Amazon's S3 cloud-based storage service. Looking a bit more closely at our data, we dug up a few more interesting angles on the impact of the failure.

Event Timeline

The issue hit suddenly – we saw an immediate spike in errors at 12:35 p.m. EST, and by 12:45 p.m. EST, error rates were 5X normal. For some specific types of timeout errors, the spike was more than 10X normal. At 3:30 p.m. EST, the error rate began dropping and by 3:50 p.m. EST, rates had returned to normal.


Web vs. API

The issue hit web pages hard, while API monitors were not noticeably impacted by this outage. Web pages and web apps often utilize content storage hosted by cloud services such as Amazon S3.

Common failure scenarios on Tuesday included page elements failing to load, which could cause either the whole web page to time out or specific content on a page might not render. Depending on the design of a given page, this partial content failure could be relatively minor or it could render a critical web journey non-functional. File uploads and downloads that rely on S3 storage endpoints were particularly hard hit.

In order to get a complete picture of application health, it's necessary to monitor your real user's journey through the application. The monitored user journeys that depended heavily on content hosted in S3 failed. Those that didn't have that dependency continued functioning. I personally experienced this with Slack – I was able to use the app, however files could not be uploaded presumably because these files are stored by Slack using S3 as the storage mechanism.

While far less pronounced than the spike in errors, some response time degradation was observed in API monitors that continued running successfully. Given that the issue affected Amazon's storage services rather than their hosting services for applications, this makes sense.

Geographic Impact

The issue was more acutely felt in the United States, but we observed impacts all over the globe. The spike in page errors was seen on websites dependent on Amazon S3, many of which are U.S.-hosted websites that are likely monitored from U.S. locations. Unsurprisingly, error counts spiked by as much as 25X in some U.S. monitoring locations. While not as significant as the U.S. locations, timeout and page error increases were also observed from Canada, Europe and Asia.

Takeaways

Much of the web is built on the backs of cloud providers. Most of the time, these cloud services provide a great user experience. Amazon will learn from the root cause of this issue and likely emerge from this outage more resilient than ever. It's impossible to control all aspects of these shared services – but here are three steps to take that are in your control.

1. Identify your business critical applications

2. Proactively monitor user journeys on these applications

3. Don't rely on your third party provider to tell you when it is down

It is key to utilize independent monitoring services to ensure your applications are up, functioning correctly and fast. Furthermore, missing content can be catastrophic or merely inconvenient to a critical user journey – it's important that your monitoring tool can be configured to know the difference.

Denis Goodwin is Director of Product Management, AlertSite, SmartBear Software.

Share this

The Latest

August 08, 2022

Hybrid and remote work environments have been growing significantly in the past few years. As individuals move away from traditional office settings in today's new remote and hybrid environments, many operational issues such as poor visibility into asset status and refreshes, unaccounted assets, and overspending on software are becoming a bigger challenge for IT departments ...

August 05, 2022

MLOps or Machine Learning Operations are a combination of best processes and practices that businesses use to run AI successfully ... While it is a relatively new field, MLOps is a collective effort that captured the interest of data scientists, DevOps engineers, AI enthusiasts, and IT ...

August 04, 2022

The data is in: enterprises are not happy with their managed service providers (MSPs) and cloud service providers (CSPs). According to the latest CloudBolt Industry Insights report, Filling the Gap: Service Providers' Increasingly Important Role in Multi-Cloud Success, 80% are so unsatisfied with their existing MSP and/or CSP, they are actively looking to replace them within 12 months ...

August 03, 2022

The last two years have accelerated massive changes in how we work, do business, and engage with customers. According to Pega research, nearly three out of four employees (71%) feel their job complexity continues to rise as customer demands increase, and employees at all levels feel overloaded with information, systems, and processes that make it difficult to adapt to these new challenges and meet their customers' growing needs ...

August 02, 2022

Investing in employees will always be smart business. And right now, investing in employees means giving people the resources — and ability — to optimize performance ... For pretty much every company, that means delivering the digital tools necessary to facilitate seamless, secure, user-friendly access and connectivity ...

August 01, 2022

Digital transformation can be the difference between becoming the next Netflix and becoming the next Blockbuster Video. With corporate survival on the line, "digital transformation" is no longer merely an impressive buzzword to throw around in boardrooms. It's the ticket for entry into the digital era, a fundamental business strategy for every modern company ...

July 29, 2022

IT infrastructure has rapidly evolved over the last decade, and as a result important specialized tools have been developed and an entire dedicated industry has grown up to serve the need for monitoring these IT systems and services in order to keep them operational and efficient ...

July 28, 2022

At Cisco AppDynamics, we recently conducted research exploring consumer attitudes and behaviors in relation to wearable technology ... In our study, 87% of global consumers claimed that trust is a critical factor when choosing a wearable medical device or application brand. And, 86% expect companies offering wearable technology and applications to demonstrate a higher standard of protection for their personal data than any other technology they use ...

July 27, 2022

You've been here before: waiting for a web page to load. You keep refreshing it, but still no luck. How many times will you try to reload the page before visiting a different site? Probably not too many. Brands today have just a few moments at most to captivate and delight potential customers ...

July 26, 2022

In the DevOps world, observability is trumpeted and lauded in many corners. However, in reading much of the coverage, there seemed to be some more fundamental issues at play. It's time to demystify the idea of observability, shedding light on what it means in a broader context. And once we break down the concept and its true value to an organization, let's answer a more important question: Are we approaching an observability tipping point? ...