AWS Middle East outage: a reminder not to rely on cloud as disaster recovery plan

AWS customers based in the Middle East (ME) are struggling to recover services following drone attacks on the cloud company’s ME data centers on March 1. Two availability zones in the UAE and one in Bahrain were impacted.

The company has been providing regular updates as it works to restore operations, but has advised customers with workloads running in the Middle East to take action now to migrate those workloads to alternate AWS Regions. “Customers should enact their disaster recovery (DR) plans, recover from remote backups stored in other regions, and update their applications to direct traffic away from the affected regions,” the company said.

The ferocity of the attacks has exposed the inadequacy of some companies’ DR plans.

Need a ‘blast radius audit’

“This attack exposes something most enterprises have been getting wrong for years,” said Cisco’s Nik Kale, principal engineer, CX engineering. “DR plans are written around the assumption that failures are localized and technical — a power outage, a cooling failure, maybe a fiber cut. What happened this week is a region-level event driven by geopolitics, not infrastructure failure. If your disaster recovery plan doesn’t account for the possibility that an entire geographic region becomes operationally hostile overnight, you don’t have a disaster recovery plan. You have a maintenance play book.”

The attacks were not the sort of failure that most companies had prepared for, Kale admitted. “[But] enterprise architects need to be running what I’d call a ‘blast radius audit,’ mapping every critical workload to its physical region, identifying which services have single-region dependencies, and pressure-testing whether failover actually works when an entire region goes dark, not just when a single zone hiccups,” he said.

“The enterprises that will come through events like this aren’t the ones with the thickest DR binders, they’re the ones who’ve actually failed over to another continent.”

Activate DR plans now

AWS ME customers who haven’t already implemented comprehensive DR responses need to activate their plans immediately, advised Brad Lassiter CEO at IT services company Last Tech. “Customers need to failover to other regions and availability zones and check DNS and routing rules. Lower time to live (TTL) wherever possible so that the network can change traffic patterns as needed,” he said, adding that enterprises also need to shift to manual operations to verify high value transactions.

Those businesses looking for legal remedies to recover costs from the outages may be disappointed, said Frank Jennings, partner at HCR Legal, a lawyer specializing in cloud law. “Most AWS users probably didn’t check their SLA for outages caused by drone strikes! Nevertheless, most cloud SLAs will expressly exclude from their uptime commitments any downtime caused by events outside the provider’s reasonable control (a ‘force majeure’ event), including natural disasters, acts of terrorism, or war,” he said.

He said, however, that definitions of “force majeure” are often vague. “Its scope depends on the specific wording of the clause in question,” he noted.

Jennings advised AWS customers (and users of other hyperscalers’ services) to check their contracts, and not to “treat cloud service agreements as low-risk commodity purchases.” The force majeure clause, the SLA exclusions and the limitation of liability provisions all warrant close scrutiny at the point of contracting, he pointed out.

Re-evaluate cloud plans

The ME attacks will certainly force many organizations to rethink their plans going forward, Kale observed. “Most enterprises pick cloud regions based on latency and pricing” he said. “Almost nobody runs a geopolitical threat model against their region selection the way they’d run a capacity model. This week proved that your cloud region is a geopolitical decision whether you treat it as one or not.”

He noted that AWS’s own guidance is telling customers to do what they should have architected for from day one: have workload portability across regions, keep remote backups stored outside the blast radius, and have application-level traffic steering that doesn’t depend on the affected region being reachable.

AWS said it is making progress restoring services. In its bulletin at 8.14 a.m. PST on March 3, it said, “For Amazon S3, we are seeing continued improvement in PUT and LIST availability. Newly written objects are now able to be successfully retrieved.” It said it was still working on DynamoDB; other services would follow when this was restored, but EC2 instances remain throttled in the region.