Hiccup in Akamai’s DDoS Mitigation Service Triggers Massive String of Outages

An hour-long outage hit airlines, banks and the Hong Kong Stock exchange. It’s thought to have been caused by a DDoS mitigation service.

UPDATE

Major financial institutions, airlines and the Hong Kong stock exchange were knocked offline by a backfiring distributed denial-of-service (DDoS) mitigation service Thursday. The hour-long outage, which was triggered at approximately 1 a.m. EST Thursday, is tied to Akamai Technology’s anti-DDoS Prolexic service.

In a statement to Threatpost at 7:44 a.m. EST, Akamai confirm a segment of its Prolexic platform was impacted and is now back up and running. “We are continuing to validate services. We will share more details of what transpired, but our first priority is ensuring all customer impact is mitigated,” wrote Chris Nicholson, senior public relations manager, Akamai.
According to reports, airline systems used by Delta, American, Southwest and United suffered outages at around 1 a.m. EST. Also impacted were financial institutions including Discover and Navy Federal Credit Union, according to a report by National Public Radio.

Australian based ITNews reported that services believed to rely on Akamai’s Prolexic were knocked temporarily offline.

“Users reported widespread problems accessing a range of internet properties and online services from the likes of CBA, Westpac, ANZ, UBank, AMP Bank, Macquarie Bank, ME Bank and more,” according to ITNews reporter Ry Crozier.

Mid-day operations of Hong Kong’s stock exchange were also impacted by the technical problems.

Virgin Australia also published a statement to its customers attributing an outage it suffered to Akamai’s Prolexic service.

“Virgin Australia was one of many organizations to experience an outage with the Akamai content delivery system today and we are working with them to ensure that necessary measures are taken to prevent these outages from reoccurring,” the airline said in a statement.

Akamai updated customers on the network incident at 12:21 p.m. EST stating

“Akamai experienced an outage for one of its Prolexic DDoS services (Routed 3.0) starting at 4:20 AM UTC. We detected the issue immediately and impacted customers received an error alert within seconds. The impact was limited to Akamai customers using version 3.0 of the Routed service.

Many of the approximately 500 customers using this service were automatically rerouted, which restored operations within a few minutes. The large majority of the remaining customers manually rerouted shortly thereafter.

The issue was not caused by a system update or a cyberattack. A routing table value used by this particular service was inadvertently exceeded. The effect was an unanticipated disruption of service. 

We restored the service by 8:47 AM UTC and customers began the process of routing back on to the service at that time. 

We recognize the impact of this issue and we extend our apologies to our customers and any of their end users who were affected.

We have taken steps to prevent a recurrence of this issue. We will also be working to make sure that every Akamai customer is set up for automatic rerouting in the future.”

Advice by Akamai posted to social media advised, “If you face the downtime, kindly turn off/route off Prolexic solution.” Early, in a tweet posted at 6:24 a.m. EST on Thursday, Akamai stated; “Akamai is aware of the issue and actively working to restore services as soon as possible.”

According to Akamai’s description, Prolexic is a DDoS mitigation service that can fend off “terabit-scale attacks”.

(This article was updated 6/17 at 2:26 p.m. EST with the updated statement from Akamai) 

Join Threatpost for “Tips and Tactics for Better Threat Hunting” — a LIVE event on Wed., June 30 at 2:00 PM ET in partnership with Palo Alto Networks. Learn from Palo Alto’s Unit 42 experts the best way to hunt down threats and how to use automation to help. Register HERE for free!

Suggested articles

Securing Your Move to the Hybrid Cloud

Infosec expert Rani Osnat lays out security challenges and offers hope for organizations migrating their IT stack to the private and public cloud environments.