UPDATE
Popular free certificate authority Let’s Encrypt said it will revoke 3 million Transport Layer Security (TLS) certificates Wednesday, because of a Certificate Authority Authorization (CAA) bug. The move could mean that millions of websites and machine identities that rely on those certificates to protect sensitive data flow could be identified as insecure, or rendered unavailable.
Certificate users contacted by Threatpost said they were notified of the revocation Tuesday and given 24 hours to resolve the issue. Certificates will be revoked March 4, 9:00 p.m. EST.
“I manage 200 domains across 20 servers and have until the end of the day to fix the problem,” said Mark Engelhardt, IT consultant with Intuitive Engineering, in Montpelier, Vt. “Let’s Encrypt did not handle this in an ideal fashion at all.”
[UPDATE: One Wednesday March 4, Let’s Encrypt said it would: Push Back The Deadline to Revoke Some TLS Certificates]
Let’s Encrypt explained on Tuesday it had to revoke the 3 million certificates because of a CAA bug that impacted the way its software checked domain ownership before issuing certificates.
While the number of certificates impacted is 3 million, the actual number of websites or machine identities affected will likely be less, because of the way certificates are reissued and the fact that others are likely not currently in use, said Pratik Savla, senior security engineer at Venafi.
Savla warned that the bug could open the door for a malicious attacker to take control of a TLS certificate on a website, allowing the hacker to eavesdrop on web traffic and gather sensitive data.
Let’s Encrypt said a preliminary investigation determined the bug was introduced into its Boulder software in July — it was detected this past Sunday and patched the same day.
Josh Aas, executive director of Let’s Encrypt, said in a statement to Threatpost, “A bug was introduced in our code during a feature flag update. Under certain conditions, this bug caused us to skip a check that we are required to perform before issuing a certificate. We determined that the bug affected about 3 million, or about 2.6 percent, of our active certificates. Unfortunately, we need to revoke these certificates, which we will be doing within the compliance timeline set forth by the Baseline Requirements.”
Kevin Bocek, vice president of security strategy and threat intelligence at Venafi, said the impact of the bug means that “millions of machines could be unavailable or be untrusted tomorrow, potentially causing harm to the companies that rely on them.”
Besides the short notice, Engelhardt said another one of his concerns was that Let’s Encrypt notification sent him scrambling to manually check 200 domains to see which of the certificates he owned would be impacted. “The message we received was vague and did not help pinpointed specific certificates.”
In an interview with Aas he pointed out that for many of Let’s Encrypt customers correcting the problem won’t be a hardship. MongoDB, for example, had 15,000 certificates that were going to be revoked. “This morning system administrators there automated the job and were done in hours.”
“There are certainly some hardships here and we recognize that. But, the timeline in which we are operating is dictated to us,” Aas said. “We have a certain amount of time after we learned about an incident to respond.”
Those requirements, he said, are part of rules dictated in a document called the Baseline Requirements, published by the CA Browser Forum working group, made up of browser makers and certificate authorities, including Let’s Encrypt.
On one of the many message boards discussing the bug, one user “Kimbo89” created a script to automate the process. Let’s Encrypt also supplied a link to its own scanner to check for impacted TLS certificates.
Let’s Encrypt described the bug this way:
“The bug: when a certificate request contained N domain names that needed CAA rechecking, Boulder would pick one domain name and check it N times. What this means in practice is that if a subscriber validated a domain name at time X, and the CAA records for that domain at time X allowed Let’s Encrypt issuance, that subscriber would be able to issue a certificate containing that domain name until X+30 days, even if someone later installed CAA records on that domain name that prohibit issuance by Let’s Encrypt.”
Last week Let’s Encrypt issued its one billionth security certificate.
(This article was updated 3/3 at 10 pm EST with comments from Josh Aas, executive director of Let’s Encrypt)
