DigiCert has delayed revocation beyond what's allowed in the Baseline Requirements a few times; most recently, https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 and https://bugzilla.mozilla.org/show_bug.cgi?id=1910805. In the former case, it seems DigiCert chose to delay revocation to appease certain clients; in the latter case they were prohibited by a Temporary Restraining Order (TRO) from performing timely revocation.
Tim Callan from Sectigo has publicly lambasted DigiCert for these delays, since in both cases it seems DigiCert hasn't pushed back hard enough on its clients. In the latter case, there's concern that measures like TROs might be employed more often to stall revocation. Sectigo (and others in the WebPKI ecosystem) seem to want DigiCert to make the revocation policies very clear to clients and to ensure that clients can actually replace their certificates in a timely manner.
Sectigo is clearly the most vocal but they don't seem to be the only ones telling DigiCert to get their delayed revocation under control. So the escalation to legal threats is really uncalled for, and DigiCert could face some very significant pushback for trying this tactic.
Reads to me like Digicert is hiding behind the TRO to protect themselves from upset customers over following the procedures they should have followed. I don't think they want to update their legal work to prevent customers from legal action over revocation, legal action has been working in their favor so far.
For a company whose entire business is verifying company names and handling the CA process, Digicert seems rather unwilling to actually stick to the process. They can't help having a TRO filed against them by customers with incompetent tech such as Alegeus Technologies LLC, but this isn't the first time they've failed to follow the appropriate processes.
Going through the courts to stop negative discourse seems rather crass for a CA. I don't understand why a company that has been subject of suspicion and doubt lime Digicert would do such a thing unless it's a last-ditch effort to get the heat off their backs.
I'm sure their clients love Digicert for not forcing them to replace their certificates at the appropriate times, but they're in for a surprise when this whole thing blows up and they suddenly need to find another cert vendor when Digicert gets delisted.
Alegeus, a client of DigiCert, filed a court motion to stop DigiCert from revoking their certificate. The courts granted the temporary restraining order.
There is no "update their legal work to prevent customers from legal action" that can avoid a temporary restraining order that is ordered by the courts to provide time to establish facts. Its simply how the legal process works. "they can't help but" seems unfair as the issue was the client Alegeus was unable to replace their certificates in the revocation timeline.
Further "Going through the courts", they didn't go to the courts a client of theirs did.
Digicert is absolutely not blameless. Outside of the TRO, they have failed to meet revocation windows before and yes, are likely using the TRO in this instance as a shield for that.
However the TRO itself is a concerning development that all CA's need to consider depending on their legal jurisdiction, as it could put companies in a bind of being legally ordered to not to complete something they are contractually and legally required to do within the required timeline and interrupt standard security practices in cases like this.
The TRO was filed for and granted without DigiCerts input. By the time they responded, both parties filed to vacate the TRO (3 days later). Judges I expect weigh, to the best of their abilities, the perceived harm in granting/not granting the TRO. Considering the technical literacy of our court system I expect that leaves much to be desired
Jul 30, 2024 - ORDER granting 2 Ex Parte Motion for TRO. IT IS HEREBY ORDERED that DigiCert is prohibited from revoking the security certificates for the Alegeus Websites for a period of seven (7) day
Jul 31, 2024 - NOTICE of Appearance by Jess M. Krannich on behalf of DigiCert
...
Aug 3, 2024 - DOCKET TEXT ORDER. 9 Joint Motion to Vacate 3 Order Granting Ex Parte Motion for TRO is GRANTED
Looking at that timeline, I can see why other CA forum members are asking "are you really taking this seriously?" The whole point is that such a restraining order was definitely a bad call by the judge and should absolutely have been contested immediately by any legal team that was interested in representing the security interests of the CA Browser Forum (which, ya know, members of the CA Browser Forum should do in such cases, hence why they're in the CA Browser forum). The fact that DigiCerts legal team did show up for that TRO and then did not act to try to defend their ability to secure their certs, is a bad thing. If you want to be a CA, you gotta be willing to defend your need to act in the name of security, and to defend that in a social, business, and legal context. The point of CAs in the world is not to make money, it is to provide security services. Responding with "hey, but my legal obligations in my country mean I can't always do that" is a valid explanation, but it also means that the rest of the CA Browser Forum should probably not trust you.
Any CA who believes that the courts in their country could not prevent them from revoking certificates is confused at best.
It's true that DigiCert could have refused to cooperate with Alegeus and fought the court order instead of cooperating with them to rotate the certificates ASAP. But that would have taken a lot longer than 5 days, even if they eventually won. If the CA Browser Forum has such a strong security interest in swift revocation, it's hard to see how further delaying the revocation in order to provoke a legal battle promotes that interest.
I'll point out notice of appearance doesn't mean they COULD have done anything. The TRO was granted for 7 days already.
From a strategic position, they probably saw they could work with the client and withdraw the TRO faster and cheaper than challenging the TRO in court. Its unlikely a challenge would have significantly decreased the time they were required to not act under the TRO.
They could have revoked the certificates of companies not listed in the TRO. Instead revocations for _all_ customers were delayed by 3 days over the time limit [1].
Additionally, it seems more prudent to me that they should drop Allegius as a customer than make the guy resign. The current situation seems like if an issue occurs again then DigiCert will not revoke the certs according to the timeline they committed to (24h).
They should - but only if a good lawyer points that out to the courts. Remember the courts are not experts in the technical details in question (and should not be - there are far more details that could come before the court than any human has time to learn), the job of lawyers is to quickly give them enough education to figure it out.
In "Common law" when something goes before a court in slightly different situations the second court will look at what the last one decided to see if it makes sense and if they agree. Then the third time looks at both the previous. After many many cases courts will have seen all the arguments and made decisions and so there is no need to educate the court anymore, they will just look at what was done before and decide the same thing.
Not all courts follow the "common law" above though, and I'm not clear how the other systems are different. (I've only lived in common law areas)
An answer for the civil law countries I know of: The laws tend to be more detailled to avoid the situation you describe. Also, there is still a lot of referencing old cases, but it is considered guiding, not binding.
Other comments have since suggested this was an emergency order where the CA didn't have their lawyer present. The courts said "don't do anything for a week while the real paperwork happens". I don't know the cases in question well enough to figure out if that is correct, but it seems reasonable. But also a week is not very long.
> Would a judge not consider potential harm to each party if the TRO is granted or not?
> If the CA could credibly point out they would face severe consequences for failure to revoke, could that have an impact on?
That would be an excellent reason to make sure to impose strict consequences on CAs without regard to whatever legal orders they're subject to, so that the next CA can point to those consequences when arguing against things like a TRO. "If you grant this TRO, the net effect will not be that you get what you want; the net effect will be to destroy our business and still not get what you want".
I dunno, seems to me in the TRO case there wasn't anything wrong with the certificates that had been issued. There wasn't any suggestion the the certificates had been issued to the wrong party or anything like that.
When performing DNS verification there's supposed to be an underscore in a record, which protects sites that let their users control arbitrary subdomains - you know, dyndns and things like that. Digicert mistakenly didn't ask the customer to include an underscore. Could have been a problem, if Alegeus was a dyndns provider - but they aren't.
Then Digicert spent a load of the 24 hour disclosure window checking things on their end, then e-mailed the customers after office hours so when they get in the next day they barely had any time to respond.
The only "incompetent tech" here is the assholes who decided on the policy that certificates be urgently revoked, as if they'd been issued to the wrong person, when they hadn't been issued to the wrong customer.
The middle ground here is DigiCert should have been able and continued with the revocation of all the other certificates not impacted by the TRO within the policies and contractual agreements it has.
The policy and its designers aren't incompetent, the intention is to ensure security first when fact-finding can take time to establish a subset of actual abuse. This is common in IT/Security and is clearly communicated in the policy customers agree to as well as something agreed on by all CA's. Security > Uptime is the baseline, and while you may disagree with that and have good arguments for the contrary, the argument is much wider than us and has been extensively hashed out in the working groups and discussions around this and other similar issues and is implemented this way by consensus while weighing valid arguments for different approaches and pros and cons.
>Could have been a problem, if Alegeus was a dyndns provider - but they aren't.
AFAIUI, Alegeus was just 1 customer of Digicert. But Digicert has tons of customers. Just because there wasn't a problem for Alegeus (due to Alegeus not being a dyndns provider), doesn't mean there weren't problems for other Digicert customers.
Revocation rules are intentionally very strict, in order to prevent CAs from weaseling out of them. If you leave open any possibility of a CA delaying or refusing revocation because of some form of "it's an administrative issue, it isn't that serious anyways", every incident will inevitably result in a weeks-long discussion about the real impact of the incident and whether this one is "serious enough" to require immediate action.
In the past CAs have been more than happy to lie and downplay incidents, because they know their customers don't like revocations and are going to consider switching to a competitor. Maybe not a big deal in practice for a technicality, but it does mean that CAs can not be trusted to always be truthful - and that in turn extends to serious incidents. This is obviously really bad for the web PKI as a whole, so the rule is that any violation of the rules, no matter how small or nitpicky, must be treated as if it were a serious security incident. Don't agree with the rules or think the rule is interpreted the wrong way? File a report afterwards to review the rules.
The entire business of a CA is to be a trusted third party. If you can't be trusted to always follow the rules exactly to the letter, you shouldn't be a CA. Behaving predictably is more important than being right about some technical minutiae regarding the security impact of some incident.
No. Our cert infrastructure needs to meet the highest standards possible, this is not an area where "welp, no harm done, don't worry about it" is acceptable.
What is the correct action when a TRO says a company cannot revoke the cert? Is it that the company will delay revocation, but will push the judicial system to resolve the issue as fast as possible?
DigiCert probably should have revoked every cert they could within 24 hours. Instead they just pushed the revocation of all 80,000+ certs out to five days.
It's quite likely that many of their other clients pushed back on the 24-hour timeline (similar to what happened in their previous incident); I believe the delayed revocation issue (https://bugzilla.mozilla.org/show_bug.cgi?id=1910322) hints at this. The TRO gave them a convenient excuse to delay all revocations without having to explain all over again why they made exemptions for their special clients.
"Any issue with domain validation is considered a serious issue by CABF and requires immediate action. Failure to comply can result in a distrust of the Certificate Authority. As such, we must revoke all impacted certificates within 24 hours of discovery. No extensions or delays are permitted. We apologize if this causes a business disruption to you and are standing by to assist you with validating your domain and issuing replacement certificates immediately."
I think the eventual correct option is pretty clear.
A TRO like that is based on the company loudly declaring that revoking will cause real damage. That means their use of certificates is incompatible with the web PKI rules and ecosystem. That means they need to be migrated out ASAP, with every certificate authority refusing to take their business.
Make that series of consequences clear, and companies will stop trying that trick.
DigiCert and other CA's need to write in their terms that failure to follow the policies and revocation timelines can/will result in termination of the contract. Alegeus should have been dropped as a customer as soon as the incident was resolved and refused further products/renewals.
Use of a TRO protects you short term, but results in having to migrate to a new CA medium term. You can't stop them from using TRO's but you can make it not worth it.
Is the CA allowed to stipulate that the customer (before being dropped) needs to pay a significant sum for, say, "expenses" if they resort to this kind of TRO?
Possibly, I'm not sure if there is precedent for charging a company for taking valid legal actions, only vexatious in contract law. Probably depends on the legal jurisdiction and courts.
I think that's pretty unrealistic, considering that X.509 is a de-facto and de-jure standard in a lot of places that also ignore that requirement. It's not always up to that company to make it possible to replace certificates easily, it's an entire chain of vendors (I know at least Salesforce's process would fail here). Unless you want those people to run self-signed/private CA certs.
The other option would be building in a way to revoke other CA's individual certificates if there's some consensus on them being compelled to not revoke them. Not sure if the status quo or this would be more dangerous, but if a TRO can compel a CA to not sign a revocation, can it also compel to sign a certificate?
A company chooses its vendors. If its vendors turn out to be incompetent, they should choose again once the damage has been done. Just because Salesforce can't get their stuff together doesn't mean the CA industry needs to bend the knee. Let Salesforce figure out how to automate certificates, they've been in the business for long enough.
If running a private CA or self-signed certificates are even viable, then there are plenty of other workarounds that can be put in place (i.e. not updating the CRL so the software doesn't know about revocations).
If a CA's terms and legal documentation aren't tight enough to prevent a court from compel them to sign a certificate, that CA clearly cannot be trusted. Hopefully that issue can be fixed by writing clearer terms and better agreements, like people have been telling Digicert to do, but perhaps that's not possible at all. In that case, either the company should move to a jurisdiction where that kind of nonsense isn't possible, or it should be removed from global trust stores all together.
> A company chooses its vendors. If its vendors turn out to be incompetent, they should choose again once the damage has been done.
You say that and yet Teams has 320 million people using it, and I bet almost none of them enjoy the experience. Sometimes you just have to work with what you're given. Given "incompetence", we might as well throw in all of Azure.
> Let Salesforce figure out how to automate certificates, they've been in the business for long enough.
That might be true for DV, but there's classes of certificates that take at least weeks to obtain, think BIMI VMC or codesigning.
> You say that and yet Teams has 320 million people using it
Yeah, there are tons of companies who chose Teams as a vendor (because it's bundled with other stuff), and inflict it on their employees. It was still absolutely their choice.
The parent was correct - it's not about the company not using x509 certificates, but not using publicly trusted certificates. There are myriad private/internal PKI solutions available from OpenSSL & bash to millions of dollars of solutions from Big Vendor.
If you can't replace the publicly-trusted certificate quickly, you probably don't need it to be publicly-trusted in the first place.
The better question is, how do you prevent this tactic from working?
For example, suppose there were required to be multiple parties who could issue a revocation, each in a different jurisdiction, and if any of them was ordered not to do it then the others would be required to do it, and would have the technical capacity to do it but not be subject to the jurisdiction of that court.
Well, you can contest the motion for a TRO, for a start. Digicert failed to do so.
You can stick with your policies and revoke the certificate within 24 hours, instead of delaying revocation until a case is open and a motion for a TRO is filed. Digicert failed to do so.
You can stick with your policies and revoke the cert in face of the legal consequences, and deal with them accordingly. Again, Digicert failed to do so.
Correction, the petition for the TRO was filed ex parte. Digicert did not have any opportunity to respond before it was granted.
They certainly could have filed a response contesting the TRO. Then their customer could have filed another motion, and eventually (7 days later in this case) the judge would have ruled on the substance of it. Their judgement was that it would be preferable to work with the customer to resolve the technical issues with revocation, and submit a joint request to dismiss the TRO. The stated reasoning behind this was that it would be significantly faster than contesting the TRO. This is true: the certs were revoked and the TRO dropped within 3 days.
I think the communication on that point was severely lacking, as they only clarified it three months later and after significant hectoring in two different bug threads: https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c43
I also think it's reasonable not to take Digicert's statements at face value, given their history. But I think both of the points you made here are wrong:
> You can stick with your policies and revoke the certificate within 24 hours, instead of delaying revocation until a case is open and a motion for a TRO is filed. Digicert failed to do so.
Let's be clear about the timeline: Digicert notified their customers that the certs would be revoked. In between the time they notified the customer and the time of revocation (less than 24 hours), the customer got the ex parte restraining order. Are you suggesting that issuers should revoke certificates without notifying their users, so that the users don't have time to get an emergency TRO? I believe that would be in violation of the BRs.
> You can stick with your policies and revoke the cert in face of the legal consequences, and deal with them accordingly. Again, Digicert failed to do so.
By "revoke the cert in face of the legal consequences" do you mean "openly defy a valid and legal court order"? Because that would also violate the BRs.
Just to be clear, the whole incident covered over 80,000 certificates.
The TRO was applicable to only those of one subscriber - just over 70 certificates, yet caused the revocations of all 80k+ to be delayed.
To add to this, 3 days after the TRO was filed both parties moved to vacate the TRO.
DOCKET TEXT ORDER. 9 Joint Motion to Vacate 3 Order Granting Ex Parte Motion for TRO is GRANTED
I'm not sure DigiCert could have done anything about the TRO or the impacted certs, but it should have been able to move forward with the revocation of all other certificates. That IMO is the real issue/failure, alongside the concern/impact of TRO's on security processes in the future.
> By "revoke the cert in face of the legal consequences" do you mean "openly defy a valid and legal court order"? Because that would also violate the BRs.
Yes, I think this would have been appropriate action. If the contractual language is extremely clear between the CA and the subscriber, there is no legal basis on which the customer can prevent revocation. The fact they found a court that doesn't understand technology is frankly irrelevant. This detail is exactly why Tim and other parties are requesting the exact language of the agreement between Digicert and the subscriber that filed the TRO. A customer acting in bad faith and abusing the legal system does not compel you to violate your own contract terms, your terms under the CAB/BR, or to take actions which are detrimental to the entire Internet. This is exactly the type of circumstance where you do what you are required to do, and then sort it out afterwards. Any appeals court would have easily overturned the TRO as it has no legal basis.
> A customer acting in bad faith and abusing the legal system does not compel you to violate your own contract terms, your terms under the CAB/BR
Yes, it absolutely does. "I think the court will agree with my view of what the contract says once the case is heard in full" is not a valid reason to disregard a TRO.
> or to take actions which are detrimental to the entire Internet
That would be harder. But a delayed revocation stemming from a flawed validation process, when the CA is responsible for the flaw and knows that the result of the validation was in fact correct, simply does not cause any detrimental effects to the entire Internet.
You could just require publishing the revocation immediately with an effective date in the future.
Of course, if that system had been in place, DigiCert would probably be facing hundreds of lawsuits from businesses disrupted through no fault of their own rather than inside baseball PKI drama.
> The better question is, how do you prevent this tactic from working?
Make it clear that if it works, it will only work once.
The CABF should adopt policies that any such legal action or any request for extension will be considered a public declaration that the customer's application is incompatible with the requirements of the Web PKI and that not only will the current CA refuse to renew the certificate but it will be publicly documented in the Bugzilla so that no other CA will issue certificates covering any of those names nor any new names for the same company without an affidavit explicitly stating that the issues preventing compliance have been resolved and the company acknowledges this and commits to never doing so again.
Existing names that were successfully revoked in time can be renewed but neither the problematic one(s) nor any new ones will be allowed.
If they then file for another TRO in the future they may still get a short-lived order but the existence of such an agreement would at least to my non-lawyer brain cause any judge who may have granted a TRO to become very displeased when the CA presented it in their response.
It’s Saturday and the courts are closed. You park in a car park I own.
I have put up a sign saying I can instantly crush any crossover vehicles I like, as I consider them ugly and lacking in character.
As I load your car into my crusher, you dispute the legality of my sign. But the court is closed until Monday, and the sign says I can crush your car instantly, no waiting.
Should I be allowed to crush your car today? Or should I have to wait until Monday, so the disputed legality can be resolved?
That’s actually a question for you. You won’t truly know if you’re allowed to until a court decides. You can choose to proceed and crush it, and then deal with the consequences of doing so if it turns out you were wrong. Likely you’ll end up owing damages to the owner of the car, perhaps even punitive damages on top.
The TRO actually means you aren't allowed, that's the point. It's an ordered injunction that legally obliges you from not acting until the courts can review the facts.
In this case, yes. That’s pretty cut and dry for the time being. However regarding the analogy I was replying to, I was pointing out that it’s less a situation of what you’re allowed to do and more one of what you believe you’re allowed to do, and weighing the consequences of being wrong against upholding your stated terms. In other words, something you should probably discuss with a lawyer.
> How can a court inhibit revocation when every CA declares their right to do so when you purchase a certificate?
A court can rule that a term of a contract is void because it contradicts public policy, and it certainly can issue a TRO pausing an action which would otherwise be allowed by a contract while resolving a dispute related to it.
Because the judge doesn't know the details of how PKI works, and either the ToS for digicert doesn't spell out that they can revoke certs at any time (which would be problematic for a CA), or the judge didn't read, or didn't understand the ToS.
The order would normally be a matter of public record, and the ecosystem should follow these events carefully and ensure never to issue a certificate to such a company again as it undermines the entire system.
No, Sectigo is the PKI business that was carved out of Comodo, back in 2017. Comodo still exists, doing whatever they do. They have been totally separate entities since then.
Web PKI drama is always astonishing to me because it is one of the only areas of the entire world where a corporation's "fucking around" is seldom ever not followed by a sobering "finding out" period. The various entities that decide what CAs to trust can effectively dismantle any CA business in the world, basically at the drop of a hat. If DigiCert decides to play this game and lose, it would make them the biggest such loser so far. DigiCert is, as far as I know, the largest CA on the Internet. It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores. Yet, there's no particular reason it couldn't happen. How exciting.
Of course, I think that is unlikely, but on the other hand, it's just as cathartic to imagine whatever idiot at DigiCert thought it was a good idea to engage legal here to have the dressing down of a lifetime. I read the thread in question. It doesn't make DigiCert look good, but this action definitely is more damning to them than anything Collan said in my opinion.
Thank you, that was an awesome laugh this morning, that request was genius:
> 2. Sub CAs Operated by 3rd Parties
> Honest Achmed's uncles may invite some of their friends to issue certificates as well, in particular their cousins Refik and Abdi or "RA" as they're known. Honest Achmed's uncles assure us that their RA can be trusted, apart from that one time when they lent them the keys to the car, but that was a one-off that won't happen again.
Someone should start a real CA business, complete with all the proper audits and everything to get it into the browser trust stores… and call it “Honest Achmed’s Used Cars and Certificates” (have it buy some random used car dealership in the middle of nowhere so the name is not a lie)
Where’s a billionaire troll when you need one? (And a form of billionaire trolling that would upset a lot less people than Musk’s version of it.)
Would be even funnier if the billionaire’s name actually was Achmed
Anyone can get a cert from lets-encrypt. No users of your website care how trustworthy your CA is. And CA's are too big to fail, so you barely need to worry about your CA being removed by the trust store. So you can only compete on price.
If we want to change this, we need a way for a certificate to be co-signed by multiple CA's. So that the certificate can be presented to the user, and they can figure out if any trusted CA of theirs has signed the certificate.
This way, revoking trust in a CA becomes easier, because people should have multiple signatures on their certificate. That means, all of a sudden, that the quality of a CA actually matters.
Whilst it might seem this is already possible, it is not. Cross-signing is only a thing for intermediate certificates, and does not work. You can also have multiple certificates for the same key, but when starting a TLS session, you must present a single certificate. (This seems to have changed for TLS 1.3, so perhaps this is already possible?)
I think TLS 1.3 still requires the end entity (server/client) cert first. All the other certs can now be in any order and the verification is suppoaed to figure out a valid path.
In theory, you could make an intermediate CA and get cross signed certs from multiple CAs (hopefully with Name Constraints), your intermediate CA signs your server cert, and you include all your cross certs and the intermediate certs for those. And the client figures out which chains it can make and if it likes any of them.
But experience has shown, verification may find a chain with signatures that line up, but the CA is expired or revoked in the local trust store, and reject the verification, even though another chain could have been found with the provided certificates.
And, because of the limited information in tls handshakes from real world clients, it's difficult (maybe impossible) to know which CAs a client would accept, so that the server could present an appropriate chain.
Time and money. Plus right now even if you bought the infra, the staff, paid for and passed the audits, and then waiting while Apple, Mozilla, Google and Oracle (at least) included your roots...Microsoft aren't taking more right now. So you have to wait for some unknown time in the future when/if they start doing that again.
You could purchase a root off an existing CA, subject to the trust stores approving it, and the boatload of cash you'd need to buy it (plus still having the staff and infra to operate it).
Sub-CAs: Not really. Operational risk to the parent CA is huge, you'd be hard pressed to get any current public CA to sign an issuing CA to be operated externally.
Cross-signing still works (though it is the stuff of nightmares in many cases) but again you have to have money and a CA willing to do it!
No, SSLCorp are hosting and managing a CA with Entrust branding. Same as Sectigo are doing. Entrust aren't doing issuance, verification - they're straight reselling from white-labeled issuing CAs.
Let's Encrypt has been around for a decade and is doing pretty well for itself. The manual validation required for EV and OV certificates precludes it from being a passion project, unless managing a call center and a squad of document reviewers is your dream job.
It wouldn't be fun if Honest Achmed’s Used Cars and Certificates didn't also issue EV and OV certificates. It wouldn't be on brand to miss out on entire segments of the certificate market.
To my understanding, the point of Honest Achmed was to demonstrate that it was possible to write a facially reasonable and compliant CA inclusion request that was also intuitively unacceptable. It successfully demonstrated that the inclusion policies at the time needed to be made more rigorous, which they were.
> It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores. Yet, there's no particular reason it couldn't happen. How exciting.
Trust stores and Browsers specifically have more options than simply remove a CA. A more reasonable approach for distrust process of a CA this size would be to stop accepting any new certificates issued after a certain date. Then existing customers will have a heads up and even if asleep will find out the bad news during scheduled renewal rather than while the responsible employees are on vacation.
It's refreshing to see folks who're obviously used to hiding behind clouds of bullshit get skewered by people who both know enough to see through it and have the time and energy to follow every thread to completion.
The most recent digicert thread smells suspiciously similar to those that lead up to the Entrust debacle.
Is there any option to turn off trust for all certificates generated after a certain date? The ideal would be to let old certificates keep working, and not trust new ones.
Yes, this has been done a few times in the past. It's unfortunate that "distrust after X date" and "distrust now" are the only penalties that currently exist but that's the reality of the current PKI world.
There are a lot of efforts to get name constraints widely supported, so a given CA can only issue for certain TLDs or even domains, but it's still not there. That could be a viable punishment for misbehaving CAs in the future, or even an intentionally chosen restriction for CAs that want less strict requirements. Government CAs could be a very useful application, if a CA cert could be name constrained to only issue for .gov.cctld at that point we could basically just let them do whatever they want without concern. Likewise for any small regional CAs, if they could be limited to only regional TLDs their blast radius is reduced which also could mean less strict requirements than those applied to CAs able to sign for .com.
It would also be really useful if I could add a "root" CA for my employer's private CA to sign certificates for say .internal or .example.com, but not allow them to sign certs for any arbitrary domain. There's decent support for constraints on intermediate certs, but then you have to trust the root won't ever be used to sign unconstrained intermediates.
It would also be useful for localhost. Create a private CA that is constrained to localhost and *.local, and you don't have to worry about it being used to MitM other sites if it gets compromised.
It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores.
How many will agree to that removal? How many will see one more reason to forever turn off automatic updates and decide what to trust themselves, having seen yet another way some faceless entity they never knew about can break things?
It will certainly send a strong message, but likely not the intended one. All it will do is increase the lack of trust in centralised PKI in general.
I'm not sure how to put this nicely, but I'll try my best: The normal people, who account for 99% of the end users of DigiCert's customers, are not going to disable updates on their web browser or their OS to "take a stand" against this decision. (And corporate users can't do this anyways, since it's not in their hands, and keeping things out of date is not a reasonable option for an organization that has security standards.) Most of them won't know what is going on here, and if they hear about it, probably won't care or know what to do about it. That's the Web PKI infrastructure working as intended, because it would be infeasible for billions of people to properly understand the gravity of all of these decisions. In order to ensure that TLS connections are at least minimally safe, it's pretty much necessary for things to work this way.
I'm not arguing that it's good that a relatively small number of entities (mainly Google, Microsoft and Mozilla) decide which CAs are trustworthy, but that's all the more reason that it's important for all of this Web PKI work to happen completely in the open, so that the few who can spare the time and effort to scrutinize what is going on can help the rest of us have a safer Internet. We don't have a better solution. That's also why DigiCert issuing legal threats because they don't like how one of these issue reports makes them look is a serious problem that can't be tolerated.
> The normal people, who account for 99% of the end users of DigiCert's customers, are not going to disable updates on their web browser or their OS to "take a stand" against this decision.
I'd imagine that they wouldn't do it to "take a stand" so much as "avoid the risk of getting their stuff broken in the short term" in this scenario, regardless of whichever party loses the blame game. See: the recent WordPress drama, which has turned customers away from both parties involved.
As I understand it, the way certificate authorities are removed from the trust store is progressive: they will announce a date after which new certificates from a given CA will no longer be trusted. I think this can be made even more progressive, by limiting the validity period of new certificates that will be trusted. DigiCert will have little recourse other than to let their customers know, and/or start providing certificates issued by another CA that follows the web PKI procedures and remains trusted. (They can still do that, of course, it's just that their own certificates issued by them directly won't be trusted anymore.)
On the flip side, for user impact, it will play out like this: Some bank or other important entity could possibly, for whatever reason, continue using a (presumably expired? unless DigiCert continues issuing anyways; note that most likely, they will not.) DigiCert certificate after the cut off date, which will lead to users receiving errors. Some of them will have HSTS setup, which will lead to an emergency situation where they have to issue a new certificate ASAP, as it will basically halt their business until they do. For places where there is no HSTS, users may be instructed to simply bypass the certificate warning temporarily, and support lines will be absolutely swamped until they actually fix the problem.
The WordPress situation is quite different. You don't have to use WordPress. Users don't even know what the Web PKI is to find an alternative to it, not that there is one or will be one.
Sure some users would keep the CA active out of fear, while everyone paying digicert would also begin to move out of fear.. What portion of the strange sort of sites that couldn't figure out how to get off but could figure out how to renew would bother with paying for a different error message?
The normal people, who account for 99% of the end users of DigiCert's customers, are not going to disable updates on their web browser or their OS to "take a stand" against this decision.
...unless they notice the breakage, and you tell them to do so to stop it.
Ignoring the policy reasons that this won't happen, not all devices can be downgraded, so by then it's too late anyway. Best you can do is bypass a non-HSTS certificate warning.
That all assumes DigiCert continues issuing certificates they know won't be trusted. I assume most likely what will actually happen is they have to start selling certificates from another CA, but failing that they're probably just going to stop issuing certificates before the cutoff.
This case literally involves the courts getting involved in what should appear in trust stores. DigiCert was ordered by the courts to not revoke certificates via a TRO.
If the browsers removed DigiCert, DigiCert could certainly sue based on the harm to their business. They could argue that the browser vendors were inconsistent with the application of their rules. They could argue that the browser vendors were unfairly competing by kicking them out.
Not saying they would win all these, but they would certainly fight it - the alternative would be the end of DigiCert, they aren't going to go down without a fight in whatever venue they can find.
Is that really the claim, though? If someone chooses to punish DigiCert for compliance with the TRO then DigiCert might have grounds to file for declarative relief from the court, as the court could very well decide to help protect DigiCert from external consequences of their demand.
Perhaps, but only in the case that the delayed revocations were scoped to those certificates covered by the TRO, and not over 1000x more from subscribers who had nothing to do with the company who filed the TRO.
Well personally, I think that'd be a terrible position for them to take. Honoring the TRO is, as far as I can tell, not the issue that is being raised.
> All it will do is increase the lack of trust in centralised PKI in general.
“In general” is a broad statement, given that the overwhelming majority of people using the Web PKI have no idea that they’re doing so. DigiCert is not a legible part of the value chain for the average users; they won’t notice that the sites they use switch CAs. This strong disfavoritism towards vendors is arguably one of the Web PKI’s greatest strengths.
Many systems do not fetch updates from the Mozilla root store, but from their (possibly Debian-derived) stable distribution of it.
Meaning two highly respected entities, known for being well aware of the wider impact of their careful enforcement of strict policies, need to agree to cause any major breakage. When that happens, I can blindly trust they did the thing needed to keep the unaffected parts of that weird system working as intended.. and then still head to bugzilla and read about the background - to laugh at whoever triggered the mess.
> Meaning two highly respected entities, known for being well aware of the wider impact of their careful enforcement of strict policies, need to agree to cause any major breakage.
Note that this is not a "both" but a "any of them". One can disagree with the other on this and still cause wide breakage.
If DigiCert were to lose browser trust (at this point, that's still a big if), it would happen the same way it happened with prior CAs, some of which were pretty big themselves (Symantec): all certificates issued after some date would not be trusted, yes. But all existing certificates would remain valid.
This gives certificate owners ample time to look for a different issuer and no certificate buyer would deliberately purchase a certificate from an issuer when they know some percentage of users will not trust that cert.
So for the end users, everything will keep working: the existing digicert certs stay valid and newly refreshed certs will be signed by a different authority. There is no need to turn off automatic updates over this.
Between Entrust and Symantec, we've already seen this happen to large well-known CAs and everything remained fine (not for the offenders, but, hey, that's the system working as intended)
"The actual reason for the underscore is so that services which allow users to create DNS records at subdomains (e.g. dynamic DNS services) can block users from registering subdomains starting with an underscore and be safe from unwanted certificate issuance. It serves the same purpose that /.well-known does for Agreed-Upon Change To Website, and that admin/administrator/webmaster/hostmaster/postmaster do for Constructed Email to Domain Contact. By using DNS records without underscores, DigiCert has violated a security-critical assumption that these services have made.
Therefore, this is truly a security-critical incident, "
That is a pretty brutal f' up of epic proportions... not sure any digicert cert should be trusted after that.
My bigger concern is that what percentage of people will realize when delegating subdomains that having a leading underscore is a security vulnerability?
Always two sides to a story but the guy who caused the validation bug at DigiCert already resigned because of it (which is extreme), the Sectigo guy wanted to prevent the bug being closed so he could keep pushing them (in a subjectively prickly fashion) for more answers about their general responsiveness.
A bit of back and forth discourse is fine and expected, but if you keep pushing someone who has their own legal dept they're eventually going to wander over to the coffee machine and have a chat about it with them, then they're going to take a look and it becomes their problem.
So the number one rule would be don't even breath the word "legal" unless you want to invoke them. This particular response is just a letter telling them to back off and it's why you have a legal dept, so they can argue with each other. This one has just found it's way into the open.
There is a understandable perspective that says CAs shouldn't be burdened with legal risk in their discussions, but that's contrary to the fact these guys are commercial entities protecting their interests, so you don't get it both ways unless all your CAs are non-commercial, and even then that would only extend so far.
Perhaps they should have also had a chat with their PR departement. And anyone responsible for company strategy. Becose the actions of the legal department just backfired.
You've given a detailed rundown like you've been following, so what is your sense about the resignation? Did the guy resign willingly, or is Digicert the kind of management that likely scapegoated him?
He assumed personal responsibility rather than institutional. At the time, everyone in the community expressed alarm that that happened, because 1) and this is obvious, there's no way a single guy would be responsible for the entire fallout and 2) because it sets a bad precedent about what companies can do with their PoC wrt web PKI. It also meant a loss in institutional knowledge about how things worked.
This is shocking to read. Even attempting to choke the legal speech of web PKI contributors with legalistic bullying is a gross inversion of the purpose and goals of the organization, and IMO warrants revoking everything to do with DigiCert on the spot.
> IMO warrants revoking everything to do with DigiCert on the spot
This is pretty heavy handed and I don't think you've thought through what the consequences of that may look like. The historic way to deal with a problematic CA is to prevent them from issuing new certificates or renewing certificates (after taking care of immediate damage, of course). There are lots of legitimate companies that use Digicert and they should have an expectation of being able to continue business in the short term while they find a different certificate provider.
I agree. "Revoking everything digicert" is a bit imprecise, "preventing digicert from issuing new certificates" is a more reasonable measure that actually only damages the parties that deserve it (digicert) and spares innocent bystanders (their customers).
No, and I'm deeply confused how you drew this conclusion from what was written.
I specifically say the historical way to deprecate a certificate authority is to prevent them from issuing new certificates. Since certificates have a finite lifetime, the certificate authority would as well and would have immediately no further revenue from certificates.
I am saying "pulling the plug" without notice is going to take down tens of thousands of bystanders and anyone using those certificates for business would be unable to conduct business securely online, which would be disastrous. For example amazon.com has a DigiCert-issued certificate.
> The public record of Alegeus Technologies LLC v. DigiCert shows no attempt by DigiCert to contest the court’s order prior to the end of its preferred period of nearly 120 hours, even though such a motion could have freed DigiCert to revoke the certificates days earlier.
and
> The other question in comment 28 was for the language establishing DigiCert’s right to revoke Alegeus Technologies certificates. DigiCert has waffled on this point, first implying that this language was to be found on its website but later refusing to confirm that the language on the site applied to Alegeus Technologies at the time.
SPECULATION: Digicert may have offered special terms to Alegeus, and possibly other customers. They may have chosen not to dispute the TRO in court because they did not have grounds to do so under those agreements. They may also have included confidentiality terms in those contracts that prevented them from speaking about it.
OPINION: I am surprised that the forum allowed the issue to be closed without the above quoted questions being satisfied, though it is possible they are addressed elsewhere, I have not done a complete reading of all the linked issues.
> Even though DigiCert’s TOU and MSA prohibited Alegeus from taking the action it did, once it filed for a TRO and the court almost immediately granted it, DigiCert’s hands were tied
> Even though DigiCert’s TOU and MSA prohibited Alegeus from taking the action it did, once it filed for a TRO and the court almost immediately granted it, DigiCert’s hands were tied
So did the judge just not read the TOU before signing the TRO?
I wonder if the CAB forum would have standing to sue Alegeus and/or that judge for interfering with the PKI process with an invalid TRO.
DigiCert posted a message fifteen days ago saying that "We have not used a legal team as a shield against accountability." (https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c74). This is clearly contradicted by their legal threat against Sectigo. Hence, Sectigo decided to post the "Threat of legal action" bug to make the community aware of what DigiCert actually did.
Maybe if DigiCert had not decided to make that comment, Sectigo would have been willing to stay quiet...
Certificate authorities have enormous trust placed upon them by every Internet user (whether they know it or not). Commensurate with this trust, they have enormous responsibilities. As the name implies, the Baseline Requirements are the minimum standard they should achieve. If they can't even do this (being unable or unwilling to revoke issued certificates within the required time-frame), then they do not deserve this trust, and it should be removed.
I understand that the TRO prevented them from revoking approximately 70 certificates, and there really is nothing they could have done differently in that case. Their other revocation failings are inexcusable.
Even taking the (one-sided) depiction of the conversations in DigiCert's letter at face value, maybe Sectigo's guy was being a git at best and intentionally trolling at worst. (I don't think he was, but let's play devil's advocate.) But even then, how did DigiCert think getting legal involved would possibly go well? Sectigo stands to gain publicity and lose nothing by going public with it to the CAB as they did here, and it's not like the CAB is going to play marriage counselor and get the two companies to make up because one of them got their feefees hurt.
Besides, this kind of hyper-polite passive-aggressive "erm akchually" conversation happens in every CAB incident discussion. I don't know why DigiCert got particularly upset about this one.
> Besides, this kind of hyper-polite passive-aggressive "erm akchually" conversation happens in every CAB incident discussion.
As somebody who doesn't spend much time scrolling CAB reports, this was jarring to me.
Digicert's legal action seems nuts, and there seems like a real, risky issue in the idea that a company's customers can use the legal system to block the company from complying with its obligations to other entities, but it's hard to see any way that could be productively addressed given the back and forth in the thread. It's like I'm watching a theatrical production staring the most stereotypical corporate drones trading comments with the most stereotypical IRC nerds, both sides doing circles around an interesting topic but too busy trading blows to ever really get to it.
> but it's hard to see any way that could be productively addressed given the back and forth in the thread
Another commenter mentioned in a sibling thread the possibility of using future-dated revocations. The CA could be mandated to publish such a revocation immediately, such that it takes effect by the 24-hour deadline. Once published, the revocations themselves should be irrevocable. This would also need to be accounted for in the CA's customer contracts.
As the comment you linked to notes, that doesn't really fix the problem so much as it ensures that the CA's legal team gets to work nights and weekends for the next year.
The legal system is almost certainly going to view future-dating and immediately publishing revocations poorly in any civil action where a customer claims harm.
It fixes the problem of a customer getting an injunction despite what the contract says. If DigiCert doesn't want to be liable for harm they should improve their performance, no try to weasel their way out of the agreements that let their business exist in the first place.
> but it's hard to see any way that could be productively addressed
It sounds to me there are things digicert could have done better, without violating the court order:
- they could have revoked all the other certs that weren't protected by the TRO. They did not.
- they could have contested the TRO sooner. But they didn't.
- Possibly, they could have given customers more warning, since they didn't notify customers until much of the 24 hours was gone
That said, IMO I don't what they did is necessarily irredeemable. But they need to come up with, and execute on, a plan to make sure something like this doesn't happen again. And threatening legal action is really not a good way to re-establish trust.
> there seems like a real, risky issue in the idea that a company's customers can use the legal system to block the company from complying with its obligations to other entities
As Digicert has repeatedly explained, this is simply how the United States legal system works. Courts have broad and indisputable power to issue temporary restraining orders, and the parties to a case must comply even if doing so violates some promise they made to a third party. (The point of the TRO is to maintain the status quo while the court figures out details like what promises have been made to who.) People in the PKI community who believe that some carefully written policy would enable CAs to reject an invalidation TRO, or convince a court that they cannot issue it, are wrong.
The reason it's never come up before is that no CA had previously attempted to enforce a widespread 24 hour revocation caused by its own error.
(I think Sectigo's argument is that Digicert did not even attempt to convince the court that it should be allowed to revoke those certificates in the mandated timeframe. If they had attempted and failed, I don't think they would be receiving criticism.)
That's their argument, yes, but it was clearly based on the incorrect belief that there's some emergency button you can press to demand that a court consider your arguments ASAP. As Digicert explained:
> The legal world does not move as fast as the demands of our CA ecosystem. Our legal approach was to work with the complainant’s legal team to get the TRO dismissed in 5 days.
The court would not have dissolved the TRO in anywhere close to 5 days without the complainant's cooperation, even if Digicert had an ironclad argument for doing so. Digicert made
the right choice to get the certificates rotated as fast as possible - and I don't think Sectigo intends to argue it would be better to stand on principle even if that makes the revocation slower.
One customer filed a TRO saying "don't revoke my certs". DigiCert then said "well, I guess we can't revoke any of the certs for the 80,000+ other customers either". That's stupid, and not acceptable. Instead, revoke the other 79,999 customers and communicate to the CABF saying "we've got one holdout that we are legally prevented from acting on." DigiCert didn't do that. They're acting not on behalf of transparently representing the interests of the CA/Browser Forum, instead they're trying to save their own skin from both sides. That's not good enough.
Should've been more specific. You can ask a court to do all kinds of things, but the individual judge who reads your filing doesn't have to (and in most cases can't) carefully analyze your arguments that day. They need time to think it over, and probably hearings where you and the other party can explain all the arguments for why certain rulings should or shouldn't be made. A contract dispute like this, where one party says they have a right to do something and the other party says they don't, is almost always going to take longer than 1 or 5 or 30 days for a court to figure out.
Temporary restraining orders are the biggest exception. If DigiCert is about to do something crazy like take down all your websites, courts are generally willing to put a temporary stop to it without understanding all the details. "Preserve the status quo" and "prevent irreparable harm" are the buzzwords.
> If DigiCert is about to do something crazy like take down all your websites, courts are generally willing to put a temporary stop to it without understanding all the details. "Preserve the status quo" and "prevent irreparable harm" are the buzzwords.
So if DigiCert's irreparable harm was great would that prevent it? Like legally requiring CAs to follow their revocation policies or pay millions in damages?
You're conflating DigiCert's argument against issuance of the TRO, with the irreparable harm the complaintant (Alegeus) is alleging will occur if the TRO is not granted.
Are there actually millions in damages being caused by delaying revocation of these certificates? Courts are generally averse to “penalty clauses” where you make up a nonsense number and call it damages. (Irreparable harm means that ordering monetary compensation can’t remediate it, so a more reasonable fee would probably not count.)
Could they not have had a clause in the contract saying “if you delay a revocation in any manner you owe us $100 million.”?
So a customer could go right ahead and get a TRO but long term it will cost them less than making sure their infrastructure can handle this rare event?
Liquidated damages are possible, but they have to have some relationship to the damages suffered, so $100 million is probably far too high. In any case, I think the complainant was correct that it was DigiCert who caused the entire scenario to happen by issuing certificates which they should have known were invalid.
I don't think I disagree with you about what courts can do. And (as shown by all the back and forth in the CAB thread, and now here), it is risky: we currently have a situation where CAs can find themselves in a position where taking court-mandated actions (or lack of actions) puts them in jeopardy with the CAB, and violating court orders puts them in jeopardy with the government.
For all the back and forth in the CAB thread, it doesn't seem we're any closer to finding an escape valve for that, which seemingly would have to come from the CAB because there isn't even really a mechanism for courts to decide not to issue TROs about cert revocation, short of somehow taking a case around this to the Supreme Court (which isn't going to happen).
It's fascinating that we've built a system that has expended perhaps several million dollars of engineering, legal and admin etc time over the issue of a single letter not being capitalized [1], without any demonstrable impact beyond a failure to meet ambiguous specifications.
I do hope that dealing with all of the underlying issues around revocation etc makes the time and effort spent useful, and the Web PKI doesn't just mire itself in squabbling that blocks progress on actually meaningful issues.
Basically the missing '_' was supposed to allow DNS providers who allow programmatic DNS record creation to filter out unauthorised certificate creation. So the certificates created without it could have been unauthorized by the owner of the domain they claim to certify.
Entrust got torpedoed basically for deploying an improvement to the requirements for its certificates slightly before the improvement got officially approved, and the browser people collectively lost their crud over the concept that Entrust didn't immediately revoke all of their... perfectly valid, secure certificates immediately.
Fundamentally, there is no accountability in the web PKI stewards. You want to talk about utter waste and incredible damage to the Internet, you can see it right here, in the people determining who is allowed to issue you sets of magic numbers that browsers have agreed are trustworthy, despite everyone involved behaving like complete children.
And of course, the browser operators all have their own root CAs, so basically have extremely motivated reasons to want to eliminate every commercial provider that isn't one of the monopoly companies.
Meanwhile:
- Compromised certificates are basically a non-issue from a threat model standpoint, every certificate error people hit are just... expired certificates people didn't rotate yet.
- Expired certificates cause issues for the majority of businesses at some point or another, making the internet increasingly fragile and unreliable.
For further reading, consider these two incidents which resulted in delayed revocation from DigiCert and a bunch of comments about how DigiCert should not be allowing delayed revocation:
Essentially, DigiCert has been delaying the revocation process (twice now) and people are unhappy about that. DigiCert has apparently attempted to silence those unhappy people (Sectigo and their representative Tim Callan) with legal action.
> Contrary to this statement, I received a letter from DigiCert’s lawyers, Wilson Sonsini, regarding posts made by Sectigo’s Chief Compliance Officer in bug 1910322. https://bugzilla.mozilla.org/show_bug.cgi?id=1910322
> We also found that the bug in the code was inadvertently remediated when engineering completed a user-experience enhancement project that collapsed multiple random value generation microservices into a single service.
To issue a cert for a client (say example.com), the CA sends a random number challenge to the client (say 12345678). The client then responds by making a DNS record for _12345678.example.com but due to a bug, DigiCert accepted 12345678.example.com instead.
Apparently there's a rule that says any such certs must be considered mis-issued and must be revoked by the issuing CA within 24 hours.
The CA talked to some clients who said it would be too much trouble to replace their cert on such short notice. One of those clients filed a lawsuit with the court system, which responded by issuing a restraining order telling the CA not to revoke the cert.
First of all, this is a dumb reason to revoke certs. It's difficult to imagine a situation where any of the mis-issued certificates correspond to an actual compromise of somebody's website by a bad actor. But we can perhaps give the benefit of the doubt: I think the intent here is a fail-safe procedure is implemented so revocation is triggered by a broad class of circumstances, which is constructed so that it's easy to check.
Second, this is a dumb thing for a website to file a lawsuit about. If you have to replace the cert on your website because your CA made a booboo, the engineer-hours needed to change out your new cert are far less than the lawyer-hours needed to try to work through the court system.
Third, it's dumb for people to blame DigiCert for the TRO. The TRO was filed by a service provider called Alegeus. If DigiCert doesn't follow the TRO, they'll be in contempt of court. Theoretically, a DigiCert employee who presses the "Revoke" button at this point could be sent to jail for it!
Fourth, a certificate revocation is basically a signed message that says "News Flash: I think this certificate might not correspond to this website." A court preventing the publication of such a message is equivalent to a court preventing a newspaper from publishing an article. This is called "prior restraint" and it has a high bar in the US. Prior restraint analysis was not performed by the court before the TRO was issued. I think the court violated DigiCert's due process rights (by not analyzing whether prior restraint applies) and DigiCert's free speech rights (by not denying the TRO as it would be prior restraint). (That browsers have a hair-trigger response to certain kinds of news flashes and will take drastic action isn't the fault of the publisher of the news.)
Fifth, it's obvious there ought to be some technical means to prevent this kind of situation in the future: What is the protocol to handle a case of a CA refusing (or court-ordered not to) revoke certs known to be compromised? I think you need to have a system that allows a quorum of CA's not including the issuing CA to speedily revoke certificates, or an entire CA. (A public real-time quorum of a dynamic set of validators passing messages they want to reach consensus on: Perhaps this is a good use case for a blockchain?) Of course you also have to deal with the problem of, what if the next TRO targets the entire validator set? You'd need to be sure those participants are jurisdictionally diverse (or anonymous) so it would be difficult for a single entity to compromise the entire system at once.
1. It's not a dumb reason to revoke certs, it's a VERY good reason to revoke certs. Lot's of hosting providers allow you to register subdomains with them e.g. mywebsite.squarespace.com; but Squarespace won't allow you to register a domain that starts with an underscore such as _123-mywebsite.squarespace.com because starting with an underscore is used for these DNS-based SSL checks. The concern was that DigiCerts system allowed you to receive an SSL cert for squarespace.com (the parent domain) by registering 12345678.squarespace.com (no underscore so as far as Squarespace knows, a normal subdomain). This is not a niche use of this; it's literally why the underscore prefix was there: https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c10
2. It is a pretty dumb thing for a website to file a lawsuit about, that is true.
3. DigiCert deserves a lot of blame for the TRO. DigiCert received one TRO for one client who had 72 certs from DigiCert[1]. What they should have done is continue with revoking the other 80,000+ certificates in the 24 hour time window and then inform the CABF that they have 72 (or however many) holdout certs caused by legally binding temporary restraining order. They should have been proactive in complying and in their communication. DigiCert did not do that though. They delayed all 80,000+ certificates for 5 days instead. That's not good enough.
I don't think DigiCert has the power to change literally every internal validating mechanism to check if a certificate is expired or not. But you seem to know more, so I'd ask you expound on that.
Fascinating. I think there was a fair amount of snark on both sides, but I do think some good points were raised by both, as well.
1) To DigiCert's point: If certs need an emergency revocation but it will impact a service which say: provides life saving services, or keeps the electricity on for the majority of a country, would it not be wise to file them as a one-off "exceptional circumstance". I think that common sense should prevail and everyone can agree that, "Yes, computer security is absolutely essential. Essential services are also essential." I wish that that was the direction the debate had gone in.
For instance, What is considered an 'exceptional circumstance'? What kind of services are covered, and what are not?
Personally, I would think that things like: health, heat, water, electricity, and physical security (prison and law enforcement) are all potentially essential areas. They are industries that ought be able to request an emergency, 48-hour exception if they know they can't meet it within 24 hours and their services will go down as a result. I feel like two days should be enough time for just about any organization to work through a certificate issue, unless it's a long holiday, or something very, very niche.
I think that, to a degree, Tim Callan (Setigo CEO) was being unreasonable in expecting DigiCert to not offer any kind of possibility for exceptions. Some services should not go down, just because it goes against the principals of computer security. It hate saying that, but it's true. Keeping the ICU running matters more than whether the hospital is following best security practices during an emergency.
Could it cause more problems by ignoring best practices? Possibly! Will enforcing best practices possibly kill someone? If the answer is anything other than a firm "No", then it is secondary to protecting that service.
2) To Sectigo's point: We should not allow any CA to hide behind Policies or poorly written MSAs. If things went the way they did because they were allowed to go that way, then that means you should learn from those things in the post mortem! Take steps to shore it up! Try and prevent other companies from following suit, otherwise more will take action whenever it meets their own best interest. It is disappointing that this part seems to fell into snarky retorts too, because there were some legitimate means to discuss this.
For instance: Instead of barring from someone from being allowed to file a TRO, simply have an agreement in place that before any legal action like a TRO is filed, the customer will meet with the CA and a emergency mediator. Just take 30 minutes to one hour to see if you can work things out before the customer submits a TRO!
It seems logical, right? If a customer has the cycles to file for a TRO, they should have the time to spare talking to the company they are filing a TRO against. Explain a clear reasons to a mediator why the TRO is needed, and why they can't get it done in time. Assuming that the customer can explain all of that in clear terms, it would then be obvious for DigiCert to acknowledge that level of criticality and "exceptional need", and offer their customer an emergency, temporary exemption.
Neither side wants a TRO! It makes DigiCert look weak during an emergency, and it makes Alegeus (the company that filed the TRO) look incompetent, desperate, and underhanded.
The crux of what Tim Callan (Sectigo) was getting at, is that there needs to be a correction to DigiCert's policies. It's blaringly obvious. DigiCert were, in a way, "legally attacked" in a manner that should be prevented in the future, as best they can prevent it.
DigiCert lackadaisically shrugging their shoulders and saying "B-But...that goes against Mozilla policy!" is just deflection and meaningless. DigiCert can go to the trouble of sending legal council after Sectigo for comments on Bugzilla, but they can't use legal council to protect DigiCert from surprise TRO's? Really? Bugzilla feedback...that is the legal issue? Not DigiCert being sucker punched by their own customers?
The whole thing is just so aggravating. Both sides need to get over themselves and try to work together. They don't need to like each other, but they should do what is best for the industry. Each side sending out daddy lawyer to fight for them completely misses the point, and kills the chance for constructive feedback.
> It seems logical, right? If a customer has the cycles to file for a TRO, they should have the time to spare talking to the company they are filing a TRO against. Explain a clear reasons to a mediator why the TRO is needed, and why they can't get it done in time. Assuming that the customer can explain all of that in clear terms, it would then be obvious for DigiCert to acknowledge that level of criticality and "exceptional need", and offer their customer an emergency, temporary exemption.
Have you read the court filings? Alegeus did indeed have this discussion, explaining that they have a detailed manual process for dozens of distinct websites, and could not complete it within the notice period even though they began working on it immediately. They escalated it all the way up to the DigiCert CEO, and filed for a TRO only after DigiCert told them - exactly as the Bugzilla discussion participants wanted - that no exemption would be offered. That's the key point they seem to be missing: taking a hard line on the revocation timeline does not deter legal action but generates more of it.
> I think that common sense should prevail and everyone can agree that, "Yes, computer security is absolutely essential. Essential services are also essential."
Privacy of personal data is also essential.
If anything, such context should warrant expediting the process of revocation.
1. Bugs happen. Critical ones, too. They didn't try to brush this under the carpet, but admitted to it, acted to resolve it and were transparent about it.
2. They worked quickly to make it happen. Would 24h been nice? Sure, but 24h is not much shorter than 120h. In general, 24h is plenty of time for some exploits and 120h doesn't open the window to many more. It would have been very different if it took them months or years to resolve it.
3. They genuinely engaged with the critics on bugzilla, even after Sectigo's CCO went completely off the rails with trying to strip customers off legal recourse and demanding to blacklist those who try to make use of it.
4. They could have taken legal actions against Sectigo's CCO directly but took the extra step to ask them to stop this nonsense. They didn't demand anything more and even outlined steps Sectigo needed to take to prevent any legal problems down the line, like affirming that their CCO did not make these statements on behalf of Sectigo, an affirmation that they would notify their employees to not make any actions that would violate the laws mentioned in their letter, affirm that their CCO would be instructed not to violate any of the laws outlined in their letter and lastly confirm that, upon consulting with their CCO, they were able to conclude that his statements were not meant to harm DigiCert.
The only ick is the short timeframe they expect a reply within, but that's sadly usual corporate US law practice...
Basically that letter is the result of asking an US law firm for help and telling them to be nice about it and helping their opponent through the process.
Certificate authorities are required by contract (the CA/Browser Forum Baseline Requirements is a contract they agree to when being granted trust as a CA) to revoke miss-issued certificates within 24 hours of the time they learn of the miss-issuance.
One of their customers filed a court case & got a temporary restraining order (TRO) preventing DigiCert from revoking ≈70 of those certs within 24 hours.
DigiCert then proceeded to delay revocation beyond the 24 hour limit *for all the certs, not just the ≈70 that the TRO covered.
That is a violation of the CA/Browser Forum Baseline Requirements. Failure to revoke ≈70 certs because of a court order could be accepted, but failure to revoke thousands of other certs (putting customers at risk) should not be.
DigiCert said they could not extricate the TRO from "critical infrastructure"
Whether you believe them, or not, there was a valid reason given. They also said they fixed this particular issue internally, so I guess we'll see.
DigiCert had over two million certs issued per the "businessEntity" case sensitivity bug thread, which was 2023.
They revoked ~84,000 out of an abundance of caution - my read was they revoked every cert issued from the (now defunct) system digicert called "OEM" - which had the bug where it wasn't prepending an underscore.
I just read the entirety of the underscore and the case sensitivity threads.
I'm not sure you can fully trust what's in the report. A lot of it seems like an attempt to obscure the truth by using internal jargon and vague statements like "Engineer did engineering stuff"—which ultimately means nothing.
Mostly correct. The issue is that only about 70 certs out of those 84k were actually covered by the TRO. They could have revoked the rest, as they were obligated to do by the Baseline Requirements. They didn't, they delayed.
Right, the "reason" for that in the report was they couldn't extricate the 70 certs from the critical infra from the other 80k OEM certs.
I'm trying very hard to be charitable. I don't know anything about digikey. But it could be incompetence; legal dept getting cold feet because someone filed a TRO in a court against the company; malice?
Incompetence from a CA isn't an excuse, it's a breach of the contract (the baseline requirements) & a reason to distrust them. Likewise for malice. The legal department getting cold feet would be a really bad reason to risk the entire business by violating the contract, only an incompetent legal department would recommend breaking the contract that the business depends on to exist. DigiCert (not DigiKey, they're an unrelated electronics component supplier) is in the business of selling certificates; that depends entirely on them remaining trusted to issue certificates.
i don't disagree with what you're saying, either; digicert never really answered why they couldn't get the other 83k revocations done faster, hoping "we couldn't do it, sorry" in the revocation bug report would suffice.
Yes, bugs happen. However, DigiCert could have handled these incidents a lot better.
For example, in the first issue (delayed revocation after incorrect capitalisation), DigiCert chose to not revoke several thousand certificates within the required 5-day window for reasons ranging from "is in use in embedded devices and can't be replaced in a timely fashion" (this is a failure of the customer's automation and planning, not a failure of DigiCert, and is not a reason for DigiCert to not revoke) to "is being pinned in a mobile app and the app needs to be rebuilt and pushed to app stores" (this is again a failure of the customer; the customer should have foreseen the event that their certificate would be revoked, and, if they really wanted to pin, should have also had a hot standby certificate from another CA pinned, for example). Further, DigiCert's own certificate policy at time included the following text (in section 1.4.2):
DigiCert strongly discourages key pinning and shall not consider it a sufficient reason to delay revocation.
DigiCert chose to ignore their own policy, and allowed this decision to delay revocation. That was entirely their own choice, to appease their own customers at the expense of complying with both their own policy and the Baseline Requirements.
The other more recent incident (the TRO) would have restrained DigiCert from revoking approximately 70 certificates. DigiCert chose to extend this delay to tens of thousands of certificates, issued to other customers who sought no restraining orders. DigiCert chose to ignore their own policy, and allowed this decision to delay revocation. That was entirely their own choice, to appease their own customers at the expense of complying with both their own policy and the Baseline Requirements.
> Sectigo's CCO went completely off the rails with trying to strip customers off legal recourse and demanding to blacklist those who try to make use of it
Can you provide a link to this? I looked Callan's comments on the cited bug[1] and none of it struck me as being even a little off the rails.
CNAME bug must have had something so bad DigiCert didn't want people poking around. Perhaps it's worth taking a closer look at the bug thread. They were also pressing really hard to close it. Something smells.
> They could have taken legal actions against Sectigo's CCO directly
You are only suggesting they could have handled it worse. Why would they take legal action against the CCO for statements on a bug report other than to squash transparency?
In a nutshell:
DigiCert has delayed revocation beyond what's allowed in the Baseline Requirements a few times; most recently, https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 and https://bugzilla.mozilla.org/show_bug.cgi?id=1910805. In the former case, it seems DigiCert chose to delay revocation to appease certain clients; in the latter case they were prohibited by a Temporary Restraining Order (TRO) from performing timely revocation.
Tim Callan from Sectigo has publicly lambasted DigiCert for these delays, since in both cases it seems DigiCert hasn't pushed back hard enough on its clients. In the latter case, there's concern that measures like TROs might be employed more often to stall revocation. Sectigo (and others in the WebPKI ecosystem) seem to want DigiCert to make the revocation policies very clear to clients and to ensure that clients can actually replace their certificates in a timely manner.
Sectigo is clearly the most vocal but they don't seem to be the only ones telling DigiCert to get their delayed revocation under control. So the escalation to legal threats is really uncalled for, and DigiCert could face some very significant pushback for trying this tactic.
Reads to me like Digicert is hiding behind the TRO to protect themselves from upset customers over following the procedures they should have followed. I don't think they want to update their legal work to prevent customers from legal action over revocation, legal action has been working in their favor so far.
For a company whose entire business is verifying company names and handling the CA process, Digicert seems rather unwilling to actually stick to the process. They can't help having a TRO filed against them by customers with incompetent tech such as Alegeus Technologies LLC, but this isn't the first time they've failed to follow the appropriate processes.
Going through the courts to stop negative discourse seems rather crass for a CA. I don't understand why a company that has been subject of suspicion and doubt lime Digicert would do such a thing unless it's a last-ditch effort to get the heat off their backs.
I'm sure their clients love Digicert for not forcing them to replace their certificates at the appropriate times, but they're in for a surprise when this whole thing blows up and they suddenly need to find another cert vendor when Digicert gets delisted.
This seems like a misunderstanding of the TRO.
Alegeus, a client of DigiCert, filed a court motion to stop DigiCert from revoking their certificate. The courts granted the temporary restraining order.
There is no "update their legal work to prevent customers from legal action" that can avoid a temporary restraining order that is ordered by the courts to provide time to establish facts. Its simply how the legal process works. "they can't help but" seems unfair as the issue was the client Alegeus was unable to replace their certificates in the revocation timeline.
Further "Going through the courts", they didn't go to the courts a client of theirs did.
Digicert is absolutely not blameless. Outside of the TRO, they have failed to meet revocation windows before and yes, are likely using the TRO in this instance as a shield for that.
However the TRO itself is a concerning development that all CA's need to consider depending on their legal jurisdiction, as it could put companies in a bind of being legally ordered to not to complete something they are contractually and legally required to do within the required timeline and interrupt standard security practices in cases like this.
Would a judge not consider potential harm to each party if the TRO is granted or not?
If the CA could credibly point out they would face severe consequences for failure to revoke, could that have an impact on?
The TRO was filed for and granted without DigiCerts input. By the time they responded, both parties filed to vacate the TRO (3 days later). Judges I expect weigh, to the best of their abilities, the perceived harm in granting/not granting the TRO. Considering the technical literacy of our court system I expect that leaves much to be desired
It was filed and granted without DigiCerts input, yes, but DigiCerts lawyer showed up within 24 hours but then did not do anything. See the courtlistener link: https://www.courtlistener.com/docket/68995396/alegeus-techno...
Timeline:
Jul 30, 2024 - ORDER granting 2 Ex Parte Motion for TRO. IT IS HEREBY ORDERED that DigiCert is prohibited from revoking the security certificates for the Alegeus Websites for a period of seven (7) day
Jul 31, 2024 - NOTICE of Appearance by Jess M. Krannich on behalf of DigiCert
...
Aug 3, 2024 - DOCKET TEXT ORDER. 9 Joint Motion to Vacate 3 Order Granting Ex Parte Motion for TRO is GRANTED
Looking at that timeline, I can see why other CA forum members are asking "are you really taking this seriously?" The whole point is that such a restraining order was definitely a bad call by the judge and should absolutely have been contested immediately by any legal team that was interested in representing the security interests of the CA Browser Forum (which, ya know, members of the CA Browser Forum should do in such cases, hence why they're in the CA Browser forum). The fact that DigiCerts legal team did show up for that TRO and then did not act to try to defend their ability to secure their certs, is a bad thing. If you want to be a CA, you gotta be willing to defend your need to act in the name of security, and to defend that in a social, business, and legal context. The point of CAs in the world is not to make money, it is to provide security services. Responding with "hey, but my legal obligations in my country mean I can't always do that" is a valid explanation, but it also means that the rest of the CA Browser Forum should probably not trust you.
Any CA who believes that the courts in their country could not prevent them from revoking certificates is confused at best.
It's true that DigiCert could have refused to cooperate with Alegeus and fought the court order instead of cooperating with them to rotate the certificates ASAP. But that would have taken a lot longer than 5 days, even if they eventually won. If the CA Browser Forum has such a strong security interest in swift revocation, it's hard to see how further delaying the revocation in order to provoke a legal battle promotes that interest.
Ah yes, I missed that note, thanks.
I'll point out notice of appearance doesn't mean they COULD have done anything. The TRO was granted for 7 days already.
From a strategic position, they probably saw they could work with the client and withdraw the TRO faster and cheaper than challenging the TRO in court. Its unlikely a challenge would have significantly decreased the time they were required to not act under the TRO.
They could have revoked the certificates of companies not listed in the TRO. Instead revocations for _all_ customers were delayed by 3 days over the time limit [1].
Additionally, it seems more prudent to me that they should drop Allegius as a customer than make the guy resign. The current situation seems like if an issue occurs again then DigiCert will not revoke the certs according to the timeline they committed to (24h).
[1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c11
Agreed on both counts (I made the same points in other comments :))
They should - but only if a good lawyer points that out to the courts. Remember the courts are not experts in the technical details in question (and should not be - there are far more details that could come before the court than any human has time to learn), the job of lawyers is to quickly give them enough education to figure it out.
In "Common law" when something goes before a court in slightly different situations the second court will look at what the last one decided to see if it makes sense and if they agree. Then the third time looks at both the previous. After many many cases courts will have seen all the arguments and made decisions and so there is no need to educate the court anymore, they will just look at what was done before and decide the same thing.
Not all courts follow the "common law" above though, and I'm not clear how the other systems are different. (I've only lived in common law areas)
An answer for the civil law countries I know of: The laws tend to be more detailled to avoid the situation you describe. Also, there is still a lot of referencing old cases, but it is considered guiding, not binding.
Well, if the CA forum had a really clear automatic sanction, I would expect the lawyer to raise that concern.
Other comments have since suggested this was an emergency order where the CA didn't have their lawyer present. The courts said "don't do anything for a week while the real paperwork happens". I don't know the cases in question well enough to figure out if that is correct, but it seems reasonable. But also a week is not very long.
hmm Isn't there an applicable standard of review for legal cases?
> Would a judge not consider potential harm to each party if the TRO is granted or not?
> If the CA could credibly point out they would face severe consequences for failure to revoke, could that have an impact on?
That would be an excellent reason to make sure to impose strict consequences on CAs without regard to whatever legal orders they're subject to, so that the next CA can point to those consequences when arguing against things like a TRO. "If you grant this TRO, the net effect will not be that you get what you want; the net effect will be to destroy our business and still not get what you want".
I dunno, seems to me in the TRO case there wasn't anything wrong with the certificates that had been issued. There wasn't any suggestion the the certificates had been issued to the wrong party or anything like that.
When performing DNS verification there's supposed to be an underscore in a record, which protects sites that let their users control arbitrary subdomains - you know, dyndns and things like that. Digicert mistakenly didn't ask the customer to include an underscore. Could have been a problem, if Alegeus was a dyndns provider - but they aren't.
Then Digicert spent a load of the 24 hour disclosure window checking things on their end, then e-mailed the customers after office hours so when they get in the next day they barely had any time to respond.
The only "incompetent tech" here is the assholes who decided on the policy that certificates be urgently revoked, as if they'd been issued to the wrong person, when they hadn't been issued to the wrong customer.
The middle ground here is DigiCert should have been able and continued with the revocation of all the other certificates not impacted by the TRO within the policies and contractual agreements it has.
The policy and its designers aren't incompetent, the intention is to ensure security first when fact-finding can take time to establish a subset of actual abuse. This is common in IT/Security and is clearly communicated in the policy customers agree to as well as something agreed on by all CA's. Security > Uptime is the baseline, and while you may disagree with that and have good arguments for the contrary, the argument is much wider than us and has been extensively hashed out in the working groups and discussions around this and other similar issues and is implemented this way by consensus while weighing valid arguments for different approaches and pros and cons.
>Could have been a problem, if Alegeus was a dyndns provider - but they aren't.
AFAIUI, Alegeus was just 1 customer of Digicert. But Digicert has tons of customers. Just because there wasn't a problem for Alegeus (due to Alegeus not being a dyndns provider), doesn't mean there weren't problems for other Digicert customers.
Revocation rules are intentionally very strict, in order to prevent CAs from weaseling out of them. If you leave open any possibility of a CA delaying or refusing revocation because of some form of "it's an administrative issue, it isn't that serious anyways", every incident will inevitably result in a weeks-long discussion about the real impact of the incident and whether this one is "serious enough" to require immediate action.
In the past CAs have been more than happy to lie and downplay incidents, because they know their customers don't like revocations and are going to consider switching to a competitor. Maybe not a big deal in practice for a technicality, but it does mean that CAs can not be trusted to always be truthful - and that in turn extends to serious incidents. This is obviously really bad for the web PKI as a whole, so the rule is that any violation of the rules, no matter how small or nitpicky, must be treated as if it were a serious security incident. Don't agree with the rules or think the rule is interpreted the wrong way? File a report afterwards to review the rules.
The entire business of a CA is to be a trusted third party. If you can't be trusted to always follow the rules exactly to the letter, you shouldn't be a CA. Behaving predictably is more important than being right about some technical minutiae regarding the security impact of some incident.
There was something wrong. They hadn't been issued in accordance with the baseline requirements.
The fact that those certificates had been issued without a secure method of verification is the thing that was wrong with those certificates.
Dyndns is not the only use case for having someone control a subdomain without having the full trust of the root domain.
No. Our cert infrastructure needs to meet the highest standards possible, this is not an area where "welp, no harm done, don't worry about it" is acceptable.
Here is the last reply from Callan:
https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c73
Seems extremely reasonable to me.
What is the correct action when a TRO says a company cannot revoke the cert? Is it that the company will delay revocation, but will push the judicial system to resolve the issue as fast as possible?
DigiCert probably should have revoked every cert they could within 24 hours. Instead they just pushed the revocation of all 80,000+ certs out to five days.
It's quite likely that many of their other clients pushed back on the 24-hour timeline (similar to what happened in their previous incident); I believe the delayed revocation issue (https://bugzilla.mozilla.org/show_bug.cgi?id=1910322) hints at this. The TRO gave them a convenient excuse to delay all revocations without having to explain all over again why they made exemptions for their special clients.
Heck, their status page (https://status.digicert.com/incidents/3sccz3v31lc9) even gives instructions for how to request a delayed revocation - even though the initial incident page (https://www.digicert.com/support/certificate-revocation-inci...) says clearly:
"Any issue with domain validation is considered a serious issue by CABF and requires immediate action. Failure to comply can result in a distrust of the Certificate Authority. As such, we must revoke all impacted certificates within 24 hours of discovery. No extensions or delays are permitted. We apologize if this causes a business disruption to you and are standing by to assist you with validating your domain and issuing replacement certificates immediately."
I think the eventual correct option is pretty clear.
A TRO like that is based on the company loudly declaring that revoking will cause real damage. That means their use of certificates is incompatible with the web PKI rules and ecosystem. That means they need to be migrated out ASAP, with every certificate authority refusing to take their business.
Make that series of consequences clear, and companies will stop trying that trick.
DigiCert and other CA's need to write in their terms that failure to follow the policies and revocation timelines can/will result in termination of the contract. Alegeus should have been dropped as a customer as soon as the incident was resolved and refused further products/renewals.
Use of a TRO protects you short term, but results in having to migrate to a new CA medium term. You can't stop them from using TRO's but you can make it not worth it.
Is the CA allowed to stipulate that the customer (before being dropped) needs to pay a significant sum for, say, "expenses" if they resort to this kind of TRO?
Possibly, I'm not sure if there is precedent for charging a company for taking valid legal actions, only vexatious in contract law. Probably depends on the legal jurisdiction and courts.
I think that's pretty unrealistic, considering that X.509 is a de-facto and de-jure standard in a lot of places that also ignore that requirement. It's not always up to that company to make it possible to replace certificates easily, it's an entire chain of vendors (I know at least Salesforce's process would fail here). Unless you want those people to run self-signed/private CA certs.
The other option would be building in a way to revoke other CA's individual certificates if there's some consensus on them being compelled to not revoke them. Not sure if the status quo or this would be more dangerous, but if a TRO can compel a CA to not sign a revocation, can it also compel to sign a certificate?
A company chooses its vendors. If its vendors turn out to be incompetent, they should choose again once the damage has been done. Just because Salesforce can't get their stuff together doesn't mean the CA industry needs to bend the knee. Let Salesforce figure out how to automate certificates, they've been in the business for long enough.
If running a private CA or self-signed certificates are even viable, then there are plenty of other workarounds that can be put in place (i.e. not updating the CRL so the software doesn't know about revocations).
If a CA's terms and legal documentation aren't tight enough to prevent a court from compel them to sign a certificate, that CA clearly cannot be trusted. Hopefully that issue can be fixed by writing clearer terms and better agreements, like people have been telling Digicert to do, but perhaps that's not possible at all. In that case, either the company should move to a jurisdiction where that kind of nonsense isn't possible, or it should be removed from global trust stores all together.
> A company chooses its vendors. If its vendors turn out to be incompetent, they should choose again once the damage has been done.
You say that and yet Teams has 320 million people using it, and I bet almost none of them enjoy the experience. Sometimes you just have to work with what you're given. Given "incompetence", we might as well throw in all of Azure.
> Let Salesforce figure out how to automate certificates, they've been in the business for long enough.
That might be true for DV, but there's classes of certificates that take at least weeks to obtain, think BIMI VMC or codesigning.
> You say that and yet Teams has 320 million people using it
Yeah, there are tons of companies who chose Teams as a vendor (because it's bundled with other stuff), and inflict it on their employees. It was still absolutely their choice.
The parent was correct - it's not about the company not using x509 certificates, but not using publicly trusted certificates. There are myriad private/internal PKI solutions available from OpenSSL & bash to millions of dollars of solutions from Big Vendor. If you can't replace the publicly-trusted certificate quickly, you probably don't need it to be publicly-trusted in the first place.
How can a court inhibit revocation when every CA declares their right to do so when you purchase a certificate?
The better question is, how do you prevent this tactic from working?
For example, suppose there were required to be multiple parties who could issue a revocation, each in a different jurisdiction, and if any of them was ordered not to do it then the others would be required to do it, and would have the technical capacity to do it but not be subject to the jurisdiction of that court.
Well, you can contest the motion for a TRO, for a start. Digicert failed to do so.
You can stick with your policies and revoke the certificate within 24 hours, instead of delaying revocation until a case is open and a motion for a TRO is filed. Digicert failed to do so.
You can stick with your policies and revoke the cert in face of the legal consequences, and deal with them accordingly. Again, Digicert failed to do so.
Correction, the petition for the TRO was filed ex parte. Digicert did not have any opportunity to respond before it was granted.
They certainly could have filed a response contesting the TRO. Then their customer could have filed another motion, and eventually (7 days later in this case) the judge would have ruled on the substance of it. Their judgement was that it would be preferable to work with the customer to resolve the technical issues with revocation, and submit a joint request to dismiss the TRO. The stated reasoning behind this was that it would be significantly faster than contesting the TRO. This is true: the certs were revoked and the TRO dropped within 3 days.
I think the communication on that point was severely lacking, as they only clarified it three months later and after significant hectoring in two different bug threads: https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c43
I also think it's reasonable not to take Digicert's statements at face value, given their history. But I think both of the points you made here are wrong:
> You can stick with your policies and revoke the certificate within 24 hours, instead of delaying revocation until a case is open and a motion for a TRO is filed. Digicert failed to do so.
Let's be clear about the timeline: Digicert notified their customers that the certs would be revoked. In between the time they notified the customer and the time of revocation (less than 24 hours), the customer got the ex parte restraining order. Are you suggesting that issuers should revoke certificates without notifying their users, so that the users don't have time to get an emergency TRO? I believe that would be in violation of the BRs.
> You can stick with your policies and revoke the cert in face of the legal consequences, and deal with them accordingly. Again, Digicert failed to do so.
By "revoke the cert in face of the legal consequences" do you mean "openly defy a valid and legal court order"? Because that would also violate the BRs.
Just to be clear, the whole incident covered over 80,000 certificates. The TRO was applicable to only those of one subscriber - just over 70 certificates, yet caused the revocations of all 80k+ to be delayed.
To add to this, 3 days after the TRO was filed both parties moved to vacate the TRO.
DOCKET TEXT ORDER. 9 Joint Motion to Vacate 3 Order Granting Ex Parte Motion for TRO is GRANTED
I'm not sure DigiCert could have done anything about the TRO or the impacted certs, but it should have been able to move forward with the revocation of all other certificates. That IMO is the real issue/failure, alongside the concern/impact of TRO's on security processes in the future.
> By "revoke the cert in face of the legal consequences" do you mean "openly defy a valid and legal court order"? Because that would also violate the BRs.
Yes, I think this would have been appropriate action. If the contractual language is extremely clear between the CA and the subscriber, there is no legal basis on which the customer can prevent revocation. The fact they found a court that doesn't understand technology is frankly irrelevant. This detail is exactly why Tim and other parties are requesting the exact language of the agreement between Digicert and the subscriber that filed the TRO. A customer acting in bad faith and abusing the legal system does not compel you to violate your own contract terms, your terms under the CAB/BR, or to take actions which are detrimental to the entire Internet. This is exactly the type of circumstance where you do what you are required to do, and then sort it out afterwards. Any appeals court would have easily overturned the TRO as it has no legal basis.
> A customer acting in bad faith and abusing the legal system does not compel you to violate your own contract terms, your terms under the CAB/BR
Yes, it absolutely does. "I think the court will agree with my view of what the contract says once the case is heard in full" is not a valid reason to disregard a TRO.
> or to take actions which are detrimental to the entire Internet
That would be harder. But a delayed revocation stemming from a flawed validation process, when the CA is responsible for the flaw and knows that the result of the validation was in fact correct, simply does not cause any detrimental effects to the entire Internet.
You could just require publishing the revocation immediately with an effective date in the future.
Of course, if that system had been in place, DigiCert would probably be facing hundreds of lawsuits from businesses disrupted through no fault of their own rather than inside baseball PKI drama.
> The better question is, how do you prevent this tactic from working?
Make it clear that if it works, it will only work once.
The CABF should adopt policies that any such legal action or any request for extension will be considered a public declaration that the customer's application is incompatible with the requirements of the Web PKI and that not only will the current CA refuse to renew the certificate but it will be publicly documented in the Bugzilla so that no other CA will issue certificates covering any of those names nor any new names for the same company without an affidavit explicitly stating that the issues preventing compliance have been resolved and the company acknowledges this and commits to never doing so again.
Existing names that were successfully revoked in time can be renewed but neither the problematic one(s) nor any new ones will be allowed.
If they then file for another TRO in the future they may still get a short-lived order but the existence of such an agreement would at least to my non-lawyer brain cause any judge who may have granted a TRO to become very displeased when the CA presented it in their response.
It’s Saturday and the courts are closed. You park in a car park I own.
I have put up a sign saying I can instantly crush any crossover vehicles I like, as I consider them ugly and lacking in character.
As I load your car into my crusher, you dispute the legality of my sign. But the court is closed until Monday, and the sign says I can crush your car instantly, no waiting.
Should I be allowed to crush your car today? Or should I have to wait until Monday, so the disputed legality can be resolved?
That’s actually a question for you. You won’t truly know if you’re allowed to until a court decides. You can choose to proceed and crush it, and then deal with the consequences of doing so if it turns out you were wrong. Likely you’ll end up owing damages to the owner of the car, perhaps even punitive damages on top.
I think the same applies here.
The TRO actually means you aren't allowed, that's the point. It's an ordered injunction that legally obliges you from not acting until the courts can review the facts.
The court IS deciding, temporarily.
In this case, yes. That’s pretty cut and dry for the time being. However regarding the analogy I was replying to, I was pointing out that it’s less a situation of what you’re allowed to do and more one of what you believe you’re allowed to do, and weighing the consequences of being wrong against upholding your stated terms. In other words, something you should probably discuss with a lawyer.
There is no TRO in the car park analogy.
But there is here, so its a flawed analogy.
I can declare my right to do something as much as I want... that doesn't mean I get to do it, certainly not if a judge decides I can't.
> How can a court inhibit revocation when every CA declares their right to do so when you purchase a certificate?
A court can rule that a term of a contract is void because it contradicts public policy, and it certainly can issue a TRO pausing an action which would otherwise be allowed by a contract while resolving a dispute related to it.
Because the judge doesn't know the details of how PKI works, and either the ToS for digicert doesn't spell out that they can revoke certs at any time (which would be problematic for a CA), or the judge didn't read, or didn't understand the ToS.
The order would normally be a matter of public record, and the ecosystem should follow these events carefully and ensure never to issue a certificate to such a company again as it undermines the entire system.
Isn't Sectigo Comodo? Extra rich coming from them.
No, Sectigo is the PKI business that was carved out of Comodo, back in 2017. Comodo still exists, doing whatever they do. They have been totally separate entities since then.
Web PKI drama is always astonishing to me because it is one of the only areas of the entire world where a corporation's "fucking around" is seldom ever not followed by a sobering "finding out" period. The various entities that decide what CAs to trust can effectively dismantle any CA business in the world, basically at the drop of a hat. If DigiCert decides to play this game and lose, it would make them the biggest such loser so far. DigiCert is, as far as I know, the largest CA on the Internet. It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores. Yet, there's no particular reason it couldn't happen. How exciting.
Of course, I think that is unlikely, but on the other hand, it's just as cathartic to imagine whatever idiot at DigiCert thought it was a good idea to engage legal here to have the dressing down of a lifetime. I read the thread in question. It doesn't make DigiCert look good, but this action definitely is more damning to them than anything Collan said in my opinion.
> It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores
Their customers would be forced to give their business to Honest Achmed[1].
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=647959
Thank you, that was an awesome laugh this morning, that request was genius:
> 2. Sub CAs Operated by 3rd Parties
> Honest Achmed's uncles may invite some of their friends to issue certificates as well, in particular their cousins Refik and Abdi or "RA" as they're known. Honest Achmed's uncles assure us that their RA can be trusted, apart from that one time when they lent them the keys to the car, but that was a one-off that won't happen again.
Someone should start a real CA business, complete with all the proper audits and everything to get it into the browser trust stores… and call it “Honest Achmed’s Used Cars and Certificates” (have it buy some random used car dealership in the middle of nowhere so the name is not a lie)
Where’s a billionaire troll when you need one? (And a form of billionaire trolling that would upset a lot less people than Musk’s version of it.)
Would be even funnier if the billionaire’s name actually was Achmed
What has prevented this (a good legit CA co) from happening before now?
Where's the value?
Anyone can get a cert from lets-encrypt. No users of your website care how trustworthy your CA is. And CA's are too big to fail, so you barely need to worry about your CA being removed by the trust store. So you can only compete on price.
If we want to change this, we need a way for a certificate to be co-signed by multiple CA's. So that the certificate can be presented to the user, and they can figure out if any trusted CA of theirs has signed the certificate. This way, revoking trust in a CA becomes easier, because people should have multiple signatures on their certificate. That means, all of a sudden, that the quality of a CA actually matters.
Whilst it might seem this is already possible, it is not. Cross-signing is only a thing for intermediate certificates, and does not work. You can also have multiple certificates for the same key, but when starting a TLS session, you must present a single certificate. (This seems to have changed for TLS 1.3, so perhaps this is already possible?)
I think TLS 1.3 still requires the end entity (server/client) cert first. All the other certs can now be in any order and the verification is suppoaed to figure out a valid path.
In theory, you could make an intermediate CA and get cross signed certs from multiple CAs (hopefully with Name Constraints), your intermediate CA signs your server cert, and you include all your cross certs and the intermediate certs for those. And the client figures out which chains it can make and if it likes any of them.
But experience has shown, verification may find a chain with signatures that line up, but the CA is expired or revoked in the local trust store, and reject the verification, even though another chain could have been found with the provided certificates.
And, because of the limited information in tls handshakes from real world clients, it's difficult (maybe impossible) to know which CAs a client would accept, so that the server could present an appropriate chain.
The obvious solution is for TLS 1.4 to mandate that all certificate chains contain at least one expired certificate.
Time and money. Plus right now even if you bought the infra, the staff, paid for and passed the audits, and then waiting while Apple, Mozilla, Google and Oracle (at least) included your roots...Microsoft aren't taking more right now. So you have to wait for some unknown time in the future when/if they start doing that again. You could purchase a root off an existing CA, subject to the trust stores approving it, and the boatload of cash you'd need to buy it (plus still having the staff and infra to operate it).
> Microsoft aren't taking more right now. So you have to wait for some unknown time in the future when/if they start doing that again.
Interesting, I hadn’t heard that, is there anywhere I can read more about it?
> You could purchase a root off an existing CA,
As well as a sub-CA, I remember in theory you can have two independent CAs with cross-signing… but do browsers actually support that?
"Nothing public to point out to" is probably accurate but it is noted publicly here: https://learn.microsoft.com/en-us/security/trusted-root/new-...
Nothing public to point to, sorry.
Sub-CAs: Not really. Operational risk to the parent CA is huge, you'd be hard pressed to get any current public CA to sign an issuing CA to be operated externally. Cross-signing still works (though it is the stuff of nightmares in many cases) but again you have to have money and a CA willing to do it!
Entrust is/was doing it with ssl.com after they were detrusted.
No, SSLCorp are hosting and managing a CA with Entrust branding. Same as Sectigo are doing. Entrust aren't doing issuance, verification - they're straight reselling from white-labeled issuing CAs.
Let's Encrypt has been around for a decade and is doing pretty well for itself. The manual validation required for EV and OV certificates precludes it from being a passion project, unless managing a call center and a squad of document reviewers is your dream job.
Let's Encrypt doesn't issue EV and OV. Honest Achmed’s Used Cars and Certificates wouldn't have to either.
It wouldn't be fun if Honest Achmed’s Used Cars and Certificates didn't also issue EV and OV certificates. It wouldn't be on brand to miss out on entire segments of the certificate market.
I'm actually curious why that wasn't approved
To my understanding, the point of Honest Achmed was to demonstrate that it was possible to write a facially reasonable and compliant CA inclusion request that was also intuitively unacceptable. It successfully demonstrated that the inclusion policies at the time needed to be made more rigorous, which they were.
[flagged]
> It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores. Yet, there's no particular reason it couldn't happen. How exciting.
Trust stores and Browsers specifically have more options than simply remove a CA. A more reasonable approach for distrust process of a CA this size would be to stop accepting any new certificates issued after a certain date. Then existing customers will have a heads up and even if asleep will find out the bad news during scheduled renewal rather than while the responsible employees are on vacation.
It's refreshing to see folks who're obviously used to hiding behind clouds of bullshit get skewered by people who both know enough to see through it and have the time and energy to follow every thread to completion.
The most recent digicert thread smells suspiciously similar to those that lead up to the Entrust debacle.
Is there any option to turn off trust for all certificates generated after a certain date? The ideal would be to let old certificates keep working, and not trust new ones.
It's been done previously, e.g. for Symantec:
https://security.googleblog.com/2017/09/chromes-plan-to-dist...
Yes, this has been done a few times in the past. It's unfortunate that "distrust after X date" and "distrust now" are the only penalties that currently exist but that's the reality of the current PKI world.
There are a lot of efforts to get name constraints widely supported, so a given CA can only issue for certain TLDs or even domains, but it's still not there. That could be a viable punishment for misbehaving CAs in the future, or even an intentionally chosen restriction for CAs that want less strict requirements. Government CAs could be a very useful application, if a CA cert could be name constrained to only issue for .gov.cctld at that point we could basically just let them do whatever they want without concern. Likewise for any small regional CAs, if they could be limited to only regional TLDs their blast radius is reduced which also could mean less strict requirements than those applied to CAs able to sign for .com.
It would also be really useful if I could add a "root" CA for my employer's private CA to sign certificates for say .internal or .example.com, but not allow them to sign certs for any arbitrary domain. There's decent support for constraints on intermediate certs, but then you have to trust the root won't ever be used to sign unconstrained intermediates.
It would also be useful for localhost. Create a private CA that is constrained to localhost and *.local, and you don't have to worry about it being used to MitM other sites if it gets compromised.
It would certainly send a strong message, and cause a lot of chaos, if the biggest CA on the Internet found itself being removed from trust stores.
How many will agree to that removal? How many will see one more reason to forever turn off automatic updates and decide what to trust themselves, having seen yet another way some faceless entity they never knew about can break things?
It will certainly send a strong message, but likely not the intended one. All it will do is increase the lack of trust in centralised PKI in general.
I'm not sure how to put this nicely, but I'll try my best: The normal people, who account for 99% of the end users of DigiCert's customers, are not going to disable updates on their web browser or their OS to "take a stand" against this decision. (And corporate users can't do this anyways, since it's not in their hands, and keeping things out of date is not a reasonable option for an organization that has security standards.) Most of them won't know what is going on here, and if they hear about it, probably won't care or know what to do about it. That's the Web PKI infrastructure working as intended, because it would be infeasible for billions of people to properly understand the gravity of all of these decisions. In order to ensure that TLS connections are at least minimally safe, it's pretty much necessary for things to work this way.
I'm not arguing that it's good that a relatively small number of entities (mainly Google, Microsoft and Mozilla) decide which CAs are trustworthy, but that's all the more reason that it's important for all of this Web PKI work to happen completely in the open, so that the few who can spare the time and effort to scrutinize what is going on can help the rest of us have a safer Internet. We don't have a better solution. That's also why DigiCert issuing legal threats because they don't like how one of these issue reports makes them look is a serious problem that can't be tolerated.
> The normal people, who account for 99% of the end users of DigiCert's customers, are not going to disable updates on their web browser or their OS to "take a stand" against this decision.
I'd imagine that they wouldn't do it to "take a stand" so much as "avoid the risk of getting their stuff broken in the short term" in this scenario, regardless of whichever party loses the blame game. See: the recent WordPress drama, which has turned customers away from both parties involved.
As I understand it, the way certificate authorities are removed from the trust store is progressive: they will announce a date after which new certificates from a given CA will no longer be trusted. I think this can be made even more progressive, by limiting the validity period of new certificates that will be trusted. DigiCert will have little recourse other than to let their customers know, and/or start providing certificates issued by another CA that follows the web PKI procedures and remains trusted. (They can still do that, of course, it's just that their own certificates issued by them directly won't be trusted anymore.)
On the flip side, for user impact, it will play out like this: Some bank or other important entity could possibly, for whatever reason, continue using a (presumably expired? unless DigiCert continues issuing anyways; note that most likely, they will not.) DigiCert certificate after the cut off date, which will lead to users receiving errors. Some of them will have HSTS setup, which will lead to an emergency situation where they have to issue a new certificate ASAP, as it will basically halt their business until they do. For places where there is no HSTS, users may be instructed to simply bypass the certificate warning temporarily, and support lines will be absolutely swamped until they actually fix the problem.
The WordPress situation is quite different. You don't have to use WordPress. Users don't even know what the Web PKI is to find an alternative to it, not that there is one or will be one.
Sure some users would keep the CA active out of fear, while everyone paying digicert would also begin to move out of fear.. What portion of the strange sort of sites that couldn't figure out how to get off but could figure out how to renew would bother with paying for a different error message?
The normal people, who account for 99% of the end users of DigiCert's customers, are not going to disable updates on their web browser or their OS to "take a stand" against this decision.
...unless they notice the breakage, and you tell them to do so to stop it.
Ignoring the policy reasons that this won't happen, not all devices can be downgraded, so by then it's too late anyway. Best you can do is bypass a non-HSTS certificate warning.
That all assumes DigiCert continues issuing certificates they know won't be trusted. I assume most likely what will actually happen is they have to start selling certificates from another CA, but failing that they're probably just going to stop issuing certificates before the cutoff.
> mainly Google, Microsoft and Mozilla
They don't have free rein to do whatever they want. If they chose to remove them, they are going to have to be ready to defend themselves in court.
Why exactly do you think certificate authorities have a legal right to appear in trust stores?
This case literally involves the courts getting involved in what should appear in trust stores. DigiCert was ordered by the courts to not revoke certificates via a TRO.
If the browsers removed DigiCert, DigiCert could certainly sue based on the harm to their business. They could argue that the browser vendors were inconsistent with the application of their rules. They could argue that the browser vendors were unfairly competing by kicking them out.
Not saying they would win all these, but they would certainly fight it - the alternative would be the end of DigiCert, they aren't going to go down without a fight in whatever venue they can find.
> DigiCert was ordered by the courts to not revoke certificates via a TRO.
... and DigiCert did not try to appeal the TRO.
Is that really the claim, though? If someone chooses to punish DigiCert for compliance with the TRO then DigiCert might have grounds to file for declarative relief from the court, as the court could very well decide to help protect DigiCert from external consequences of their demand.
Perhaps, but only in the case that the delayed revocations were scoped to those certificates covered by the TRO, and not over 1000x more from subscribers who had nothing to do with the company who filed the TRO.
Well personally, I think that'd be a terrible position for them to take. Honoring the TRO is, as far as I can tell, not the issue that is being raised.
Both Google and Microsoft tend to act like they have enough money and lawyers to defend whatever behaviour they feel like engaging in.
Those companies are always ready to defend themselves in court, and this is a country with extremely strong free speech rights.
What country?
> All it will do is increase the lack of trust in centralised PKI in general.
“In general” is a broad statement, given that the overwhelming majority of people using the Web PKI have no idea that they’re doing so. DigiCert is not a legible part of the value chain for the average users; they won’t notice that the sites they use switch CAs. This strong disfavoritism towards vendors is arguably one of the Web PKI’s greatest strengths.
Many systems do not fetch updates from the Mozilla root store, but from their (possibly Debian-derived) stable distribution of it. Meaning two highly respected entities, known for being well aware of the wider impact of their careful enforcement of strict policies, need to agree to cause any major breakage. When that happens, I can blindly trust they did the thing needed to keep the unaffected parts of that weird system working as intended.. and then still head to bugzilla and read about the background - to laugh at whoever triggered the mess.
> Meaning two highly respected entities, known for being well aware of the wider impact of their careful enforcement of strict policies, need to agree to cause any major breakage.
Note that this is not a "both" but a "any of them". One can disagree with the other on this and still cause wide breakage.
If DigiCert were to lose browser trust (at this point, that's still a big if), it would happen the same way it happened with prior CAs, some of which were pretty big themselves (Symantec): all certificates issued after some date would not be trusted, yes. But all existing certificates would remain valid.
This gives certificate owners ample time to look for a different issuer and no certificate buyer would deliberately purchase a certificate from an issuer when they know some percentage of users will not trust that cert.
So for the end users, everything will keep working: the existing digicert certs stay valid and newly refreshed certs will be signed by a different authority. There is no need to turn off automatic updates over this.
Between Entrust and Symantec, we've already seen this happen to large well-known CAs and everything remained fine (not for the offenders, but, hey, that's the system working as intended)
From the bugzilla
"The actual reason for the underscore is so that services which allow users to create DNS records at subdomains (e.g. dynamic DNS services) can block users from registering subdomains starting with an underscore and be safe from unwanted certificate issuance. It serves the same purpose that /.well-known does for Agreed-Upon Change To Website, and that admin/administrator/webmaster/hostmaster/postmaster do for Constructed Email to Domain Contact. By using DNS records without underscores, DigiCert has violated a security-critical assumption that these services have made.
Therefore, this is truly a security-critical incident, "
That is a pretty brutal f' up of epic proportions... not sure any digicert cert should be trusted after that.
Direct link to the comment: https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c10
The author of that comment is Andrew Ayer, who also does great writeups about CA incidents and processes on his blog:
https://www.agwa.name/blog/index
My bigger concern is that what percentage of people will realize when delegating subdomains that having a leading underscore is a security vulnerability?
Always two sides to a story but the guy who caused the validation bug at DigiCert already resigned because of it (which is extreme), the Sectigo guy wanted to prevent the bug being closed so he could keep pushing them (in a subjectively prickly fashion) for more answers about their general responsiveness.
A bit of back and forth discourse is fine and expected, but if you keep pushing someone who has their own legal dept they're eventually going to wander over to the coffee machine and have a chat about it with them, then they're going to take a look and it becomes their problem.
So the number one rule would be don't even breath the word "legal" unless you want to invoke them. This particular response is just a letter telling them to back off and it's why you have a legal dept, so they can argue with each other. This one has just found it's way into the open.
There is a understandable perspective that says CAs shouldn't be burdened with legal risk in their discussions, but that's contrary to the fact these guys are commercial entities protecting their interests, so you don't get it both ways unless all your CAs are non-commercial, and even then that would only extend so far.
Perhaps they should have also had a chat with their PR departement. And anyone responsible for company strategy. Becose the actions of the legal department just backfired.
You've given a detailed rundown like you've been following, so what is your sense about the resignation? Did the guy resign willingly, or is Digicert the kind of management that likely scapegoated him?
He assumed personal responsibility rather than institutional. At the time, everyone in the community expressed alarm that that happened, because 1) and this is obvious, there's no way a single guy would be responsible for the entire fallout and 2) because it sets a bad precedent about what companies can do with their PoC wrt web PKI. It also meant a loss in institutional knowledge about how things worked.
Do you have any resources regarding the resignation I could read?
Here's the long message he left stating in part his resignation. The comments which follow spend quite some time speaking about his resignation: https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c17
His resignation was merely a tactic to diffuse the situation. He didn’t actually leave—he's still there, active as ever.
Thanks for pointing it out - I couldn't really follow the full 'drama' between the two threads.
> It also meant a loss in institutional knowledge about how things worked.
And the experience earned from the mistake.
He is still here. Far from a scapegoat. Wait and see. He will be back in no time.
[dead]
No he didn't truly resign. Retained as contractor and most likely waiting to be reinstated. You got misled.
This is shocking to read. Even attempting to choke the legal speech of web PKI contributors with legalistic bullying is a gross inversion of the purpose and goals of the organization, and IMO warrants revoking everything to do with DigiCert on the spot.
> IMO warrants revoking everything to do with DigiCert on the spot
This is pretty heavy handed and I don't think you've thought through what the consequences of that may look like. The historic way to deal with a problematic CA is to prevent them from issuing new certificates or renewing certificates (after taking care of immediate damage, of course). There are lots of legitimate companies that use Digicert and they should have an expectation of being able to continue business in the short term while they find a different certificate provider.
I agree. "Revoking everything digicert" is a bit imprecise, "preventing digicert from issuing new certificates" is a more reasonable measure that actually only damages the parties that deserve it (digicert) and spares innocent bystanders (their customers).
Are you saying it was okay to distrust EnTrust but DigiCert is too big to fail?
No, and I'm deeply confused how you drew this conclusion from what was written.
I specifically say the historical way to deprecate a certificate authority is to prevent them from issuing new certificates. Since certificates have a finite lifetime, the certificate authority would as well and would have immediately no further revenue from certificates.
I am saying "pulling the plug" without notice is going to take down tens of thousands of bystanders and anyone using those certificates for business would be unable to conduct business securely online, which would be disastrous. For example amazon.com has a DigiCert-issued certificate.
Agreed. That was my bad.
Looking at the original report ( https://bugzilla.mozilla.org/show_bug.cgi?id=1910322 ) I can see a couple of questions that DigiCert appears to be avoiding:
> The public record of Alegeus Technologies LLC v. DigiCert shows no attempt by DigiCert to contest the court’s order prior to the end of its preferred period of nearly 120 hours, even though such a motion could have freed DigiCert to revoke the certificates days earlier.
and
> The other question in comment 28 was for the language establishing DigiCert’s right to revoke Alegeus Technologies certificates. DigiCert has waffled on this point, first implying that this language was to be found on its website but later refusing to confirm that the language on the site applied to Alegeus Technologies at the time.
SPECULATION: Digicert may have offered special terms to Alegeus, and possibly other customers. They may have chosen not to dispute the TRO in court because they did not have grounds to do so under those agreements. They may also have included confidentiality terms in those contracts that prevented them from speaking about it.
OPINION: I am surprised that the forum allowed the issue to be closed without the above quoted questions being satisfied, though it is possible they are addressed elsewhere, I have not done a complete reading of all the linked issues.
EDIT to add: DigiCert has a response in a different thread here: https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c43 that would appear to contradict my speculation. Specifically
> Even though DigiCert’s TOU and MSA prohibited Alegeus from taking the action it did, once it filed for a TRO and the court almost immediately granted it, DigiCert’s hands were tied
> Even though DigiCert’s TOU and MSA prohibited Alegeus from taking the action it did, once it filed for a TRO and the court almost immediately granted it, DigiCert’s hands were tied
So did the judge just not read the TOU before signing the TRO?
I wonder if the CAB forum would have standing to sue Alegeus and/or that judge for interfering with the PKI process with an invalid TRO.
So what's happened in the ... two months and a bit since the dates mentioned for the letters that this is apparently in reference to?
DigiCert posted a message fifteen days ago saying that "We have not used a legal team as a shield against accountability." (https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c74). This is clearly contradicted by their legal threat against Sectigo. Hence, Sectigo decided to post the "Threat of legal action" bug to make the community aware of what DigiCert actually did.
Maybe if DigiCert had not decided to make that comment, Sectigo would have been willing to stay quiet...
Continued back-and-forth on Bugzilla: https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c63
Bug has been updated with a DigiCert response.
Obviously draw your own conclusions, but I actually laughed out loud at the following statement from DigiCert:
"In reality, our letter to you was consistent with our desire to promote open and honest dialogue."
Certificate authorities have enormous trust placed upon them by every Internet user (whether they know it or not). Commensurate with this trust, they have enormous responsibilities. As the name implies, the Baseline Requirements are the minimum standard they should achieve. If they can't even do this (being unable or unwilling to revoke issued certificates within the required time-frame), then they do not deserve this trust, and it should be removed.
I understand that the TRO prevented them from revoking approximately 70 certificates, and there really is nothing they could have done differently in that case. Their other revocation failings are inexcusable.
Even taking the (one-sided) depiction of the conversations in DigiCert's letter at face value, maybe Sectigo's guy was being a git at best and intentionally trolling at worst. (I don't think he was, but let's play devil's advocate.) But even then, how did DigiCert think getting legal involved would possibly go well? Sectigo stands to gain publicity and lose nothing by going public with it to the CAB as they did here, and it's not like the CAB is going to play marriage counselor and get the two companies to make up because one of them got their feefees hurt.
Besides, this kind of hyper-polite passive-aggressive "erm akchually" conversation happens in every CAB incident discussion. I don't know why DigiCert got particularly upset about this one.
> Besides, this kind of hyper-polite passive-aggressive "erm akchually" conversation happens in every CAB incident discussion.
As somebody who doesn't spend much time scrolling CAB reports, this was jarring to me.
Digicert's legal action seems nuts, and there seems like a real, risky issue in the idea that a company's customers can use the legal system to block the company from complying with its obligations to other entities, but it's hard to see any way that could be productively addressed given the back and forth in the thread. It's like I'm watching a theatrical production staring the most stereotypical corporate drones trading comments with the most stereotypical IRC nerds, both sides doing circles around an interesting topic but too busy trading blows to ever really get to it.
> but it's hard to see any way that could be productively addressed given the back and forth in the thread
Another commenter mentioned in a sibling thread the possibility of using future-dated revocations. The CA could be mandated to publish such a revocation immediately, such that it takes effect by the 24-hour deadline. Once published, the revocations themselves should be irrevocable. This would also need to be accounted for in the CA's customer contracts.
https://news.ycombinator.com/item?id=43169867
As the comment you linked to notes, that doesn't really fix the problem so much as it ensures that the CA's legal team gets to work nights and weekends for the next year.
The legal system is almost certainly going to view future-dating and immediately publishing revocations poorly in any civil action where a customer claims harm.
It fixes the problem of a customer getting an injunction despite what the contract says. If DigiCert doesn't want to be liable for harm they should improve their performance, no try to weasel their way out of the agreements that let their business exist in the first place.
> but it's hard to see any way that could be productively addressed
It sounds to me there are things digicert could have done better, without violating the court order:
- they could have revoked all the other certs that weren't protected by the TRO. They did not.
- they could have contested the TRO sooner. But they didn't.
- Possibly, they could have given customers more warning, since they didn't notify customers until much of the 24 hours was gone
That said, IMO I don't what they did is necessarily irredeemable. But they need to come up with, and execute on, a plan to make sure something like this doesn't happen again. And threatening legal action is really not a good way to re-establish trust.
> there seems like a real, risky issue in the idea that a company's customers can use the legal system to block the company from complying with its obligations to other entities
As Digicert has repeatedly explained, this is simply how the United States legal system works. Courts have broad and indisputable power to issue temporary restraining orders, and the parties to a case must comply even if doing so violates some promise they made to a third party. (The point of the TRO is to maintain the status quo while the court figures out details like what promises have been made to who.) People in the PKI community who believe that some carefully written policy would enable CAs to reject an invalidation TRO, or convince a court that they cannot issue it, are wrong.
The reason it's never come up before is that no CA had previously attempted to enforce a widespread 24 hour revocation caused by its own error.
(I think Sectigo's argument is that Digicert did not even attempt to convince the court that it should be allowed to revoke those certificates in the mandated timeframe. If they had attempted and failed, I don't think they would be receiving criticism.)
That's their argument, yes, but it was clearly based on the incorrect belief that there's some emergency button you can press to demand that a court consider your arguments ASAP. As Digicert explained:
> The legal world does not move as fast as the demands of our CA ecosystem. Our legal approach was to work with the complainant’s legal team to get the TRO dismissed in 5 days.
The court would not have dissolved the TRO in anywhere close to 5 days without the complainant's cooperation, even if Digicert had an ironclad argument for doing so. Digicert made the right choice to get the certificates rotated as fast as possible - and I don't think Sectigo intends to argue it would be better to stand on principle even if that makes the revocation slower.
One customer filed a TRO saying "don't revoke my certs". DigiCert then said "well, I guess we can't revoke any of the certs for the 80,000+ other customers either". That's stupid, and not acceptable. Instead, revoke the other 79,999 customers and communicate to the CABF saying "we've got one holdout that we are legally prevented from acting on." DigiCert didn't do that. They're acting not on behalf of transparently representing the interests of the CA/Browser Forum, instead they're trying to save their own skin from both sides. That's not good enough.
> the incorrect belief that there's some emergency button you can press to demand that a court consider your arguments ASAP
... isn't that exactly the legal button the complainant pushed to prevent the revocation?
Should've been more specific. You can ask a court to do all kinds of things, but the individual judge who reads your filing doesn't have to (and in most cases can't) carefully analyze your arguments that day. They need time to think it over, and probably hearings where you and the other party can explain all the arguments for why certain rulings should or shouldn't be made. A contract dispute like this, where one party says they have a right to do something and the other party says they don't, is almost always going to take longer than 1 or 5 or 30 days for a court to figure out.
Temporary restraining orders are the biggest exception. If DigiCert is about to do something crazy like take down all your websites, courts are generally willing to put a temporary stop to it without understanding all the details. "Preserve the status quo" and "prevent irreparable harm" are the buzzwords.
> If DigiCert is about to do something crazy like take down all your websites, courts are generally willing to put a temporary stop to it without understanding all the details. "Preserve the status quo" and "prevent irreparable harm" are the buzzwords.
So if DigiCert's irreparable harm was great would that prevent it? Like legally requiring CAs to follow their revocation policies or pay millions in damages?
You're conflating DigiCert's argument against issuance of the TRO, with the irreparable harm the complaintant (Alegeus) is alleging will occur if the TRO is not granted.
Are there actually millions in damages being caused by delaying revocation of these certificates? Courts are generally averse to “penalty clauses” where you make up a nonsense number and call it damages. (Irreparable harm means that ordering monetary compensation can’t remediate it, so a more reasonable fee would probably not count.)
Could they not have had a clause in the contract saying “if you delay a revocation in any manner you owe us $100 million.”?
So a customer could go right ahead and get a TRO but long term it will cost them less than making sure their infrastructure can handle this rare event?
Liquidated damages are possible, but they have to have some relationship to the damages suffered, so $100 million is probably far too high. In any case, I think the complainant was correct that it was DigiCert who caused the entire scenario to happen by issuing certificates which they should have known were invalid.
I don't think I disagree with you about what courts can do. And (as shown by all the back and forth in the CAB thread, and now here), it is risky: we currently have a situation where CAs can find themselves in a position where taking court-mandated actions (or lack of actions) puts them in jeopardy with the CAB, and violating court orders puts them in jeopardy with the government.
For all the back and forth in the CAB thread, it doesn't seem we're any closer to finding an escape valve for that, which seemingly would have to come from the CAB because there isn't even really a mechanism for courts to decide not to issue TROs about cert revocation, short of somehow taking a case around this to the Supreme Court (which isn't going to happen).
It's fascinating that we've built a system that has expended perhaps several million dollars of engineering, legal and admin etc time over the issue of a single letter not being capitalized [1], without any demonstrable impact beyond a failure to meet ambiguous specifications.
I do hope that dealing with all of the underlying issues around revocation etc makes the time and effort spent useful, and the Web PKI doesn't just mire itself in squabbling that blocks progress on actually meaningful issues.
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1894560
The bug that prompted this was more serious: https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c74
Basically the missing '_' was supposed to allow DNS providers who allow programmatic DNS record creation to filter out unauthorised certificate creation. So the certificates created without it could have been unauthorized by the owner of the domain they claim to certify.
I think that the Van Halen Brown M&M anecdote is relevant here.
> and the Web PKI doesn't just mire itself in squabbling that blocks progress on actually meaningful issues.
In your view, are there any meaningful issues going un-addressed currently?
Entrust got torpedoed basically for deploying an improvement to the requirements for its certificates slightly before the improvement got officially approved, and the browser people collectively lost their crud over the concept that Entrust didn't immediately revoke all of their... perfectly valid, secure certificates immediately.
Fundamentally, there is no accountability in the web PKI stewards. You want to talk about utter waste and incredible damage to the Internet, you can see it right here, in the people determining who is allowed to issue you sets of magic numbers that browsers have agreed are trustworthy, despite everyone involved behaving like complete children.
And of course, the browser operators all have their own root CAs, so basically have extremely motivated reasons to want to eliminate every commercial provider that isn't one of the monopoly companies.
Meanwhile:
- Compromised certificates are basically a non-issue from a threat model standpoint, every certificate error people hit are just... expired certificates people didn't rotate yet.
- Expired certificates cause issues for the majority of businesses at some point or another, making the internet increasingly fragile and unreliable.
Entrust didn’t get nuked over a single incident, they were nuked because of a pattern of behavior that extended back several years. https://webpki.substack.com/p/entrust-considered-harmful-par...
Can someone point to where i can read some context?
DigiCert's legal threat, while obviously biased towards DigiCert, gives some context: https://bug1950144.bmoattachments.org/attachment.cgi?id=9468...
For further reading, consider these two incidents which resulted in delayed revocation from DigiCert and a bunch of comments about how DigiCert should not be allowing delayed revocation:
- Incident report https://bugzilla.mozilla.org/show_bug.cgi?id=1894560, delayed revocation report https://bugzilla.mozilla.org/show_bug.cgi?id=1896053 - incident due to the issuance of some certificates with incorrectly-capitalized phrases in the certificate's Business Category field; baseline requirements require revocation within five days but DigiCert dragged that out much further
- Incident report https://bugzilla.mozilla.org/show_bug.cgi?id=1910322, delayed revocation report https://bugzilla.mozilla.org/show_bug.cgi?id=1910805, DigiCert information page https://www.digicert.com/support/certificate-revocation-inci... - incident due to incorrect CNAME-based domain validation (failure to check that the CNAME started with an underscore); baseline requirements require revocation within 24 hours but DigiCert was stopped by the TRO and revoked after five days.
Essentially, DigiCert has been delaying the revocation process (twice now) and people are unhappy about that. DigiCert has apparently attempted to silence those unhappy people (Sectigo and their representative Tim Callan) with legal action.
I don't believe that that's a legal filing. DigiCert never filed anything with the courts. That's just a letter to Sectigo threatening to sue.
Whoops, sorry, should've said legal threat not filing - fixed.
Can someone link to (or post copies of) the offending questions/statements?
In the second paragraph:
> Contrary to this statement, I received a letter from DigiCert’s lawyers, Wilson Sonsini, regarding posts made by Sectigo’s Chief Compliance Officer in bug 1910322. https://bugzilla.mozilla.org/show_bug.cgi?id=1910322
> The code worked in our original monolithic system but was not implemented properly when we moved to our micro-services systems
:chefs_kiss:
it gets even better:
> We also found that the bug in the code was inadvertently remediated when engineering completed a user-experience enhancement project that collapsed multiple random value generation microservices into a single service.
Some might see horror but I see a few genius level engineers 'working' 1hour a month rebuilding the same getrandom syscall.
They hiring? :]
> multiple random value generation microservices
Wot. They had _multiple_ services generating random numbers?
It might mean multiple instances of the same code (which would explain why there might be commonalities)?
As we all know, software isn't software unless it can immediately scale to having 12 billion users.
This is a really stupid situation all around.
To issue a cert for a client (say example.com), the CA sends a random number challenge to the client (say 12345678). The client then responds by making a DNS record for _12345678.example.com but due to a bug, DigiCert accepted 12345678.example.com instead.
Apparently there's a rule that says any such certs must be considered mis-issued and must be revoked by the issuing CA within 24 hours.
The CA talked to some clients who said it would be too much trouble to replace their cert on such short notice. One of those clients filed a lawsuit with the court system, which responded by issuing a restraining order telling the CA not to revoke the cert.
First of all, this is a dumb reason to revoke certs. It's difficult to imagine a situation where any of the mis-issued certificates correspond to an actual compromise of somebody's website by a bad actor. But we can perhaps give the benefit of the doubt: I think the intent here is a fail-safe procedure is implemented so revocation is triggered by a broad class of circumstances, which is constructed so that it's easy to check.
Second, this is a dumb thing for a website to file a lawsuit about. If you have to replace the cert on your website because your CA made a booboo, the engineer-hours needed to change out your new cert are far less than the lawyer-hours needed to try to work through the court system.
Third, it's dumb for people to blame DigiCert for the TRO. The TRO was filed by a service provider called Alegeus. If DigiCert doesn't follow the TRO, they'll be in contempt of court. Theoretically, a DigiCert employee who presses the "Revoke" button at this point could be sent to jail for it!
Fourth, a certificate revocation is basically a signed message that says "News Flash: I think this certificate might not correspond to this website." A court preventing the publication of such a message is equivalent to a court preventing a newspaper from publishing an article. This is called "prior restraint" and it has a high bar in the US. Prior restraint analysis was not performed by the court before the TRO was issued. I think the court violated DigiCert's due process rights (by not analyzing whether prior restraint applies) and DigiCert's free speech rights (by not denying the TRO as it would be prior restraint). (That browsers have a hair-trigger response to certain kinds of news flashes and will take drastic action isn't the fault of the publisher of the news.)
Fifth, it's obvious there ought to be some technical means to prevent this kind of situation in the future: What is the protocol to handle a case of a CA refusing (or court-ordered not to) revoke certs known to be compromised? I think you need to have a system that allows a quorum of CA's not including the issuing CA to speedily revoke certificates, or an entire CA. (A public real-time quorum of a dynamic set of validators passing messages they want to reach consensus on: Perhaps this is a good use case for a blockchain?) Of course you also have to deal with the problem of, what if the next TRO targets the entire validator set? You'd need to be sure those participants are jurisdictionally diverse (or anonymous) so it would be difficult for a single entity to compromise the entire system at once.
1. It's not a dumb reason to revoke certs, it's a VERY good reason to revoke certs. Lot's of hosting providers allow you to register subdomains with them e.g. mywebsite.squarespace.com; but Squarespace won't allow you to register a domain that starts with an underscore such as _123-mywebsite.squarespace.com because starting with an underscore is used for these DNS-based SSL checks. The concern was that DigiCerts system allowed you to receive an SSL cert for squarespace.com (the parent domain) by registering 12345678.squarespace.com (no underscore so as far as Squarespace knows, a normal subdomain). This is not a niche use of this; it's literally why the underscore prefix was there: https://bugzilla.mozilla.org/show_bug.cgi?id=1910322#c10
2. It is a pretty dumb thing for a website to file a lawsuit about, that is true.
3. DigiCert deserves a lot of blame for the TRO. DigiCert received one TRO for one client who had 72 certs from DigiCert[1]. What they should have done is continue with revoking the other 80,000+ certificates in the 24 hour time window and then inform the CABF that they have 72 (or however many) holdout certs caused by legally binding temporary restraining order. They should have been proactive in complying and in their communication. DigiCert did not do that though. They delayed all 80,000+ certificates for 5 days instead. That's not good enough.
[1] - https://www.courtlistener.com/docket/68995396/2/alegeus-tech...
Thank you for explaining what all of this was about. Everyone else keeps talking in abstract and apocalyptic terms.
An underscore. WTF.
What is the actual impact on infrastructure? Is it just some old certificates being valid past expiration?
I don't think DigiCert has the power to change literally every internal validating mechanism to check if a certificate is expired or not. But you seem to know more, so I'd ask you expound on that.
Streisand effect 101?
Fascinating. I think there was a fair amount of snark on both sides, but I do think some good points were raised by both, as well.
1) To DigiCert's point: If certs need an emergency revocation but it will impact a service which say: provides life saving services, or keeps the electricity on for the majority of a country, would it not be wise to file them as a one-off "exceptional circumstance". I think that common sense should prevail and everyone can agree that, "Yes, computer security is absolutely essential. Essential services are also essential." I wish that that was the direction the debate had gone in.
For instance, What is considered an 'exceptional circumstance'? What kind of services are covered, and what are not?
Personally, I would think that things like: health, heat, water, electricity, and physical security (prison and law enforcement) are all potentially essential areas. They are industries that ought be able to request an emergency, 48-hour exception if they know they can't meet it within 24 hours and their services will go down as a result. I feel like two days should be enough time for just about any organization to work through a certificate issue, unless it's a long holiday, or something very, very niche.
I think that, to a degree, Tim Callan (Setigo CEO) was being unreasonable in expecting DigiCert to not offer any kind of possibility for exceptions. Some services should not go down, just because it goes against the principals of computer security. It hate saying that, but it's true. Keeping the ICU running matters more than whether the hospital is following best security practices during an emergency.
Could it cause more problems by ignoring best practices? Possibly! Will enforcing best practices possibly kill someone? If the answer is anything other than a firm "No", then it is secondary to protecting that service.
2) To Sectigo's point: We should not allow any CA to hide behind Policies or poorly written MSAs. If things went the way they did because they were allowed to go that way, then that means you should learn from those things in the post mortem! Take steps to shore it up! Try and prevent other companies from following suit, otherwise more will take action whenever it meets their own best interest. It is disappointing that this part seems to fell into snarky retorts too, because there were some legitimate means to discuss this.
For instance: Instead of barring from someone from being allowed to file a TRO, simply have an agreement in place that before any legal action like a TRO is filed, the customer will meet with the CA and a emergency mediator. Just take 30 minutes to one hour to see if you can work things out before the customer submits a TRO!
It seems logical, right? If a customer has the cycles to file for a TRO, they should have the time to spare talking to the company they are filing a TRO against. Explain a clear reasons to a mediator why the TRO is needed, and why they can't get it done in time. Assuming that the customer can explain all of that in clear terms, it would then be obvious for DigiCert to acknowledge that level of criticality and "exceptional need", and offer their customer an emergency, temporary exemption.
Neither side wants a TRO! It makes DigiCert look weak during an emergency, and it makes Alegeus (the company that filed the TRO) look incompetent, desperate, and underhanded.
The crux of what Tim Callan (Sectigo) was getting at, is that there needs to be a correction to DigiCert's policies. It's blaringly obvious. DigiCert were, in a way, "legally attacked" in a manner that should be prevented in the future, as best they can prevent it.
DigiCert lackadaisically shrugging their shoulders and saying "B-But...that goes against Mozilla policy!" is just deflection and meaningless. DigiCert can go to the trouble of sending legal council after Sectigo for comments on Bugzilla, but they can't use legal council to protect DigiCert from surprise TRO's? Really? Bugzilla feedback...that is the legal issue? Not DigiCert being sucker punched by their own customers?
The whole thing is just so aggravating. Both sides need to get over themselves and try to work together. They don't need to like each other, but they should do what is best for the industry. Each side sending out daddy lawyer to fight for them completely misses the point, and kills the chance for constructive feedback.
> It seems logical, right? If a customer has the cycles to file for a TRO, they should have the time to spare talking to the company they are filing a TRO against. Explain a clear reasons to a mediator why the TRO is needed, and why they can't get it done in time. Assuming that the customer can explain all of that in clear terms, it would then be obvious for DigiCert to acknowledge that level of criticality and "exceptional need", and offer their customer an emergency, temporary exemption.
Have you read the court filings? Alegeus did indeed have this discussion, explaining that they have a detailed manual process for dozens of distinct websites, and could not complete it within the notice period even though they began working on it immediately. They escalated it all the way up to the DigiCert CEO, and filed for a TRO only after DigiCert told them - exactly as the Bugzilla discussion participants wanted - that no exemption would be offered. That's the key point they seem to be missing: taking a hard line on the revocation timeline does not deter legal action but generates more of it.
> I think that common sense should prevail and everyone can agree that, "Yes, computer security is absolutely essential. Essential services are also essential."
Privacy of personal data is also essential.
If anything, such context should warrant expediting the process of revocation.
[flagged]
Why do you all think DigiCert handled this badly?
1. Bugs happen. Critical ones, too. They didn't try to brush this under the carpet, but admitted to it, acted to resolve it and were transparent about it.
2. They worked quickly to make it happen. Would 24h been nice? Sure, but 24h is not much shorter than 120h. In general, 24h is plenty of time for some exploits and 120h doesn't open the window to many more. It would have been very different if it took them months or years to resolve it.
3. They genuinely engaged with the critics on bugzilla, even after Sectigo's CCO went completely off the rails with trying to strip customers off legal recourse and demanding to blacklist those who try to make use of it.
4. They could have taken legal actions against Sectigo's CCO directly but took the extra step to ask them to stop this nonsense. They didn't demand anything more and even outlined steps Sectigo needed to take to prevent any legal problems down the line, like affirming that their CCO did not make these statements on behalf of Sectigo, an affirmation that they would notify their employees to not make any actions that would violate the laws mentioned in their letter, affirm that their CCO would be instructed not to violate any of the laws outlined in their letter and lastly confirm that, upon consulting with their CCO, they were able to conclude that his statements were not meant to harm DigiCert.
The only ick is the short timeframe they expect a reply within, but that's sadly usual corporate US law practice...
Basically that letter is the result of asking an US law firm for help and telling them to be nice about it and helping their opponent through the process.
DigiCert miss-issued thousands of certs.
Certificate authorities are required by contract (the CA/Browser Forum Baseline Requirements is a contract they agree to when being granted trust as a CA) to revoke miss-issued certificates within 24 hours of the time they learn of the miss-issuance.
One of their customers filed a court case & got a temporary restraining order (TRO) preventing DigiCert from revoking ≈70 of those certs within 24 hours.
DigiCert then proceeded to delay revocation beyond the 24 hour limit *for all the certs, not just the ≈70 that the TRO covered.
That is a violation of the CA/Browser Forum Baseline Requirements. Failure to revoke ≈70 certs because of a court order could be accepted, but failure to revoke thousands of other certs (putting customers at risk) should not be.
DigiCert said they could not extricate the TRO from "critical infrastructure"
Whether you believe them, or not, there was a valid reason given. They also said they fixed this particular issue internally, so I guess we'll see.
DigiCert had over two million certs issued per the "businessEntity" case sensitivity bug thread, which was 2023.
They revoked ~84,000 out of an abundance of caution - my read was they revoked every cert issued from the (now defunct) system digicert called "OEM" - which had the bug where it wasn't prepending an underscore.
I just read the entirety of the underscore and the case sensitivity threads.
I'm not sure you can fully trust what's in the report. A lot of it seems like an attempt to obscure the truth by using internal jargon and vague statements like "Engineer did engineering stuff"—which ultimately means nothing.
Mostly correct. The issue is that only about 70 certs out of those 84k were actually covered by the TRO. They could have revoked the rest, as they were obligated to do by the Baseline Requirements. They didn't, they delayed.
Right, the "reason" for that in the report was they couldn't extricate the 70 certs from the critical infra from the other 80k OEM certs.
I'm trying very hard to be charitable. I don't know anything about digikey. But it could be incompetence; legal dept getting cold feet because someone filed a TRO in a court against the company; malice?
Incompetence from a CA isn't an excuse, it's a breach of the contract (the baseline requirements) & a reason to distrust them. Likewise for malice. The legal department getting cold feet would be a really bad reason to risk the entire business by violating the contract, only an incompetent legal department would recommend breaking the contract that the business depends on to exist. DigiCert (not DigiKey, they're an unrelated electronics component supplier) is in the business of selling certificates; that depends entirely on them remaining trusted to issue certificates.
i merely typoed, i'm your GGP to your comment \^
i don't disagree with what you're saying, either; digicert never really answered why they couldn't get the other 83k revocations done faster, hoping "we couldn't do it, sorry" in the revocation bug report would suffice.
I fully agree. Malice and trust are fundamentally incompatible.
Yes, bugs happen. However, DigiCert could have handled these incidents a lot better.
For example, in the first issue (delayed revocation after incorrect capitalisation), DigiCert chose to not revoke several thousand certificates within the required 5-day window for reasons ranging from "is in use in embedded devices and can't be replaced in a timely fashion" (this is a failure of the customer's automation and planning, not a failure of DigiCert, and is not a reason for DigiCert to not revoke) to "is being pinned in a mobile app and the app needs to be rebuilt and pushed to app stores" (this is again a failure of the customer; the customer should have foreseen the event that their certificate would be revoked, and, if they really wanted to pin, should have also had a hot standby certificate from another CA pinned, for example). Further, DigiCert's own certificate policy at time included the following text (in section 1.4.2):
DigiCert chose to ignore their own policy, and allowed this decision to delay revocation. That was entirely their own choice, to appease their own customers at the expense of complying with both their own policy and the Baseline Requirements.The other more recent incident (the TRO) would have restrained DigiCert from revoking approximately 70 certificates. DigiCert chose to extend this delay to tens of thousands of certificates, issued to other customers who sought no restraining orders. DigiCert chose to ignore their own policy, and allowed this decision to delay revocation. That was entirely their own choice, to appease their own customers at the expense of complying with both their own policy and the Baseline Requirements.
Are you starting to see a pattern?
Yes, they're really good at making excuses. It seems like they could get away with anything.
[dead]
> Sectigo's CCO went completely off the rails with trying to strip customers off legal recourse and demanding to blacklist those who try to make use of it
Can you provide a link to this? I looked Callan's comments on the cited bug[1] and none of it struck me as being even a little off the rails.
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1910322
CNAME bug must have had something so bad DigiCert didn't want people poking around. Perhaps it's worth taking a closer look at the bug thread. They were also pressing really hard to close it. Something smells.
[dead]
[dead]
From here on:
https://bugzilla.mozilla.org/show_bug.cgi?id=1910805#c30
> They could have taken legal actions against Sectigo's CCO directly
You are only suggesting they could have handled it worse. Why would they take legal action against the CCO for statements on a bug report other than to squash transparency?