BackgroundIn late January 2001, VeriSign erroneously issued two Class 3 code-signing certificates to someone falsely claiming to represent Microsoft. The certificates were issued in Microsoft's name, specifically "Microsoft Corporation". After issuing the certificates, a routine VeriSign audit uncovered the error in mid-March, about 6 weeks later. VeriSign then did three things:
- It notified Microsoft of the error.
- It posted a public notice.
- It revoked the certificates in its normal Certificate Revocation List (CRL)
Microsoft's immediate response was to announce the problem and describe what users could do to actively recognize and avoid trusting the fraudulent certificates. Microsoft's ultimate response was to issue a patch, described in Microsoft Security Bulletin MS01-017.
It's important to understand that the two certificates at the center of this incident are real VeriSign certificates. not fakes masquerading as real. The fundamental error is that VeriSign issued those certificates to somebody other than Microsoft, so somebody can then create and sign software that appears to come from Microsoft.
It's also important to understand that VeriSign's normal procedure for revoking ANY of its certificates is to publish the certificate serial-numbers in its CRL. VeriSign maintains many CRLs, one for each class and category of its certificates. You can see a list of all the available CRLs simply by visiting VeriSign's CRL home.
Stirring Up Hornets
Bruce Schneier, a well-known and widely respected security expert, then wrote about the episode in the April issue of his monthly newsletter Crypto-Gram, in an article entitled Fake Microsoft Certificates . In that article, Bruce made several statements about the nature of Microsoft's security infrastructure. One statement was: [. ] there is no way to revoke the certificates (Windows has no CRL features), [. ]By coincidence, a letter I wrote to Crypto-Gram about a previous article also appeared in the same issue, and I specifically cited the bogus certificate episode in one of my examples. I also made a statement about Microsoft's security infrastructure, specifically: [. ] no Microsoft software is capable of automatically obtaining the Certificate Revocation List (CRL) listing those two bogus certificates, because there's no revocation infrastructure. Microsoft did not agree with either one of these characterizations. Two things then happened:
- Late on 14 April 2001, the program manager of .NET security, David B. Cross. sent me an email containing his argument against my characterization, ending with:
"I believe your statement was made in error and should be corrected. "
I did not accept his argument, and we engaged in a few responses and counter-responses. I will not present his specific argument here, because it essentially mirrored the argument that would appear later in a public Microsoft response.
The issue is not whether Microsoft provided a patch. It plainly did.
The issue is not whether the patch is effective in blocking the two certificates or or not. It presumably is.
The specific point at issue here is whether Windows, via its CryptoAPI, does or does not have a viable revocation infrastructure. We can be even more specific and confine ourselves to the viability of the revocation infrastructure just for VeriSign's Class 3 code-signing certificates. since those are the kind of certificates at the center of this incident. They are also the most likely kind of certificates that developers will buy and sign their code with.
This may seem to be a minor quibble, but revocation is a crucial part of any certificate-based security infrastructure. That's because certificates can be revoked for any number of reasons prior to their expiration date.
Issuing and Revoking Certificates
Any organization that issues certificates is called a Certificate Authority (CA). As a rule, every certificate issued by a CA contains an expiration date (expiry), placed directly into the certificate by the CA when the certificate is made. Each certificate also contains its own class and category, e.g. a class 3 code-signing certificate, so anyone who validates certificates can tell exactly what privilege or capability should be granted to the certificate's holder.
Every CA that issues certificates will also need to revoke them. There can be many reasons for this. One reason is that the private keys corresponding to the certificate have been compromised or lost. Then the certificate holder contacts the CA and asks that the certificate be revoked. The CA authenticates this request before actually listing the certificate on its CRL. A CA itself may also decide to revoke a certificate. For example, if the certificate holder violates the issuing agreement, or if the certificate was issued in error, such as happened with the bogus Microsoft certificates. Whatever the reason, the certificate's serial-number appears on the CRL of the issuing CA, and anyone who receives the certificate should decline to trust it.
Each CA decides how long its certificates will last, and customers know the length of this period (typically one year) before they buy the certificate. The expiry cannot be modified without making the certificate invalid. Since the expiry is right there in the certificate, anyone reading the certificate to determine its trustworthiness can determine the expiry for themselves. They don't need to be told by some other source when any given certificate will expire.
Things are not so easy when a certificate must be revoked before its normal expiry. Then, anyone reading the certificate needs additional data: the Certificate Revocation List (CRL) corresponding to the certificate. CA's typically provide a CRL at some well-known and publically accessible location on the Internet, i.e. at some URL. CA's also update their CRL's when new revocations occur. Each CRL is digitally signed and dated by the issuing CA, so everyone having the CRL knows it was really generated by the CA rather than some random interloper.
CA's typically issue CRLs at some periodic rate, whether there are changes or not. This can be daily or weekly in typical cases. It may be more or less frequently, according to the CA's policies. The reason for revocation, hence the rationale for the update frequency may also vary. The key phrase used to describe the timeliness of CRLs is suitably-recent and can be found in RFC 2459: In short, CAs decide how often to issue CRLs, and users of those certificates ultimately decide how often to obtain a suitably-recent CRL. This does not remove the need for the CRL. That is, "never" is not an allowable value for "suitably-recent".All certificate users, i.e. anyone who examines certificates to determine its trustworthiness, need all the following pieces of information in order to determine the complete trustworthiness of any certificate:
- The certificate in question, i.e. the one whose trust is being evaluated.
- The already trusted root certificate of the CA that issued the certificate in question.
- A suitably-recent CRL issued by the CA covering the class and category of the certificate in question.
So where do you obtain all these three elements? The trusted root CA certificate is something you already have. It has to be, because you have to trust it already. The certificate in question is presented to you each time. But where do CRLs come from? Ultimately from a CA, but how?
CRLs are obtained using a revocation infrastructure. This just means that for every CA you trust, and whose root CA certificate you use, you have a corresponding URL to download a CRL, or some other means to obtain one. On its face, this seems simple enough, and it is. The difficulties come in how a particular CA tells you where to obtain its CRLs, and whether your computer does anything about it. This is also the basis for Bruce's and my assertions that Windows has no revocation infrastructure.
When a CA tells you where to obtain CRLs, it must do so in a trustworthy manner. That is, it must be effectively impossible for any random interloper to provide a fake, outdated, or otherwise bogus CRL. If you used a fake or outdated CRL, someone could convince you a certificate was valid when it really wasn't.Some things that CA's do to ensure authentic CRLs are:
- Every CRL is digitally signed by the CA using its root certificate. This ensures that CRLs can't be faked.
- Every CRL is dated by the CA. This ensures that every CRL eventually becomes stale or expires and you obtain the latest one.
- Every CRL has a higher sequence number than the one issued before it. This ensures that no one can slip you a CRL earlier than the one you have now, even one that has not yet expired.
- The Manual Method -- You could make users manually download a CRL from a certain URL, install it in a specific place on the local machine, and then use the local file. This has the same disadvantage as every manually maintained security update or fix: users don't do it.
- The Well-Known-URL Method -- Since you already have a trusted root certificate for every CA, you can also have a specific known and trusted URL associated with every CA. Every CA would publish this URL in a well-known location, so everyone would know it. When you need a CRL, you just look up a URL in your trusted list of associations, retrieve the CRL, and you're ready. ("Well-known" primarily means well-known to software developers, since they create the default trusted associations, though users can update or edit them.)
- The Root-Certificate-URL Method -- Every CA could write the URL of its current CRL in its trusted root certificate. Then when you need the CRL you just read its URL from the trusted root certificate, retrieve the CRL, and you're ready. (This is actually just a simple variation on the next method.)
- The RFC 2459 Method -- Rather than writing a URL only in its root certificate, every CA could write the URL of a CRL into every certificate it issues. The advantage of this over the Root-Certificate-URL method is that it allows a CA to add new URLs over time. RFC 2459 is the document where this method is described.
- You must know where to obtain a suitably-recent CRL.
- You must actually obtain that CRL.
- You must authenticate the contents of the CRL.
- You must actually use the CRL.
Now that you know what a revocation infrastructure is, and why it's important, let's return to the VeriSign/Microsoft case. We already know that Windows accepts VeriSign certificates. Microsoft made that decision for all its users by including VeriSign's trusted root certificate in its product. This means Windows should also be obtaining VeriSign's CRLs and checking VeriSign's certificates against those CRLs. If it doesn't, then Windows users will unwittingly accept and trust certificates already revoked by VeriSign. Remember that revocations can occur for many reasons, including lost or compromised keys, violations of VeriSign policy, etc. and not just erroneously issued certificates. Also remember that VeriSign alone revokes VeriSign certificates, not Microsoft. While Microsoft can request that VeriSign revoke one or more of Microsoft's own certificates, Microsoft cannot request that anyone else's certificates be revoked. Well, it can, but VeriSign should not actually revoke any such certificate unless Microsoft presents very good justification to VeriSign that a significant enough infraction has actually occurred.
So exactly which method does VeriSign use to publish its CRLs? And exactly how does Windows obtain and use those VeriSign CRLs?
First, let's examine how VeriSign publishes its CRLs.
Does VeriSign use the RFC 2459 method? According to Microsoft, and we have no reason to doubt them on this point, VeriSign does not. The erroneously issued certificates definitely don't list any CRL locations (also called CRL Distribution Points or CDPs). You can even check this with any program that decodes and dumps X.509v3 certificates, so you don't have to take Microsoft's word for it.
Does VeriSign use the Root-Certificate method? No, there are no CRLs in VeriSign's Class 3 code-signing root certificate, either.
Are VeriSign's CRLs available from some well-known URL? Yes. While ordinary users may not consider this a "well-known" URL, developers of security products that accept VeriSign certificates should.
Are VeriSign's CRLs manually retrievable? Yes, you can download any of them. with any browser or other HTTP client.
Clearly, VeriSign is leaving it to the actual crypto-library developers to either retrieve a CRL from its well-known URL, or to somehow give users a means to manually install a VeriSign CRL after it's been manually downloaded. The crypto-library developers for Windows is Microsoft, who provides the CryptoAPI that all its code-signing security is built on. Yes, third parties can provide additional modules and plug-ins, but the entire basic capability is defined and provided by Microsoft.
Now let's look more closely at RFC 2459. Is VeriSign violating a standard by not providing a CRL location in every issued certificate, as RFC 2459 presumably states? Given the Microsoft rhetoric, the rather surprising answer is "No". In fact, RFC 2459 says that giving a CRL location within a certificate is optional. The exact relevant quote is this: The meaning of "non-critical" appears earlier in RFC 2459: In other words, the words "critical" and "non-critical" refer to how a certificate should be handled when a field within the certificate is actually marked "critical" in a specific certificate. not to whether a particular extension defined in RFC 2459 is required or optional. Criticality is a way to ensure that new extensions are treated with proper consequences when evaluated by old software that may not recognize the meaning or uses of the extension. Marking the new extension as critical ensures that old software will reject such certificates, thereby preserving the overall security and integrity of what the certificates are protecting.
The precise meaning of the capitalized words SHOULD, MUST, and OPTIONAL is given in RFC 2119: One may question whether VeriSign has "understood and carefully weighed" the full implications, but that's a completely different question than whether they followed RFC 2459 or not. The plain and simple fact is that VeriSign did not provide the CRL distribution point extension of RFC 2459, and it was within standardized practice to omit it. If Microsoft objected to this omission, it was entirely free to omit VeriSign's Class 3 code-signing root certificate from its product, and prevent all such certificates issued by VeriSign from being accepted. No one forced Microsoft to use VeriSign certificates lacking the CRL extension. Microsoft alone chose to do so.
Obtaining VeriSign CRLs
We've seen how VeriSign publishes its CRLs, and examined whether its certificates and CRLs are within RFC 2459's requirements. Now let's revisit the question of exactly how Windows obtains and uses VeriSign CRLs. In particular, does Microsoft's CryptoAPI support the Well-Known-URL method, or does it rely on the user to provide the Manual method? The rather astonishing conclusion one must arrive at is "Neither".
To be specific, Microsoft's CryptoAPI, as shipped by Microsoft, only handles CRLs when they are listed in certificates that have the CRL Distribution Point extension of RFC 2459.
At least this is the only conclusion one can draw from Microsoft's public responses to the incident. You can see this for yourself by looking at the Response to Inaccurate Crypto-Gram Article on VeriSign Certificates. searching for the string "2459", and reading the nearby "explanation": There is a way to revoke the certificates -- via the Certificate Revocation List (CRL) mechanism defined in the relevant industry standard, RFC 2459. In fact, Microsoft makes a big deal about RFC 2459 in its response, and how the certificates don't list a CRL location. It even goes so far as to include a screen shot showing that the certificates don't have the RFC 2459
CRL Distribution Point extension in them.
But no one ever said they did.
No one is saying that now.
This is all a red herring. Microsoft is simply evading the real question of how it could design and ship a revocation infrastructure relying entirely on a feature it must have known didn't exist in the VeriSign certificates it was accepting.
A similar "explanation" can be seen in Microsoft Security Bulletin MS01-017. after the first occurrence of the string "CRL": However, because VeriSign's code-signing certificates do not specify a CRL Distribution Point (CDP), it is not possible for any browser's CRL-checking mechanism to locate and use the VeriSign CRL. Microsoft has developed an update that rectifies this problem. The update package includes a CRL containing the two certificates, and an installable revocation handler that consults the CRL on the local machine, rather than attempting to use the CDP mechanism. Also, the bulletin ends with the assertion: Without this data, no system -- Microsoft or otherwise -- could obtain and check the CRL. Again, Microsoft is basically saying, "If only VeriSign certificates had conformed to RFC 2459 and included a CRL location (or CDP), everything would have worked fine". It's also saying that no other browser or security infrastructure can obtain or use a VeriSign CRL either.
On it's face, though, this new assertion is absurd. Any system -- even Microsoft's -- could have an internal Well-Known URL for VeriSign, from which it automatically obtains suitably-recent CRLs for any VeriSign certificates it cares to accept. As an unautomated alternative, any system -- even Microsoft's -- could have a simple Manual procedure for obtaining and using CRLs. Which systems actually do this is a different question entirely. Indeed, one system we certainly know cannot and does not: Microsoft's.But maybe a simple Manual process WILL work with MSIE. Here's what one user wrote to me about the process:
- Go to the VeriSign CRL site and click on the link named Class3SoftwarePublishers.crl to download the CRL.
- Go to Tools | Internet Options, then click on the Contents Tab.
- Click the Certificates button.
- Click Import, then use the Certificate Manager Import Wizard to import the CRL.
Now it's possible that I've done something wrong, but if I have, it's something that a very computer-literate person (though admittedly not a techie) managed to miss. Either way, something is very wrong.
So maybe Microsoft has a working Manual procedure and maybe they don't -- it seems to do something but we can't confirm it. And however we might characterize the process, it can hardly be called simple or obvious (unlike, say, downloading JPEG images or Windows media files). Complexity and obscurity are the enemies of effective security. If users can't confirm that the Manual process has worked and is effective, why should they trust it? And why should they consider it to be a working revocation infrastructure?
Whether Microsoft's assertion that "no system. could obtain and check the CRL" is valid in any practical sense or not, it is a hollow one. It is at best wishful thinking, since it ignores the concrete realities of VeriSign's actual certificates and CRLs. The really interesting thing is that VeriSign has been publishing CRLs under its current mechanisms FOR YEARS. indeed, for years before RFC 2459 was ever written. Either it was wasting time and money doing so, since those CRLs were never used by anyone, or someone somewhere managed to figure out a way to get the VeriSign CRLs and use them properly. That Microsoft was unable to do so is the real issue here.
We've already seen that the CRL extension in RFC 2459 is OPTIONAL. even though recommended. We've also seen that VeriSign does not issue Class 3 code-signing certificates containing CRL locations, and apparently never has. Whether it does in the future is irrelevant. There are countless Class 3 code-signing certificates already issued and deployed in all kinds of software. All that software will stop working unless Microsoft provides something like an automatic Well-Known-URL method as part of the way its revocation infrastructure handles VeriSign CRLs.
This leaves a really big unanswered question: Why would Microsoft base its entire CRL and revocation infrastructure on an optional feature that was absent from certificates issued by one of its principal certificate providers? At the very least, there are some gravely flawed design assumptions here, or there is a major breakdown in communication and requirements. Microsoft alone knows the real answer, and Microsoft alone should be held accountable.
Amazingly, this condition (unobtainable and/or unusable VeriSign CRLs) must have persisted for some time. That is, the revocation problem itself has existed since CryptoAPI first shipped. Windows has NEVER been able to obtain and use a VeriSign CRL, at least for any practical meaning of the words "obtain" or "use". If it had, then Microsoft would not have had to issue ANY patch or update. All it would have had to do was let the normal revocation infrastructure work in the normal way, and all Windows users would eventually have received VeriSign's normally published CRL using the normal revocation infrastructure. Even if Microsoft had a clearly working Manual method for obtaining CRLs, all it would have had to do was tell its customers to obtain a new VeriSign CRL. After all, that's the whole point of having an infrastructure in the first place -- to avoid continually reinventing wheels or blazing new trails through wilderness each and every time. Whatever else would have been necessary, it is certain that no uniquely issued and manually installed Microsoft patch or "update" would have been needed. The infrastructure would have worked exactly the way it was intended to work, whether that was automatically or manually.
Security is a chain. Break one link and the whole chain fails. The critical broken link in this incident was Microsoft's sole reliance on a feature that simply didn't appear in the certificates it was accepting. Microsoft says this still means it has a working revocation infrastructure. But if their principal provider of certificates, VeriSign, doesn't include the optional feature pivotal to their revocation infrastructure, then they are left with an utterly vacuous meaning for "working", much less for "revocation infrastructure".
So Microsoft has issued a patch, er, I mean "update", and now Windows programs can obtain and use future VeriSign CRLs automatically, and everything will be fine from here on out.
Or will it? Remember that CRLs are periodically reissued with new entries added and outdated ones removed. It is necessary to have a suitably-recent CRL. not just any old CRL from some past period. So how does Windows obtain and install one of the normally issued and suitably-recent VeriSign CRLs?
According to the Microsoft Security Bulletin MS01-017. the patch installs a "private CRL" containing only three revoked VeriSign certificates. Two of the revoked certificates are the bogus certificates. The third one is a test certificate that Microsoft specifically revoked so it could test its patch. The "private CRL" does not contain any of the other VeriSign revocations. Why not?
There also appears to be no mechanism in the patch that obtains, installs, or uses any CRL other than the "private CRL" installed by the patch. That "private CRL" is not even a complete VeriSign CRL listing all revoked Class 3 code-signing certificates at the time it was issued. The "private CRL" lists only those certificates that Microsoft deemed to be worthy of revocation. In short, from the moment the patch was issued, it WASN'T EVEN A SUITABLY-RECENT CRL since it didn't contain all the revoked certificates, only a Microsoft-selected subset. Why aren't all of VeriSign's other revoked certificates listed? Or has Microsoft determined that accepting any of those other revoked certificates poses no risk to users?
What's going on here? Could it be that Microsoft has simply created an expedient patch that doesn't really fix the larger CRL and revocation infrastructure problem, but merely connects into the CryptoAPI using a CRL plug-in mechanism? Could it be that future VeriSign-issued CRLs will continue to be unknown and unused by Microsoft, just as they always have been?
Or perhaps someday Microsoft will eventually issue a complete fix for VeriSign CRLs, that will always automatically obtain the actual VeriSign-published CRL and use it instead of some "private CRL" installed by a unique manual patch.
We can only hope.
But until that time comes, it seems fairly obvious to conclude that Microsoft does not have a working revocation infrastructure. At least not one that works for one of its principal suppliers of code-signing certificates, VeriSign.
On Certificate ExpirationMicrosoft's Response to Inaccurate Crypto-Gram Article on VeriSign Certificates contains another interesting "clarification", this time regarding how long the bogus certificates will last. Bruce Schneier wrote: Some news reports claim that the certificates will expire in a year, others claim that they will be good forever. Microsoft's response to this simple statement is: Fact: It's no mystery when the certificates will expire. The FAQ section of the bulletin provides screen shots of the certificates, and it can plainly be seen that one expires on 30 January 2002, and the other expires on 31 January 2002. This data is corroborated by information on the VeriSign web site. There are two points to rebut here:
- No one questioned what the expiration dates of the certificates are;
- Schneier was simply saying that "Some news reports. claim that they will be good forever". He was not saying whether these reports were correct or not.
The short answer that Schneier should have given is that some browsers are perfectly willing to accept expired certificates. In fact, it is not unknown for browsers to accept expired SSL certificates, perhaps with a warning dialog to the user, perhaps not. It is also common for older versions of browsers to actually be relying on expired root certificates. For example, some CA's had root certificates that expired on 31 Dec 1999. Those CA's had also issued countless SSL certificates that also expired on 31 Dec 1999. But on 01 Jan 2000, the world didn't come to a halt. All that happened is that some people visiting SSL sites were asked whether to continue trusting the expired certificates or not. Most people did. Were they complete fools to do so? Were they falling into a security hole? Were they exposing themselves to needless risk? Not necessarily.
Quoting from Schneier's book "Secrets and Lies" (Wiley, 2000), where he discusses the expiration of real-world credentials like passports, credit cards, and driver's licenses: Expirations provide a safety net. A bad credential can be out there for only so long, because it will expire eventually. (p. 229) But as noted above, and also described at more length in "Secrets and Lies", the credentials we use in the digital world (i.e. certificates) are, in practice, used in ways very different from the ways we use real-world credentials. For one thing, expiration dates on real-world credentials are usually enforced much more strictly. That's not saying that laxity in enforcing digital certificate expiration dates is a good thing, just that it happens regularly and the digital world hasn't come crashing down as a result.
In any case, certificate expirations only provide a safety net. The situation for signed code is not substantially different. If you trusted a code-signing certificate when you first received the code, you really don't need to stop trusting it just because the certificate expires. You decided to trust the certificate at the time you obtained the code, so what changed? Time may have marched on in its petty pace from day to day, but you have exactly the same certificate and the same code, so the trustworthiness of both are no different. What difference does the certificate's expiration make to you? Has your trust of the code's signer or the CA changed in any way between the last day before the certificate expires and the first day after it expires?
As far as trustworthiness is concerned, the critical point in time was when you first obtained and executed the signed code, not all the subsequent times you executed it or checked the certificate. If you were justified in deciding to trust the code the first time you ran it, then you were equally justified in trusting it all the other times. Put in the opposite way, if your trust was misplaced the first time you ran the code, then it was equally misplaced all the other times as well. Nothing else about the certificate or the code changes either result, because the fundamental decision has already been made. The basic relationship -- trust the code or not -- has not changed no matter how many other times you evaluate the same certificate and run the same code. Expiration does not change this relationship.
Note that you CANNOT apply the same logic to revocation. That is, if a certificate is revoked after you've checked it and chosen to trust it, that certificate's revocation should make you re-evaluate your trust decision. In particular, you may want to stop trusting the certificate, depending on the reason for the revocation. For example, if the certificate was revoked because the keys were compromised, then some imposter could have those keys and be able to fool you into trusting some new malicious code. Or maybe the person controlling the certificate's keys has left the company represented by the certificate, and the company revokes the old certificate and has a new certificate issued with new keys. In both cases, something important about the relationship changed, and it could be very risky to continue with the same trust relationship as before the revocation occurred.
Thus, we can see that having a working revocation infrastructure is just as important as having certificate expiration dates. You definitely do not want to be left without a revocation infrastructure when it comes to code-signing. Revocation may even be MORE important than expiration, though I would not want to be without expiration. Doing so would just make certificate management harder.
Certificate expiration is simply the passing of an arbitrary deadline imposed on the certificate by the issuing CA, which may have at least as much to do with a revenue stream for the CA as with any fundamental question of trustworthiness. This is not to denigrate CAs. They take on important risks and perform valuable services. Their business is trust. But it's always worthwhile to understand the relationships as well as the trustworthiness of any business you rely upon. At the very least, it helps you to see latent conflicts of interest.
- Microsoft says it supports CRLs identified by the CRL Distribution Point field of a certificate defined according to RFC 2459. Windows ships with a significant number of built-in trusted root CA certificates. There may even be updates that include additional built-in trusted root certificates. Of all these built-in trusted root certificates, which CAs provide the CRL Distribution Point feature in their code-signing certificates? Any prudent developer aware of this issue would want to be sure that any code-signing certificate they bought is protected by a CRL that's automatically obtained and used by all Windows machines. So please list the built-in trusted CAs that meet the criterion.