Issue 5323075: Script to generate CAs
|
Created:
12 years, 5 months ago by jaraco Modified:
12 years, 1 month ago CC:
httplib2-dev_googlegroups.com Visibility:
Public. |
Patch Set 1 #MessagesTotal messages: 5
This patch addresses issue170 in the issue tracker. It also fixes 157 and 156.
Sign in to reply to this message.
On 2011/11/03 20:49:06, jaraco wrote: > This patch addresses issue170 in the issue tracker. It also fixes 157 and 156. Hi jaraco, There are a couple of concerns regarding an approach that bundles a large root CA file with httplib2, as well as with the patch as is in particular: * As a practical matter, this approach might have significant performance issues. The python 2.x SSL API is severely lacking, and one cannot create and get a handle for a reusable SSL context (in 3.x this is available afaik). Which means that openssl will read in and parse the provided CA file on *every connection*, which is a good reason to keep the CA file small. * The mozilla source contains certs that are marked as untrusted; IIUC these are used to black-list CAs in mozilla. The import script would have to skip those. Which is what for example, debian/ubuntu does (see scripts in the source of the ca-certificates package). It looks like that in addition, debian explicitly blacklists/excludes some CAs. * In many uses, pulling in all these roots adds unnecessary risk exposure due to potentially compromised CAs. In particular since there is no automated update mechanism which would propagate the deletion or blacklisting of a compromised CA in the Mozilla source. There have been a number of events in recent past where CAs have been black-listed (see, e.g., http://en.wikipedia.org/wiki/DigiNotar#Issuance_of_fraudulent_certificates), so this is not a theoretical concern. After conferring with jcgregorio, we'd like to propose the following approach: * By default, install with the existing, minimal root CA cert. * On systems (e.g. unix) where a system provided CA cert file is available, provide an install-time option to instead use a symlink to (not a copy of) this file. Since the CA certs file will be updated by the distribution, httplib won't be responsible for updates. As far as security considerations are concerned, it would be reasonable to use the symlink by default; however the performance concern still stands. Cert file locations include (from http://weekly.golang.org/src/pkg/crypto/tls/root_unix.go): "/etc/ssl/certs/ca-certificates.crt", // Debian Linux "/etc/pki/tls/certs/ca-bundle.crt", // Fedora/RHEL "/etc/ssl/ca-bundle.pem", // OpenSUSE * Otherwise, provide an option to fetch and parse the mozilla cert source at install time, and print a warning that the user will be responsible for tracking changes to the mozilla source, and re-installing if necessary. The parsing script would have to be modified to take untrusted/blacklisted certs into account. If its license permits, perhaps the debian script could be used. Does this sound workable? Thanks, --xtof
Sign in to reply to this message.
Thanks xtof for the thoughtful reply. I see now that it's not just a technical matter of including the appropriate certs. By reading the source code history and bug reports, my impression was that CAs were being added incrementally as they were needed, which seemed like a terrible approach. Furthermore, the CA file that was included suggested that it was generated from the Mozilla source, even though it clearly did not have all of the certificates. I agree that parsing the Mozilla source at any time is probably undesirable, if it does indeed include certificates which are known to not be trustworthy. However, I believe there are some issues with the proposed approach as well. The proposal is to keep a minimal subset of trusted certificate authorities bundled with httplib2, but it doesn't define which authorities or how an authority is included or excluded from the list. It would be wise to have a standard metric which an authority must meet to be included, rather than deferring to the potentially biased whimsy of some developers. Otherwise, it becomes unclear which authorities should or shouldn't be included. Furthermore, technical limitations of the underlying libraries should not be a factor affecting inclusion or exclusion of an authority's certificate. If there are performance limitations, those should be addressed directly, not by excluding legitimate authorities. Additionally, it seems unreasonable to exclude certificate authorities based on the premise that they might at some point become compromised. Each of these authorities has undergone the same verification process through the Mozilla organization. I don't think it's in the best interest of the httplib2 project to second-guess those audits and instead establish their own arbitrary criteria for a trustable authority. If httplib2 is going to be in the business of verifying the validity of certificates by default (and failing if they fail to validate), then in my opinion, it should be able to effectively validate most certificates issued by legitimate, public CAs. While you suggest that adding additional CAs adds unnecessary exposure, omitting those CAs has the potential of adding even greater exposure. If the library cannot cover a similar security space as common browsers, then consumers of the application will find themselves forced for practical reasons to disable certificate validation, in which case httplib2 has encouraged diminished security. I suggest that httplib2 implement a fair and objective measure for inclusion of CAs bundled with the library (whether this be all, none, some objectively-determined subset, or an industry-standard such as Mozilla's set), but then implement a solution similar to the one you described: 1) httplib2 first attempts to load trusted certificates from the system. I don't believe there's any need to copy/install/symlink the certs; just load them from the system on demand. Avoiding a post-install step dramatically increases the simplicity of packaging. This approach might work well with the Windows approach where system certificates are stored in the registry. 2) If system certificates are not available (or are disabled by a user preference), certificates will be loaded from the library. 3) (optional) The application could have the option to downselect a subset of the certificates to be used in validation. This option could be used by a security-sensitive application to restrict the available certificates or to improve performance (if necessary). This approach has several advantages. 1) It prefers system certificates, which defers the security and updates to the user's system, which is much more likely to be regularly updated and gets updates without changes to httplib2. 2) It provides a decent fallback which can provide a best-effort verification of certificates expected to be found in the wild. 3) It addresses the needs of the bugs as filed in the issue tracker. 4) It does not add post-install steps or other user configuration for the vast majority of deployments. 5) It better leverages the capability already present in the community/industry. 6) It provides a fair and equitable metric for CA inclusion. 7) It is more platform friendly. 8) It puts a priority of security and capability and defers optimization. I look forward to your thoughts on my suggestions. Regards, Jason -----Original Message----- From: xtof@google.com [mailto:xtof@google.com] Sent: Wednesday, 16 November, 2011 18:39 To: Jason R. Coombs; joe.gregorio@gmail.com Cc: httplib2-dev@googlegroups.com; reply@codereview.appspotmail.com Subject: Re: Script to generate CAs (issue 5323075) On 2011/11/03 20:49:06, jaraco wrote: > This patch addresses issue170 in the issue tracker. It also fixes 157 and 156. Hi jaraco, There are a couple of concerns regarding an approach that bundles a large root CA file with httplib2, as well as with the patch as is in particular: * As a practical matter, this approach might have significant performance issues. The python 2.x SSL API is severely lacking, and one cannot create and get a handle for a reusable SSL context (in 3.x this is available afaik). Which means that openssl will read in and parse the provided CA file on *every connection*, which is a good reason to keep the CA file small. * The mozilla source contains certs that are marked as untrusted; IIUC these are used to black-list CAs in mozilla. The import script would have to skip those. Which is what for example, debian/ubuntu does (see scripts in the source of the ca-certificates package). It looks like that in addition, debian explicitly blacklists/excludes some CAs. * In many uses, pulling in all these roots adds unnecessary risk exposure due to potentially compromised CAs. In particular since there is no automated update mechanism which would propagate the deletion or blacklisting of a compromised CA in the Mozilla source. There have been a number of events in recent past where CAs have been black-listed (see, e.g., http://en.wikipedia.org/wiki/DigiNotar#Issuance_of_fraudulent_certificates), so this is not a theoretical concern. After conferring with jcgregorio, we'd like to propose the following approach: * By default, install with the existing, minimal root CA cert. * On systems (e.g. unix) where a system provided CA cert file is available, provide an install-time option to instead use a symlink to (not a copy of) this file. Since the CA certs file will be updated by the distribution, httplib won't be responsible for updates. As far as security considerations are concerned, it would be reasonable to use the symlink by default; however the performance concern still stands. Cert file locations include (from http://weekly.golang.org/src/pkg/crypto/tls/root_unix.go): "/etc/ssl/certs/ca-certificates.crt", // Debian Linux "/etc/pki/tls/certs/ca-bundle.crt", // Fedora/RHEL "/etc/ssl/ca-bundle.pem", // OpenSUSE * Otherwise, provide an option to fetch and parse the mozilla cert source at install time, and print a warning that the user will be responsible for tracking changes to the mozilla source, and re-installing if necessary. The parsing script would have to be modified to take untrusted/blacklisted certs into account. If its license permits, perhaps the debian script could be used. Does this sound workable? Thanks, --xtof http://codereview.appspot.com/5323075/
Sign in to reply to this message.
I'm going to tentatively agree with the approach, but it depends on how many platforms for which we can consistently find and use local CA files on. If we can cover Windows, Mac and Ubuntu then I'll be fine with it, if we can't then I'd like to have the fallback position be the small CA file we ship today as opposed to leaving then with an empty CA file. On Wed, Nov 30, 2011 at 11:37 PM, Jason R. Coombs <jaraco@jaraco.com> wrote: > Thanks xtof for the thoughtful reply. > > I see now that it's not just a technical matter of including the appropriate > certs. By reading the source code history and bug reports, my impression was > that CAs were being added incrementally as they were needed, which seemed > like a terrible approach. Furthermore, the CA file that was included > suggested that it was generated from the Mozilla source, even though it > clearly did not have all of the certificates. > > I agree that parsing the Mozilla source at any time is probably undesirable, > if it does indeed include certificates which are known to not be > trustworthy. > > However, I believe there are some issues with the proposed approach as well. > > The proposal is to keep a minimal subset of trusted certificate authorities > bundled with httplib2, but it doesn't define which authorities or how an > authority is included or excluded from the list. It would be wise to have a > standard metric which an authority must meet to be included, rather than > deferring to the potentially biased whimsy of some developers. Otherwise, it > becomes unclear which authorities should or shouldn't be included. > > Furthermore, technical limitations of the underlying libraries should not be > a factor affecting inclusion or exclusion of an authority's certificate. If > there are performance limitations, those should be addressed directly, not > by excluding legitimate authorities. > > Additionally, it seems unreasonable to exclude certificate authorities based > on the premise that they might at some point become compromised. Each of > these authorities has undergone the same verification process through the > Mozilla organization. I don't think it's in the best interest of the > httplib2 project to second-guess those audits and instead establish their > own arbitrary criteria for a trustable authority. > > If httplib2 is going to be in the business of verifying the validity of > certificates by default (and failing if they fail to validate), then in my > opinion, it should be able to effectively validate most certificates issued > by legitimate, public CAs. > > While you suggest that adding additional CAs adds unnecessary exposure, > omitting those CAs has the potential of adding even greater exposure. If the > library cannot cover a similar security space as common browsers, then > consumers of the application will find themselves forced for practical > reasons to disable certificate validation, in which case httplib2 has > encouraged diminished security. > > I suggest that httplib2 implement a fair and objective measure for inclusion > of CAs bundled with the library (whether this be all, none, some > objectively-determined subset, or an industry-standard such as Mozilla's > set), but then implement a solution similar to the one you described: > > 1) httplib2 first attempts to load trusted certificates from the system. I > don't believe there's any need to copy/install/symlink the certs; just load > them from the system on demand. Avoiding a post-install step dramatically > increases the simplicity of packaging. This approach might work well with > the Windows approach where system certificates are stored in the registry. > 2) If system certificates are not available (or are disabled by a user > preference), certificates will be loaded from the library. > 3) (optional) The application could have the option to downselect a subset > of the certificates to be used in validation. This option could be used by a > security-sensitive application to restrict the available certificates or to > improve performance (if necessary). Here I would say that #3 should be #1 and that it is not an optional feature, that a user must be able to pass in a cert filename to be used instead of the system certs. > > This approach has several advantages. > > 1) It prefers system certificates, which defers the security and updates to > the user's system, which is much more likely to be regularly updated and > gets updates without changes to httplib2. > 2) It provides a decent fallback which can provide a best-effort > verification of certificates expected to be found in the wild. > 3) It addresses the needs of the bugs as filed in the issue tracker. > 4) It does not add post-install steps or other user configuration for the > vast majority of deployments. > 5) It better leverages the capability already present in the > community/industry. > 6) It provides a fair and equitable metric for CA inclusion. > 7) It is more platform friendly. > 8) It puts a priority of security and capability and defers optimization. > > I look forward to your thoughts on my suggestions. > > Regards, > Jason > > -----Original Message----- > From: xtof@google.com [mailto:xtof@google.com] > Sent: Wednesday, 16 November, 2011 18:39 > To: Jason R. Coombs; joe.gregorio@gmail.com > Cc: httplib2-dev@googlegroups.com; reply@codereview.appspotmail.com > Subject: Re: Script to generate CAs (issue 5323075) > > On 2011/11/03 20:49:06, jaraco wrote: >> This patch addresses issue170 in the issue tracker. It also fixes 157 > and 156. > > Hi jaraco, > > There are a couple of concerns regarding an approach that bundles a large > root CA file with httplib2, as well as with the patch as is in > particular: > > * As a practical matter, this approach might have significant performance > issues. The python 2.x SSL API is severely lacking, and one cannot create > and get a handle for a reusable SSL context (in 3.x this is available > afaik). Which means that openssl will read in and parse the provided CA file > on *every connection*, which is a good reason to keep the CA file small. > > * The mozilla source contains certs that are marked as untrusted; IIUC these > are used to black-list CAs in mozilla. The import script would have to skip > those. Which is what for example, debian/ubuntu does (see scripts in the > source of the ca-certificates package). It looks like that in addition, > debian explicitly blacklists/excludes some CAs. > > * In many uses, pulling in all these roots adds unnecessary risk exposure > due to potentially compromised CAs. In particular since there is no > automated update mechanism which would propagate the deletion or > blacklisting of a compromised CA in the Mozilla source. There have been a > number of events in recent past where CAs have been black-listed (see, e.g., > http://en.wikipedia.org/wiki/DigiNotar#Issuance_of_fraudulent_certificates), > so this is not a theoretical concern. > > > After conferring with jcgregorio, we'd like to propose the following > approach: > > * By default, install with the existing, minimal root CA cert. > > * On systems (e.g. unix) where a system provided CA cert file is available, > provide an install-time option to instead use a symlink to (not a copy of) > this file. Since the CA certs file will be updated by the distribution, > httplib won't be responsible for updates. As far as security considerations > are concerned, it would be reasonable to use the symlink by default; however > the performance concern still stands. > Cert file locations include (from > http://weekly.golang.org/src/pkg/crypto/tls/root_unix.go): > "/etc/ssl/certs/ca-certificates.crt", // Debian Linux > "/etc/pki/tls/certs/ca-bundle.crt", // Fedora/RHEL > "/etc/ssl/ca-bundle.pem", // OpenSUSE > > * Otherwise, provide an option to fetch and parse the mozilla cert source at > install time, and print a warning that the user will be responsible for > tracking changes to the mozilla source, and re-installing if necessary. The > parsing script would have to be modified to take untrusted/blacklisted certs > into account. If its license permits, perhaps the debian script could be > used. > > > Does this sound workable? > > Thanks, > --xtof > > http://codereview.appspot.com/5323075/ > -- Joe Gregorio http://bitworking.org
Sign in to reply to this message.
I was made aware today of the Requests library support for certificate verification using the Mozilla certificate set via 'certifi'. I have yet to investigate if they do anything to downselect or otherwise customize the trusted authority set.
Sign in to reply to this message.
|
