👁 homepage

Python's ssl.match_hostname() does sub string matching as specified in RFC 2818:

 Names may contain the wildcard
 character * which is considered to match any single domain name
 component or component fragment. E.g., *.a.com matches foo.a.com but
 not bar.foo.a.com. f*.com matches foo.com but not bar.com.

The RFC doesn't specify how internationalized domain names shoould be handled because it predates RFC 5890 for IDNA by many year. IDNA are prefixed with "xn--", e.g. u"götter.example.de".encode("idna") == 
"xn--gtter-jua.example.de". This can result into false positive matches for a rule like "x*.example.de".

Chrome has special handling for IDN prefix in X509Certificate::VerifyHostname()
http://src.chromium.org/viewvc/chrome/trunk/src/net/cert/x509_certificate.cc

Also see #17980

Actually, I don't this is a bug: match_hostname() expects str data, and therefore IDNA-decoded domain names:

>>> b"xn--gtter-jua.example.de".decode("idna")
'götter.example.de'

Doing the matching on the decoded domain name should be safe.
Then it very much depends on whether the data you've got was IDNA-decoded, or naïvely ASCII-decoded, and I don't think the Python stdlib is very consistent here. Looking at the socket module, gethostbyaddr and getnameinfo seem to use ASCII decoding...

It's called "internationalized domain name for APPLICATIONS". ;) It's up to the application to interpret the ASCII text as IDNA encoded FQDNs. As far as I know DNS, SSL's CNAME and OS interfaces etc. always use ASCII labels. It's an elegant solution. Just the UI part of an application needs to understand IDNA. 

http://tools.ietf.org/html/rfc6125#section-6.4.2

 If the DNS domain name portion of a reference identifier is an
 internationalized domain name, then an implementation MUST convert
 any U-labels [IDNA-DEFS] in the domain name to A-labels before
 checking the domain name. In accordance with [IDNA-PROTO], A-labels
 MUST be compared as case-insensitive ASCII. Each label MUST match in
 order for the domain names to be considered to match, except as
 supplemented by the rule about checking of wildcard labels
 (Section 6.4.3; but see also Section 7.2 regarding wildcards in
 internationalized domain names).


Coincidentally the same RFC contains matching rules for wild card certs 
http://tools.ietf.org/html/rfc6125#section-6.4.3

 If a client matches the reference identifier against a presented
 identifier whose DNS domain name portion contains the wildcard
 character '*', the following rules apply:

 1. The client SHOULD NOT attempt to match a presented identifier in
 which the wildcard character comprises a label other than the
 left-most label (e.g., do not match bar.*.example.net).

 2. If the wildcard character is the only character of the left-most
 label in the presented identifier, the client SHOULD NOT compare
 against anything but the left-most label of the reference
 identifier (e.g., *.example.com would match foo.example.com but
 not bar.foo.example.com or example.com).

 3. The client MAY match a presented identifier in which the wildcard
 character is not the only character of the label (e.g.,
 baz*.example.net and *baz.example.net and b*z.example.net would
 be taken to match baz1.example.net and foobaz.example.net and
 buzz.example.net, respectively). However, the client SHOULD NOT
 attempt to match a presented identifier where the wildcard
 character is embedded within an A-label or U-label [IDNA-DEFS] of
 an internationalized domain name [IDNA-PROTO].

> It's called "internationalized domain name for APPLICATIONS". ;) It's 
> up to the application to interpret the ASCII text as IDNA encoded
> FQDNs. As far as I know DNS, SSL's CNAME and OS interfaces etc. always 
> use ASCII labels. It's an elegant solution. Just the UI part of an
> application needs to understand IDNA. 

The socket module already decodes to/encodes from IDNA in places (e.g. gethostname()). We need a consistent policy in the stdlib; I would like Martin's advice on this.

I finally found the correct RFC for wildcard matching. I think our implementation violates some recommendations.

http://tools.ietf.org/html/rfc6125#section-6.4.2
http://tools.ietf.org/html/rfc6125#section-6.4.3

As a policy, the standard library should accept non-ASCII host names ("U-labels") wherever possible. I.e the hostname parameter of match_hostname should allow for U-labels (as well as A-labels).

When returning names, it should always return the data "as-is", which typically means A-labels. Anybody wanting to display U-labels will need to decode them explicitly.

I believe that the matching of IDNA names doesn't currently happen according to 6.4.2 of RFC 6125, however, this is not actually the issue that Christian reported (which was only about wildcard matching). I suggest to create a separate issue for that.

As for 6.4.3: I find the text to be quite ill-formulated. Specifically, I'm referring to the sentence

 However, the client SHOULD NOT
 attempt to match a presented identifier where the wildcard
 character is embedded within an A-label or U-label [IDNA-DEFS] of
 an internationalized domain name [IDNA-PROTO].

First, in the context of X.509, a wildcard *cannot* be embedded "with an ... U-label"; the certificate can only possibly contain A-labels (because the datatype of dNSName is IA5String).

Second, as written, it *does* allow to match 'götter.example.de' against "x*.example.de", since "x*.example.de" is not an A-label. An A-label is defined as

 An "A-label" is the ASCII-Compatible Encoding (ACE, see
 Section 2.3.2.5) form of an IDNA-valid string. It must be a
 complete label: IDNA is defined for labels, not for parts of them
 and not for complete domain names. This means, by definition,
 that every A-label will begin with the IDNA ACE prefix, "xn--"
 (see Section 2.3.2.5), followed by a string that is a valid output
 of the Punycode algorithm [RFC3492] and hence a maximum of 59
 ASCII characters in length. The prefix and string together must
 conform to all requirements for a label that can be stored in the
 DNS including conformance to the rules for LDH labels
 (Section 2.3.1). If and only if a string meeting the above
 requirements can be decoded into a U-label is it an A-label.

Since an A-label is required to conform to the LDH label syntax, it cannot possibly contain the asterisk (LDH labels can only contain letters, digits, and the hyphen. Hence, the entire requirement is irrelevant (as literally written). They might mean something else, but I cannot guess what it is that they mean.

I disagree with the classification of this issue as critical. It does not involve a crash, a serious regression, or a breakage of a very important API.

Ryan Sleevi of the Google Chrome Security Team has informed us about another issue that is caused by our failure to implement RFC 6125 wildcard matching rules. RFC 6125 allows only one wildcard in the left-most fragment of a hostname. For security reasons matching rules like *.*.com should be not supported.

For wildcards in internationalized domain names I have followed 	the piece of advice "In the face of ambiguity, refuse the temptation to guess.". A substring wildcard does no longer match an IDN A-label fragment. '*' still matches a full punycode fragment but 'x*' no longer matches 'xn--foo'. I copied the idea from Chrome's matching code:

http://src.chromium.org/viewvc/chrome/trunk/src/net/cert/x509_certificate.cc?revision=212341#l640

 // * must not match a substring of an IDN A label; just a whole fragment.
 if (reference_host.starts_with("xn--") &&
 !(pattern_begin.empty() && pattern_end.empty()))
 continue;

The relevant RFC section for the patch are

 http://tools.ietf.org/html/rfc6125#section-6.4.3
 http://tools.ietf.org/html/rfc2818#section-3.1
 http://tools.ietf.org/html/rfc2459#section-4.2.1.7
 http://tools.ietf.org/html/rfc5280#section-7

Affected versions:

- Python 3.2 (< 3.2.5)
- Python 3.3 (< 3.3.3)
- Python 3.4a1
- requests < 1.2.3
https://pypi.python.org/pypi/requests
- backports.ssl_match_hostname (<3.2a3)
https://pypi.python.org/pypi/backports.ssl_match_hostname/
- urllib3 < 1.6
https://github.com/shazow/urllib3
- bzr
- setuptools
- tornado
- pip

So, is this a security issue? I've been wondering if I should apply the attached patch to the backports-ssl_match_hostname module on pypi. I was hoping there'd be some information here as to whether this will be going into the stdlib in the future.

Thus far, ssl_match_hostname has just been a backport of the match_hostname function but if this is a security problem, I could press for us to diverge from the python3 stdlib. It would be easier to make the case if this is seen as a critical problem that will need to be fixed even if the current patch might not be the eventual fix.

Yes, it's a security issue. But the patch would changes the behavior of the function. The current function conforms to RFC 2818. The patch implements RFC 6125, which is more restrictive.

New changeset 10d0edadbcdd by Georg Brandl in branch '3.3':
Issue #17997: Change behavior of ``ssl.match_hostname()`` to follow RFC 6125,
http://hg.python.org/cpython/rev/10d0edadbcdd

Also merged to default.

Python 3.2 hasn't been fixed yet.

Should acquire a CVE for the issue?

Just to clarify the status of this issue: it *only* blocks 3.2.

For future reference how do I find out if this has been applied to 3.2?

Since it's been out in 3.2.x for so long, I won't apply this for 3.2 since at this point a behavior change might do more harm than good.