Netflix, have you heard about RFC 952?

Having had a firewall melt-down last week (the subject of another post, when I get around to it), I rebuilt our entire firewall ruleset from scratch, because the config backups simply brought back the undesired effects (clearly something in the 45,000 lines of config disagreed with it...).

As a result, I've ended up "experimenting" with some options, with sometimes unintended, or non-obvious effects.

Netflix. It's broken. Oh noes!

One early casualty was Netflix - one of the few video streaming things we allow (because: bandwidth [available on plentiful and cheap national traffic] and legality [has a legitimate presence in South Africa]). Children love copyright infringing content; I try to make it easy to access legal stuff instead.



As an interim "hack", I put in a custom rule that just allowed all outgoing HTTPS traffic from client networks to two /24 netblocks which were clearly part of Netflix's ZA CDN architecture. This worked, but made me uncomfortable, so I spent some time digging deeper into this. Despite trying every conceivable combination of allowing various Netflix services through the various options open to trial in the Fortigate ecosystem, Netflix just kept giving me errors (until I put the dirty hack in place).

Whoops, something went wrong...
Internet Connection Problem
There was an unexpected error. Please reload the page and try again.
Error Code: M7031-1101

Turning to the firewall logs, one could see matching deny results.

"The Server Name Indication for the HTTPS session contained an invalid domain name."
is not exactly a very helpful error message, until you've understood what this actually means.

It turns out that the somewhat cryptic errors in the block message within the Fortigate logs actually ultimately mean "Netflix don't read RFCs".

Enabling "Block Invalid URLs" in Security Profiles> Web Filter>(some profile) under Static URL filter seems innocent enough - surely an invalid URL is a bad thing? It turns out that invalid URLs are somewhat popular on the Internet, so you possibly don't want this.

Be careful what you wish for...

Netflix just happened to be the first affected service that anyone complained about.

The root problem is that RFC952 only allows a very restricted set of characters - which notably excludes the underscore ( _ ) - which if you've ever done much tracerouting, you've probably seen a lot of; not all people read RFCs, which is a problem. I can't say that before today, I read either of the applicable RFCs, but you'd expect an internet giant to do so.

It's become common practice in describing certain services (e.g. _service._tcp.tld is quite familiar to many) - however, that does not violate the RFCs, because those aren't hostnames (see for example RFC2181 - DNS by design is not only about hostnames). Netflix, however, makes use of underscores in hostnames, for example in:

ipv4_1.lagg0.c006.jnb001.ix.nflxvideo.net. 

tsk tsk. So, when you name your hosts, avoid underscores!

The updated RFC1123 doesn't change this restriction on the valid character set in hostnames (although it does allow for longer names and relaxes the conditions on the first character) - neither do subsequent RFCs, from what I can see. This means that technically any hostname with an underscore in it is invalid, and should (strictly) be ignored.

The distinction between valid DNS records (which some people might conflate with "domain names") and hostnames seems initially subtle, but it is an important one.

Take home message: DNS records allow underscores; however, a hostname record MUST NOT contain underscores.

At least I know why Netflix is broken, and what I can do about it - other than using dirty hack workarounds.

There is some interesting discussion of this valid character problem on stackoverflow.