Robby Pedrica's Tech Blog

The double-edged sword of ECH

—

by

Robby Pedrica
in ,

Definition: ECH: Encrypted Client Hello

  • a newly proposed standard (ie. draft)
  • prevents networks from snooping on which websites a user is visiting

Encrypted Client Hello (ECH) is a successor to ESNI and masks the Server Name Indication (SNI) that is used to negotiate a TLS handshake. This means that whenever a user visits a website that has ECH enabled, no one except for the user, and the website owner can determine which website was visited.

In simple terms, no 3rd party will be able to see the address that you’re going to. ECH is an extension on the TLS protocol, and the idea with this is to provide additional privacy for encrypted traffic.

A quick graphic on the flow of an ECH request:

https://blog.cloudflare.com/announcing-encrypted-client-hello

All good and well. We like privacy. For private users.

But hang on a moment

Owners of corporate networks (eg. SMB, enterprise, corporate, etc.) have a right and duty to see what’s happening on their networks, and along with policies, have used a number of tools to enable interception and content checking of the traffic on their networks. There is a huge security blackhole with encrypted traffic – malicious attack, data exfiltration and compromise are the potential results of not being able to monitor and check network traffic, so this aspect of network security is vitally important, if not absolutely critical for the protection of corporate networks and assets.

MiTM (man-in-the-middle) or DPI (deep packet inspection) gateways are the standard ways of doing this kind of monitoring. Network users generally have no expectation of privacy (when enforced with corporate network access policies) so that is not an issue.

What is an issue is if there is something in place which could block or obscure the ability to check your own network traffic.

ECH is one of these; by hiding the target of a secure encrypted request, you could potentially block network owners from seeing what users on their network are doing.

DoH/DoT/QUIC

ECH is not the only new protocol to be released recently that could cause monitoring issues.

QUIC

QUIC (or HTTP/3 as its final published derivative is called) is a protocol developed by Google as early as 2012, and submitted as a draft recommendation to the IETF in 2013.

QUIC works hand-in-hand with HTTP/3‘s multiplexed connections, allowing multiple streams of data to reach all the endpoints independently, and hence independent of packet losses involving other streams.

QUIC’s secondary goals include reduced connection and transport latency, and bandwidth estimation in each direction to avoid congestion.

Essentially QUIC is a performance-enhancing and traffic optimisation tool.

But MiTM/DPI gateways do not play well with QUIC, so the only option is to use DPI to identify QUIC traffic and block it. If a browser attempts a QUIC connection and is blocked, it will essentially downgrade the request to http/2 or http/1.1 and go on with its business – a neat trick.

DoT

DNS over TLS is an extension to TLS that allows for DNS queries to be TLS encrypted between the client and the DNS server (using TCP port 853). The idea is to provide privacy for your DNS queries. And yes, privacy is very important in certain cases.

Not in the case of corporate networks though, just as with QUIC above.

This is a simple fix – just block TCP port 853 in your gateways. Clients will automatically attempt the DNS query then using standard non-encrypted DNS and your gateways now have access to those queries.

DoH

As with QUIC, DoH is a little more tricky to deal with seeing as it uses SSL/TLS encrypted HTTPS to carry the DNS query. It’s normally impossible to distinguish DoH traffic from any other HTTPS-encrypted traffic. Because it’s encrypted.

But just as with QUIC, MiTM/DPI can be used to unpack the encrypted traffic where now the DoH traffic is easily identified, and blocked.

Again the client will downgrade to a standard non-encrypted DNS query to move forward.

I’ll have a draft, thank you

Didn’t you say that ECH was in draft mode? Yes I did.

But that hasn’t stopped CloudFlare from moving ahead and enabling this on their ENTIRE web hosting estate in the last few weeks. And causing chaos.

There has been no public notice of this change by CF except for the following in their developers documentation:

Starting in August, 2024, ECH will be gradually released on free zones. It will not be possible to disable it. A toggle will be added to the Cloudflare Dashboard at a later point before ECH is made available for other zone plans.

https://developers.cloudflare.com/ssl/edge-certificates/ech/

Normally, browsers require a user to have DoH enabled to facilitate the ECH capability however:

  • Apparently Firefox 130 does not require DNS-over-HTTPS for ECH to work
  • Cloudflare uses a single ECH key & configuration for all the domains
  • cloudflare-ech.com is used as a canary domain (SNI) for TLS requests

The above means that for public/non-corp internet users, ECH will most likely work with CloudFlare-hosted websites (and other service providers that implement ECH) without any special requirements on the end-user’s side.

For corporates however, this is a minefield.

ECH-enabled sites will either be blocked by default causing website failures for end-users, or these sites will be allowed and corps will not have insight into that traffic.

So, what to do

There are a number of options for corps, none of which are quite ideal but …

  1. allow cloudflare-ech.com (and possibly DoH as well) – this will allow ECH-enabled sites to work but blind corps to the related traffic
  2. ask the website owner to disable TLS 1.3 or ECH support – very little chance of this happening
  3. enable QUIC in your DPI gateways – makes corps blind to the related traffic
  4. disable TLS 1.3 Kyber support in the browser (eg. Chrome) – this requires either a GPO or manual intervention and can be a big admin overhead
  5. allow open access to the site – which once again makes you blind to the related traffic

Thankfully firewall (and other gateway) vendors are starting to add varying support for dealing with ECH.

As an example, Fortinet (with FortiOS 7.4 and greater) includes the ability to block ECH directly in standard certificate inspection profiles, automatically forcing browsers to downgrade/not use ECH and keeping the ability to inspect traffic.

Other vendors are following suit.

Et tu, Brute?

Sometimes, some disruption is acceptable and inevitable to bring about a sea change in a particular area. In this case however, I think that disruption is too much, and CloudFlare (CF) has not thought this through properly.

CF had already disabled ECH in 2023 due to implementation issues. Let’s hope they rethink their plans for ECH, disable it again and engage with the industry to look at options for enabling ECH privacy while simultaneously allowing corporates the critical function of monitoring their networks.

Links

https://blog.cloudflare.com/announcing-encrypted-client-hello

https://developers.cloudflare.com/ssl/edge-certificates/ech

https://support.mozilla.org/en-US/kb/faq-encrypted-client-hello