TLS/ESNI/ECH/DoT/DoH/JA3 – Talking about the last hole in the security of browsing experience under HTTPS network

Original link: https://hijiangtao.github.io/2022/10/09/Secure-Your-Browsing-Experience-with-More-Encrypted-Tools/

Start with TLS

TLS, the full name of Transport Layer Security, is a secure transport layer protocol, formerly known as SSL (Secure Socket Layer, Secure Socket Layer), located between the TCP and the application layer. Compared with HTTP, HTTPS does not change the protocol itself, but adds a layer of TLS between TCP and HTTP for encryption to ensure information security.

From the perspective of TLS, it hopes to solve several types of risks in the transmission of network traffic in plaintext: eavesdropping, tampering , and forgery. A summary of class risk:

Eavesdropping : The information exchange during the handshake process is clear text, but it is only about the exchange of cipher suite and certificate information, and it is useless for a third party to obtain it. The key pre-master key that can obtain the key is transmitted through asymmetric encryption. After the handshake is completed, symmetric encryption is used, and the eavesdropper cannot obtain the key and therefore cannot eavesdrop.
Tampering : During the handshake process, the hash of the certificate cannot be tampered with, and other information is tampered with, which will prevent the communication between the two parties afterward. After the handshake is completed, because the key cannot be obtained for encryption, although the content of the package can be modified, it cannot become the content expected by the attacker.
Forgery : The pre-master key in the key negotiation is encrypted with the public key of the certificate. The certificate is determined to be credible, and only the certificate owner can decrypt it, so the key is only owned by both parties of the communication, so it cannot be forged. .

But is it perfect? The data packets in the TLS handshake Client Hello phase contain information such as the addresses that the client wants to access, and these information are transmitted in clear text, which allows third parties to know which URLs you want to visit through monitoring and other means.

This article will introduce the encryption method in reverse order, first introduce the SNI related information about the TLS handshake phase, then talk about the DNS security layer, and finally introduce the TLS handshake fingerprint. Through this article, you may be able to have a deeper understanding of the following information learn:

Learn why ESNI is essential for privacy and security
Learn how ESNI works using public key encryption and DNS records
Learn what DNS security layer is and what the DoT/DoH implementation standards are
Learn what ECH is and how it differs from ESNI
Understand how the TLS handshake fingerprint and JA3/JA3S/JARM are calculated

This article is organized based on personal understanding and learning. If there is any inaccuracy, please point out that the reference materials are listed in the reference chapter at the end of the article. In the last chapter, this article summarizes the content of the full text, and students who already have the relevant foundation can jump to view it directly.

Server Name Indication (SNI)

Why you need SNI

When multiple websites are hosted on a single server and share a single IP address, and each website has its own SSL certificate, the server may not know which SSL certificate to display when a client device tries to connect securely to one of the websites. This is because the SSL/TLS handshake occurs before the client device is instructed to connect to a website via HTTP.

What is SNI

SNI (Server Name Indication, Server Name Indication) is to solve the TLS extension that a server uses multiple domain names and certificates, and mainly solves the disadvantage that a server can only use one certificate. SNI is an extension to the TLS protocol (formerly known as the SSL protocol), which is used in HTTPS. It is included in the TLS/SSL handshake process to ensure client devices see the correct SSL certificate for the website they are trying to access. This extension makes it possible to specify the hostname or domain name of a website during the TLS handshake, rather than when opening an HTTP connection after the handshake.

How SNI works

After SNI is enabled, the client is allowed to submit the requested domain name information when initiating the SSL handshake request. After the load balancer receives the SSL request, it will search for the certificate according to the domain name. If it finds the certificate corresponding to the domain name, it will return the certificate; if not found The certificate corresponding to the domain name returns the default certificate. The load balancer supports this function when the HTTPS listener is configured, that is, it supports binding multiple certificates.

The practice of TLS is to add the host name to the ClientHello (client hello) message in the first phase of the TLS handshake. SNI is supported since TLSv1.2. Supported since OpenSSL version 0.9.8.

SNI adds the domain name to the TLS handshake process so that the TLS program reaches the correct domain name and receives the correct SSL certificate, allowing the rest of the TLS handshake to work.

Encrypted Server Name Indication (ESNI)

Why you need ESNI

Although TLS can encrypt the entire communication process, there are still many privacy-sensitive parameters that have to be transmitted in clear text during the negotiation process. The most important and tricky one is the domain name to be accessed, namely SNI.

Encrypted SNI (Encrypted Server Name Indication, ESNI) is added to the SNI extension by encrypting the SNI portion of the client hello, which further protects by preventing anyone snooping between the client and server from seeing which certificate the client is requesting client. Cloudflare and Mozilla Firefox introduced support for ESNI in 2018.

ESNI ensures that a listening third party cannot monitor the TLS handshake process and use this to determine which websites a user is visiting.

How ESNI works

ESNI protects the privacy of SNI by encrypting the SNI portion of the client hello message (and only this portion) with the public key. Encryption only works if both sides of the communication (client and server in this case) have keys to encrypt and decrypt information, just like two people can only use the same locker if they both have locker keys like a cabinet.

Since the client hello message is sent before the client and server negotiate the TLS encryption key, the ESNI encryption key must be transported in some other way.

A Web server can add a public key to its DNS record so that when a client looks up the correct server address, it can also find the server’s public key. It’s a bit like putting the house key in a lockbox outside the house so that visitors can enter the house safely. The client can then use the public key to encrypt the SNI record so that only a specific server can decrypt it.

Encrypted Client Hello (ECH)

Problems still exist with ESNI

There is a problem with ESNI. The first is the distribution of keys. Cloudflare rotates keys every hour during deployment, which can reduce the loss caused by key leakage. However, DNS has a caching mechanism, and clients are likely to obtain outdated keys. Clients cannot continue to connect using ESNI. Secondly, the DNS request to the website may return several IP addresses, each of which represents a different CDN server. However, there is only one TXT record in ESNI, which may send the key to the wrong server and cause the handshake to fail.

What is ECH

The goal of ECH (Encrypted Client Hello, also known as Encrypted Client Hello) is to overcome the shortcomings of ESNI. At the same time, as its name suggests, ECH has greater ambitions not only to encrypt SNI, but to encrypt the entire Client Hello.

How ECH works

ECH also uses DoH (which will be described in detail later, it can be temporarily understood as an implementation of the DNS security layer) for key distribution, but the distribution process has been improved. If the decryption fails, the ESNI server will abort the connection, and the ECH server will provide the client with a public key for the client to retry the connection in hopes of completing the handshake.

The ECH protocol actually involves two ClientHello messages: ClientHelloOuter, which is sent in clear text as usual, and ClientHelloInner, which is encrypted and sent as an extension to ClientHelloOuter. The server completes the handshake with only one of the ClientHellos: if the decryption is successful, it continues with the ClientHelloInner; otherwise, it just uses the ClientHelloOuter.

ClientHelloInner consists of the handshake parameters the client wants to use for the connection. This includes the SNI, ALPN list, etc. of the origin server it wants to access. ClientHelloOuter, while also a full-fledged ClientHello message, is not used for the intended connection. Instead, the handshake is done by the ECH service provider, and in the event that the decryption cannot be completed, it signals to the client that its intended destination cannot be reached because the decryption failed. At this point, the service provider also sends the correct ECH public key, which the client can use to retry the handshake, thereby “correcting” the client’s configuration.

DNS security layer

Why DNS needs an extra layer of security

DNS is the Internet’s phone book; DNS resolvers convert human-readable domain names into machine-readable IP addresses. By default, DNS queries and responses are sent in clear text (over UDP), which means they can be read by the network, ISP, or anyone who can monitor the transmission. Even if the website uses HTTPS, the DNS queries required to navigate to the website are displayed.

This lack of privacy has huge security implications, and if DNS queries are not private, attackers can easily track a user’s web browsing behavior.

DNS over TLS standard (DoT)

DNS over TLS (DNS over TLS, or DoT for short) is a standard for encrypting DNS queries to keep them secure and private. DoT uses the security protocol TLS, the same protocol that HTTPS websites use to encrypt and authenticate communications. DoT adds TLS encryption on top of the User Datagram Protocol (TCP) used for DNS queries, which ensures that DNS requests and responses cannot be tampered with or forged by in-transit attacks .

Slightly different from DoT’s processing on the TCP protocol, there is another standard encrypted on the UDP protocol, DNS over DTLS, that is, the UDP-based DNS TLS encryption protocol standard.

DNS over HTTPS (DoH)

DNS over HTTPS or DoH is an alternative to DoT. With DoH, DNS queries and responses are encrypted, but they are sent over the HTTP or HTTP/2 protocol, not directly over UDP. Like DoT, DoH ensures that attackers cannot forge or tamper with DNS traffic. From a network administrator’s perspective, DoH traffic behaves like other HTTPS traffic, such as regular user interactions with websites and web applications.

Difference Between DoT and DoH

Both standards were developed separately and have their own RFC documents, but the most important difference between DoT and DoH is the ports they use. DoT only uses port 853, DoH uses port 443, which is also the port used by all other HTTPS traffic.

Because DoT has a dedicated port, anyone with network visibility can spot DoT traffic back and forth, even though the request and response themselves are encrypted. DoH is the opposite, DNS queries and responses are masqueraded as other HTTPS traffic because they are both coming in and out of the same port.

Other ways to encrypt DNS requests

In addition to the standards mentioned above, we still have a variety of other implementations to achieve the purpose of DNS request encryption:

VPN: With VPN, you can not only encrypt DNS requests, but even encrypt all network traffic;
DNSCurve: DNSCurve uses the Curve25519 elliptic curve encryption algorithm to create the key used by Salsa20, coupled with the MAC function Poly1305, to encrypt and authenticate DNS network packets between the resolver and the authentication server. Its proposers hope to replace DNSSEC with DNSCurve;
DNSCrypt: DNSCrypt is a network protocol designed by Frank Denis and Yecheng Fu for authentication of Domain Name System (DNS) communication between user computers and recursive domain name servers. DNSCrypt packages unmodified DNS queries and responses in a cryptographic structure to detect forgery;
IPsec: IPsec (Internet Protocol Security) is a collection of protocols and services that provide security for IP networks. It is a commonly used technology in VPN (Virtual Private Network). Since IP packets themselves do not integrate any security features, the transmission of IP packets in public networks such as the Internet may face the risk of being forged, stolen or tampered with. The two communicating parties establish an IPsec tunnel through IPsec, and the IP data packets are encrypted and transmitted through the IPsec tunnel, which effectively ensures the security of data transmission in an insecure network environment such as the Internet;

TLS handshake fingerprint

The TLS handshake fingerprint, also known as the SSL fingerprint, or the JA3 fingerprint, is the hash value information calculated from some fields in the Client Hello message sent by the client to the server.

In addition, in the TLS handshake, there is also the Server Hello that the server responds to. This type of information also has characteristics. According to a similar idea, the JA3S fingerprint can be obtained. And because the server will respond to different Server Hellos according to different Client Hellos, JARM fingerprints can be obtained according to this.

What is JA3/JA3S

Because the TLS/SSL negotiation is carried in clear text, the client application can be fingerprinted using the details in the SSL Client Hello packet. JA3 collects decimal byte values from the following fields in the Client Hello packet:

 SSLVersion , Cipher , SSLExtension , EllipticCurve , EllipticCurvePointFormat // 分别代表TLS/SSL 版本，可接受的密码，扩展列表，椭圆曲线，椭圆曲线格式

Then concatenate the values together in order , separate each field with ,, separate each value in each field with - , leave blank if there are no TLS/SSL extensions, and finally MD5 hash these strings to generate A 32-character fingerprint that is easy to use and share, this is the JA3 client fingerprint.

For the server, the same server will create its Server Hello message in different ways based on the Client Hello message and its content. Therefore, it cannot be fingerprinted here just based on the server’s Hello message, as the client and JA3 do.

JA3S is the server-side JA3 for SSL/TLS communication, the fingerprint of the server’s response to a specific client.

JA3S uses the following fields to generate fingerprints: SSLVersion,Cipher,SSLExtension , in the application, the entire encryption negotiation between the client and its server needs to be fingerprinted by combining JA3 + JA3S . Because the server will respond differently to different clients, but the response to the same client is always the same.

The above two types of fingerprints can be used to mark the client and server features in the SSL handshake. For the detailed definition of JA3/JA3S, please refer to https://github.com/salesforce/ja3

What is JARM

The TLS Server returns different Server Hello packets according to the parameters in the TLS Client Hello. The parameters of Client Hello can be manually specified and modified, so by sending multiple carefully constructed Client Hellos (such as specifying different TLS versions, support suites, etc. to test the response of the server), the corresponding special Server Hello (including the server’s return) is obtained. cipher suite, the TLS protocol version selected by the server, TLS extension ALPN protocol information, TLS extension list and other information), and finally form the JARM fingerprint information to mark the characteristics of the server. Specific parameters that can have an impact include but are not limited to:

Operating system and its version
Third-party libraries such as OpenSSL and their versions
Calling sequence of third-party libraries
User-defined configuration

Unlike JA3/JA3S, which is obtained through passive monitoring of traffic, JARM is obtained through active detection.

Detailed definitions can be found at https://github.com/salesforce/jarm

Purpose of the TLS Handshake Fingerprint

Both are simplified representations of TLS handshake information (version, support suite). So JA3/JA3S can be used to mark the traffic generated by the characteristic client and server.

The JA3S fingerprint varies according to the different environments of the client (system version, etc.). JARM can be used for threat hunting.

Summarize

The TLS protocol has become the most popular protocol on the Internet to keep network communications free from interference and eavesdropping. However, in the handshake phase, since the SNI information is transmitted in clear text, the client’s access URL and other information are easily eavesdropped.

ESNI is an important step in protecting network privacy and security, but other new protocols and capabilities are also important. The Internet was not designed with security and privacy in mind, so many steps in the process of visiting a website are non-private. However, there are various new protocols that help encrypt each step, making it safe from malicious attackers.

The Domain Name System, or DNS, matches human-readable website addresses, such as www.example.com, with alphanumeric IP addresses. It’s like looking up someone’s address in a large address book that everyone uses. However, plain DNS is unencrypted, which means that anyone can see which address someone is looking up, and anyone can pretend to be an address book. Even with ESNI installed, attackers can still see the DNS records that users are querying and determine which websites they are visiting.

Three other add-on agreements are designed to fill these gaps:

TLS-based DNS
DNS over HTTPS
DNSSEC

Both TLS-based DNS and HTTPS-based DNS do the same thing: use TLS encryption to encrypt DNS queries. The main difference between them is which layer of the network is used and which network port is used. DNSSEC confirms that DNS records are authentic and originate from legitimate DNS servers and not from attackers posing as DNS servers (for example, in a DNS cache poisoning attack ).

After talking about the guarantee of browsing privacy, let’s look at the analysis and detection of malicious traffic. Since network administrators can identify and block custom protocols, many malicious tools have turned to using existing protocols. The popularity of TLS provides a good choice for these malicious tools. Malicious tools using the TLS protocol can hide their traffic in a large number of Web browsers and other legitimate TLS overlay traffic to evade detection.

Extract some specific fields from the TLS connection (Client hello phase) to generate fingerprints. By analyzing the difference between malicious tools and conventional tools in protocol information, further analysis or research can be performed to detect malicious traffic through connection mode or time characteristics.

Concept Interpretation and Others

Server name : The server name is the name of the computer. For web servers, this name is usually not visible to end users unless the server hosts only one domain and the server name is equivalent to that domain name.
Hostname : The hostname is the name of the device connected to the network. In the context of the Internet, a domain name or website name is a type of hostname. Both are separate from the IP address associated with the domain name.
Virtual hostname : A virtual hostname is a hostname that does not have its own IP address, it is hosted on a server along with other hostnames. It’s “virtual” because it doesn’t have dedicated physical servers, just like virtual reality exists only in digital form, not in the physical world.
AdGuard Progress : AdGuard is a cross-platform content filtering ad blocking software. Its support for DoH, ESNI, etc. is still in the works https://github.com/AdguardTeam/CoreLibs/issues/722
Browser support for ESNI : Currently, only Firefox supports ESNI.
Browser support for DoH/DoT: All major browsers support it .

refer to

This article is reprinted from: https://hijiangtao.github.io/2022/10/09/Secure-Your-Browsing-Experience-with-More-Encrypted-Tools/
This site is for inclusion only, and the copyright belongs to the original author.