Understanding HTTP Request Smuggling with Hop-to-Hop Headers

We have seen HTTP request smuggling attacks by modifying the Content-Length and Transfer-Encoding header. These methods exploit the execution of the headers on the client-side and server side. However, there’s also another way of performing this attack by using hop-to-hop headers. In this blog, we will discuss this technique in complete detail.

This requires intermediate proxy servers between the client and the main server. We will also briefly discuss server-side cache poisoning using the techniques mentioned above. This way of exploiting the server-side cache was found by Francesco Mariani and Jacopo Tediosi in a private bug bounty program of Akamai CDN.

What are Hop-to-Hop Headers?

Hop-to-hop headers are HTTP request headers which are designed in such a way that they are consumed by the proxy which is handling the request. The request forwarded by the proxy doesn’t contain the hop-to-hop headers. It is different from an end-to-end header, which is designed to be present in the request till the end.

HTTP/1.1 treats the following headers as the hop-to-hop header:

• Keep-Alive
• Transfer-encoding

• Connection  
• Trailer
• Upgrade
• Proxy-authorization
• Proxy-authenticate

Further, it is also possible to define some custom hop-to-hop headers. This could be done by adding them as valuable to the connection header.  

To check for the presence of a proxy, you must see which header is creating a change in the response. A header is added in the ‘Connection’ header value, and the difference in response is analyzed. Let’s say if the response to the request (with ‘Cookie’ as the value of the connection header) and the request (without the ‘Cookie’ header) matches; it indicates the presence of a proxy deleting the ‘Cookie’ header (a custom hop-to-hop header present as a value of Connection header).  

However, in some cases, the first proxy does not delete the hop-to-hop header specified in the connection header and forwards it to the next proxy. The second proxy could then delete the specified hop-to-hop header. This can confuse the server into thinking that the request originated from the second proxy because what the second proxy forwards looks like a normal request without any custom hop-to-hop header.  

Using Content-Length as Hop-to-Hop Header

According to the above discussion, ‘Content-Length’ can also be used as a hop-to-hop header by specifying it in the value of ‘Connection’ header. This can lead to Request Smuggling attack as described below.  

If a proxy is present, it will remove the content-length header from the request and forward it. Again, if a second proxy is present, it would consider the incoming request as two separate requests because the content-length header is removed from the original request headers. The body of the original request would be considered as a separate request by the second proxy and thus two requests are being sent to the server.

From the server side, the second proxy receives two responses and forwards it to the first proxy which was expecting only one. Therefore, the second response gets queued at the first proxy and is sent as response to subsequent requests from other clients.

Server-side Caching at Edge Nodes

What is Server-Side Caching?

Caching is a concept of saving frequently used data in memory and using them instead of actually sourcing the data.  

Server-side caching means temporarily storing the web files and data on the original server to be reused later. It can also be used to reduce the load on the database when multiple servers are trying to access the database. When the user first requests a webpage, the website goes under the normal process of retrieving data from the server and constructing the website’s webpage. After the request has happened and the response is returned, the server copies the webpage and stores it as a cache. The next time the user revisits the website, it loads the already saved or cached copy of the webpage, thus making it faster.

What are Edge Nodes?

Edge computing brings data, insights, and decision-making closer to the things that act upon them. Rather than relying on a central location that can be thousands of miles away, the edge is as near to the “thing” as possible. There are lots of daily interactions where it is taken for granted that the response would be instantaneous.  

An edge node is a computer that acts as an end-user portal for communication with other nodes in cluster computing. Edge servers were developed during the 1990s to serve web and video content faster by deploying near users. If the resource requested by the user is present in the cache, i.e., edge server, then the response is served from the edge server, or else, the request is forwarded to the original server.

Let’s go through the complete exploit from beginning.

The researchers observed that the application was sending back a header in response. It was ‘Connection’ heder set with ‘Transfer-Encoding’. This would indicate the possibility of backend using transfer-encoding to analyze the request body and therefore the HTTP request smuggling vulnerability.  

So, HTTP Request Smuggling attacks with the ‘Connection’ header value set to ‘Content-Length’ were performed on the application. This made ‘Content-Length’ as a hop-to-hop header which got consumed on passing the first proxy. The original request split into two different requests because without the ‘Content-Length’ header the request body would be considered as a separate request.  

The second proxy that was present received two requests and forwarded it to the server. Two responses were generated from the two different servers specified in the hostname of the requests. The second proxy again forwarded these responses to the first proxy.

However, the first proxy only expected a single response as it had received a single request. The second response was therefore cached and sent in further requests. Sending the request twice, the response showed some kind of DNS error or an invalid URL message which was unexpected.  

Upon more research, it was found that the proxy only could resolve the domains internal to the Akamai network, and any request with an outside domain would give back such kind of error. So, request smuggling was really working, and the assumptions about the proxy servers being used were also correct. Using a domain that uses Akamai CDN correctly gave back the response without error.  

Further, the researchers tested for cache poisoning. As Akami CDN was using an edge server, the request-response pair was stored as cache data. The edge server sends back the stored response for the same request to all the edge nodes connected to it.

For testing this, the researchers used VPN and changed the IP address to nearby regions, and surprisingly, the same response was observed for requests from all the different IP addresses. The queued response for the second malicious request (passed as the original request body) was sent to further requests from different IPs. This vulnerability could be used to put some unauthorized data on the Akamai server, leading to a loss of confidentiality.

Conclusion

In this way, it was possible to put/host the data of any domain using Akamai CDN on the Akamai CDN’s cache and further reflect that data as a response to the users.  

Akamai acknowledged the finding and immediately fixed it by not allowing the use of Content-Length as the Connection header value. They also reported that Akamai was no longer vulnerable to this attack.

‍