NGINX may be protecting your applications from traversal attacks without you even knowing
By Danny Robinson and Rotem Bar
As a security team within a rapidly growing company, we encounter lots of different types of vulnerabilities. We have numerous in-house development teams, all running full speed to build new features into their products, which inevitably also leads to the introduction of security bugs.
Our mission is to prevent, detect and remediate all security issues within our environment. Our goal is to enable rather than block our developers’ work. In order to achieve this, our strategy has been to share our knowledge through building new processes and systems that help our developers release their latest feature safely without introducing vulnerabilities into our production systems.
This is a story about a traversal vulnerability, how we handled it and why it can also impact your organization.
BACKGROUND:
We found a Traversal Vulnerability in one of our application services through our in-house scanner Jirachi (more on this in another post). Although it was a simple traversal attack and seemed easy to exploit, for some reason when testing through our staging site our Proof Of Concept kept failing.
We were able to exploit the directory traversal vulnerability when communicating directly with the service, and thought that this should be an easy Jira ticket to write up. However, when we went to write the full end-to-end POC, the exploit kept failing. We looked at the only other component that had changed between the two tests, and we found that the service was behind an AWS Application Load Balancer (ALB). However, there were no rules, no Web Application Firewall (WAF), nothing — so we couldn’t figure out why the load balancer was preventing our POC.
Investigating the AWS ALB further, we found an HTTP 400 error response that had similarities with an error response in NGINX that we found in a git commit. We quickly downloaded the latest version of NGINX, and digging through the commit history, we found that there was a configuration option called merge_slashes. When we set merge_slashes to “off”, we saw the same behaviour as we did with the ALB. We came to the realisation that the AWS ALB url parsing shares similar functionality as NGINX. Furthermore it was this functionality that was responsible for sending the http 400 error response and not allowing the request to reach the vulnerable application.
We couldn’t give up there though. We had to find a way to get our request to the vulnerable application!
Technical POC: Traversal Attack through NGINX URL parser
To demonstrate how we can get the malicious request through the NGINX url parser, we put together a sample application.
As you can see below, We created a sample web server code which has a Local File Inclusion vulnerability
const express = require('express');
const app = express();
const port = 80;
const fs = require('fs');
app.get('*', (req, res) => {
const data = fs.readFileSync('secure/' + req.url)
res.send(data);
})
app.listen(port, () => {
***console***.log(`Vulnerable App listening at http://localhost:${port}`)
})
The server accepts a GET request. The user-supplied url is concatenated with the ‘secure’ string and that file is then sent back to the user.
As an example sending a request to http://server/1.txt will cause the server to send back a file from <CurrentWorkingDirectory>/secure/1.txt
In order to demonstrate this, we send a GET request directly to the server and in the response we get the content of file 1.txt.
Vulnerable application
In order to exploit the traversal vulnerability in this app, we can request a file from a directory one level up by using the ‘../’ notation. Here we request the index.js file directly from the server .
POC —
Carrying out the same traversal attack through AWS ALB, we received a 400 bad request
However, after some research we found that we can get our malicious request through the url parser by using multiple slashes (‘/’).
In order to demonstrate this point further, the below image shows an attempt at retrieving the /etc/passwd file through an AWS ALB. Once again, we get a 400 bad request.
As before, we’re able to reach the vulnerable application by appending more slashes (‘/’).
Root Cause Analysis
How did we know AWS ALB shares similar functionality in url parsing as NGINX?
When the requests were blocked we saw a certain pattern.
The 400 bad request response from the AWS ALB above had a distinctive html signature:
<body bgcolor=”white”> …
After simple Google search and digging in GitHub we found that this signature is used by NGINX.
Specifically, we’re able to find which commit and version this error signature was removed in.
Fun fact:
body bgcolor=white was pretty common in the past as IE rendered pages by default in light gray, other browsers let the user select a default background such as black and there was large inconsistency between browsers. Putting bgcolor=white ensured that all browsers display the background as white.
How did we determine that merge_slashes is ‘off’:
Within the NGINX documentation, we are able to see that the merge_slashes default value is “On”. Therefore two or more ‘/’ characters will be normalized into one ‘/’ character.
http://NGINX.org/en/docs/http/ngx_http_core_module.html#merge_slashes
Enables or disables compression of two or more adjacent slashes in a URI into a single slash.
Syntax: merge_slashes on | off;
Default: merge_slashes on;
Context: http, server
When the merge_slashes configuration is turned on, using multiple slashes ‘///’ did not allow us to exploit that vulnerability successfully.
Trying out our bypass on latest version of NGINX:
The following shows the POC carried out on the latest stable version of NGINX (v1.18.0) with merge_slashes ‘off’.
Identification of NGINX issue:
In order to better understand how NGINX handles the url, we built a program to test the parsing function within the ngx_http_parse file:
ngx_http_parse_complex_uri(ngx_http_request_t *r, ngx_uint_t merge_slashes)
Within this function, the handling of state == sw_dot_dot at https://github.com/NGINX/NGINX/blob/master/src/http/ngx_http_parse.c#L1581 is what our tests focus on.
This shows the code written for testing the parsing function.
Here we can see when we run the test with a regular traversal attack, we get an error and return code 11.
Here we can see that when we run the test with extra slashes, we return 0.
What is the Impact?
The impact here is complicated
Looking at CVE’s we saw about 4000 known directory traversal CVE’s dating from 1999 to 2020. Directory Traversal attacks are here to stay!
Furthermore, companies testing themselves behind any ALB or NGINX solution configured with merge_slashes ‘off’ will probably not find this bug so easily. However, more knowledgable or dedicated attacks would know how to get their malicious request through the url parsing mechanism.
In order to get a better understanding of how wide-spread this issue is, we ran a search on github for merge_slashes ‘off’;. There are 549 Open Source Projects that use this configuration and unknowingly may be blocking malicious requests to applications that are vulnerable.
Could this blocking behaviour be changed?
Not really as we see it.
If traversal is blocked within NGINX, then applications relying on “safe traversal as a feature” will start failing. This is critical for some businesses and therefore, we don’t see this behaviour being changed.
Is this a vulnerability?
NGINX as itself is not a protection solution and AWS ALB is not intended to protect your application. If you are looking for a protection solution, consider using a WAF.
What do we recommend?
If possible, run your tests on the application directly without any load balancers in the middle. If you must test through load balancers, you should ensure that your testing takes into account this behaviour so you don’t miss any possible vulnerabilities.
Summary
This exercise showed us why it is important to always look around for additional attack vectors, never rely on one protection mechanism and always continue ramping up our security methods.
We do not consider this a vulnerability within NGINX nor AWS Application Load Balancers. By taking this information public, we are hoping that we are able to make organizations more aware of the behaviour displayed in NGINX and ALB and allow them to better test their application to reduce their risks
Stay tuned for more information about our security efforts. In the following weeks, we will be releasing more security tools we built in house and would love to share with the community.
Want to be part of our security adventures? We’re hiring! Contact us at security@appsflyer.com
TL;DR
Add forward slashes at the beginning of your requests when testing directory traversal attacks through load balancers.