After an interesting adventure, it's now possible to announce a new CVE-2021-31828 which effects Open Distro for ElasticSearch (ODFE) , versions until 1.12.0.2.
Open Distro is a plugin for ElasticSearch that enhances security, alerting, SQL querying, and gives more advanced capabilities.
Jump directly to how it was found
The Vulnerability
If you are running ODFE 1.12.0.2 and below, It may be possible for existing privileged users to enumerate listening services or interact with configured resources via HTTP requests in your environment. (SSRF)
Do note that the severity and probability may change according to the usage and environment of the specific installation.
For example, If only known administrators can access the elastic instance and the elastic instance is isolated from reaching any network then the risk is low.
On the other hand, if the elastic instance is reachable by all users in the company and open to clients when no network hardening was done this vulnerability severity may be high and sometimes critical.
For starters, before you read about how this CVE was found, if you are using Elasticsearch and using Open Distro as part of your installation, now is a good time to go and check the version, and upgrade the plugin to the latest version, if you are not up to date.
If you are not sure if you are using Open Distro you can go to your ElasticSearch instance and run the command https://{YOUR_IP}:9200/_cat/plugins?v to see if you have the Open Distro plugin installed. If you are running Open Distro, you will see the following packages in the response:
opendistro-job-scheduler
opendistro-sql
opendistro_index_management
opendistro_security
Mitigation
- Upgrade to the latest version of ODFE (1.13.1.0 and above)
- Configure the alerting module to access only the needed resources.
First thing you’ll need to do is check your alerting default by running the following command which will show you the default configuration:
curl -u "admin:your_password" -k "https://{YOUR_IP}:9200/_cluster/settings?include_defaults=true" | \
jq ".defaults.opendistro.alerting"
Make sure the destination.allow_list enables only the required actions, some installations don’t need all features of ODFE and can pose a risk
{
"alerting": {
/**...**/
"destintation": {
"allow_list": [
"chime",
"slack",
"custom_webook",
"email",
"test_action"
]
}
}
}
In the latest patch, ODFE added the field destination.host.deny_list which allows an administrator to control which IP addresses the plugin is denied access to. By default this list is empty. It is highly recommended to configure at least all known internal networks in order to deny traffic to your internal resources.
Example:
{
"alerting": {
/**...**/
"destintation": {
"allow_list": [
"custom_webook",
"test_action"
],
"host": {
"deny_list": [
"127.0.0.0/8",
"10.0.0.8/8",
"172.16.0.0/12",
"192.168.0.0/16"
]
}
}
}
}
Ok, now that required a lot of trust…. now that we got the important part out of the way, let’s actually talk about the adventure itself.
The Story
Bug bounty and penetration tests are a part of my way of life, Every project I have taken on, I have had the privilege of learning something new, sometimes it’s just small stuff, and other times when I really get lucky, mind-blowing tricks.
The Open Distro CVE adventure actually started off, after the kick-off with a client, as a seemingly frustrating penetration test. This was because there wasn’t a single line of code written in the actual project itself. The project basically consisted of a supplier combining different open source solutions into one big tech stack, configuring and supporting the system for the needs of the customer.
This actually happens a lot, especially in government contracts, banks, insurance companies, and other large corporates where projects are outsourced to external system integrators. These system integrators then combine different components into one cohesive solution (really “templatey” type stacks to solve specific problems rapidly).
Back to our story, the purpose of this specific solution was to provide a friendly Kibana interface for a system that does a lot of processing on the backend and ultimately enables users to easily fetch and process data. (As an aside, this is already the second system I’ve seen this year that essentially “ditched” the custom user interface and provides direct access to a highly customized Kibana interface.)
After ticking the regular boxes, and checking the different configurations, and running the required hardening tests, I still wanted to do better. This is where it gets interesting, I started to investigate what I could potentially do with the permissions granted in the system to a typical user.
The OpenDistro plugin
OpenDistro is an AWS-maintained open-source plugin and is actually pretty cool. It provides many capabilities which are missing in ElasticSearch. The experience is like using a native and complementary add-on to ElasticSearch...but by now we know very well, with great power comes great responsibility.
As part of my investigation, I came upon the alerting module. This module allows the user to create an "open distro monitor" (read more about these here ), define a trigger as a webhook to any resource, control the method type, as well as have full control of the body.
The first thing that came to my mind when seeing this was a potential SSRF vulnerability. Server Side Request Forgery is a common attack method that exploits one service to make a request on your behalf from other servers. You can read more about this here .
After creating a trigger in the alerting module, you can then test it. This is where the SSRF option comes in. When testing your request it is sent to the requested server, which returns some kind of response. Sweet!
Crafting some script magic I was able to edit a trigger, invoke it, and read the results. With this ability, I was able to start scanning the network, access the metadata API (which specifically in the project was unhardened), access the Kubernetes API, as well as other local resources.
The next thing an attacker would do in such a scenario is scanning the available resources and cross-reference them with known and reported vulnerabilities, or other open and vulnerable/unauthenticated endpoints. So that’s what I did too.
The alerting module has support for POST, PUT and PATCH access, and because of this POST access, I was able to request a resource from another server and exploit a different known vulnerability in the connected server (out of the scope of this article) to achieve remote code execution (AKA RCE).
Investigating the source of the problem brought me to this source code:
github.com/opendistro-for-elasticsearch/ale..
CustomWebhookMessage customWebhookMessage = (CustomWebhookMessage) message;
uri = buildUri(customWebhookMessage.getUrl(), customWebhookMessage.getScheme(), customWebhookMessage.getHost(),
customWebhookMessage.getPort(), customWebhookMessage.getPath(), customWebhookMessage.getQueryParams());
httpRequest = constructHttpRequest(((CustomWebhookMessage) message).getMethod());
This line of code essentially creates HTTP requests based upon user input and sends them without any validations.
Remediation Process
After searching for how and who to submit this issue, I found that the OpenDistro plugin actually belongs to AWS, and started the disclosure process.
The AWS security team was very responsive, and I want to thank all of the team members involved who took the issue seriously, started the triage process immediately, and kept me in the loop during the whole process.
The fix has now been deployed, and now administrators can upgrade their ODFE plugin and configure a list of allowed hosts to communicate with. (See mitigation section above)
From An Attacker’s Perspective
Many times attackers that are unable to access resources directly will find the “proxy” resource that will provide them the access they need to your sensitive internal resources and data.
Once they have a good understanding of the technologies in the stack - it’s a simple search to find vulnerabilities in the different versions, and try to exploit those known vulnerabilities.
Lessons Learned
Don’t underestimate yourself in any pentest/bounty program.
Don’t leave any stone unturned, especially in areas you think are battle-hardened. You might surprise yourself.
Be aware of the plugin ecosystem.
When we come to secure/attack our stacks and systems we can’t only look only at the application itself - but we also need to take into account the different third-party resources they integrate with, and in this case plugins. Each and every one of these integrations need to be examined individually, as they can be your single point of failure. Take the SolarWinds example, where an old unpatched version was the key to the kingdom.
Test carefully webhooks
Because most systems born today use multiple microservices, webhooks and the environment are easily exploited into doing stuff they shouldn't.
Any time your service requests external HTTP resources you should validate that users cannot change the target of the server. In the event that this is by design, it will be your responsibility to add the required validations needed to properly secure your other services.
Disclosure & Remediation Timeline:
24 January - Reported to AWS
24-26 January - First response from AWS and start of triage process
26 February - Fix committed into github repo in version 1.13.1.0
5 March - Dockerhub updated with version 1.13.1
26 April - CVE created
6 May - CVE released
11 May - Blog published