Diving Into JumpServer: Attacker’s Gateway to Internal Networks (2/2)

Welcome back to our exploration of JumpServer security. In our previous post, Diving Into JumpServer: Attacker’s Gateway to Internal Networks (1/2) we laid the groundwork by covering the basics of JumpServer's architecture and core components. This post builds directly upon that foundation, so be sure to check out the first post if you have not done so yet.

To emphasize the impact of our discoveries, we'll continue the attack narrative from part one. We'll demonstrate how an attacker, having bypassed authentication, can leverage subsequent vulnerabilities to completely compromise the JumpServer infrastructure and internal hosts.

Impact

Building on our discovery of authentication bypass vulnerabilities, this second blog focuses on the critical authenticated code execution flaws we uncovered. By chaining these vulnerabilities, an unauthenticated attacker can fully compromise the JumpServer environment and its associated internal hosts. 

These vulnerabilities were fully addressed in JumpServer versions 3.10.12, 4.0.0 and are tracked as CVE-2023-43650, CVE-2023-43651, CVE-2023-43652, CVE-2023-42818, CVE-2023-46123, CVE-2024-29201, CVE-2024-29202, CVE-2024-40628, CVE-2024-40629.

In this part of the series, we will focus on CVE-2023-43651, CVE-2024-29201, CVE-2024-29202, CVE-2024-40628, and CVE-2024-40629.

Technical Details

Background

An authenticated user of JumpServer will see the available instances they can access. More often than not, users will have limited privileges and access only to certain services/instances.

Choosing to run a shell command on an instance involves a straightforward input window and the subsequent display of the command's result:

This output reveals a key detail that tells us about an underlying technology that is used: Interpreter Discovery. This indicates Ansible’s attempt to identify the appropriate Python interpreter on the target host. But what exactly is Ansible, and why is it relevant here?

Ansible

At its core, Ansible is an open-source automation engine designed to streamline IT tasks like configuration management, application deployment, and task automation. Leveraging a simple, agentless architecture, Ansible employs YAML-based playbooks to define desired system states and orchestrate changes across a diverse infrastructure. 

In JumpServer, users can choose the simple command input shown before, or directly write Ansible YAML playbook file.

But how is Ansible integrated into JumpServer on an architectural level?

Architecture

The user requests to execute a task using the HTTP API, which is then forwarded by the core API to the Celery service. Upon availability, Ansible executes the task and receives the output from the internal host. After receiving the results, Celery uploads it to the database, allowing it to be visible to the user via a different API call.

This means that the Celery container also has access to the database. Given that the database holds sensitive information, like host credentials, it becomes a prime target for malicious actors. Let’s take a look at the Celery container from a security standpoint.

Celery 

Ansible playbook validation bypass (CVE-2024-29201)

Ansible provides various ways within the playbook syntax to execute tasks on the Ansible host itself. According to the Ansible threat model, playbooks are trusted and can run code on both local and remote machines. This is at odds with JumpServer's threat model, which allows authorized users to run code on remote machines but not on the JumpServer host itself. 

Trying to execute local tasks using the built-in features within the playbook file, as shown in the official documentation, will trigger an exception in JumpServer:

This is due to Fit2Cloud, the maintainers of JumpServer, being aware of this risk and implementing a playbook verification using a blocklist of dangerous keywords within the YAML file content:

dangerous_keywords = (
   'hosts:localhost',
   'hosts:127.0.0.1',
   'hosts:::1',
   'delegate_to:localhost',
   'delegate_to:127.0.0.1',
   'delegate_to:::1',
   'local_action',
   'connection:local',
   'ansible_connection'
)
f = open(playbook_file)
for line in f:
    for keyword in dangerous_keywords:
        if keyword in normalize(line): // simplified
            block()

This verification is purely string-based, without parsing the YAML file, but there is a quirk within YAML parsing. YAML parsers accept JSON as a valid format of input. Knowing that attackers can now write playbooks in JSON format and use the Unicode feature within JSON to bypass the simple string check JumpServer enforces. The Unicode will be normalized during parsing, and eventually execute arbitrary code on the Celery container.

// YAML = blocked
all:
 ...
  localhost:
   ansible_connection: local


// JSON + Unicode =  not blocked
{
    ...
    "vars": {
        "ansible_\u0063onnection": "local"
    }
}

Ansible Jinja template injection (CVE-2024-29202)

In JumpServer, running “shortcut commands” or “quick jobs” utilizes Ansible’s ad-hoc command feature, which allows the execution of a non-recurring command without writing an entire playbook. These have several variables related to the job available and can be interpolated into the script using double curly braces. 

# script
echo my username: {{ jms_username }}

# output
my username: admin

This feature uses Ansible's Jinja2 templating which has full Python code capabilities. Rendering user-controlled templates is considered unsafe and is called Server-Side Template Injection. An attacker can simply execute Python code where the template is rendered, which is the Celery container.

Interestingly, creating this playbook in the Job > Template will pass it to Ansible regardless of any host availability. This means that even in the rare case of a compromised user without access to any host at all, this attack is still possible.

Arbitrary File Write in Ansible Playbooks leads to Code Execution (CVE-2024-40629)

Following the inherent mismatch in the threat model between JumpServer and Ansible, we opted to search for some new ways attackers might leverage Ansible’s features to exploit JumpServer. We discovered that the ansible.builtin.fetch module downloads a file from a remote host and saves it on the local host file system at a specified path. This can be exploited further to gain code execution in multiple ways such as overwriting Python scripts that are executed during the normal lifecycle of the container.

Arbitrary File Read in Ansible Playbooks (CVE-2024-40628)

Additionally, Ansible Playbooks can read local files in a number of ways, such as:

• The ansible.builtin.copy module copies a file from the local host to the remote host. It can then be read using the intended command execution features on the remote host.
• The ansible.builtin.file lookup plugin can read a file from the local host and use it in other commands, such as a debug print statement.

Compromised Celery impact

Since the Celery container has access to the database and the internal hosts, as we covered in the first blog post, compromising the Celery microservice would also provide access to the database and subsequently all internal hosts and JumpServer users:

Koko

Although the Celery findings are critical, an attacker who exploits these vulnerabilities won't be able to maintain persistence on the JumpServer instance. Since each microservice operates as a Docker container, updating or reinstalling JumpServer would remove the attacker's foothold. To gain persistence, the attacker would need to escape the container and execute code on the host machine.

Container Escape via Koko’s mongosh (CVE-2023-43651)

Looking at the “all-in-one” docker-compose file, we noticed an interesting setting of the Koko service:

services:
    …
    koko:
        image: jumpserver/koko:${VERSION}
        container_name: jms_koko
        restart: always
        privileged: true
        tty: true

This specific container is configured using the privileged flag, which is considered unsafe and essentially discards any container isolation. It allows access to the host machine in various ways, such as mounting the host file system or registering a kernel module. If an attacker manages to execute code on the Koko container, it will offer a path to persist their holdings from the JumpServer infrastructure to the host itself.
We previously demonstrated vulnerabilities in the Celery microservice, but how does the Koko service hold up to security standards? 

JumpServer's Web Terminal, powered by the Koko container, provides remote server and database access via SSH and database REPL tools. This functionality, which proxies commands from the web interface, inherently carries the risk of arbitrary code execution due to potential malicious user input. To mitigate this, JumpServer replaced the root shell with a highly restricted nobody user effectively limiting file system access and preventing unauthorized actions on the Koko host machine, thereby minimizing the impact of any potentially malicious code execution.

However, these mitigations are not consistently applied across all connection types. We identified MongoDB to be missing these mitigations:

The MongoDB connection is enabled by making the user communicate with the web proxy using a web socket, which is then handled by Koko using a new mongosh subprocess that connects to the database itself

If the connection is successful, the WebSocket connection is directed to this subprocess, and any user input in the web terminal is directed to mongosh, acting as if it is installed locally on the user’s machine. 

The MongoDB shell is actually a Node.js shell with certain variables in scope that can be used to access the remote database. As the shell is normally run on a developer's machine, it is not restricted and is just as capable as any Node.js process.

This means an attacker can access not only the mongodb features but also all the Node.js functionalities. From there, executing code is as simple as using a basic childProcess.execSync() function.

Compromised Koko impact

The impact of this vulnerability is a complete compromise of the JumpServer host machine, granting full control over logs, internal hosts, and persistence.

Key Takeaways for Developers

1. Microservice Architecture and API Security: Microservice architectures can inadvertently expose API properties and endpoints, increasing the attack surface. Rigorous API security testing and access control are essential.
2. Threat Model Alignment: Address threat model gaps by ensuring consistent security assumptions across all integrated components. Understanding how these components interact with your system, and addressing potential conflicts is crucial for a secure environment. 
3. Container Best Practices: Avoid running privileged containers. Implement the principle of least privilege to minimize the potential impact of container compromises.

Patch

The vulnerabilities discussed here were fixed in various ways and versions:

Celery - Ansible:

• Initially, Fit2Cloud addressed CVE-2024-29201 and CVE-2024-29202 in version 3.10.7 by forking Ansible and enabling only SuperPlaybookRunner to use local connections. Jijnja2 was changed to use SandboxedEnvironment instead of NativeEnvironment.
• Following our CVE-2024-40628 and CVE-2024-40629 reports in versions 3.10.12 and 4.0.0, Fit2Cloud decided to address this issue at its root by running Ansible in an isolated process mode.

Koko:

• CVE-2023-43651: fixed in versions 2.28.20 and 3.7.1 by executing Mongodb shell using the restricted nobody user. A subsequent privilege escalation bypass to this fix was addressed by removing the BOOTSTRAP_TOKEN environment variables when constructing the command runtime. 

These vulnerabilities were fully addressed in JumpServer versions 3.10.12, 4.0.0

Timeline

Summary

This blog series has showcased critical security vulnerabilities within JumpServer, an open-source Privileged Access Management (PAM) application. We demonstrated how attackers can exploit vulnerabilities stemming from diverse root causes. At Sonar, we are committed to equipping developers with tools to write clean, secure code, and with the knowledge necessary to prevent similar vulnerabilities in their own applications. Understanding the fundamental issues that led to these exploits is important for building robust and secure software.

We would like to thank Fit2Cloud, the vendor behind JumpServer, for their responsiveness and professionalism throughout the vulnerability disclosure and remediation process.

Related Blog Posts

• Diving Into JumpServer: Attacker’s Gateway to Internal Networks (1/2)
• Micro Services, Major Headaches: Detecting Vulnerabilities in Erxes' Microservices
• Beware the Cookie Monster: Cyberhaven Extension Vulnerability Allowed Cookie Theft
• The Tainted Voyage: Uncovering Voyager's Vulnerabilities