Apache Kylin 3.0.1 Command Injection Vulnerability

Apache Kylin is an open source, distributed Analytical Data Warehouse for Big Data written in Java. It was originally developed by eBay and is used by global enterprises such as Cisco, Baidu and Xiaomi to analyze extremely large datasets. After a SQL injection (CVE-2020-1937) was announced in Apache Kylin on 23 Feb 2020, our team @ RIPS Technologies (who is now joining forces with SonarSource) decided to evaluate what our static analysis engine could find in this project. This is how we discovered another, even more severe vulnerability (CVE-2020-1956) in the Kylin code base that allows malicious users to execute arbitrary OS commands and to take over the host system. In this blog post we will analyze the root cause of such vulnerabilities and how to prevent these in your Java applications.

Impact

The vulnerability was introduced in March 2018 with Apache Kylin version 2.3.0. It affects all releases up to version 2.6.5 and 3.0.1. An authenticated user with MANAGEMENT or ADMIN permissions on any project can inject arbitrary system commands during a Cube migration via the Kylin web interface. The attacker’s system commands are then executed on the targeted web server and allow to fully compromise the system and its data. Apache rates the severity of this vulnerability as important.

Technical Analysis

Apache Kylin handles large data sets in Cubes. The vulnerability hides in the Cube migration feature which is located in the migrateCube() method of the CubeService class code. A Cube migration is initiated via REST API endpoint in the CubeController (/kylin/api/cubes/{cube}/{project}/migrate). The CubeController handles the migrate POST request and passes a project name from the URL to the CubeService.

/server-base/src/main/java/org/apache/kylin/rest/controller/CubeController.java

 @RequestMapping(value="/{cube}/{project}/migrate", method={ RequestMethod.POST })
 // ...
 public void migrateCube(@PathVariable String cube, @PathVariable String project) {
 // ...
    cubeService.migrateCube(cubeInstance, project);

In the CubeService, the project name from the URL is concatenated unsanitized into a system command via a format string. This allows authenticated attackers to malform the API request and to inject malicious commands into the project name which are then executed on the system.

/server-base/src/main/java/org/apache/kylin/rest/service/CubeService.java

public void migrateCube(CubeInstance cube, String projectName) {
// ...
       String srcCfgUri = config.getAutoMigrateCubeSrcConfig();
       String dstCfgUri = config.getAutoMigrateCubeDestConfig();
       // ...
       String stringBuilder = ("%s/bin/kylin.sh org.apache.kylin.tool.CubeMigrationCLI %s %s %s %s %s %s true true");
       String cmd = String.format(Locale.ROOT, stringBuilder,  
               KylinConfig.getKylinHome(),
               srcCfgUri, 
               dstCfgUri, 
               cube.getName(), 
               projectName,
               config.isAutoMigrateCubeCopyAcl(),
               config.isAutoMigrateCubePurge());
       // ...
       exec.execute(cmd, patternedLogger); public void migrateCube(@PathVariable String cube, @PathVariable String project) {

For example, the attacker can invoke a separate system command by injecting backtick characters into the project name:

http://target/kylin/api/cubes/kylin_streaming_cube/ `sleep+10`/migrate

When looking at the stringBuilder above we can see that additional data is concatenated into the system command. In the first lines, a source and destination URI for a config file (srcCfgUri and dstCfgUri) is retrieved and then appended to the kylin.sh command. These configuration settings can be permanently modified by using the Cube Designer as shown in the Figure below. When system commands are injected into the configuration settings by a malicious user, these are executed during Cube migration as well.

Patch

In order to mitigate this vulnerability, all inputs have to be validated which can be modified by a malicious user and are used in a security-sensitive operation, such as a system command.

The initial patch of the Apache Kylin team based on a denylist approach. It removes malicious characters that could be used for exploitation in input parameters. However, it is difficult to define all malicious characters for all different kinds of OS environments. A special character is easily missed and hence this approach is error prone and should be avoided whenever possible. For example, the Windows operating system allows a newline character \n to separate two system commands which would bypass this denylist.

Error-prone patch - denylist

public static final String COMMAND_DENY_LIST = "[ &`>|{}()$;\\-#~!+*”\\\\]+";

public static String checkParameter(String commandParameter) {
        String repaired = commandParameter.replaceAll(COMMAND_DENY_LIST, "");
        if (repaired.length() != commandParameter.length()) {
            logger.info("Detected illegal character in command.");
        }
        return repaired;
}

An alternative patch has been implemented which uses an allowlist approach. Here, a fixed set of allowed characters is defined. Ideally, this list should contain only alpha-numerical characters but in the case of Kylin project names additional characters are required.

Corrected patch - allowlist

public static final String COMMAND_ALLOW_LIST = "[^\\w%,@/:=?.\"\\[\\]]";

public static String checkParameter(String commandParameter) {
        String repaired = commandParameter.replaceAll(COMMAND_ALLOW_LIST, "");
        if (repaired.length() != commandParameter.length()) {
            logger.info("Detected illegal character in command.");
        }
        return repaired;
}

One important thing to keep in mind is that the parameters are now sanitized against breaking out of the current command and invoking new commands. But the original command kylin.sh is still executed with these user-controlled parameters. Thus the developer needs to ensure that the shell script kylin.sh itself does not perform security-sensitive operations with these parameters. For example, the allowlist allows the character sequence ../ which could be used for a path traversal attack when the project name is used in a file path.

The patch was implemented in Apache Kylin 3.0.2 and 2.6.6 and all users are encouraged to upgrade. Alternatively, Kylin administrators can set the configuration kylin.tool.auto-migrate-cube.enabled to false in order to disable Cube migrations and to prevent exploitation.

Timeline

Summary

In this blog post we analyzed a security vulnerability in Apache Kylin that allows malicious, authenticated users to compromise the underlying system by abusing features of the Kylin web application. We looked at the root cause of this code vulnerability which can be easily introduced in any code base and evaluated different ways how to patch such an issue. With the help of static code analysis, these types of injection flaws can be automatically found early in the development lifecycle. The security vulnerability was reported to the vendor who quickly released a fixed version to protect its users. We would like to thank the Apache Security and Apache Kylin Team for the professional collaboration on fixing this issue in a timely manner.

You can join the discussion about this vulnerability in our community forum.