As part of our continuous effort to improve our Clean Code technology and the security of the open-source ecosystem, our R&D team is always on the lookout for new 0-day security vulnerabilities in prominent software.
We recently uncovered two critical code vulnerabilities in a personal cloud solution named CasaOS. CasaOS can be installed on any machine thanks to Docker and comes with end-user NAS devices like the ZimaBoard or the X86Pi. Users deploy CasaOS to store their personal data on devices they trust and access it from anywhere.
CasaOS is developed by IceWhale in Go and has close to 17,000 stars on GitHub as we're writing this article.
A CasaOS dashboard.
These security vulnerabilities, tracked as CVE-2023-37265 and CVE-2023-37266, allow attackers to get around authentication requirements and gain full access to the CasaOS dashboard. From here, attackers can access the data stored on the device, but that's not all.
Because of CasaOS' extensibility and support for third-party applications, they can also execute arbitrary commands on the system to gain persistent access to the device or pivot into internal networks. There are reports that an exploit for Plex Media Server, another personal cloud system, was used in the LastPass breach, and this initial foothold to get access to the employee's internal network.
While we now are releasing the technical details of our findings several months after the vendor addressed them, we were made aware of public exploits based on the study of the patch only 10 days after the security release. That means that all unpatched instances are already at risk. We urge all CasaOS users to upgrade their instances to the latest available release (v0.4.4-1 at the time of writing this article).
Let's dive into the technical details of these security vulnerabilities and see what we can learn from them!
Pretending to be an internal service with CVE-2023-37265
To follow this vulnerability, we must first understand that CasaOS is not a standalone software but a set of services you install on top of a distribution like Ubuntu or Debian. That means that by default, CasaOS has control over all components processing incoming HTTP(S) requests.
A world of microservices
The first component to receive users' requests is casaos-gateway, the only service to be directly exposed to the network. In a common fashion in the Go ecosystem, it forwards requests to other local microservices depending on the request path.
Many services are listening only on localhost, waiting for casaos-gateway to send them traffic:
casaoscasaos-message-buscasaos-user-servicecasaos-local-storagecasaos-app-management
This can be confirmed by reading the configuration of casaos-gateway. The entry runtimepath defines where it will store the route that services later declare by notifying casaos-gateway on its management port.
root@casaos-dev:~# cat /etc/casaos/gateway.ini
[common]
runtimepath=/var/run/casaos
[gateway]
logfileext=log
logpath=/var/log/casaos
logsavename=gateway
port=80
wwwpath=/var/lib/casaos/wwwThese routes are persisted in /var/run/casaos/routes.json:
root@casaos-dev:~# cat /var/run/casaos/routes.json
{"/":"http://127.0.0.1:46351","/.well-known/jwks.json":"http://127.0.0.1:36915","/doc/v2/app_management":"http://127.0.0.1:41401","/doc/v2/casaos":"http://127.0.0.1:45277", [...], "/v3/file":"http://127.0.0.1:45277"}Internally, the gateway uses this list and Go's net/http/httputil/ReverseProxy to forward incoming requests to the right service.
Image taken from https://wiki.casaos.io/en/contribute/development.
Where is it coming from?
A common problem caused by such reverse proxies is that the final service will see all requests coming from the reverse proxy; in this case, the source IP address at the network layer level would always be localhost! To solve this problem, it's commonly agreed that the reverse proxy uses a header named X-Forwarded-For to give this information to the application.
Two scenarios can arise:
- The client's request doesn't have an
X-Forwarded-Forheader: the reverse proxy will create one, and put the client's IP address in it, i.e.,X-Forwarded-For: 1.2.3.4. - The client's request already has an
X-Forwarded-Forheader because of another reverse proxy placed in front ofcasaos-gateway. In this case, it will forward the header after appending the previous proxy's IP address, i.e.X-Forwarded-For: 1.2.3.4, 192.168.42.42. The current proxy won't add its own IP address because the next hop can get it from the network layer.
The HTTP RFC does not specify how one should deal with invalid X-Forwarded-For headers, so most implementations will simply copy the value found in the header when relaying requests.
RFC 7239 introduced a new header named Forwarded, intending to replace many headers of the X-Forwarded-* family. Still, its deployment stays very sparse and has the same implications security-wise.
There's a very common foot gun here: the application should only trust this family of headers if it knows there's a reverse proxy in front. Otherwise, the header could come directly from the client and can contain anything!
In our case, there's always casaos-gateway in front, so it's all good, but we've already found such issues in many applications, such as OneDev and Cacti
So far, so good–now it's up to the application to handle this header to find the client's IP address.
Your IP address is in another layer!
But why would we need to know the client's real IP address? Logging is the first thing that comes to most developers' minds. If you put your security hat on, you may also want to use it for rate limiting, access control, or authentication.
Looking at the code of the microservices listed above, there are several cases of security decisions based on the client's IP address. For instance, in the repository CasaOS-Common, used by casaos and casaos-local-storage, there's this snippet:
func ExceptLocalhost(publicKeyFunc func() (*ecdsa.PublicKey, error)) gin.HandlerFunc {
return func(c *gin.Context) {
if c.ClientIP() == "::1" || c.ClientIP() == "127.0.0.1" {
c.Next()
return
}
JWT(publicKeyFunc)(c)
}
}The authentication middleware is skipped if a request comes from 127.0.0.1 or ::1! That means that gin.Context.ClientIP() probably has logic to handle application-level IP address information (e.g., through X-Forwarded-For). Otherwise, all requests wouldn't require authentication since they come from casaos-gateway through the loopback interface.
Digging further into Gin's implementation, we see the following documentation around the ClientIp() implementation:
// ClientIP implements one best effort algorithm to return the real client IP.
// It calls c.RemoteIP() under the hood, to check if the remote IP is a trusted proxy or not.
// If it is it will then try to parse the headers defined in Engine.RemoteIPHeaders (defaulting to [X-Forwarded-For, X-Real-Ip]).
// If the headers are not syntactically valid OR the remote IP does not correspond to a trusted proxy,
// the remote IP (coming from Request.RemoteAddr) is returned.
func (c *Context) ClientIP() string {
// [...]
}The important bit is "If the headers are not syntactically valid [...] the remote IP (coming from Request.RemoteAddr) is returned". Here is the validation function applied to all comma-separated values of X-Forwarded-For starting from the rightmost one:
// validateHeader will parse X-Forwarded-For header and return the trusted client IP address
func (engine *Engine) validateHeader(header string) (clientIP string, valid bool) {
if header == "" {
return "", false
}
items := strings.Split(header, ",")
for i := len(items) - 1; i >= 0; i-- {
ipStr := strings.TrimSpace(items[i])
ip := net.ParseIP(ipStr)
if ip == nil {
break // <==== [1]
}
// X-Forwarded-For is appended by proxy
// Check IPs in reverse order and stop when find untrusted proxy
if (i == 0) || (!engine.isTrustedProxy(ip)) {
return ipStr, true
}
}
return "", false
}Interestingly, at [1], the code bails out of the loop if it finds an invalid IP address, even if it previously found valid IP addresses.
That means that if we're sending an invalid X-Forwarded-For header with our request, for instance with the value foobar, it gets relayed by casaos-gateway after appending its IP address, and the validation of this header in Gin fails. It then falls back to the source IP address of the client–casaso-gateway, talking from 127.0.0.1!
Requests from this IP address do not require authentication, resulting in an authentication bypass on most API endpoints. It is trivial to demonstrate it on our test instance. Our first request without the X-Forwarded-For header gets an error 401, while a second request with an invalid X-Forwarded-For gets through:
$ curl -v 'http://192.168.64.3/v1/sys/logs'
> GET /v1/sys/logs HTTP/1.1
> Host: 192.168.64.3
> User-Agent: curl/8.2.1
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< [...]
<
* Connection #0 to host 192.168.64.3 left intact
$ curl -v -H 'X-Forwarded-For: x' 'http://192.168.64.3/v1/sys/logs'
> GET /v1/sys/logs HTTP/1.1
> Host: 192.168.64.3
> User-Agent: curl/8.2.1
> Accept: */*
> X-Forwarded-For: x
>
< HTTP/1.1 200 OK
< [...]
{"success":200,"message":"ok","data":"2023-06-30T13:09:16.882Z\tinfo\tNotified systemd that casaos main service is ready\t{\"func\": \"main.main\"[...]We'll now look into our second finding, CVE-2023-37266, before discussing post-exploitation risks.
Creating arbitrary JWTs with CVE-2023-37266
While investigating CVE-2023-37265, we noticed a strange behavior of the session JWT. Modifying the token's claims and signature did not result in errors, and something was likely wrong with it.
How can we validate this theory? There are now tools aimed at the Bug Bounty community to detect weak secrets, like Ian Carroll's cookiemonster, but even the venerable John the Ripper is good for this job. It immediately confirms the use of an empty HMAC-SHA256 secret:
$ john --format=HMAC-SHA256 jwt.txt
Using default input encoding: UTF-8
Loaded 1 password hash (HMAC-SHA256 [password is key, SHA256 128/128 ASIMD 4x])
Proceeding with single, rules:Single
Press 'q' or Ctrl-C to abort, almost any other key for status
Almost done: Processing the remaining buffered candidate passwords, if any.
Proceeding with wordlist:/nix/store/15mz0y4nyxl4apy0w1562bw8kd4f8wps-john-1.9.0-jumbo-1/share/john/password.lst, rules:Wordlist
(?)
1g 0:00:00:00 DONE 2/3 (2023-06-30 15:28) 100.0g/s 25600p/s 25600c/s 25600C/s 123456..franklin
Use the "--show" option to display all of the cracked passwords reliably
Session completedIf the secret isn't really secret, anybody can craft an arbitrary JWT as they please. By crafting valid-looking but unsigned tokens, attackers could bypass the authentication and gain administration privileges on vulnerable CasaOS instances.
Now what? Post-exploitation considerations
As always and out of caution, we won't be sharing exploitation scripts but we believe that it is important to share the full extent of these findings' impact to help users protect themselves. It is also a good opportunity to discuss security best practices when designing software features.
One of the most appealing features of CasaOS is its support of third-party applications through the internal application store or manually through the web interface and their Custom Install wizard.
The Custom Install wizard.
These applications are simply Docker containers deployed on the same host as CasaOS. Docker containers only provide thin isolation between the application and other processes, and CasaOS users can mount devices and folders from the host in the context of the application. Both Docker and the application run as root, the former on the default user namespace and the latter on its own, so malicious applications can compromise the host through these mounts.
Such abusable features are widespread, especially in software aimed at tech-savvy users who always like to have control over what they use. Attackers can also leverage these features to compromise the system. It grants them persistent access to the device, even across software updates, and helps them to pivot into the victim's internal network.
A good practice is to allow users to opt-out, or even better opt-in, of these features in a configuration file. It lets advanced users turn off dangerous features if they don't need them, resulting in a reduction of the overall attack surface.
How CasaOS addressed our findings
Shortly after discovering, confirming, and documenting these security vulnerabilities, our Vulnerability Researchers responsibly disclosed them to the CasaOS maintainers through GitHub's Security Advisories feature. We've had great discussions with them to identify and validate the robustness of their patches before releasing CasaOS v0.4.4.
Preventing spoofing of local addresses (CVE-2023-37265)
While we initially recommended stripping all incoming X-Forwarded-For and similar headers, maintainers wanted to keep the support of potential reverse proxies in front of CasaOS instances.
Another solution was found in 391dd7f, where they decided to rewrite outgoing X-Forwarded-For headers in a way they would never contain 127.0.0.1 or ::1:
func rewriteRequestSourceIP(r *http.Request) {
// we may receive two kinds of requests. a request from reverse proxy. a request from client.
// in reverse proxy, X-Forwarded-For will like
// `X-Forwarded-For:[192.168.6.102]`(normal)
// `X-Forwarded-For:[::1, 192.168.6.102]`(hacked) Note: the ::1 is inject by attacker.
// `X-Forwarded-For:[::1]`(normal or hacked) local request. But it from browser have JWT. So we can and need to verify it
// `X-Forwarded-For:[::1,::1]`(normal or hacked) attacker can build the request to bypass the verification.
// But in the case. the remoteAddress should be the real ip. So we can use remoteAddress to verify it.
ipList := strings.Split(r.Header.Get("X-Forwarded-For"), ",")
r.Header.Del("X-Forwarded-For")
r.Header.Del("X-Real-IP")
// Note: the X-Forwarded-For depend the correct config from reverse proxy.
// otherwise the X-Forwarded-For may be empty.
remoteIP := r.RemoteAddr[:strings.LastIndex(r.RemoteAddr, ":")]
if len(ipList) > 0 && (remoteIP == "127.0.0.1" || remoteIP == "::1") {
// to process the request from reverse proxy
// in reverse proxy, X-Forwarded-For will container multiple IPs.
// if the request is from reverse proxy, the r.RemoteAddr will be 127.0.0.1.
// So we need get ip from X-Forwarded-For
r.Header.Add("X-Forwarded-For", ipList[len(ipList)-1])
}
// to process the request from client.
// the gateway will add the X-Forwarded-For to request header.
// So we didn't need to add it.
}From a security design standpoint, this solution is not entirely satisfactory because internal services still don't require authentication for requests from localhost. Attackers could still gain arbitrary code execution from a simple Server-Side Request Forgery on the same host. In the same way, if any of these services are exposed to untrusted users by mistake, attackers would not have to provide credentials to compromise the server.
Enforcing JWT validation for CVE-2023-37266
After validating this finding on our local instance, we wanted to confirm it on the public demonstration instance in a non-intrusive way. We collected a valid JWT but noticed that the signature seemed to use another secret, and unleashing John didn't yield any result.
In fact, CasaOS maintainers already addressed this security vulnerability in the development branch with the commit 705bf1f, a few weeks before our research. The patch updates both route groups to use the configuration's private key and ensure the token's signature is valid.
diff --git a/route/v1.go b/route/v1.go
index da317eb4..98117604 100644
--- a/route/v1.go
+++ b/route/v1.go
// [...]
@@ -39,7 +41,11 @@ func InitV1Router() *gin.Engine {
r.GET("/v1/recover/:type", v1.GetRecoverStorage)
v1Group := r.Group("/v1")
- v1Group.Use(jwt.ExceptLocalhost())
+ v1Group.Use(jwt.JWT(
+ func() (*ecdsa.PublicKey, error) {
+ return external.GetPublicKey(config.CommonInfo.RuntimePath)
+ },
+ ))
{
v1SysGroup := v1Group.Group("/sys")diff --git a/route/v2.go b/route/v2.go
index 4c4a1fb5..d07e0629 100644
--- a/route/v2.go
+++ b/route/v2.go
// [...]
@@ -74,11 +76,14 @@ func InitV2Router() http.Handler {
// return true
},
ParseTokenFunc: func(token string, c echo.Context) (interface{}, error) {
- claims, code := jwt.Validate(token) // TODO - needs JWT validation
- if code != common_err.SUCCESS {
+ // claims, code := jwt.Validate(token) // TODO - needs JWT validation
+ // if code != common_err.SUCCESS {
+ // return nil, echo.ErrUnauthorized
+ // }
+ valid, claims, err := jwt.Validate(token, func() (*ecdsa.PublicKey, error) { return external.GetPublicKey(config.CommonInfo.RuntimePath) })
+ if err != nil || !valid {
return nil, echo.ErrUnauthorized
}
-
c.Request().Header.Set("user_id", strconv.Itoa(claims.ID))
return claims, nilWe still reported our finding because we didn't find an official security advisory for it, and CasaOS decided to assign a CVE since it has been vulnerable for quite some time now.
Timeline
| Date | Action |
| 2023-07-03 | We report all issues to CasaOS maintainers. |
| 2023-07-03 | The vendor confirms the issues and creates GitHub private advisories to coordinate disclosure and collaborate on the patches. |
| 2023-07-14 | CasaOS v0.4.4 is released. |
| 2023-07-17 | CVE-2023-37265 and CVE-2023-37266 are assigned to our findings. |
Summary of CasaOS Vulnerabilities
In this article, we came back to the details behind CVE-2023-37265 and CVE-2023-37266, two critical code vulnerabilities in CasaOS. These are fairly simple security vulnerabilities to identify and exploit, and we are aware of public exploits for them; we encourage you again to update all your CasaOS instances.
Interestingly, even with modern languages such as Go, whole classes of design bugs are still likely to arise in new software. Education is essential to security, and we hope this article helps you spot and prevent similar pitfalls in your code.
In general, identifying IP addresses at the application layer is risk-prone and shouldn't be relied on for security decisions. Many different headers may transport this information (X-Forwarded-For, Forwarded, etc.), and the language APIs sometimes need to interpret nuances of the HTTP protocol the same way. Similarly, all frameworks have their own quirks and can be tricky to navigate without expert knowledge of these common security footguns. Again, the first finding was caused by a documented feature: the framework does what it claims to do.
If you would like to learn more about this topic and how it became very hard for developers to make sensible decisions based on these headers, we strongly recommend reading Adam Pritchard's The perils of the “real” client IP–we only scratched the surface of what can go wrong in our article.
We also recommend users of any personal NAS solutions to consider restricting their network exposure, for instance, with a VPN tunnel. These devices often contain personal data and are connected to our internal networks–they are goldmines to attackers and should be secured as such.
As a final note, we thank CasaOS maintainers, especially CorrectRoadH and tigerinus, for their very efficient handling of our reports and interesting discussions.