Introduction

Google Cloud Shell is a product of the Google Cloud Platform (GCP) that provides a number of interesting features, such as an interactive shell environment and an integrated code editor, with the intent of making experimenting and administration of GCP products easier while requiring very little setup. A particularly interesting feature is the option to customize the used Cloud Shell environment, for example by installing additional toolkits. This can be done by creating a custom Docker image which uses the default environment's Docker image as base image. Any Cloud Shell user can make use of such custom Cloud Shell images when they are pushed to the Google Cloud Registry publicly, simply by adding the image name as an URL parameter when connecting to the Cloud Shell environment, for example https://ssh.cloud.google.com/cloudshell/editor?cloudshell_image=gcr.io/custom-project/custom-image.

In an effort to protect users from loading and executing malicious code, whenever a Cloud Shell user requests a custom environment in such a way they will be presented with the following two choices:

  1. Trust the image, so that the user's virtual machine is automatically authorized to access the Cloud Shell user's GCP resources.
  2. Do not trust the image, thus not making such credentials available within the environment.

custom-image.png

Figure 1: A prompt shown when a user requests loading a custom Cloud Shell image.

As part of the Google Vulnerability Reward Program (VRP) I discovered an information leakage vulnerability that would allow the creator of a malicious custom image, which when loaded by another Cloud Shell user in an untrusted way, to later on regain access to that user's Cloud Shell environment once that user had switched back to using the default environment in a trusted way.

Discovery

I was trying to understand how Google Cloud Shell worked so I tried out the different features. One of those features was web preview, which allows web applications running inside Google Cloud Shell's virtual machine to be previewed in the user's browser. To test the feature I just clicked on the button to preview the default port 8080. Not having started any server on that port I was presented with an adequate error message.

preview.png

Figure 2: A first test of the web preview feature.

I noticed that the domain I was taken to was of the form 8080-dot-123456789-dot-devshell.appspot.com. Having previously analyzed how other parts of Cloud Shell worked it appeared to me that the domain would be of the form $PORT-dot-$ID-dot-devshell.appspot.com where port was the port requested for the web preview feature and ID was a unique ID being assigned to each Cloud Shell user so that only the respective users could access their domains. I quickly confirmed this assumption by requesting a web preview for port 9000, and this time was taken to a domain of the form 9000-dot-123456789-dot-devshell.appspot.com.

Eventually I decided to test this feature with a server running. Not wanting to serve any real web pages I just started netcat on port 8080 and activated the web preview again. As expected, the HTTP request arrived in netcat, which I answered with a HTTP response and my browser ended up showing the body of that response.

request-cookie.png

Figure 3: A request and response handled by the web preview feature, together with a leaked cookie.

Curios about which route that request took within the virtual machine I decided to analyze that in more detail. Analyzing the internals of the virtual machine provided to a Cloud Shell user is possible since a Cloud Shell user can escape from the Docker container to the underlying virtual machine, which is however not a security risk as each user gets assigned their own virtual machine. It turned out that the request seemed to be routed from the user- and port-specific endpoint ($PORT-dot-$ID-dot-devshell.appspot.com) through Google's internal network to the virtual machine associated to the specific user. There, an nginx-based gateway service would route the request to the specified port. The response from the respective service would then take the same route back to the user.

architecture.png

Figure 4: An overview of the assumed relevant part of the Cloud Shell service's architecture.

Having tried to understand the relevant part of the Cloud Shell service's architecture, I took another look at the request the proxy service had forwarded. I then realized that the HTTP headers in the received request also contained a cookie (devshell-proxy-session), for which I did not have any explanation yet. The gateway service did neither perform nor require any authentication since it was only connected to an internal network. I decided to check whether my browser had this cookie set and this was indeed the case.

cookie.png

Figure 5: A devshell-proxy-session cookie stored in the browser.

During further analysis it turned out that initial requests to endpoints of the form $PORT-dot-$ID-dot-devshell.appspot.com would be sent through Google's OAuth process and eventually the described cookie would be set as an authentication token for the domain if the logged in user happened to be associated with Cloud Shell ID. Next, I wanted to check for how long the cookie would be valid. I requested a new Cloud Shell instance and tried to access the proxy endpoint using the old cookie which still worked. Next I switched to an untrusted custom image, and determined that the cookie was still valid. Even after going back to the trusted default image the cookie could still be used to access the endpoint.

Vulnerability and exploitability

The problem which can be derived from the above description is that a potential attacker could trick a Cloud Shell user into loading their custom Docker image. Even if that image were to be loaded as an untrusted environment the underlying virtual machine could still be accessed through the proxy service's endpoint $PORT-dot-$ID-dot-devshell.appspot.com by the legitimate Cloud Shell user. Since a potential attacker would control the untrusted environment's image that implies they would be able to also read any network traffic on the underlying virtual machine, and therefore could extract any devshell-proxy-session cookie routed through the virtual machine's gateway service. Since these cookies would not expire, a potential attacker could in this way intercept them while the untrusted environment was loaded and use them to access the new virtual machine of the same Cloud Shell user once they had switched back to the trusted default environment.

Impact

At first, the impact of this vulnerability might not seem very high, since it would only allow a potential attacker to access web servers a Cloud Shell user started for testing purposes. Further, the attacker would have to guess which port a user would be using for their web services which makes an attack seem implausible.

However, there is a special web service which is being made available on every Cloud Shell's virtual machine, even when using a custom environment. The Cloud Shell's editor, which is based on the Theia IDE, runs on port 970, can be accessed at 970-dot-$ID-dot-devshell.appspot.com and is also routed through the virtual machine's gateway service. This editor, being an IDE, can also be used to spawn a shell on the underlying container, which allows complete access to the virtual machine and thus, when running in a trusted environment, also to the Cloud Shell user's GCP resources. Consequently, this gives a potential attacker a port known to be accessible and useful on any Cloud Shell virtual machine.

Attack scenario

A potential attack scenario for this vulnerability consists of the following steps:

  1. The attacker creates and publishes a custom Docker image. The image contains code, which upon startup manipulates the gateway service to automatically extract cookies from received requests. Such requests, in an attempt to preload the editor, are automatically sent to the editor's service when a Cloud Shell user initially connects to a new Cloud Shell instance. These requests also contain the user-specific endpoint address, which is necessary in addition to the cookie for making a connection to the endpoint. Extracted cookies and domains are forwarded to the attacker by the malicious image.
  2. A Cloud Shell user decides to load the provided image as an untrusted environment. The malicious code gets executed, forwarding cookies and domains to the attacker.
  3. The attacker can periodically try to connect to the received domains using the received cookies. Once a connection is established it is possible to check if the user has already switched back to the trusted default environment.

Fix

Once a proof of concept had been created and submitted to Google the vulnerability was fixed. That was done by having the proxy service strip the relevant cookie from requests before they are forwarded to the virtual machine. This can be easily shown by repeating the initially described experiment. The Cloud Shell environment being targeted at developers, other cookies still continue to be forwarded through the proxy, as these might be required by the web services being tested by the Cloud Shell user and cannot cause any harm.

request.png

Figure 6: A request forwarded by the Cloud Shell's proxy service no longer containing a devshell-proxy-session cookie.

Take away message

Any system should be provided as little information as possible but as much information as necessary to operate. In the described case, this principle was implemented correctly on one side of the connection, as the cookie was set with the Secure and HttpOnly flags, thus eliminating Cross-Site Scripting (XSS) attacks attempting to steal the cookie. On the other side of the connection, the principle however was violated by continuing to forward the relevant cookie past the instance it was targeted at, which is the Cloud Shell proxy service. The combination of the server side being temporarily accessible by an attacker, a cookie which would not expire across the runtime of virtual machine instances and the permanent user-specific domain name endpoints made this issue exploitable.