Introduction
Google Cloud Shell is a product of the Google Cloud Platform (GCP) that provides a number of interesting features, such
as an interactive shell environment and an integrated code editor, with the intent of making experimenting and
administration of GCP products easier while requiring very little setup. A particularly interesting feature is the
option to customize the used Cloud Shell environment, for example by installing additional toolkits. This can be done
by creating a custom Docker image which uses the default environment's Docker image as base image. Any Cloud Shell user
can make use of such custom Cloud Shell images when they are pushed to the Google Cloud Registry publicly, simply by
adding the image name as an URL parameter when connecting to the Cloud Shell environment, for example
https://ssh.cloud.google.com/cloudshell/editor?cloudshell_image=gcr.io/custom-project/custom-image
.
In an effort to protect users from loading and executing malicious code, whenever a Cloud Shell user requests a custom environment in such a way they will be presented with the following two choices:
- Trust the image, so that the user's virtual machine is automatically authorized to access the Cloud Shell user's GCP resources.
- Do not trust the image, thus not making such credentials available within the environment.
Figure 1: A prompt shown when a user requests loading a custom Cloud Shell image.
As part of the Google Vulnerability Reward Program (VRP) I discovered an information leakage vulnerability that would allow the creator of a malicious custom image, which when loaded by another Cloud Shell user in an untrusted way, to later on regain access to that user's Cloud Shell environment once that user had switched back to using the default environment in a trusted way.
Discovery
I was trying to understand how Google Cloud Shell worked so I tried out the different features. One of those features was web preview, which allows web applications running inside Google Cloud Shell's virtual machine to be previewed in the user's browser. To test the feature I just clicked on the button to preview the default port 8080. Not having started any server on that port I was presented with an adequate error message.
Figure 2: A first test of the web preview feature.
I noticed that the domain I was taken to was of the form 8080-dot-123456789-dot-devshell.appspot.com
. Having
previously analyzed how other parts of Cloud Shell worked it appeared to me that the domain would be of the form
$PORT-dot-$ID-dot-devshell.appspot.com
where port was the port requested for the web preview feature and ID was a
unique ID being assigned to each Cloud Shell user so that only the respective users could access their domains. I
quickly confirmed this assumption by requesting a web preview for port 9000, and this time was taken to a domain of the
form 9000-dot-123456789-dot-devshell.appspot.com
.
Eventually I decided to test this feature with a server running. Not wanting to serve any real web pages I just started netcat on port 8080 and activated the web preview again. As expected, the HTTP request arrived in netcat, which I answered with a HTTP response and my browser ended up showing the body of that response.
Figure 3: A request and response handled by the web preview feature, together with a leaked cookie.
Curios about which route that request took within the virtual machine I decided to analyze that in more detail.
Analyzing the internals of the virtual machine provided to a Cloud Shell user is possible since a Cloud Shell user can
escape from the Docker container to the underlying virtual machine, which is however not a security risk as each user
gets assigned their own virtual machine. It turned out that the request seemed to be routed from the user- and
port-specific endpoint ($PORT-dot-$ID-dot-devshell.appspot.com
) through Google's internal network to the virtual
machine associated to the specific user. There, an nginx-based gateway service would route the request to the specified
port. The response from the respective service would then take the same route back to the user.
Figure 4: An overview of the assumed relevant part of the Cloud Shell service's architecture.
Having tried to understand the relevant part of the Cloud Shell service's architecture, I took another look at the
request the proxy service had forwarded. I then realized that the HTTP headers in the received request also contained a
cookie (devshell-proxy-session
), for which I did not have any explanation yet. The gateway service did neither
perform nor require any authentication since it was only connected to an internal network. I decided to check whether
my browser had this cookie set and this was indeed the case.
Figure 5: A devshell-proxy-session
cookie stored in the browser.
During further analysis it turned out that initial requests to endpoints of the form
$PORT-dot-$ID-dot-devshell.appspot.com
would be sent through Google's OAuth process and eventually the described
cookie would be set as an authentication token for the domain if the logged in user happened to be associated with Cloud
Shell ID. Next, I wanted to check for how long the cookie would be valid. I requested a new Cloud Shell instance and
tried to access the proxy endpoint using the old cookie which still worked. Next I switched to an untrusted custom
image, and determined that the cookie was still valid. Even after going back to the trusted default image the cookie
could still be used to access the endpoint.
Vulnerability and exploitability
The problem which can be derived from the above description is that a potential attacker could trick a Cloud Shell user
into loading their custom Docker image. Even if that image were to be loaded as an untrusted environment the underlying
virtual machine could still be accessed through the proxy service's endpoint $PORT-dot-$ID-dot-devshell.appspot.com
by
the legitimate Cloud Shell user. Since a potential attacker would control the untrusted environment's image that
implies they would be able to also read any network traffic on the underlying virtual machine, and therefore could
extract any devshell-proxy-session
cookie routed through the virtual machine's gateway service. Since these cookies
would not expire, a potential attacker could in this way intercept them while the untrusted environment was loaded and
use them to access the new virtual machine of the same Cloud Shell user once they had switched back to the trusted
default environment.
Impact
At first, the impact of this vulnerability might not seem very high, since it would only allow a potential attacker to access web servers a Cloud Shell user started for testing purposes. Further, the attacker would have to guess which port a user would be using for their web services which makes an attack seem implausible.
However, there is a special web service which is being made available on every Cloud Shell's virtual machine, even when
using a custom environment. The Cloud Shell's editor, which is based on the Theia IDE, runs on port 970, can be
accessed at 970-dot-$ID-dot-devshell.appspot.com
and is also routed through the virtual machine's gateway service.
This editor, being an IDE, can also be used to spawn a shell on the underlying container, which allows complete access
to the virtual machine and thus, when running in a trusted environment, also to the Cloud Shell user's GCP resources.
Consequently, this gives a potential attacker a port known to be accessible and useful on any Cloud Shell virtual
machine.
Attack scenario
A potential attack scenario for this vulnerability consists of the following steps:
- The attacker creates and publishes a custom Docker image. The image contains code, which upon startup manipulates the gateway service to automatically extract cookies from received requests. Such requests, in an attempt to preload the editor, are automatically sent to the editor's service when a Cloud Shell user initially connects to a new Cloud Shell instance. These requests also contain the user-specific endpoint address, which is necessary in addition to the cookie for making a connection to the endpoint. Extracted cookies and domains are forwarded to the attacker by the malicious image.
- A Cloud Shell user decides to load the provided image as an untrusted environment. The malicious code gets executed, forwarding cookies and domains to the attacker.
- The attacker can periodically try to connect to the received domains using the received cookies. Once a connection is established it is possible to check if the user has already switched back to the trusted default environment.
Fix
Once a proof of concept had been created and submitted to Google the vulnerability was fixed. That was done by having the proxy service strip the relevant cookie from requests before they are forwarded to the virtual machine. This can be easily shown by repeating the initially described experiment. The Cloud Shell environment being targeted at developers, other cookies still continue to be forwarded through the proxy, as these might be required by the web services being tested by the Cloud Shell user and cannot cause any harm.
Figure 6: A request forwarded by the Cloud Shell's proxy service no longer containing a devshell-proxy-session
cookie.
Take away message
Any system should be provided as little information as possible but as much information as necessary to operate. In the
described case, this principle was implemented correctly on one side of the connection, as the cookie was set with the
Secure
and HttpOnly
flags, thus eliminating Cross-Site Scripting (XSS) attacks attempting to steal the cookie. On
the other side of the connection, the principle however was violated by continuing to forward the relevant cookie past
the instance it was targeted at, which is the Cloud Shell proxy service. The combination of the server side being
temporarily accessible by an attacker, a cookie which would not expire across the runtime of virtual machine instances
and the permanent user-specific domain name endpoints made this issue exploitable.
Further information
Google Cloud Shell architecture and further vulnerabilities in Google Cloud Shell
- a video by @LiveOverflow about the architecture and a vulnerability found by @wtm_offensi
- architecture overview and four more vulnerabilities by @wtm_offensi
- information about escaping from Docker container to virtual machine by @SpenGietz
- unrelated vulnerabilites in Google Cloud Shell