Inspect code deployed on App Engine

With Google Cloud Debugger sunsetted [1], inspecting what files were deployed on a given version of a given App Engine project is harder than before.

We are going to take advantage of the build and deploy system used by App Engine to dive into our app.

Downloading an App Engine version

You need to know that any time you use gcloud app deploy to deploy your application from your computer or CI system to App Engine, the gcloud CLI packages all your files and sends them to Cloud Storage.
From there, a lot of magic involving Cloud Build happens under the hood: your files are first built into a container image using Buildpacks [2], then this image is turned into a new version of your App Engine service, and finally the version starts an instance, App Engine migrates traffic and ta-dam! your new code is now reachable.

The interesting bit for us today is the container image that is being built with your files stays in the project registry and you can find it years later under Container Registry.
Open the repository “app-engine-tmp” then follow the path app/ttl-2h/default/buildpack-app until you reach all previous versions of your App Engine service.
Filter the table by Uploaded date to find what specific image you want.

From here you can copy the full image path and pull it somewhere we are going to inspect it, locally or in Cloud Shell for instance:

docker pull eu.gcr.io/${PROJECT_ID}/app-engine-tmp/app/ttl-2h/default/buildpack-app@sha256:f342dd7…

Now tag that image with a simpler name for next steps:

docker tag $_ my-gae-app

Inspecting the code

Now that we have downloaded the container image for a given App Engine version, we still need to find a way to inspect the code because it’s not exactly in the same form than previously when we used gcloud to deploy it to App Engine.
Here are a few options:

1. Start a container from the image and browse the file;
2. Use dive [3] to inspect each layer from the image;
3. Extract the whole image into a TAR archive, after all container images are nothing but big and strange zip files;

By starting the container

Simply starting the a container from the image will allow us to browse the files and use any *nix command like cat and such.
Your application code is available under the /workspace folder.

docker run -it --rm my-gae-app sh
cd /workspace

By using dive

Dive is «a tool for exploring a docker image, layer contents, and discovering ways to shrink the size of your Docker/OCI image».

Now remember, what we pulled from the Container Registry is an App Engine version container image.
What this means is that before running our code onto a GAE Service, Google had to bake a container image specifying all runtime and environment parameters ready to receive any user-uploaded code and run it on App Engine.

So by using dive on this container image, we also get to inspect all these layers as an added bonus.

alias dive="docker run -ti --rm  -v /var/run/docker.sock:/var/run/docker.sock wagoodman/dive"
dive my-gae-app

Now you can navigate between layers on the left and file system on the right, your application code will be available on one of the latest layers (bottom of the list).

With this approach you can’t open the files only list them, but it can be a quick way to check if one of your files has been included in the build for instance. (see gcloudignore)

By extracting the image into an archive

This last method will extract all of the container image filesystem into a TAR archive on your machine.
This may be a long process, and will definitely use a lot of disk space so only use this method sparsely.

docker image save my-gae-app > extract.tar
tar -xvf extract.tar