RBE with Firecracker MicroVMs

BuildBuddy Cloud has experimental support for running remote build actions within Firecracker microVMs, which are lightweight VMs that are optimized for fast startup time.

MicroVMs remove some of the restrictions imposed by the default Docker container-based Linux execution environment. In particular, microVMs can be used to run Docker, which means that actions run on BuildBuddy can spawn Docker containers in order to easily run apps that require lots of system dependencies, such as MySQL server.

BUILD configuration

Let's say we have a BUILD file like this:

BUILD

sh_test(
    name = "docker_test",
    srcs = ["docker_test.sh"],
)

And an executable shell script docker_test.sh that looks like this:

docker run --rm ubuntu:20.04 echo 'PASS' || exit 1

This test would normally fail when run using BuildBuddy's shared Linux executors, since running Docker inside RBE actions is only supported when using self-hosted executors.

But we can instead run this test using Docker-in-Firecracker by adding a few exec_properties to the test runner action:

BUILD

sh_test(
    name = "docker_test",
    srcs = ["docker_test.sh"],
    exec_properties = {
        # Tell BuildBuddy to run this test using a Firecracker microVM.
        "test.workload-isolation-type": "firecracker",
        # Tell BuildBuddy to ensure that the Docker daemon is started
        # inside the microVM before the test starts, so that we don't
        # have to worry about starting it ourselves.
        "test.init-dockerd": "true",
    },
)

note

The test. prefix on the exec_properties keys ensures that the properties are only applied to the action that actually runs the test, and not the actions which are building the test code. See execution groups for more info.

And that's it! This test now works on BuildBuddy's shared Linux executors.

However, it's a bit slow. On each action, a fresh microVM is created. This is normally fine, because microVMs start up quickly. But the Docker daemon also has to be re-initialized, which takes a few seconds. Worse yet, it will be started from an empty Docker image cache, meaning that any images used in the action will need to be downloaded and unpacked from scratch each time this action is executed.

Fortunately, we can mitigate both of these issues using runner recyling.

Preserving microVM state across actions

MicroVM state can be preserved across action runs by enabling the recycle-runner exec property:

BUILD

sh_test(
    name = "docker_test",
    srcs = ["docker_test.sh"],
    exec_properties = {
        "test.workload-isolation-type": "firecracker",
        "test.init-dockerd": "true",
        # Tell BuildBuddy to preserve the microVM state across test runs.
        "test.recycle-runner": "true",
    },
)

Then, subsequent runs of this test should be able to take advantage of a warm microVM, with Docker already up and running, and the ubuntu:20.04 image already cached. This is an optimization, and should not be relied upon for correctness. Finding a warm VM can fail for various reasons, like because it fell out of the cache or the machine that has it is fully loaded.

This works across test targets, as long as their platform properties, including exec_properties are identical. To create separate microVMs, add any unique execution property. You should do this to separate tests with different environments, so the VMs do not become too large. For example, if you have some tests that use only MySQL, and others that only use PostgreSQL, they should have different exec_properties.

A single microVM snapshot may be used by many test runs, and can survive for very long. To start a fresh instance, add an execution property or modify an existing one.

For the above reasons, we recommend using test.runner-recycling-key to encode dependency and version information. For example, a test that keeps MySQL running between runs should have "test.runner-recycling-key": "mysql@v1.2.3". When you switch to a new version of MySQL, you would update this to guarantee a new microVM.

tip

When using runner recycling, the entire microVM state is preserved—not just the disk contents. You can think of it as being put into "sleep mode" between actions.

This means that you can leave Docker containers and other processes running to be reused by subsequent actions, which is helpful for eliminating startup costs associated with heavyweight processes.

For example, instead of starting MySQL server with docker run mysql on each test action (which is quite slow), you can leave MySQL server running at the end of each test, and instead re-connect to that server during test setup of the next test. You can use docker container inspect to see if it the server is already running, and SQL queries like DROP DATABASE IF EXISTS followed by CREATE DATABASE to get a clean DB instance.

See BuildBuddy's test MySQL implementation for an example in Golang.

Using custom images

If you are using a custom RBE image, you do not need to do anything special to make it work with Firecracker. BuildBuddy will automatically convert your Docker image to a disk image compatible with Firecracker. The container-image execution property is specified using the same docker:// prefix, like: docker://some-registry.io/foo/bar.

To run Docker containers in your microVM (Docker-in-Firecracker), you will need to make sure your container image has Docker installed. BuildBuddy's default RBE image already has Docker installed, but when using a custom image, you may need to install Docker yourself.

See Install Docker Engine for the commands that you'll need to add to your Dockerfile in order to install Docker.

Once you've built your custom image, test that Docker is properly installed by running:

docker run --rm -it --privileged --name=docker-test your-image.io/foo dockerd --storage-driver=vfs

Then, once Docker is finished booting up, run the following command from another terminal. You should see "Hello world!" printed if Docker is properly installed:

docker exec -it docker-test docker run busybox echo "Hello world!"