Caching CI on Ephemeral Runners (Without a Docker Daemon)

A previous post built flare-runner: a Cloudflare Worker that turns a workflow_job webhook into a one-shot Cloudflare Container running a single CI job, then throws the container away. One container per job, scale to zero, no standing runner box.

The catch with one-shot runners is the same thing that makes them nice: every job starts on a clean machine. A persistent runner accumulates a warm module cache, a warm compiler cache, and a local Docker layer cache between jobs. Throw the machine away after each job and all of that is gone. Naively, every build is a cold build.

The fix is to move the cache off the machine - into the GitHub Actions cache (plain HTTPS) for files, and into your container registry for image layers. Neither needs anything special on the runner. Here are two example workflows that show the whole pattern, plus the one mistake that quietly defeats it.

Example 1: build and test, with cache, no Docker

A compiled-language job spends most of its time in two places: downloading dependencies and compiling. Both are cacheable, and actions/cache works fine on a self-hosted ephemeral runner because it talks to the cache service over HTTPS - it does not care that the disk is fresh.

For Go, the two directories that matter are the module cache (~/go/pkg/mod) and the build cache (~/.cache/go-build). The second one is the interesting part: it is the compiler’s incremental cache, and restoring it is what turns a cold compile into a near-instant one.

jobs:
  test:
    runs-on: [self-hosted, cloudflare]
    steps:
      - uses: actions/checkout@v4

      - name: Restore Go build + module cache
        uses: actions/cache@v4
        with:
          path: |
            ~/.cache/go-build
            ~/go/pkg/mod
          key: go-${{ runner.os }}-${{ hashFiles('go.sum') }}-${{ github.sha }}
          restore-keys: |
            go-${{ runner.os }}-${{ hashFiles('go.sum') }}-
            go-${{ runner.os }}-

      - uses: actions/setup-go@v5
        with:
          go-version-file: go.mod
          cache: false        # we cache explicitly - see the gotcha below

      - run: go test ./... -count=1

On a real backend (about 100 packages) this took the compile from 136 seconds cold to 8 seconds warm - roughly a 17x difference, and almost all of the job’s wall-clock. -count=1 keeps the test results from being cached (the tests still actually run) while the build cache still does its job.

The gotcha that silently disables the cache

actions/setup-go has a built-in cache (cache: true, the default). It is keyed on your lockfile hash. That sounds right, but it has a property that quietly breaks the warm path: actions/cache (which it uses underneath) will not overwrite an entry that already exists for a key. Once an entry exists for your current go.sum, setup-go restores that same frozen entry on every later run and never saves the build cache your job just produced.

The symptom is maddening: the logs say “Cache restored successfully” on every run, yet the compile is slow every time. It restored a cache - just not one with your build objects in it.

The fix is the key in the example above: include ${{ github.sha }} so each commit writes a fresh entry, and use restore-keys to fall back to the most recent prefix match. Now every run restores the previous build cache and saves an updated one. Disable setup-go’s own cache (cache: false) so the two do not fight over the same directories.

The same shape works for other ecosystems - cache ~/.npm or the pnpm store for Node, ~/.cache/pip (or the wheel cache) for Python. The principle is identical: name the directories, key them so they refresh, let the cache service ship them over HTTPS.

Example 2: build an image, with cache, still no Docker

Cloudflare Containers cannot nest Docker, so image builds use buildah, which builds and pushes an OCI image with no daemon. Two settings turn a naive buildah build into a cached one.

The first is the storage driver. buildah’s default vfs driver makes a full copy of the filesystem for every layer - no copy-on-write. On a constrained runner a multi-stage build will simply run the disk out (“no space left on device”). Switch to overlay, which is copy-on-write, and the same build fits comfortably and runs faster.

The second is the layer cache. A fresh container has no local layer cache, so point buildah at a cache stored in the registry itself with --layers --cache-from / --cache-to. It pushes intermediate layers to a cache repository and pulls them back on the next run. (This needs buildah 1.24+; the older 1.23 that ships on Ubuntu 22.04 does not have --cache-to. Ubuntu 24.04 ships 1.33.)

jobs:
  image:
    runs-on: [self-hosted, cloudflare]
    steps:
      - uses: actions/checkout@v4

      - run: echo "$REGISTRY_PASSWORD" | buildah login -u "$REGISTRY_USER" --password-stdin "$REGISTRY"

      - name: Build + push with a registry layer cache
        run: |
          buildah build \
            --layers \
            --cache-from "$REGISTRY/myapp-buildcache" \
            --cache-to   "$REGISTRY/myapp-buildcache" \
            --isolation chroot --storage-driver overlay \
            -t "$REGISTRY/myapp:$GITHUB_SHA" .
          buildah push --storage-driver overlay "$REGISTRY/myapp:$GITHUB_SHA"

For a small image (a static binary copied onto a base, a config file or two) this cached the base and setup layers and took the build from 54 seconds cold to 35 warm. The win scales with how much of your Dockerfile is stable setup - apt-get, pip install, a base image with system libraries - because those are exactly the layers that hit the cache.

When the layer cache is a trap: compiling inside the image

There is one case where the registry layer cache looks like it should help and mostly does not: a multi-stage Dockerfile that compiles your code inside the build. The deps layers cache fine, but the RUN go build (or cargo build, or npm run build) layer is invalidated the moment your source changes - which is every commit. So that layer, the expensive one, runs cold every time. A heavy backend image built this way took about 400 seconds even with the cache warm, because the compile re-ran inside the layer on each commit.

The fix is to stop compiling inside the image. You already have a fast, cached compiler from Example 1 - so compile on the runner, then package the result into a thin runtime image:

      # 1. compile natively, using the restored Go build cache (~8s warm)
      - run: CGO_ENABLED=0 go build -o /tmp/myapp ./cmd/myapp

      # 2. package the binary into a tiny runtime image - no compiler, no cache miss
      - run: |
          d=$(mktemp -d); cp /tmp/myapp "$d/"
          cat > "$d/Dockerfile" <<'EOF'
          FROM alpine:3.23
          RUN apk add --no-cache ca-certificates
          COPY myapp /usr/local/bin/myapp
          ENTRYPOINT ["/usr/local/bin/myapp"]
          EOF
          buildah build --isolation chroot --storage-driver vfs -t "$REGISTRY/myapp:$GITHUB_SHA" "$d"
          buildah push --storage-driver vfs "$REGISTRY/myapp:$GITHUB_SHA"

Now the compile is the cached 8-second step, and the image build is a trivial copy. The same multi-stage build that ran ~400 seconds becomes a compile plus a package in well under a minute - and the runtime image is smaller, because it never contained a toolchain. (If your runtime is genuinely heavy - an image built on a large base with its own system packages - keep building its real Dockerfile with --storage-driver overlay and the registry cache; the thin trick is specifically for “static binary on a small base”.)

The pattern

Ephemeral runners do not remove the need for a cache; they move it off the machine. Files go to the Actions cache over HTTPS, image layers go to your registry. Once both caches live somewhere the next clean container can reach, a one-job-then-gone runner is as warm as a standing box - without leaving a box running, holding your secrets, between jobs.

Both examples, and the runner that runs them, are at github.com/andrius/flare-runner.