Andrius Kairiukstis
← Blog

Running GitHub Actions on Debian Jessie and Stretch

Dec 15, 2023 4 minutes read
github-actionsdebiandockerci-cdnodejs

The team’s repositories include a number of services that still build and ship on Debian Jessie and Stretch. The new CI on GitHub Actions had to keep building them, even though the underlying Node.js compatibility between GitHub’s actions and the old glibc on those distributions was the central problem. This post is about the stack of CI images we built, and the workarounds we use to keep actions/checkout running on systems that GitHub no longer expects you to use.

Why Old Debian Was Still There

Several production services are pinned to Jessie or Stretch because the business has not yet funded the rebuild against a newer Debian release. Rebuilding the package on Bookworm is straightforward; what is not straightforward is the regression testing that follows, especially for the Kamailio and SEMS components that talk to the carrier network. Until the product side gives the budget for that work, the build pipelines have to keep producing Jessie and Stretch packages from the same source as the modern ones.

The Image Stack

The CI images are organised in three layers per Debian distribution. A small base image installs the distribution itself with the right sources.list and a stable apt configuration. A generic image on top of that adds the tools the workflows need: build-essential, devscripts, debhelper, lintian, javahelper, maven, the JFrog CLI, the GitHub CLI, and a usable Node.js. Per-service variants on top of the generic image add service-specific dependencies for components like Kamailio or SEMS.

The full set covers Jessie, Stretch, Buster, Bullseye and Bookworm. There is also a separate “runner image” used by the workflow shell itself, based on node:lts-alpine, which carries bash, git, gh, the JFrog CLI, jq, yq, xmllint, docker and docker-cli-compose. That image is what the workflow steps execute in when they are not building a package; the per-distribution images are what the package build steps run in.

The Node.js Problem

Most of GitHub’s first-party actions are JavaScript actions: GitHub ships a Node binary inside the runner and the action’s index.js runs against it. When actions/checkout@v3 was the current version it shipped Node 16. When actions/checkout@v4 released, it jumped to Node 20. Node 20’s binary requires a newer glibc than Debian Jessie has, so @v4 simply fails to start on a Jessie container. The workaround is to pin actions/checkout to @v3 in every workflow that targets Jessie or Stretch, with a comment explaining why so the next person to bump dependencies does not break the build.

Inside the per-distribution image, the system Node is also too old to run modern actions. The Jessie image installs Node 16 through NVM at build time, because Node 16 is the latest line that still runs against the glibc shipped with Jessie. Node 18 and Node 20 do not. The image is careful to install Node into a path that the workflow PATH picks up, because the action runner shells out to node and expects to find a working interpreter.

Smaller Workarounds

A handful of small fixes accumulated over the year:

  • The gh CLI is installed from the upstream apt repository, not from Debian, because Jessie’s archive stops well before gh existed.
  • libxml2-utils is added to the per-distribution images because the workflows parse pom.xml with xmllint.
  • default-mysql-client is added to every distribution because some workflow steps need to talk to a MySQL server.
  • The slim runner image carries GNU grep. The Alpine busybox grep does not support regular expressions in a way the workflows expect.
  • git config --global --add safe.directory . runs early in every workflow, because modern git refuses to operate on a working tree that is not owned by the current user, and the container runs as root.

Added Later

The Bookworm distribution was added to the stack in November, so the modern services now have a current Debian image to build against. The Jessie and Stretch images stay where they are and continue to use actions/checkout@v3. When upstream releases @v5 against Node 22 (or newer), the same pattern will apply: the modern services move forward, the legacy services stay pinned to whichever version still runs against their glibc, and the comment in the workflow file explains why.