Ephemeral (single use) runner registrations

[MatisseHack]

Describe the bug
When starting a self hosted runner with ./run.cmd --once, the runner sometimes accepts a second job before shutting down, which causes that second job to fail with the message:

The runner: [runner-name] lost communication with the server. Verify the machine is running and has a healthy network connection.

This looks like the same issue recently fixed here: microsoft/azure-pipelines-agent#2728

To Reproduce
Steps to reproduce the behavior:

1. Create a repo, enable GitHub Actions, and add a new workflow

2. Configure a new runner on your machine

3. Run the runner with ./run.cmd --once

4. Queue two runs of your workflow

5. The first job will run and the runner will go offline

6. (Optionally) configure and start a second runner

7. The second job will time out after several minutes with the message:

   The runner: [runner-name] lost communication with the server. Verify the machine is running and has a healthy network connection.
   

   (where [runner-name] is the name of the first runner)

8. Also: trying to remove the first runner with the command ./config.cmd remove --token [token] will result in the following error until the second job times out:

   Failed: Removing runner from the server
   Runner "[runner-name]" is running a job for pool "Default"
   

Expected behavior
The second job should run on (and wait for) any new runner that comes online rather than try to run as a second job on the, now offline, original runner.

Runner Version and Platform

2.262.1 on Windows

Runner and Worker's Diagnostic Logs

_diag.zip

[TingluoHuang]

@MatisseHack
How did you find the flag? 😆
the run once flag --once is not in the official actions doc and better not be used in production, there is a known issue that we haven't fix. 😄

[bryanmacfarlane]

Yeah, this is why it's not part of the documentation or command line help. Because the partially implemented code doesn't work and has races so we didn't ship it.

We have work on the backlog to register a runner with the service as ephemeral. It has to be a service side feature so the service is cooperative and doesn't assign it another job in the window when the runner or outside / in automation decides to tear it down.

[MatisseHack]

I've used the --once flag extensively with Azure Pipelines to create an automatically scaling pool of single use agents. I'm trying to convert that system to work with GitHub Actions. I was pleased to find that the flag still worked (at least partially)!

Makes sens why it wasn't shipped 😄 . I look forward to using it when it's supported though!

[rclmenezes]

We solved this issue by adding a timeout to our ephemeral workers. The timeout will deregister and kill the worker.

#!/bin/bash -x
set -eux

/usr/bin/actions/register

# Automatically de-register in 30 min
echo "sudo /usr/bin/actions/deregister && sudo shutdown -h now" | at now + 30 minutes

sudo -i -u actions /home/actions/run.sh --once
/usr/bin/actions/deregister
shutdown -h now

Actions team: please support --once when you have a chance!

We currently have ~16x parallelization on our CI to keep builds under 20 minutes. When moving to self-hosted, we had the following options:

1. Have an autoscaling group that keeps 16 instances around at the minimum and deploy more at peak times (this is a huge waste of money, it's hard to scale out and we have no clue how to scale-in without affecting existing jobs).

2. Make a Github-hosted job that deploys spot instances to perform the other jobs! This is substantially cheaper, avoids queues at peak times and allows us to parallelize even more when we need to!

Unfortunately, the second strategy only works well with --once...

So please consider supporting --once officially! If --once stops working, we'll probably have to consider an alternative CI provider.

[emilyalbini]

We also use the --once flag as we need a fresh build environment for each build (on AArch64, so we can't use GitHub hosted runners), and while we haven't encountered this issue yet it'd be really nice to get proper support!

[rclmenezes]

Also @bryanmacfarlane - two more requests that would make ephermeral workers easier:

1. Have --once workers automatically de-register themselves on completion / timeout. Currently, if a worker errors out before cleanup code, you'll have a ton of zombie "offline" registered workers. I think this needs to be solved on the backend.

2. If no workers are registered with the proper labels, Github will error out the job immediately. This is a problem if you're spinning up ephemeral workers, because the job will error before they boot up.

Our hackish solution is to keep a registered worker that does no work:

It simply re-registers itself once a day to avoid the 30 day cleanup 😂

Thanks!

[nabilcode]

👋 there,

Do we have a date as to which the --once would be supported?

[zacker150]

Our hackish solution is to keep a registered worker that does no work:

It looks GitHub changed the runner service to error out the actions if there are no live runners. This hackish solution no longer works ☹️.

[rclmenezes]

This really sucks. It's the last nail in the coffin for ephemeral self-hosted workers.

We're going to switch to permanent self-hosted workers and then probably switch to another CI provider :(

[bryanmacfarlane]

@rclmenezes ack on #1 and #2. we're currently designing and working on it. The plan is exactly what you laid out, register the runner ephemeral with the backend service so the service auto cleans it up after the job is complete and the runner / container exits.