runner/examples/lxc-systemd
Earl Warren b79578d98e
feat: forgejo-runner-service.sh safeguard to avoid overheating
re-creating the LXC container from scratch is expensive. When
rebooting or when multiple units start at the same time, it may cause
an execessive load.

Use a global lock to guard this operation so they happen in sequence
and not in parallel. They typically take around one minute to complete
which means that in the event of a reboot, it will take around 1
minutes * the number of runners for the unlucky one to start.

During this interval workflows will have to wait.
2025-01-27 14:23:58 +01:00
..
README.md feat: systemd unit example for managing LXC containers (#451) 2025-01-25 12:24:31 +00:00
forgejo-runner-service.sh feat: forgejo-runner-service.sh safeguard to avoid overheating 2025-01-27 14:23:58 +01:00

README.md

forgejo-runner-service.sh installs a Forgejo runner within an LXC container and runs it from a systemd service.

Quickstart

  • Install: sudo wget -O /usr/local/bin/forgejo-runner-service.sh https://code.forgejo.org/forgejo/runner/raw/branch/main/examples/lxc-systemd/forgejo-runner-service.sh && sudo chmod +x /usr/local/bin/forgejo-runner-service.sh
  • Obtain a runner registration token ($TOKEN)
  • Choose a serial number that is not already in use in /etc/forgejo-runner
  • Create a runner INPUTS_SERIAL=30 INPUTS_TOKEN=$TOKEN INPUTS_FORGEJO=https://code.forgejo.org forgejo-runner-service.sh
  • Start systemctl enable --now forgejo-runner@$INPUTS_SERIAL
  • Monitor with:
    • systemctl status forgejo-runner@$INPUTS_SERIAL
    • tail --follow=name /var/log/forgejo-runner/$INPUTS_SERIAL.log

Installation or upgrade

  • sudo wget -O /usr/local/bin/forgejo-runner-service.sh https://code.forgejo.org/forgejo/runner/raw/branch/main/examples/lxc-systemd/forgejo-runner-service.sh && sudo chmod +x /usr/local/bin/forgejo-runner-service.sh

Description

  • Each runner is assigned a unique serial number ($INPUTS_SERIAL)
  • The configuration is in /etc/forgejo-runner/$INPUTS_SERIAL
  • The environment variables are in /etc/forgejo-runner/$INPUTS_SERIAL/env
  • The cache is in /var/lib/forgejo-runner/runner-$INPUTS_SERIAL
  • The systemd service unit is forgejo-runner@$INPUTS_SERIAL
  • The logs of the runner daemon are in /var/log/forgejo-runner/$INPUTS_SERIAL.log

How it works

  • Creating a runner (for instance with INPUTS_SERIAL=30 INPUTS_TOKEN=$TOKEN INPUTS_FORGEJO=https://code.forgejo.org forgejo-runner-service.sh) will:
    • use $INPUTS_TOKEN to register on $INPUTS_FORGEJO and save the result in the /etc/forgejo-runner/$INPUTS_SERIAL/.runner file
    • generate a default configuration file in the /etc/forgejo-runner/$INPUTS_SERIAL/config.yml file which can then be manually edited
  • Each runner is launched in a dedicated LXC container named runner-$INPUTS_SERIAL-lxc with the following bind mounts:
    • /etc/forgejo-runner/$INPUTS_SERIAL
    • /var/lib/forgejo-runner/runner-$INPUTS_SERIAL/.cache/actcache
  • systemctl start forgejo-runner@$INPUTS_SERIAL will do the following when it starts and every $INPUTS_LIFETIME interval after that:
    • attempt to gracefully stop (SIGTERM) the runner, waiting for all jobs to complete
    • forcibly kill the runner if it does not stop within 6h
    • shutdown the LXC container and delete it (the volumes bind mounted are preserved)
    • create a brand new LXC container (with the specified $INPUTS_LXC_CONFIG)
    • install and run a Forgejo runner daemon in the LXC container using /etc/forgejo-runner/$INPUTS_SERIAL/config.yml
    • redirect the output of the runner to /var/log/forgejo-runner/$INPUTS_SERIAL.log
  • systemctl stop forgejo-runner@$INPUTS_SERIAL will stop the runner but keep the LXC container running

Creation

The creation of a new runner is driven by the following environment variables:

  • INPUTS_SERIAL: unique number in the range [10-100] (check /etc/forgejo-runner)
  • INPUTS_TOKEN: a runner registration token obtained from the web UI
  • INPUTS_FORGEJO: the Forgejo instance from which INPUTS_TOKEN was obtained (e.g. https://code.forgejo.org)
  • INPUTS_RUNNER_VERSION: the version of the Forgejo runner as found in https://code.forgejo.org/forgejo/runner/releases (e.g. 6.2.0)
  • INPUTS_LXC_CONFIG: the value of the --config argument of lxc-helpers used when creating the LXC container for the runner (e.g. docker)
  • INPUTS_LIFETIME: the LXC container is re-created when its lifetime expires (e.g. 7d)

Hacking

  • An existing LXC configuration will not be modified. If lxc-ls exists, it is assumed that LXC is configured and ready to be used.
  • Migrating an existing runner:
    serial=10
    mkdir /etc/forgejo-runner/$serial
    cp .runner config.yml /etc/forgejo-runner/$serial
    INPUTS_SERIAL=$serial INPUTS_FORGEJO=https://code.forgejo.org forgejo-runner-service.sh
    systemctl status forgejo-runner@$serial
    
  • Set debug by adding VERBOSE=true in /etc/forgejo-runner/$INPUTS_SERIAL/env