GitLab CI for edox-ops

This guide explains how continuous integration works when the repository is hosted on GitLab: which files define the pipeline, how to run it, and why Docker (including Docker-in-Docker) is required for integration tests.

Pipeline definition: .gitlab-ci.yml

First-time setup (create project, push code, open the first pipeline): gitlab-getting-started.md.

SSH keys (multiple keys, Host gitlab-ai, remotes): gitlab-ssh-keys.md.

Agent access to pipeline logs (glab, API, paste): gitlab-ci-agent-access.md.

Python package releases (build, TestPyPI, PyPI, GitLab PyPI): package-release.md.

Pipeline overview

On each merge request, push to the default branch, push to develop (the working branch), or tag, GitLab runs these jobs in order:

Working branch (develop)

Day-to-day commits go on develop, not on master. GitLab CI runs on every push to develop. When the pipeline is green, fast-forward master to develop — no merge commit. Do not update master while CI is failing.

git checkout develop
# … commit …
git push gitlab develop
# After pipeline success (fast-forward only):
git checkout master && git pull gitlab master
git merge --ff-only gitlab/develop
git push gitlab master

If git merge --ff-only refuses, rebase develop onto master, push develop, wait for CI, then retry:

git checkout develop && git fetch gitlab
git rebase gitlab/master
git push --force-with-lease gitlab develop
# CI green again, then merge --ff-only as above

To catch up develop with new commits on master, use git rebase gitlab/master, not git merge.

Open a merge request from develop → master if you want review; set the project to Fast-forward merge so GitLab does not create a merge commit.

When CI fails on develop

Do not add a separate fix(...) commit on top. Fold the correction into the commit that introduced the problem so history stays one commit per logical step.

Situation	What to do
The failing commit is HEAD	Change files, then git commit --amend (same message or reword).
The problem is an earlier commit on develop	git commit --fixup <bad-commit-hash>, then git rebase -i --autosquash <parent-of-bad-commit>.
Several fixups for one step	Multiple --fixup commits, one autosquash rebase.

After rewriting history on develop, push with lease (safe force on the working branch only):

git push --force-with-lease gitlab develop

Never force-push master. Update master only after the develop pipeline is green, and only with git merge --ff-only (or GitLab fast-forward merge).

Stage	Job	What it runs
lint	ruff	ruff check and ruff format --check
lint	audit:python	pip-audit on runtime dependencies (pip install -e . only)
lint	audit:website	npm audit --audit-level=critical in website/ (Docusaurus lockfile)
test	unit	pytest with coverage on edox_ops (fails below 80% per pyproject.toml); Cobertura report for GitLab UI
integration	integration:ubuntu	Docker build + Ubuntu 24.04 systemd integration harness
integration	integration:debian	Same harness on Debian 12 (BASE_IMAGE=debian:12)
integration	integration:arm64	Raspberry Pi proxy: Debian 12 smoke on saas-linux-small-arm64 (native ARM)

On master (default branch), after ruff and unit:

Stage	Job	What it runs
build	docs:build	assemble_docs.py (Sphinx -W), verify_docs_build.py; artifact website/build/
pages	pages	Copy build to public/ → esysdox-ops.org (production environment)
pages	docs:smoke	Curl key URLs on esysdox-ops.org after pages

On develop, docs:build and pages run automatically after unit (pages does not block on integration). The build uses DOCS_SITE_URL=https://esysdox-ops-d340b4.gitlab.io and publishes preview docs to esysdox-ops-d340b4.gitlab.io without replacing the production site on esysdox-ops.org (custom domain stays on master).

On version tags (vX.Y.Z, matching src/edox_ops/__version__), the automated release pipeline runs:

Stage	Job	What it runs
build	package:build	verify_release_version.py, python -m build, twine check; artifact dist/
release	package:smoke	Install dist/*.whl in a fresh venv; edox-ops --version + --help
publish	publish:gitlab	Twine upload to GitLab PyPI (CI_JOB_TOKEN)
publish	publish:testpypi	TestPyPI upload (automatic after package:smoke)
publish	package:smoke-testpypi	pip install from TestPyPI (+ PyPI for deps)
publish	publish:pypi	PyPI upload (manual — only maintainer click)
publish	package:smoke-pypi	pip install from PyPI

See release-pipeline.md for the maintainer checklist and failure recovery.

flowchart LR
  trigger[Push or merge request] --> lint[ruff + pip-audit + npm audit]
  lint --> unit[pytest + coverage]
  unit --> integration[Docker DinD job]
  integration --> script[run_docker.sh]
  script --> nested[Privileged Ubuntu container with systemd]
  nested --> tests[bootstrap, nginx, project hosting]

• lint (ruff, audit:python) and unit use the python:3.11-bookworm image. audit:website uses node:20-bookworm and npm ci in website/. Any normal GitLab runner that can pull container images is enough. audit:python needs outbound HTTPS to PyPI advisory data (same as pip install).

Coverage in the GitLab UI

The unit job publishes coverage in two ways:

1. Pipeline coverage % — parsed from the pytest TOTAL … % line (coverage: regex in .gitlab-ci.yml). Shown on the pipeline list and latest pipeline widget when the job succeeds.
2. Merge request coverage report — Cobertura coverage.xml uploaded as a coverage_report artifact. On MRs, open Changes to see line coverage (green/red gutters) when the report is present.

Download the raw report from the unit job artifacts if needed.

• integration uses the Docker-in-Docker (DinD) pattern: a Docker client in the job talks to a Docker daemon started as a CI service, then runs the same script as on a Linux workstation or GitHub Actions.

How to run CI on GitLab

1. Push the repository

Create a GitLab project, add a remote, and push:

git remote add gitlab git@gitlab.com:your-group/esysdox-ops.git
git push -u gitlab master

Use your instance URL and default branch name (main vs master) as appropriate.

2. Ensure a runner is available

• GitLab.com: Open the project → Settings → CI/CD → Runners. Shared runners are usually enabled for new projects.
• Self-managed GitLab: Install GitLab Runner on a Linux host with Docker, then register it to the project (see Self-hosted runner below).

3. Watch the pipeline

After a push or merge request, open Build → Pipelines. Click a pipeline to see job logs for ruff, unit, and integration.

No CI/CD variables are required for lint or unit. Integration needs a runner that supports Docker-in-Docker and privileged nested containers (see below).

Docker on GitLab — why and how

Integration tests do not execute bootstrap or nginx directly on the runner VM. They mirror production by using an Ubuntu 24.04 image with systemd inside Docker, exactly like local test.ps1 / run_docker.sh.

The CI job must therefore:

1. Build an image from tests/integration/Dockerfile (default ubuntu:24.04; set BASE_IMAGE=debian:12 for Debian).
2. Run a nested container with --privileged and cgroup mounts (see tests/integration/run_docker.sh).
3. Exec /usr/local/bin/run-integration inside that container.

That is not the same as “run tests in a single CI image.” The harness starts a second, long-lived container where systemctl manages nginx and gitlab-runner.

Raspberry Pi proxy (integration:arm64)

integration:arm64 runs a smoke script (run_integration_arm64_smoke.sh), not the full run_integration.sh. It validates Debian-family distro detection, bootstrap, doctor (host:platform), and prepare-ci-host Docker install on ARM64.

The job is tagged saas-linux-small-arm64 so it runs on GitLab’s native ARM64 hosted runners (not QEMU on amd64). DinD uses --mtu=1400 on ARM to avoid network issues (gitlab#473739).

Local fallback on an amd64 machine:

BASE_IMAGE=debian:12 \
DOCKER_PLATFORM=linux/arm64 \
INTEGRATION_SCRIPT=/usr/local/bin/run-integration-arm64-smoke \
bash tests/integration/run_docker.sh

Docker-in-Docker (DinD) in .gitlab-ci.yml

The integration job is configured as:

Piece	Image / setting	Role
Job image	docker:27-cli	Provides the docker CLI
Service	docker:27-dind	Runs the Docker daemon the CLI talks to
Variables	DOCKER_HOST, DOCKER_TLS_CERTDIR	Connect client to the DinD service
Script	bash tests/integration/run_docker.sh	Build, run privileged container, test

Flow inside the job:

GitLab Runner
  └── CI job container (docker:27-cli)
        └── DinD service (docker daemon)
              └── docker build  →  edox-ops-integration image
              └── docker run --privileged  →  systemd + tests

What the runner must support

Requirement	Reason
Docker executor (or shell + Docker on the host)	The job must invoke docker build and docker run
Docker-in-Docker service	Supplies a daemon inside the job; the runner host’s Docker is not used by default
Privileged nested containers	run_docker.sh uses docker run --privileged and cgroup mounts for systemd

GitLab.com shared runners

GitLab.com Linux shared runners typically provide Docker and DinD. The integration job often works with no project-specific configuration. If you see errors about --privileged or cgroups, register a self-hosted runner with privileged = true.

Self-hosted runner (recommended for production)

On a fresh Ubuntu 22.04/24.04 VPS, use edox-ops to install Docker, GitLab Runner, and privileged Docker-in-Docker settings required by the integration job:

# After edox-ops is installed on the host (pip, GitLab PyPI, or git checkout)
sudo edox-ops gitlab-runner prepare-ci-host --yes
sudo edox-ops gitlab-runner register \
  --profile ci \
  --url https://gitlab.com \
  --token "$EDOX_RUNNER_TOKEN"
edox-ops gitlab-runner status
edox-ops doctor

prepare-ci-host runs these steps (logged to /var/log/edox-ops/gitlab-runner-ci-host.log):

Step	What it does
docker.install	Docker Engine from the official apt repository
runner.user.create / runner.install / runner.service.install	Same as gitlab-runner install
runner.docker.access	Adds gitlab-runner to the docker group
runner.docker.config	Patches config.toml for privileged DinD when runners exist

register --profile ci sets:

Setting	Value
Executor	docker
Default image	python:3.11-bookworm
Tags	docker-privileged
Privileged	true
Volumes	/cache, /certs/client

Create the registration token under Settings → CI/CD → Runners → New project runner, or export EDOX_RUNNER_TOKEN in the shell (never commit it).

Optional: dedicate a runner to integration with tags. In .gitlab-ci.yml:

integration:
  tags: [docker-privileged]

The ci profile already registers docker-privileged; add the YAML tag only when you want jobs to run exclusively on self-hosted runners.

Manual alternative (without edox-ops): install Docker and GitLab Runner, register with docker executor, then edit /etc/gitlab-runner/config.toml:

[[runners]]
  [runners.docker]
    privileged = true
    volumes = ["/certs/client", "/cache"]

Restart with sudo gitlab-runner restart.

Shell executor alternative

If the runner machine is Ubuntu and Docker runs on the host, register a shell executor, grant the gitlab-runner user access to docker, and run bash tests/integration/run_docker.sh in the job without DinD image/services. DinD is still preferred because it isolates CI from the host daemon.

Run the same checks locally

pip install -e ".[dev]"
ruff check src tests scripts/pre_commit scripts/gitlint_rules.py
ruff format --check src tests scripts/pre_commit scripts/gitlint_rules.py
python -m pytest -q --cov=edox_ops
pip install pip-audit && pip install -e . && pip-audit --desc on .
bash tests/integration/run_docker.sh   # Linux

On Windows, use .\test.ps1 instead of run_docker.sh (same Docker workflow).

Troubleshooting

Cannot connect to the Docker daemon

• The pipeline waits up to 30 seconds for DinD in before_script. If it still fails, check that the docker:27-dind service started.
• Self-hosted: confirm privileged = true and volumes includes /certs/client for TLS to DinD.

docker run --privileged or cgroup errors

• Set privileged = true under [runners.docker] in config.toml.
• Avoid runners that block privileged nested containers.

Integration job stuck on systemctl

• Same as local Docker: the nested container needs --privileged and the cgroup volume from run_docker.sh. The runner must allow that.

Run lint/unit only (temporary)

Disable or set integration to when: manual while debugging runners. Do not rely on that for long-term merges to the default branch.

GitHub Actions (optional)

If the repository is also mirrored to GitHub, .github/workflows/ci.yml runs the same checks. When the canonical host is GitLab, treat this guide and .gitlab-ci.yml as the source of truth; both CI configs may coexist.