Fedora Bootc & PXE CI/CD Pipeline

This project is a comprehensive infrastructure-as-code solution for generating immutable, container-native operating systems. It treats the entire OS as an OCI artifact, built via GitLab CI, configured via Ansible, and deployed via virtual disk or PXE/Network Boot.

The core philosophy is strict immutability: configuration management (Ansible) runs exclusively during the build phase. The resulting image is deployed atomically, with no post-provisioning drift.

Architectural Overview

The system is split into three logical layers handled by a chained CI pipeline:

1. Base Layer (base/): Extends fedora-bootc. It installs system-level dependencies (SSH keys, certificates, repos) using Ansible playbooks executed within the build context.
2. Application Layers (apps/): Modular definitions for workloads. These define Podman Quadlets (.container files) and systemd units that are baked into the final image.
3. Release & Extraction (live/): A specialized pipeline that transmutes the OCI image into bootable artifacts (Kernel, Initramfs, SquashFS) for bare-metal distribution.

Build Strategy: Ansible-Driven Immutability

Unlike traditional infrastructure where Ansible runs against live hosts, this project runs Ansible inside the Containerfile. This ensures the configuration is versioned, cacheable, and reproducible.

Containerfile Pattern:

FROM harbor.example.com/remote/quay/fedora/fedora-bootc:41

# 1. Install Ansible
# 2. Execute Playbook (applies configs, copies templates)
# 3. Cleanup Ansible to keep layer size down
RUN dnf -y install ansible && \
    ansible-playbook playbook.yml --extra-vars "@vars/default_vars.yml" && \
    dnf clean all

Playbook Logic

The playbooks heavily utilize Jinja2 templating to generate systemd units and configuration files. For example, generating Quadlet definitions:

- name: Template /usr/lib/*
    ansible.builtin.template:
    src: "{{ item }}"
    dest: "{{ item | replace(playbook_dir + '/templates', '') | regex_replace('\\.j2$', '') }}"
    owner: root
    group: root
    mode: '0644'
    with_fileglob:
      - "templates/usr/lib/systemd/system/*"

Workload Management & Embedded Images

Workloads are managed via Podman Quadlets, and embedded in the OS image using bootc's logically-bound images feature.

Service Definition

Example caddy.container:

[Unit]
Description=Caddy container
Wants=network-online.target
After=network-online.target
Wants=awx-callback.service
After=awx-callback.service
StartLimitIntervalSec=0

[Container]
ContainerName=caddy
Image=docker.io/caddy:{{ caddy_image_tag }}
# This is necessary to allow the container to use the embedded image
GlobalArgs=--storage-opt=additionalimagestore=/usr/lib/bootc/storage
UserNS=auto
Mount=type=bind,src=/var/caddy,dst=/data,relabel=private,chown=true
Mount=type=bind,src=/etc/caddy/index.html,dst=/usr/share/caddy/index.html,ro=true
PublishPort=80:80

[Service]
Restart=always
RestartSec=30

[Install]
WantedBy=multi-user.target default.target

Logically-Bound Images

To ensure the OS is truly self-contained (air-gapped capable), we do not rely on pulling images at runtime.

The Ansible playbook templates .image files and symlinks them to /usr/lib/bootc/bound-images.d/.

• Mechanism: When bootc detects a file in bound-images.d, it pulls that specific container image during the OS build process and stores it in the OSTree storage.

• Result: The final OS image contains the application image inside it. When the node boots, the image is already present on disk, requiring no network traffic to start the workload.

The "Live" Boot Extraction Pipeline

The most complex component of this repository is the artifact extraction logic found in the bootc-live component. To support PXE/Network booting of these custom images, the CI pipeline performs the following low-level operations:

• Image Build: Uses bootc-image-builder to convert the OCI container into a raw disk image (.raw).

• Loop Mounting: A privileged CI runner mounts the raw image partitions using losetup.

• Kernel Extraction: Identifies the current OSTree deployment and extracts vmlinuz and initramfs.img.

• SquashFS Generation: Creates a compressed squashfs.img of the root filesystem for the Live OS layer.

• OSTree Symlinking: Manipulates the OSTree structure to ensure kernel arguments point to a static path rather than a dynamic hash.

A reusable Gitlab CI/CD component is used for this step. It can be found here.

Continuous Integration Details

The project uses a complex GitLab CI configuration:

• Trigger Logic: Changes in base/ trigger downstream builds of apps/ and live/. Changes in apps/ result in a rebuild of only the affected application.

• Digest Pinning: hack/parse_digest.sh ensures strict immutability by resolving tags to SHA256 digests before build.

• S3 Upload: Final boot artifacts are versioned and uploaded to S3-compatible storage for consumption by the booting infrastructure (iPXE).

• Renovate Integration: Automated dependency updates for OCI images via renovate.json.

Repository Structure

. ├── base/ # The base OS definition │ ├── Containerfile # bootc base + Ansible execution │ └── playbook.yml # System config (users, ssh, repos) ├── apps/ # Application overlays │ └── ... ├── live/ # Live OS specific configuration ├── .gitlab-ci/ # CI Components and templates └── hack/ # Utility scripts