Dan Davis

Incus for Ansible Development

At work I write a lot of Ansible playbooks and roles targeting RHEL8. We do not have a great method for writing and testing those playbooks and roles locally though. The normal workflow is to run a CI job in Gitlab against a dev branch to spin up an ec2 instance and run the playbook there, but I am not a fan of how long that takes. I want my Ansible DX to be like my python DX:

  1. Make a change
  2. Run it locally
  3. Get immediate feedback
  4. Repeat

I initially thought to use Podman/Docker, but that does not quite work since I usually need systemd and getting systemd running in a container is not straightfoward. A full blown VM seemed like overkill and I was a little wary of trying to get KVM/QEMU working from within my WSL instance. After some initial testing it turned out Incus was exactly what I was looking for. Incus is a manager/hypervisor for LXC and is the successor to LXD which used to serve the same purpose. The linuxcontainers.org website describes LXC containers as:

LXC containers are often considered as something in the middle between a chroot
and a full fledged virtual machine. The goal of LXC is to create an environment
as close as possible to a standard Linux installation but without the need for a separate kernel.

If you want to run Incus on Fedora be sure to follow the extra setup steps.

I've created a helper script to allow me to spin up new dev containers as needed. It sets up the container with the minimal dependencies needed to start running Ansible playbooks. The script is also included below:

#!/usr/bin/env bash

# For all available distros see https://images.linuxcontainers.org/
DISTRO="${DISTRO:-almalinux}"
RELEASE="${RELEASE:-8}"
CONTAINER_NAME="${CONTAINER_NAME:-$DISTRO$RELEASE}"
# TODO: allow specifying a username instead of hardcoded 'deploy' user

# Create an almalinux8 container
incus launch images:"${DISTRO}/${RELEASE}" "${CONTAINER_NAME}"
sleep 4
# Create a user to run the playbook
incus exec "${CONTAINER_NAME}" -- adduser -ms /bin/bash deploy 
# Install minimum packages needed to run ansible playbooks
incus exec "${CONTAINER_NAME}" -- dnf install openssh-server sudo python3 firewalld -y
sleep 1
# Setup deploy user to run sudo w/o password
incus exec "${CONTAINER_NAME}" -- sh -c "echo 'deploy ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/deploy-user"
incus exec "${CONTAINER_NAME}" -- chmod 0440 /etc/sudoers.d/deploy-user
# Enable + start the sshd service
incus exec "${CONTAINER_NAME}" -- systemctl enable --now sshd.service
# Workaround for bugged nftables on almalinux 8
# without this, running firewall-cmd --reload will fail because of an exception in python-nftables
if [[ "$DISTRO" == "almalinux" ]] && [[ "$RELEASE" == "8" ]]; then
    incus exec "${CONTAINER_NAME}" -- sed -i 's/FirewallBackend=nftables/FirewallBackend=iptables/' /etc/firewalld/firewalld.conf
fi
# Enable + start the firewalld service
incus exec "${CONTAINER_NAME}" -- systemctl enable --now firewalld.service
# Create ssh key pair for deploy user
ssh-keygen -t rsa -b 2048 -N "" -q -f ./deploy-key
# Make sure .ssh dir exists
incus exec "${CONTAINER_NAME}" -- mkdir -p /home/deploy/.ssh
# Copy pub key to container
incus file push ./deploy-key.pub "${CONTAINER_NAME}/home/deploy/.ssh/authorized_keys"
# Fix permissions on .ssh dir
incus exec "${CONTAINER_NAME}" -- chown -R deploy:deploy /home/deploy/.ssh
incus exec "${CONTAINER_NAME}" -- chmod 0700 /home/deploy/.ssh
incus exec "${CONTAINER_NAME}" -- chmod 0600 /home/deploy/.ssh/authorized_keys
# Create a snapshot of the system in pristine condition
incus snapshot create "${CONTAINER_NAME}" clean

ip=$(incus list | grep "${CONTAINER_NAME}" | awk '{print $6}')
echo ""
echo "Setup complete! Here's how to ssh and run ansible playbooks:"
echo ""
echo "  ssh -i ./deploy-key deploy@${ip}"
echo "  ansible-playbook -u deploy -i ${ip}, --key-file ./deploy-key playbook.yaml"
echo ""
echo "If you bork something you can restore to a clean slate by running:"
echo ""
echo "  incus snapshot restore ${CONTAINER_NAME} clean"
echo ""
echo "To delete the container run:"
echo ""
echo "  incus delete ${CONTAINER_NAME} --force"