Chapter 1: Build Your Practice Lab

Troubleshooting and fast rebuild

Fixes for everything that went wrong while this lab was built and tested, a command to record your versions, and the whole build as a script for when you want to rebuild from scratch.

ufw on the host

Ubuntu's ufw firewall is off by default, but if you have enabled it, its default policy blocks three things the lab needs. It doesn't show up until the VMs first try to use the network.

sudo ufw status                                             # "Status: active" means you need this
sudo ufw allow in on rhcebr0 to any port 67 proto udp       # DHCP: VMs get their IPs
sudo ufw allow in on rhcebr0 to any port 53                 # DNS: lab names resolve
sudo ufw route allow in on rhcebr0                          # forwarding: VMs reach the internet
sudo ufw reload
lxc restart --project rhce --all                            # VMs ask for their IPs again

These rules are scoped to rhcebr0, so nothing changes for the rest of the host. They don't open the lab to anyone either: ufw only adds permissions, and the rhce_isolate seal still drops traffic from the lab to your local network, and everything coming in.

sudo ufw status verbose | grep rhcebr0
lxc list --project rhce          # every VM has its 172.25.250.x address again

General problems

Repair an existing lab

For a lab whose first boot missed its packages, or one built before the profile in section 1.4 switched on SELinux, firewalld and chrony, or added the manual pages and everyday tools the system administration path uses. It brings running VMs up to date in place, keeps everything on them, and takes about five minutes.

for vm in workstation servera serverb serverc serverd utility; do
  lxc info --project rhce "$vm" >/dev/null 2>&1 || continue
  lxc exec --project rhce "$vm" -- bash -c '
    dnf install -y python3 python3-libselinux vim-enhanced bash-completion lvm2 tar rsync \
      man-db man-pages tree tmux lsof nano bind-utils \
      selinux-policy-targeted policycoreutils-python-utils firewalld chrony
    semanage fcontext -a -t bin_t "/var/run/lxd_agent(/.*)?" 2>/dev/null || true
    systemctl enable --now firewalld chronyd' </dev/null
done
lxc exec --project rhce utility -- bash -c 'firewall-cmd --permanent --add-service=http && firewall-cmd --reload' 2>/dev/null
lxc restart --project rhce --all      # each VM boots twice while SELinux labels its disk

Then install the current rht-vmctl from section 1.7 (it also snapshots the extra disks), and on workstation run steps 1, 2, 5, 6 and 7 of section 1.6 (packages, collections, SSH keys for devops, student and root, and the lab command), and if utility serves collections, download the pinned versions there too. After about two minutes, run the verification in section 1.7. When every line passes, replace the baseline with rht-vmctl save: the old clean doesn't have these fixes.

Record your versions

Handy when comparing notes or filing an issue. Run it on the host once the lab is built:

echo "== host";  grep PRETTY /etc/os-release; uname -r; snap list lxd; lxc version
echo "== image"; lxc config get --project rhce servera image.description
for vm in workstation servera; do
  echo "== $vm"; lxc exec --project rhce "$vm" -- bash -c 'cat /etc/rocky-release; getenforce; rpm -q selinux-policy-targeted'
done
echo "== workstation tools"
lxc exec --project rhce workstation -- su - student -c '
  which -a ansible; ansible --version | head -1
  rpm -q ansible-core rhel-system-roles podman python3.11
  ansible-navigator --version
  ansible-galaxy collection list 2>/dev/null | grep -E "^(ansible|community|redhat)\."'

Rebuild everything fast

Everything in this chapter as commands, once you know what each step does. It assumes the host is prepared and the seal is enabled (sections 1.2 and 1.3, plus the ufw rules if ufw is active), and that the profile YAML from section 1.4 is saved as rhce-profile.yaml on the host. Afterwards, continue with sections 1.6 and 1.7 as normal: prepare workstation (including the collections), add utility if you want it, run the verification, then save clean.

Download the script. Read it before running it. The expandable example below matches the download.

#!/usr/bin/env bash
set -euo pipefail

# Network (Phase 03)
lxc network create rhcebr0 --project default \
  ipv4.address=172.25.250.254/24 ipv4.nat=true \
  ipv4.dhcp.ranges=172.25.250.100-172.25.250.199 \
  ipv6.address=none dns.domain=lab.example.com
lxc network set rhcebr0 --project default \
  raw.dnsmasq="$(printf 'host-record=content.example.com,172.25.250.8\nhost-record=materials.example.com,172.25.250.8')"

# Project (Phase 05)
lxc project create rhce -c features.images=false -c features.profiles=true \
  -c features.storage.volumes=true -c features.networks=false
lxc project switch rhce

# Profile (Phase 06)
lxc profile edit default < rhce-profile.yaml

# Disks and VMs (Phases 07–08)
declare -A IP=( [workstation]=9 [servera]=10 [serverb]=11 [serverc]=12 [serverd]=13 )
for vm in workstation servera serverb serverc serverd; do
  lxc init "${IMAGE:-images:rockylinux/9/cloud}" "$vm" --vm    # IMAGE=rocky9-lab to use a frozen copy
  lxc config device override "$vm" eth0 ipv4.address=172.25.250.${IP[$vm]}
  if [[ $vm == server* ]]; then
    lxc storage volume create default "$vm-disk2" --type=block size=5GiB
    lxc config device add "$vm" disk2 disk pool=default source="$vm-disk2"
  fi
done
lxc config set workstation limits.cpu=2 limits.memory=2GiB

sudo systemctl restart rhce-isolate     # make sure the seal is on before first boot
lxc start --all
sleep 240                                # first boot + the SELinux relabel reboot
lxc list

Tear the whole lab down

The same commands remove the lighter system administration lab, which has only workstation, servera and serverb.

lxc project switch rhce
for vm in workstation servera serverb serverc serverd utility; do
  lxc delete --force "$vm" 2>/dev/null || true
done
for v in servera serverb serverc serverd; do
  lxc storage volume delete default "$v-disk2" 2>/dev/null || true
done
lxc project switch default
lxc project delete rhce
lxc network delete rhcebr0
sudo systemctl disable --now rhce-isolate