Chapter 1: Build Your Practice Lab

Troubleshooting and fast rebuild

Fixes for everything that went wrong while this lab was built and tested, a command to record your versions, and the whole build as a script for when you want to rebuild from scratch.

ufw on the host

Ubuntu's ufw firewall is off by default, but if you have enabled it, its default policy blocks three things the lab needs. It doesn't show up until the VMs first try to use the network.

sudo ufw status                                             # "Status: active" means you need this
sudo ufw allow in on rhcebr0 to any port 67 proto udp       # DHCP: VMs get their IPs
sudo ufw allow in on rhcebr0 to any port 53                 # DNS: lab names resolve
sudo ufw route allow in on rhcebr0                          # forwarding: VMs reach the internet
sudo ufw reload
lxc restart --project rhce --all                            # VMs ask for their IPs again

These rules are scoped to rhcebr0, so nothing changes for the rest of the host. They don't open the lab to anyone either: ufw only adds permissions, and the rhce_isolate seal still drops traffic from the lab to your local network, and everything coming in.

sudo ufw status verbose | grep rhcebr0
lxc list --project rhce          # every VM has its 172.25.250.x address again

General problems

Repair an existing lab

For a lab whose first boot missed its packages, or one built before the profile in section 1.4 switched on SELinux, firewalld and chrony. It brings running VMs up to date in place, keeps everything on them, and takes about five minutes.

for vm in workstation servera serverb serverc serverd utility; do
  lxc info --project rhce "$vm" >/dev/null 2>&1 || continue
  lxc exec --project rhce "$vm" -- bash -c '
    dnf install -y python3 vim-enhanced bash-completion man-db man-pages tree tmux lsof nano bind-utils \
      lvm2 tar rsync selinux-policy-targeted policycoreutils-python-utils firewalld chrony
    semanage fcontext -a -t bin_t "/var/run/lxd_agent(/.*)?" 2>/dev/null || true
    systemctl enable --now firewalld chronyd' </dev/null
done
lxc exec --project rhce utility -- bash -c 'firewall-cmd --permanent --add-service=http && firewall-cmd --reload' 2>/dev/null
lxc restart --project rhce --all      # each VM boots twice while SELinux labels its disk

Then install the current rht-vmctl from section 1.7 (it also snapshots the extra disks). After about two minutes, run the verification in that section. When every line passes, replace the baseline with rht-vmctl save: the old clean doesn't have these fixes.

Record your versions

Handy when comparing notes or filing an issue. Run it on the host once the lab is built:

echo "== host";  grep PRETTY /etc/os-release; uname -r; snap list lxd; lxc version
echo "== image"; lxc config get --project rhce servera image.description
for vm in workstation servera; do
  echo "== $vm"; lxc exec --project rhce "$vm" -- bash -c 'cat /etc/rocky-release; getenforce; rpm -q selinux-policy-targeted'
done
echo "== workstation tools"
lxc exec --project rhce workstation -- su - student -c 'rpm -q man-db tree tmux bind-utils lsof'

Rebuild everything fast

Everything in this chapter as commands, once you know what each step does. It assumes the host is prepared and the seal is enabled (sections 1.2 and 1.3, plus the ufw rules if ufw is active), and that the profile YAML from section 1.4 is saved as sysadmin-profile.yaml on the host. Afterwards, continue with sections 1.6 and 1.7 as normal: prepare workstation and the lab tools, run the verification, then save clean.

Download the script. Read it before running it. The expandable example below matches the download.

#!/usr/bin/env bash
# Builds the system administration practice lab: workstation, servera and serverb.
# Same network, project and names as the Ansible lab, with fewer machines.
set -euo pipefail

# Network (Phase 03)
lxc network create rhcebr0 --project default \
  ipv4.address=172.25.250.254/24 ipv4.nat=true \
  ipv4.dhcp.ranges=172.25.250.100-172.25.250.199 \
  ipv6.address=none dns.domain=lab.example.com
lxc network set rhcebr0 --project default \
  raw.dnsmasq="$(printf 'host-record=content.example.com,172.25.250.8\nhost-record=materials.example.com,172.25.250.8')"

# Project (Phase 05)
lxc project create rhce -c features.images=false -c features.profiles=true \
  -c features.storage.volumes=true -c features.networks=false
lxc project switch rhce

# Profile (Phase 06)
lxc profile edit default < sysadmin-profile.yaml

# Disks and VMs (Phases 07–08)
declare -A IP=( [workstation]=9 [servera]=10 [serverb]=11 )
for vm in workstation servera serverb; do
  lxc init "${IMAGE:-images:rockylinux/9/cloud}" "$vm" --vm    # IMAGE=rocky9-lab to use a frozen copy
  lxc config device override "$vm" eth0 ipv4.address=172.25.250.${IP[$vm]}
  if [[ $vm == server* ]]; then
    lxc storage volume create default "$vm-disk2" --type=block size=5GiB
    lxc config device add "$vm" disk2 disk pool=default source="$vm-disk2"
  fi
done
lxc config set workstation limits.cpu=2 limits.memory=2GiB

sudo systemctl restart rhce-isolate     # make sure the seal is on before first boot
lxc start --all
sleep 240                                # first boot + the SELinux relabel reboot
lxc list

Tear the whole lab down

The same commands remove either lab, the lighter one from this chapter or the Ansible lab:

lxc project switch rhce
for vm in workstation servera serverb serverc serverd utility; do
  lxc delete --force "$vm" 2>/dev/null || true
done
for v in servera serverb serverc serverd; do
  lxc storage volume delete default "$v-disk2" 2>/dev/null || true
done
lxc project switch default
lxc project delete rhce
lxc network delete rhcebr0
sudo systemctl disable --now rhce-isolate

Moving to the Ansible path

The Ansible path needs the larger lab: four servers, an optional utility server, an automation account on every machine, and the Ansible tools on workstation. It also runs every exercise in this path, so you never need both labs.

Copy anything you want from workstation to the host with lxc file pull.

Run the commands in Tear the whole lab down above. LXD, its storage pool, the seal's rules file and rht-vmctl stay on the host.

Follow the Ansible path's practice lab chapter from its network section onwards. The seal's rules file already exists, so in that phase you only need to turn the service on again. The Ansible lab's profile is different (it adds the automation account), which is why the machines must be created again rather than changed.