Chapter 1: Build Your Practice Lab

Snapshots and rht-vmctl

Phase 11. Training labs reset their machines with rht-vmctl. Here you install a home-lab version with the same syntax. It runs on the Ubuntu host and uses LXD snapshots underneath.

The reset loop

Build the lab once, save a clean snapshot, then break things freely: any machine can go back to clean in seconds.

When to take the clean snapshot: clean is the state every reset returns to. Take it once the machines and tools are ready, but before you create any practice files.

Install and use it

Phase 11 · Snapshots and rht-vmctl (~10 min)

  • Install rht-vmctl.
  • Verify the build and take the clean baseline.

Host: install the script

The script is a single Bash file of about 300 lines. Read it first, then download and install it. The same commands replace an older copy.

curl -fsSL https://kernelpath.dev/lab/setup/rht-vmctl -o /tmp/rht-vmctl
less /tmp/rht-vmctl                            # read it; press q to leave
sudo install -m 755 /tmp/rht-vmctl /usr/local/bin/rht-vmctl

Host: try it

rht-vmctl version                        # "rht-vmctl 3"
rht-vmctl status all                     # workstation, servera, serverb: RUNNING

Host: verify the whole build

The clean snapshot is what every reset returns to, so check everything first. This changes nothing: it prints PASS or FAIL for every requirement in this chapter, including that workstation has no practice files yet. Every line should pass.

P=rhce; ok(){ printf '  \e[32mPASS\e[0m %s\n' "$1"; }; ko(){ printf '  \e[31mFAIL\e[0m %s\n' "$1"; }
chk(){ if eval "$2" >/dev/null 2>&1; then ok "$1"; else ko "$1"; fi; }
chk_not(){ if eval "$2" >/dev/null 2>&1; then ko "$1"; else ok "$1"; fi; }
X(){ lxc exec --project $P "$1" -- bash -c "$2" </dev/null; }
S(){ lxc exec --project $P workstation -- su - student -c "$1" </dev/null; }
GW=$(ip route | awk '/^default/{print $3; exit}')
LANIP=$(hostname -I | awk '{print $1}')

echo "== host"
chk "LXD on 5.21/stable and held"   "snap list lxd | grep -q '5.21/stable.*held'"
chk "seal service active"           "systemctl is-active --quiet rhce-isolate"
chk "rhcebr0 up at .254"            "ip -4 addr show rhcebr0 | grep -q 172.25.250.254/24"
chk_not "host cannot ping servera"  "ping -c1 -W2 172.25.250.10"
chk "rht-vmctl 3 or newer"          "[ \$(rht-vmctl version | awk '{print \$2}') -ge 3 ]"

echo "== VMs"
for vm in workstation servera serverb; do
  chk "$vm running"                 "lxc list --project $P ^$vm\$ -c s --format csv | grep -q RUNNING"
  chk "$vm FQDN hostname"           "X $vm 'hostname | grep -q ^$vm.lab.example.com\

Loading…

#39;" chk "$vm profile packages" "X $vm 'rpm -q lvm2 tar rsync vim-enhanced man-db tree firewalld chrony policycoreutils-python-utils'" chk "$vm SELinux enforcing" "X $vm 'test \$(getenforce) = Enforcing'" chk "$vm firewalld running" "X $vm 'systemctl is-active --quiet firewalld'" done echo "== servers" for vm in servera serverb; do chk "$vm sdb present and empty" "X $vm 'test -b /dev/sdb && [ -z \"\$(lsblk -no FSTYPE /dev/sdb)\" ] && [ \$(lsblk -no NAME /dev/sdb | wc -l) -eq 1 ]'" done echo "== workstation" chk "student can use sudo" "S 'echo student | sudo -S true'" chk "tools installed" "S 'which tree tmux dig man'" chk "SSH key: student and root on both" "S 'for h in servera serverb; do for u in student root; do ssh -o BatchMode=yes \$u@\$h true || exit 1; done; done'" chk "lab command ready" "S 'lab check'" chk "no practice files yet (clean)" "S 'test -z \"\$(ls -A ~/practice 2>/dev/null)\"'" chk "names resolve inside lab" "S 'getent hosts serverb.lab.example.com'" chk "internet via NAT" "S 'curl -sfI -m5 https://rockylinux.org'" chk_not "lab cannot reach host LAN IP" "S 'ping -c1 -W2 $LANIP'" chk_not "lab cannot reach router" "S 'ping -c1 -W2 $GW'"
If this failsGo to
LXD on 5.21/stable and heldSection 1.2, sudo snap refresh --hold lxd
seal service / host cannot ping / lab cannot reachSection 1.3, Phase 04
rht-vmctl 3 or newerStep 1 above
FQDN hostname, sdbSection 1.4 profile; recreate the VM
profile packages, SELinux, firewalldRepair an existing lab (no need to recreate)
tools installed, sudoSection 1.6, step 1
SSH keySection 1.6, steps 5 and 6 (running them again is safe)
lab command readySection 1.6, step 7; lab check on workstation says which part fails
no practice files yet"Already created files?" below
names resolve / internetThe ufw step in section 1.3, then troubleshooting

Host: take the clean baseline

When everything passes:

rht-vmctl save          # snapshot 'clean' on every VM; asks before replacing an existing one

Then confirm each VM has exactly one snapshot, clean. Remove any older ones with rht-vmctl rmsnap VM NAME, so that clean is always the newest (see the ZFS rule below).

for vm in workstation servera serverb; do
  n=$(lxc query "/1.0/instances/$vm/snapshots?project=rhce" | sed -n 's#.*/snapshots/\([^?"]*\).*#\1#p' | paste -sd,)
  [ "$n" = clean ] && echo "  PASS $vm: $n" || echo "  FAIL $vm: $n"
done
for vm in servera serverb; do          # the extra disks have their own snapshots
  n=$(lxc query "/1.0/storage-pools/default/volumes/custom/$vm-disk2/snapshots?project=rhce" | sed -n 's#.*/snapshots/\([^?"]*\).*#\1#p' | paste -sd,)
  [ "$n" = clean ] && echo "  PASS $vm-disk2: $n" || echo "  FAIL $vm-disk2: $n"
done

What the baseline contains

Command reference

A VM is a name like servera, or servers (servera and serverb here), or all (every VM).

CommandWhat it does
rht-vmctl status allState of each VM
rht-vmctl start allStart VMs and wait until they have booted
rht-vmctl stop allClean shutdown
rht-vmctl poweroff serverbForce off
rht-vmctl reset serveraBack to the clean snapshot, extra disk (sdb) included, booted and ready
rht-vmctl reset serversReset the servers, never workstation. Run it before each exercise
rht-vmctl reset allReset everything; asks first because of workstation
rht-vmctl fullreset serverbSame as reset in this lab
rht-vmctl view serveraText console; leave with Ctrl+a then q
rht-vmctl saveCreate/replace the clean baseline on every VM and its extra disk
rht-vmctl save workstation -n ch04-doneNamed checkpoint
rht-vmctl snapsList snapshots (UTC times)
rht-vmctl restore VM NAMEGo back to a named snapshot
rht-vmctl rmsnap VM NAMEDelete a snapshot (and the extra disk's snapshot of the same name)
rht-vmctl wsLog in as student@workstation
rht-vmctl login servera rootShell on any VM (default user: student)
rht-vmctl versionWhich version is installed

The ZFS snapshot rule

ZFS stores a VM's snapshots as a chain and can only roll back to the newest one. Going back to an older snapshot would mean deleting every snapshot taken after it, so LXD refuses. Try it:

SituationWhat happens
clean is the newest snapshot on a VMrht-vmctl reset always works. This is the normal case for the servers.
You took another snapshot after cleanReset to clean fails with the error above
You want an older snapshot anywayRestore the newer one instead, or delete the newer snapshots first with rht-vmctl rmsnap VM NAME