Exercise: Troubleshooting Ansible

The playbook secure-web.yml should set up Apache with TLS on the web servers, but several problems stop it: in the playbook, in the inventory and in how it connects. Find and fix each one until the playbook runs cleanly. Solutions are hidden under each task. Try each one yourself first.

The project ~/troubleshoot-review has an ansible.cfg, an inventory, the playbook secure-web.yml, the Apache configuration vhosts.conf and the page index.html. serverb.lab.example.com should be the only host in the webservers group. Ansible can reach it as devops, with the SSH keys already in place, and devops can become root without a password.

Troubleshooting Ansible

Troubleshoot playbooks.
Troubleshoot managed hosts.

Prerequisites: Complete control-node setup and the preceding lessons in this chapter. Run commands as student on workstation unless a step names another machine.

From the project directory, run the relevant home-lab check. For an exercise with several checkpoints, grade each state before changing or removing it:

lab grade troubleshoot-review

The grader reads project files and host state. A passing report covers the listed checks; also perform the task’s independent connection, repeat-run, and reboot checks where requested.

Starter files contain deliberate YAML, inventory, or task errors. Repair them before the final checkpoint.

Use the wrong inventory group in a copy of the playbook. Explain why zero hosts processed is not a successful deployment.

Record the published result first. The grader checks the original requirements, so a changed name or value may intentionally fail those checks.

Repair secure-web.yml so serverb runs the intended TLS web service.

Correct the playbook's syntax, its target inventory, and the connection/escalation settings as needed.
All intended deployment tasks must remain and complete. The configured Apache service must be running and the supplied TLS content reachable.
Record one diagnostic observation for each correction. A syntax check alone is insufficient evidence of a working service.
Repeat the deployment and verify both host state and HTTPS behavior.

Check the syntax

From ~/troubleshoot-review, check the syntax of secure-web.yml and fix the problem it reports.

[student@workstation ~]$ cd ~/troubleshoot-review
[student@workstation troubleshoot-review]$ ansible-navigator run \
> -m stdout secure-web.yml --syntax-check
...output omitted...
Syntax Error while loading YAML.
  mapping values are not allowed in this context

The error appears to be in '/home/student/troubleshoot-review/secure-web.yml': line 7, column 30, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  vars:
    random_var: This is colon: test
                             ^ here

A colon followed by a space inside a value must be protected by quotes:

  vars:
    random_var: "This is colon: test"

Check the syntax again

It still has a problem. Fix the next one.

[student@workstation troubleshoot-review]$ ansible-navigator run \
> -m stdout secure-web.yml --syntax-check
...output omitted...
Syntax Error while loading YAML.
  did not find expected key

The error appears to be in '/home/student/troubleshoot-review/secure-web.yml': line 44, column 7, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

      - name: Start and enable web services
      ^ here

The task is indented two spaces too far. Remove the extra spaces from all its lines:

    - name: Start and enable web services
      ansible.builtin.service:
        name: httpd
        state: started
        enabled: true

And a third time

[student@workstation troubleshoot-review]$ ansible-navigator run \
> -m stdout secure-web.yml --syntax-check
...output omitted...
  found unacceptable key (unhashable type: 'AnsibleMapping')

The error appears to be in '/home/student/troubleshoot-review/secure-web.yml': line 12, column 16, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

      ansible.builtin.dnf:
        name: {{ item }}
               ^ here
...output omitted...

A value that starts with {{ must be quoted:

    - name: Install web server packages
      ansible.builtin.dnf:
        name: "{{ item }}"
        state: latest
      notify:
        - Restart services
      loop:
        - httpd
        - mod_ssl

[student@workstation troubleshoot-review]$ ansible-navigator run \
> -m stdout secure-web.yml --syntax-check
playbook: /home/student/troubleshoot-review/secure-web.yml

Fix the connection

Run the playbook. Ansible cannot connect to serverb.lab.example.com. Two problems prevent it; fix both. (You may well find them in a different order, or both at once.)

[student@workstation troubleshoot-review]$ ansible-navigator run -m stdout secure-web.yml

PLAY [Create secure web service] ***********************************************

TASK [Gathering Facts] *********************************************************
fatal: [serverb.lab.example.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: students@serverc.lab.example.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).", "unreachable": true}

For serverb, Ansible connected to serverc, as a user called students. With -vvv the connection details confirm it:

[student@workstation troubleshoot-review]$ ansible-navigator run -m stdout secure-web.yml -vvv
...output omitted...
<serverc.lab.example.com> ESTABLISH SSH CONNECTION FOR USER: students
<serverc.lab.example.com> SSH: EXEC ssh -C -o ControlMaster=auto ... -o 'User="students"' ... serverc.lab.example.com ...
<serverc.lab.example.com> (255, b'', b'students@serverc.lab.example.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).\r\n')
...output omitted...

The inventory sets ansible_host for serverb, redirecting the connection. Remove it:

[webservers]
serverb.lab.example.com

The play sets the wrong remote_user. Change it to devops:

---
# start of secure web server playbook
- name: Create secure web service
  hosts: webservers
  remote_user: devops

Fix the next failure

Run the playbook again. The connection works now, but a task fails. Fix it.

[student@workstation troubleshoot-review]$ ansible-navigator run -m stdout secure-web.yml
...output omitted...
TASK [Install web server packages] *********************************************
failed: [serverb.lab.example.com] (item=httpd) => {"ansible_loop_var": "item", "changed": false, "item": "httpd", "msg": "This command has to be run under the root user.", "results": []}
failed: [serverb.lab.example.com] (item=mod_ssl) => {"ansible_loop_var": "item", "changed": false, "item": "mod_ssl", "msg": "This command has to be run under the root user.", "results": []}

Nothing in the configuration or the play turns on privilege escalation. Add it to the play:

- name: Create secure web service
  hosts: webservers
  remote_user: devops
  become: true

Run it to the end

Run the playbook once more. It must complete; then confirm with an ad hoc command that httpd is running on serverb.

[student@workstation troubleshoot-review]$ ansible-navigator run -m stdout secure-web.yml
...output omitted...
TASK [Install web server packages] *********************************************
changed: [serverb.lab.example.com] => (item=httpd)
changed: [serverb.lab.example.com] => (item=mod_ssl)
...output omitted...
TASK [Httpd_conf_syntax variable] **********************************************
ok: [serverb.lab.example.com] => {
    "msg": "The httpd_conf_syntax variable value is {'changed': False, 'stdout': '', 'stderr': 'Syntax OK', 'rc': 0, 'cmd': ['/sbin/httpd', '-t'], ...output omitted...
}
...output omitted...
RUNNING HANDLER [Restart services] *********************************************
changed: [serverb.lab.example.com]

PLAY RECAP *********************************************************************
serverb.lab.example.com    : ok=11   changed=8    unreachable=0    failed=0  ...
[student@workstation troubleshoot-review]$ ansible webservers -u devops -b \
> -m command -a 'systemctl status httpd'
serverb.lab.example.com | CHANGED | rc=0 >>
● httpd.service - The Apache HTTP Server
     Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; preset: disabled)
     Active: active (running) since ...
...output omitted...

-u devops and -b make the ad hoc command connect and escalate the same way the play now does. A second run of the playbook reports changed=0.

Grade and finish

Check your work with lab grade troubleshoot-review, then put the exercise away with lab finish troubleshoot-review.