Device Readiness Checks

Virtualization providers often report that a lab device has started long before it can be configured. For example, containerlab declares success as soon as the containers are running, even when they contain virtual machines that need minutes to boot, and some devices accept SSH sessions before they detect their data-plane interfaces.

netlab initial (and netlab config –reload) thus starts with a sequence of device readiness checks. This document describes how those checks are selected, executed, and configured, and how you can add readiness checks for a new device.

When Are the Readiness Checks Executed?

The readiness checks are executed:

As the first step of the configuration deployment process in netlab initial (and thus netlab up) unless the (internal) --skip-ready flag is set.
Before reloading saved device configurations with netlab config –reload.
As the only action of the netlab initial –ready command. You can use that command in CLI hooks or when testing the readiness checks of a new device.

The readiness checks are executed only on nodes that will be configured (the nodes specified with the --limit parameter, excluding unprovisioned nodes).

The readiness checks are implemented in the netsim/cli/initial/ready package.

Selecting the Readiness Checks

The readiness checks executed on a node are specified in the netlab_ready group variable, a list of readiness steps. As with other group variables, you can set it for a device (group_vars.netlab_ready), for a device/provider combination (provider.group_vars.netlab_ready), or on an individual node.

These readiness steps are recognized by netlab:

Step	Description	Handler
health	Wait for the virtualization provider to report healthy nodes	internal (Python)
ssh	Wait for a working SSH server	internal or Ansible playbook
command	Wait for a CLI command to return the expected result	internal (Python)
ansible	Execute a device-specific Ansible task list	Ansible playbook

Notes:

netlab performs no readiness checks on devices that do not have the netlab_ready variable. That’s perfectly fine for devices started with Vagrant (Vagrant waits for the SSH server) or for Linux containers that are configured with docker exec.
The order of the steps in the netlab_ready list is irrelevant. The readiness checks are always executed in this order: health, ssh, command, and finally the Ansible playbook (that includes the ansible step).
Every step is executed in parallel on all nodes that need it. The next step starts after all the nodes have completed the previous step.
A failed readiness check aborts the lab configuration process.
Some devices set netlab_ready to [ ansible ] without having an Ansible readiness task list. That’s a trick to skip the SSH readiness check that would otherwise be inherited from another device or a provider-specific setting.

Example: Cisco Nexus OS uses the command readiness check on all virtualization providers. It also has to wait for the SSH server when running in a vrnetlab container:

group_vars:
  netlab_ready: [ command ]
  netlab_check_retries: 50
  netlab_check_delay: 10
clab:
  group_vars:
    netlab_ready: [ ssh, command ]
  features:
    initial:
      healthcheck: true

Internal and Ansible Handlers

The defaults.netlab.initial.ready dictionary specifies which readiness steps are executed with internal (Python) code:

netlab:
  initial:
    ready:
      ssh: internal
      health: internal
      command: internal

The readiness steps that are not handled internally (including the ansible step) are executed with the device-ready.ansible playbook, with Ansible tags set to ready_step for every step that has to be executed.

The device-ready.ansible playbook implements only the ssh and ansible steps. Changing the health or command handler to anything but internal thus disables those checks.

The playbook selects the target hosts with the netlab_ready_step Ansible inventory groups. The ansible output module (netsim/outputs/ansible.py) creates those groups from the netlab_ready values of individual nodes.

Wait Times

All readiness checks use these group variables (that can also be set on individual nodes) to control the wait time:

Variable	Default	Meaning
netlab_check_retries	20	Number of retries
netlab_check_delay	5	Delay between retries/timeout of a single attempt (seconds)

The internal handlers compute the maximum wait time as netlab_check_retries × netlab_check_delay (100 seconds with the default values) and keep retrying the check until it succeeds or the wait time expires. The Ansible task lists should use the same variables in their retries and delay parameters.

Provider Health Check

The health step calls the health_check method of all virtualization providers used by the nodes in the waitlist. The method receives the list of nodes that need the health check and the lab topology. The default implementation (in netsim/providers/__init__.py) does nothing.

Only the containerlab provider implements the health check:

The provider init hook (health_ready_setup in netsim/providers/clab/labops.py) adds the health step to the clab.group_vars.netlab_ready list of every device with the clab.features.initial.healthcheck feature (copying the device group_vars.netlab_ready value into the provider-specific group variables first).
The health_check method executes containerlab inspect every second until all specified containers report the healthy state or until the defaults.providers.clab.healthy_timeout (default: 600 seconds) expires.
The health check is disabled when defaults.providers.clab.healthy_timeout is set to zero.

To use the health check on a new containerlab device (usually a vrnetlab container), set the clab.features.initial.healthcheck device feature to True:

clab:
  features:
    initial:
      healthcheck: true

Tip

A healthy vrnetlab container does not imply a working SSH server. You’ll probably have to add the ssh step to the clab.group_vars.netlab_ready list.

SSH Server Check

The ssh step waits until netlab can log into the device using SSH and execute a CLI command. The internal implementation is in netsim/cli/initial/ready/ssh.py:

netlab uses netmiko to connect to the device and execute the command when the netmiko library is installed and the device has the netmiko.device_type feature (see Deploying Device Configurations with Netmiko for details). All netmiko timeouts are set to netlab_check_delay.
Otherwise, netlab executes the ssh command (prefixed with sshpass when the ansible_ssh_pass variable is set) with a netlab_check_delay timeout.
Both methods connect to the node management IPv4 address (or IPv6 address if the node has no IPv4 management address) using the ansible_user username. The SSH check fails if the node has no management IP address.
The check is retried (once per second) until it succeeds or the maximum wait time expires.
The maximum wait time is increased to defaults.providers.provider.ssh_timeout if that value is set and larger than the computed wait time.

These group variables control the SSH server check:

Variable	Meaning
ansible_user	SSH username
ansible_ssh_pass	SSH password (requires `sshpass` when netmiko cannot be used)
netlab_check_command	CLI command to execute (default: `show version`). Use a different command if your device does not support it (example: `who` on cRPD)
netlab_ssh_args	Extra arguments for the `ssh` command (for example, legacy key exchange algorithms)

The Ansible implementation of the SSH check (netsim/ansible/tasks/readiness-check/ssh.yml) is used when defaults.netlab.initial.ready.ssh is set to ansible. It executes the same ssh command with a 10-second timeout, retrying it netlab_check_retries times with a netlab_check_delay delay.

Command Check

The command step waits until a CLI command executed on the device returns the expected output (for example, until the device detects its first data-plane interface). The internal implementation is in netsim/cli/initial/ready/command.py; it uses the netlab connect code to execute the command, so it works with SSH and docker exec connections (selected with the netlab_console_connection or ansible_connection group variable).

The command check is configured with the command.ready device feature, a dictionary with these keys:

Key	Meaning
execute	The command to execute on the device
expect	A string that must appear in the command output
when	Optional Python expression. The command check is skipped when the expression evaluates to False

The when expression and the execute and expect strings (evaluated as Python f-strings) can use all node attributes and the phy_interfaces variable, the list of node interfaces without the virtual_interface attribute.

Example: Cisco Nexus OS command readiness check (netsim/devices/nxos.yml) waits for the first physical interface to appear in the show interface brief printout:

features:
  command:
    ready:
      when: phy_interfaces
      execute: show interface brief
      expect: '{phy_interfaces[0].ifname.replace("Ethernet","Eth")}'

The command is executed every netlab_check_delay seconds (at least once per second) until its output contains the expect string or until the maximum wait time expires. If the command execution fails (for example, the SSH session cannot be established), the check is retried, but only for a quarter of the maximum wait time; in that case netlab reports the last error it encountered.

Warning

A device with the command step in its netlab_ready list must have the command.ready feature; netlab reports an internal error otherwise.

Ansible Readiness Task List

The ansible step executes a device-specific Ansible task list. Use it when the device needs a readiness check that cannot be implemented with the generic mechanisms described above, for example:

Waiting for a Linux service (cumulus_nvue.yml)
Waiting for a HTTPS (fortios-clab.yml) or gNMI (sros.yml) server
Executing a device-specific command over a NETCONF or REST API connection (junos.yml, netscaler.yml)

The device-ready.ansible playbook executes the tasks/wait-for-ready.yml task list on all hosts in the netlab_ready_ansible inventory group. That task list finds the device readiness task list using the paths.ready.dirs directory search path and the paths.ready.files filename patterns (more details):

ready:
  dirs:
  - "topology:readiness-check"
  - "readiness-check"
  - "~/.netlab/readiness-check"
  - "/etc/netlab/readiness-check"
  - "package:ansible/tasks/readiness-check"
  files:
  - "{{netlab_device_type}}-{{node_provider}}.yml"
  - "{{netlab_device_type}}.yml"
  - "{{ansible_network_os}}-{{node_provider}}.yml"
  - "{{ansible_network_os}}.yml"

The system readiness task lists are thus stored in the netsim/ansible/tasks/readiness-check directory, and the task list name is based on the netlab_device_type variable (or ansible_network_os when netlab_device_type is not set), optionally combined with the node provider (example: vyos-clab.yml). The task list is skipped if no matching file is found.

Warning

The Ansible readiness task list is executed only on nodes that have the ansible step in their netlab_ready list.

The readiness task list is executed with gather_facts set to false. The playbook sets the netlab_device_type, netlab_interfaces, and node_provider variables before executing the task list. Use netlab_check_retries and netlab_check_delay variables (with device-specific defaults) to control the wait time, for example:

- name: Check {{ inventory_hostname }} nscli readiness
  ansible.builtin.raw: nscli -U '127.0.0.1:{{ ansible_user }}:{{ ansible_ssh_pass }}' 'show ns hostName'
  changed_when: False
  register: command_out
  until: command_out.rc == 0 and "Done" in command_out.stdout
  retries: "{{ netlab_check_retries | default(60) }}"
  delay: "{{ netlab_check_delay | default(5) }}"

Adding Readiness Checks to a New Device

Use these guidelines to select the readiness checks for a new device:

Devices started by Vagrant usually don’t need the ssh step (Vagrant waits for the SSH server).
vrnetlab containers usually need the health check (set clab.features.initial.healthcheck) and the ssh step (in clab.group_vars.netlab_ready).
Use the command step for devices that accept SSH sessions before they’re ready to be configured (for example, before they detect data-plane interfaces). Prefer it over an Ansible task list; it’s faster and does not depend on Ansible collections.
Use the ansible step only when you cannot implement the readiness check with the internal handlers.
Set netlab_check_retries and netlab_check_delay if the device needs more than 100 seconds to become ready.

Troubleshooting

Use netlab initial –ready to execute just the readiness checks.
Use --debug ssh to display the SSH commands, wait times, and SSH errors (every five seconds).
Use --debug ready to display the command check wait times, the command outputs, and the expected results.
Use -v to display the readiness check progress even when the devices become ready in less than five seconds.