Identify when an issue was first introduced using Invariant

Use Invariant snapshot history to track down the specific snapshot where a configuration change first introduced an issue.

---

Scenario: A critical interface (dist-1[Ethernet3]) was shut down, triggering a minor outage. You aren't sure when the interface was first shut down. You can scan your Invariant snapshot history to locate the last snapshot where the interface was still up.

Check the latest snapshot

Observe that the interface is currently down.

$ invariant show interfaces --json --network live-IAD
[
  {
    "Hostname": "dist-1",
    "Interface": "Ethernet3",
    /* ... */
    "Admin_Up": false,
    "Active": false,
    "Inactive_Reason": "Administratively down"
  },
]

List historical snapshots

Get a list of snapshot UUIDs for your network, ordered chronologically (newest first is typical output).

$ invariant snapshots --tsv --network live-IAD
f17e9368-67fd-4094-b191-4da3f837eb60    2024-01-15 10:30:00 ...
d1dff68d-7da6-4438-87d8-a12086073823    2024-01-15 10:00:00 ...

Scan historical snapshots

Configure and use this bash script to scan for the most recent snapshot where dist-1[Ethernet3] was up.

TARGET_HOSTNAME="dist-1"
TARGET_INTERFACE="Ethernet3"
NETWORK_NAME="live-IAD"

# Get snapshot UUIDs, skipping the header row, newest first
SNAPSHOT_UUIDS=$(invariant snapshots --tsv --network "$NETWORK_NAME" | tail -n +2 | cut -f1)

LAST_GOOD_SNAPSHOT_UUID=""
FIRST_BAD_SNAPSHOT_UUID=""

for SNAPSHOT_UUID in $SNAPSHOT_UUIDS; do
#   echo "Checking snapshot: $SNAPSHOT_UUID"
  IS_UP=$(invariant show interfaces --snapshot "$SNAPSHOT_UUID" --json | \
            jq --arg hn "$TARGET_HOSTNAME" --arg iface "$TARGET_INTERFACE" \
            '.[] | select(.Interface.hostname == $hn and .Interface.interface == $iface) | .Admin_Up')

  if [ "$IS_UP" = "true" ]; then
    LAST_GOOD_SNAPSHOT_UUID=$SNAPSHOT_UUID
    echo "Interface $TARGET_HOSTNAME[$TARGET_INTERFACE] was UP in snapshot $LAST_GOOD_SNAPSHOT_UUID."
    break # Found the last good state
  else
    # This snapshot is "bad" or the interface was already down
    echo "Interface $TARGET_HOSTNAME[$TARGET_INTERFACE] was DOWN in snapshot $LAST_GOOD_SNAPSHOT_UUID."
    FIRST_BAD_SNAPSHOT_UUID=$SNAPSHOT_UUID
  fi
done

Run the script:

$ invariant snapshots --tsv --network live-IAD | tail -n +2 | cut -f1
# f17e9368-67fd-4094-b191-4da3f837eb60
# d1dff68d-7da6-4438-87d8-a12086073823
# c1c0a0b0-1234-5678-9abc-def012345678
# b1b2b3b4-abcd-ef01-2345-6789abcdef01
# a0a1a2a3-1111-2222-3333-444455556666

$ # Paste configured code here
Interface dist-1[Ethernet3] was DOWN in snapshot f17e9368-67fd-4094-b191-4da3f837eb60.
Interface dist-1[Ethernet3] was DOWN in snapshot d1dff68d-7da6-4438-87d8-a12086073823.
Interface dist-1[Ethernet3] was DOWN in snapshot c1c0a0b0-1234-5678-9abc-def012345678.
Interface dist-1[Ethernet3] was UP in snapshot b1b2b3b4-abcd-ef01-2345-6789abcdef01.

Confirm: Directly confirm the status of the suspected interface (dist-1[Ethernet3]) in both the "last good" and "first bad" snapshots using the interfaces report.

# LAST_GOOD_SNAPSHOT_UUID shows the interface was up
$ invariant show interfaces --snapshot $LAST_GOOD_SNAPSHOT_UUID --json | \
    jq '.[] | select(.Interface.hostname == "dist-1" and .Interface.interface == "Ethernet3")'
[
  {
    "Hostname": "dist-1",
    "Interface": "Ethernet3",
    /* ... */
    "Admin_Up": true,
    "Active": true,
    "Inactive_Reason": null,
  },
]

# FIRST_BAD_SNAPSHOT_UUID shows the interface was shut down
$ invariant show interfaces --snapshot $FIRST_BAD_SNAPSHOT_UUID --json | \
    jq '.[] | select(.Interface.hostname == "dist-1" and .Interface.interface == "Ethernet3")'
[
  {
    "Hostname": "dist-1",
    "Interface": "Ethernet3",
    /* ... */
    "Admin_Up": false,
    "Active": false,
    "Inactive_Reason": "Administratively down"
  },
]

See API > CLI for complete CLI documentation and Output Overview for details on available reports.