Can't deploy new TACACS config on NXOS

[decoupca]

Due to inherent limitations on NXOS, it's not possible to deploy new TACACS config from a local (non-AAA) account, as NXOS will only fail over to local accounts when TACACS/RADIUS servers are not reachable.

This means we must deploy new AAA config from a TACACS account. But, as soon as the new TACACS config is sent, the current AAA session immediately becomes invalid, and any further commands produce an error. NAPALM (understandably) interprets this as a merge failure, and attempts a rollback, which also fails due to the invalid session.

The only way to deploy new TACACS config I've found (apart from some kind of console-based OOB magic) is to send the config from one TACACS account, which will partially succeed and partially fail due to the broken session. Then, using different TACACS credentials, merge the same config again, which will clean up any remaining negation or other non-breaking changes that the first attempt left behind. This second pass also issues the final copy run start.

I'm hoping I've missed something, open to any and all suggestions. If I have missed something, we may be able to build it into NAPALM.

[decoupca]

After further testing, it looks like it's not possible to merge new TACACS config with NAPALM, at least with nxos_ssh, thanks to the way NXOS invalidates the current session. Given the limitations of NXOS (no local accounts unless all TACACS servers are unreachable) and the design goals of NAPALM (unified API that handles merge/commit in the same call), I'm not sure it would ever be possible.

It's not obvious to me that NXAPI would change things, but I haven't tested it. Best solution I've found: merge TACACS config with a library like netmiko or scrapli, which deploys the config but registers a failure due to invalidated session, then log in again with a different TACACS account to issue the final copy run start.

[ktbyers]

Could you do this still using a NAPALM merge (merge TACACS config change)? NAPALM will commit the change and then get disconnected so it won't be able to rollback the change.

You would then have to login again with NAPALM and force the copy run start to happen (after re-logging in).

Obviously, not a great solution i.e. would want some other way back into the device if it all went bad.

[decoupca]

I tried doing a 2-phase NAPALM deploy using different AAA creds for each run. They both failed citing the same error: couldn't rollback due to AAA failure.

I didn't dive into exactly why, but my guess is that re-deploying the config the second time has the same effect as the first--invalidating the AAA session--even though the same AAA settings were in place. IOS/NXOS may re-initialize AAA sessions when those commands are entered, even if they haven't changed.

AFAICT there is no way to use NAPALM only for the copy run start, since that's all rolled into commit_config().

For anyone in the same boat, here's what worked:

- block:
  # Deploys the config but save fails due to broken AAA session. I guess you could do save_when: never to get around it.
  - cisco.nxos.nxos_config:
      src: "{{ remediation_config_path }}"
      save_when: changed
    tags: deploy
    register: deploy_result

  rescue:
    # Saves config using a different AAA account, since the first one is now pinned to a broken cached session
    - cisco.nxos.nxos_config:
        save_when: always
      tags: deploy
      register: deploy_result
      vars:
        ansible_user: "{{ secondary_user }}"
        ansible_password: "{{ secondary_password }}"

[ktbyers]

@decoupca We should solve it in Python first...trying to layer on Ansible will make a lot, lot harder. There is a good chance you would need a custom Ansible module just for this use case.

[decoupca]

Can't test it now, but based on what I've seen, this would work:

new_tacacs.txt:

tacacs-server host 10.2.3.4 key 7 "key-hash"
tacacs-server host 10.5.6.7 key 7 "key-hash"
tacacs-server host 10.8.9.10 key 7 "key-hash"
tacacs-server host 10.11.12.13 key 7 "key-hash"
aaa group server tacacs+ TACACS-NEW
  server 10.2.3.4
  server 10.5.6.7
  server 10.8.9.10
  server 10.11.12.13
  source-interface mgmt0
aaa authentication login default group TACACS-NEW
aaa authorization commands default group TACACS-NEW local
aaa accounting default group TACACS-NEW
no tacacs-server host 10.0.0.1
no tacacs-server host 10.0.0.2
no aaa group server tacacs+ TACACS-OLD1
no aaa group server tacacs+ TACACS-OLD2
no aaa group server tacacs+ TACACS-OLD3

from netmiko import ConnectHandler
from getpass import getpass

primary_username = input("Primary username: ")
primary_password = getpass("Primary password: ")
secondary_username = input("Secondary username: ")
secondary_password = getpass("Secondary username: ")

NETMIKO_ARGS = {
    "host": "nxos-lab1",
    "username": primary_username,
    "password": primary_password,
    "device_type": "cisco_nxos",
    "ssh_config_file": "~/.ssh/config",
}

conn1 = NETMIKO_ARGS.copy()

conn2 = NETMIKO_ARGS.copy()
conn2.update({
    'username': secondary_username,
    'password': secondary_password,
})

with ConnectHandler(**conn1) as conn:
    conn.send_config_from_file("new_tacacs.txt")

with ConnectHandler(**conn2) as conn:
    conn.send_command("copy run start")