failed to replace PIV token

[blackwood821]

I'm encountering an issue where kbmadm recover zones fails to replace the PIV token:

kbmadm: recover command failed
  Caused by KbmdError: kbmd returned an error
    in check_error() at kbmadm/kbmadm.c:1032
  Caused by RecoveryError: failed to create ebox template with new PIV token
    in do_piv_replace() at kbmd/recover.c:812
  Caused by ReplaceError: failed to replace PIV token
    in register_replace_common() at kbmd/plugin.c:715
  Caused by OutputError: output had 0 lines; expected at least 2
    in parse_register_output() at kbmd/plugin.c:657

More info at https://smartdatacenter.topicbox.com/groups/sdc-discuss/Tb40c40a62c5ce3e3-M25ff99472735e8f154e99d52

[blackwood821]

@arekinath thought it may have been fixed by b4af574 since this CN was originally on PI 20201203T165910Z but I just upgraded to 20210325T002528Z and I get the same error and stack.

[blackwood821]

I looked in the kbmapi service log and saw that the kbmapi_recovery_tokens request for this CN returned two results which I was not expecting since there is only one recovery YubiKey in the recovery config. I couldn't figure out how to call:

GET /pivtokens/:guid/recovery-tokens

Since I don't see a sdc-kbmapi wrapper script on the head node and when I run curl from inside the kbmapi zone it says the request needs to be authenticated so I logged into the postgres database in the manatee zone and queried the kbmapi_recovery_tokens table. I saw that there were two records for this CN's YubiKey whereas there was only one on another CN that is working so I deleted the most recent one for this CN and then kbmadm recover zones worked.

So now my questions are:

1. Are there ever supposed to be more than one record for a CN's YubiKey in the kbmapi_recovery_tokens table?
2. If not, then how would multiple records get created in there? Up to the point of encountering the issue I only used kbmadm on the CN global and kbmctl on the KBMAPI zone so I didn't do anything manually that would have done this.
3. If it's valid for there to be more than one record in this table for a CN's YubiKey then it seems like the kbmapi plugin code is not handling it correctly.