In helping a customer who was having an issue with getting TOTP working, I came upon an interesting situation with their ID Vault. When issuing ‘show idvault’, the following error was displayed.
Invalid or nonexistent document: Vault replica list inconsistency for vault /ID_Vault
The really strange thing about this situation was the replica of the ID Vault was on both the primary and secondary server, however only the primary server was listed as a Vault Server in the ID Vault itself. Additionally, the Public Encryption Key for the RecoveryKeys document only had the primary server listed, the secondary server was missing.
It was unclear how this situation happened as the customer had created the vault replica on the secondary server via the Manage Vault interface and reported they received no errors performing this operation.
To debug and remedy the situation, I added the following debug variables to the notes.ini of both the primary and secondary servers.
- CONSOLE_LOG_ENABLED = 1
- DEBUG_IDV_CONNECT = 1
- DEBUG_IDV_TRUSTCERT = 1
- DEBUG_IDV_UPDATE = 1
I then deleted the secondary server via Manage Vault, ensured ‘show idvault’ no longer reported any errors, deleted the ID_Vault.nsf file on the secondary server. I then ended and restarted the secondary server and added the secondary server back in via Manage Vault.
Addition of the secondary server via Manage Vault failed with this error:
COULD NOT ADD SERVER02/ACME (FILE ALREADY EXISTS)
This was very strange as I had deleted the ID_Vault.nsf file on the secondary server and restarted it. In reviewing the console log, it became very clear what happened. The following errors show that Cluster Symmetry Repair had created the “missing” file on the secondary server.
In reviewing the cluster symmetry configuration, “All Folders” was selected for ‘Maintain symmetry’. As a temporary remediation until the Cluster Configuration could be thoroughly reviewed, I disabled repair on the ID_Vault.nsf database on both servers.
At this point, I was able to repeat the steps of removing the secondary server as a vault server, manually deleting the vault from the secondary server, restarting the server and then adding the secondary as a vault server via Manage Vault.
This remedied this situation and allowed the Manage Vault process to properly create the replica on the secondary server.
Moral of the story:
Make sure the IBM_ID_VAULT directory is excluded from Cluster Symmetry Repair!