Thanks for the info Scott. We are currently running 11.9 patch 5. Looks like the associated UCCE credential changes introduced with either patch 4 or 5 have resulted in a defect. Below is what I encountered.
We have multiple pre-existing UCCE instances. As with previous Prognosis versions, each UCCE instance has one set of credentials used by all devices that are added via the first device add. This set of credentials is reflected by a single entry in the password configuration listed under the instance name. I recently added a new UCCE instance to Prognosis via the WebUI. Per the changes introduced in patch 4 and 5, the credentials had to be entered for each device within the instance. This resulted in an entry being inserted into the password configuration for each new UCCE device added.
Once I added the new instance and all associated devices, I noticed Prognosis lost contact to all my pre-existing UCCE instances (minus the device that was used to create the instance which is when the single credentials were previously added). What I found was that when I added the new UCCE instance, Prognosis also automatically added new entries for every single device in all the pre-existing instances. These newly added entries were obviously added default credentials that did not match was used to create those original instances, thus why Prognosis lost contact to all those pre-existing UCCE devices.
I have since tested and replicated this issue multiple times in our lower environments. What I found is that on our 11.9 patch 5 version, the first time a UCCE device change is made that touches the password configuration, Prognosis automatically adds a default credential entry for every pre-existing UCCE devices. This can be triggered by adding a new instance, adding a device to an existing instance or simply making a change to the credentials of an existing device.
If the individual UCCE device entries are deleted from the password configuration, the device resorts back to using the original correct credentials configured under the instance entry and thus becomes contactable again. That is only a temporary fix as all individual device entries are re-inserted to the password configuration the next time a UCCE update is made. It looks like the correct fix is to manually update each of the newly added default device entries in the password configuration with the correct credentials for all pre-existing UCCE devices. Once done, the pre-existing UCCE devices become contactable again.
I am happy to open a support case to provide details of the finding along with any supporting logs, files, etc. as this is something that probably needs to be addressed in a future patch. Others should be aware of this as well though, as I learned the hard way on our production system.
