vCenter Server Applicance certificate expiration

APPLIES TO

VMware vCenter Server Applicance - 6.5.x,6.7.x,7.0.x

SYMPTOMS

Logging in through the Web client display errors similar to:

And services start fails with operation time out.

~]# service-control --start --all
Operation not cancellable. Please wait for it to finish...
Performing start operation on service lwsmd...
Successfully started service lwsmd
Performing start operation on service vmafdd...
Successfully started service vmafdd
Performing start operation on service vmdird...
Successfully started service vmdird
Performing start operation on service vmcad...
Successfully started service vmcad
Performing start operation on profile: ALL...
Successfully started service vmware-vmon
Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start sps, content-library, vstats, vpxd, updatemgr, vsan-health, wcp services. Error: Operation timed out

CAUSE

Some certificates expired.

SOLUTION

~]# for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done
STORE MACHINE_SSL_CERT
Alias : __MACHINE_CERT
            Not After : Nov 29 06:22:16 2023 GMT
STORE TRUSTED_ROOTS
Alias : 5d9dad30bc64732db4cfb1d7b14339fd147bb431
            Not After : Mar  5 08:23:29 2033 GMT
Alias : ba1cb7fd2c908b633ce8e8b573af8261827c91bc
            Not After : Nov 24 06:32:14 2033 GMT
STORE TRUSTED_ROOT_CRLS
Alias : 1c270b74eef59ba889dbc6f00b191fe50d1fc562
Alias : 5b45d7645175d0a41548a17b2a9e6b8ba59ffa8d
STORE machine
Alias : machine
            Not After : Nov 29 06:22:42 2025 GMT
STORE vsphere-webclient
Alias : vsphere-webclient
            Not After : Nov 29 06:22:43 2025 GMT
STORE vpxd
Alias : vpxd
            Not After : Nov 29 06:22:45 2025 GMT
STORE vpxd-extension
Alias : vpxd-extension
            Not After : Nov 29 06:22:46 2025 GMT
STORE hvc
Alias : hvc
            Not After : Nov 29 06:22:49 2025 GMT
STORE data-encipherment
Alias : data-encipherment
            Not After : Mar  5 08:23:29 2033 GMT
STORE APPLMGMT_PASSWORD
STORE SMS
Alias : sms_self_signed
            Not After : Mar 11 08:33:14 2033 GMT
STORE wcp
Alias : wcp
            Not After : Nov 29 06:22:50 2023 GMT

# for Hostname
~]# hostname -f
localhost

# for VMCA Name
~]# /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
192.168.3.199

~]# cat /var/tmp/vmware/certool.cfg

~]# /usr/lib/vmware-vmca/bin/certificate-manager
                 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
                |                                                                     |
                |      *** Welcome to the vSphere 7.0 Certificate Manager  ***        |
                |                                                                     |
                |                   -- Select Operation --                            |
                |                                                                     |
                |      1. Replace Machine SSL certificate with Custom Certificate     |
                |                                                                     |
                |      2. Replace VMCA Root certificate with Custom Signing           |
                |         Certificate and replace all Certificates                    |
                |                                                                     |
                |      3. Replace Machine SSL certificate with VMCA Certificate       |
                |                                                                     |
                |      4. Regenerate a new VMCA Root Certificate and                  |
                |         replace all certificates                                    |
                |                                                                     |
                |      5. Replace Solution user certificates with                     |
                |         Custom Certificate                                          |
                |         NOTE: Solution user certs will be deprecated in a future    |
                |         release of vCenter. Refer to release notes for more details.|
                |                                                                     |
                |      6. Replace Solution user certificates with VMCA certificates   |
                |                                                                     |
                |      7. Revert last performed operation by re-publishing old        |
                |         certificates                                                |
                |                                                                     |
                |      8. Reset all Certificates                                      |
                |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-D to exit.
Option[1 to 8]: 8
Do you wish to generate all certificates using configuration file : Option[Y/N] ? : N

Please provide valid SSO and VC privileged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:
Enter password:
certool.cfg file exists, Do you wish to reconfigure : Option[Y/N] ? : Y

Press Enter key to skip optional parameters or use Previous value.

Enter proper value for 'Country' [Previous value : US] :

Enter proper value for 'Name' [Previous value : CA] :

Enter proper value for 'Organization' [Previous value : VMware] :

Enter proper value for 'OrgUnit' [Previous value : VMware Engineering] :

Enter proper value for 'State' [Previous value : California] :

Enter proper value for 'Locality' [Previous value : Palo Alto] :

Enter proper value for 'IPAddress' (Provide comma separated values for multiple IP addresses) [optional] : 192.168.3.199

Enter proper value for 'Email' [Previous value : email@acme.com] :

Enter proper value for 'Hostname' (Provide comma separated values for multiple Hostname entries) [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : localhost

Enter proper value for VMCA 'Name' :192.168.3.199
Continue operation : Option[Y/N] ? : Y

You are going to reset by regenerating Root Certificate and replace all certificates using VMCA
Continue operation : Option[Y/N] ? : Y
Status : 60% Completed [Reset vpxd-extension Cert...]
2024-03-14T04:09:34.584Z  Updating certificate for "com.vmware.vim.eam" extension


2024-03-14T04:09:35.119Z  Updating certificate for "com.vmware.rbd" extension


2024-03-14T04:09:35.607Z  Updating certificate for "com.vmware.imagebuilder" extension

Reset status : 100% Completed [Reset completed successfully]