A Quick one to Update the vCenter thumbprint in TCA and TKG after a vCenter certificate has been updated.

Symptoms-

In Telco Cloud Automation (TCA) 2.X, all cluster, nodepool, and Network Function (NF) operations require a secure trust to vCenter via a vSphere certificate thumbprint.

If a vCenter's certificate has been updated, the following steps must be followed to restore functionality to TCA 2.X

This Impacts Telco Cloud Automation 2.X

Resolution-

Update the vCenter thumbprint in TCA and TKG after a vCenter certificate has been updated.

1. Obtain the vSphere certificate thumbprint.

2. SSH into each TCA-CP that is attached to the vCenter as the admin user (for minikube).

3. Run the following command and take note of the Custom Resource (CR) name and namespace: kubectl get VCenterPrime -A

4. Run the following command to edit the CR of the target vCenter environment: kubectl edit VCenterPrime -n <namespace> <cr_name>

5. Change the thumbprint field to match the vSphere certificate thumbprint from step 1.

6. Synchronize the Cell Site Domain Data. Note: Wait until the Parent domain's status changes to PROVISIONED before proceeding.

7. Synchronize the Cell Site Host Data. Note: Only resync the host(s) that have failed to provision as resyncing all hosts is a time consuming process. Note: Ensure the partial resync checkbox is NOT checked to enable a full resync of the cell site host.

For each management cluster:

1. SSH into the management cluster control plane VIP with capv

2. Run the following command and note the CR name and namespace:
   

   • kubectl get VCenterPrime –A

1. Edit the CR for the target vCenter enviornment with the following command:
   

   • kubectl edit VCenterPrime -n <namespace> <cr_name>

1. Change the "thumbprint" field to match the correct thumbprint from step 1.

If the thumbprint of a market VC changes:

1. Run the following command and note the CR name and namespace.
   

   • kubectl get VCenterSub -A

1. Edit the CR for the target vCenter enviornment with the following command:
   

   • kubectl edit VCenterSub -n <namespace> <cr_name>

1. Change the "thumbprint" field to match the correct thumbprint from step 1.

Ensure the correct vCenter is being updated by verifying the IP address field in the CR.

apiVersion: telco.vmware.com/v1alpha1 
kind: VCenterPrime 
metadata: 
  name: vcprime-mgmt-cluster07 
  namespace: tca-system 
spec: 
  server: 
    address: 10.208.70.139  ===> verify the correct vCenter 
    credentialRef: 
      kind: Secret 
      name: vcprime-mgmt-cluster07-secret 
      namespace: tca-system 
  subConfig: 
    datacenter: tcpscale-VMCCloudDC 
    thumbprint: FB:3A:8E:E1:B3:23:DD:FF:F3:6E:19:BE:FE:01:E1:18:E8:24:88:F3 ===> change here 

To re-import the TLS thumbprint for TCA-M/TCA-CP (vCenter self-signed certificates only)

1. Login to TCA-M/TCA-CP manage page with tca-m/tca-cp:9443

2. Re-import the new vCenter TLS thumbprint by “Administration” -> Certificate -> Trusted CA Certificate -> Click the “Import” button

To update the TLS thumbprint for TCA TKG Management Clusters

If vCenter Certificate was changed for a stretch cluster environment:

1. SSH into management cluster control plane VIP with capv and update the {mgmt-cluster-name}-vsphere-cpi-addon secret in the management cluster context
   

   • kubectl get secret -A | grep cpi-addon

1. Save original CPI vSphere configure to temporary file
   

   • kubectl get secret -n tkg-system {mgmt-cluster-name}-vsphere-cpi-addon -o jsonpath='{.data.vsphereconf-custom\.lib\.txt}' |base64 -d >/tmp/vsphereconf.txt

1. Update the CPI vSphere config with new thumbprint in the temporary file
   

   • Sample as following of CPI vsphere config: [root@tca /home/admin]# vim /tmp/vsphereconf.txt ((@def vsphere_conf(): -@) [Global] user = "administrator@vsphere.local" password = "Admin!23" port = "443" datacenters = "os-test-dc, cellsite-dc" [VirtualCenter "10.185.11.97"] datacenters = "os-test-dc" thumbprint = "13:C1:98:D9:E2:DF:A9:5A:95:4C:6A:96:FA:8D:FE:CF:56:6C:D3:1C" ===> change to new thumbprint ip-family = "ipv4" [VirtualCenter "sc2-10-185-10-130.eng.vmware.com"] datacenters = "cellsite-dc" thumbprint = "FD:89:0D:8D:B6:A6:FA:EB:E2:B7:15:CF:D3:F0:57:EB:8C:E3:96:70" ip-family = "ipv4" [Workspace] server = "10.185.11.97" datacenter = "os-test-dc" thumbprint = "13:C1:98:D9:E2:DF:A9:5A:95:4C:6A:96:FA:8D:FE:CF:56:6C:D3:1C" ===> change to new thumbprint ip-family = "ipv4"

1. Encode the CPI vsphere configure with new thumbprint
   

   • export encoded_vsphereconf_content=`base64 -w 0 /tmp/vsphereconf.txt`

1. Update the secret {mgmt-cluster-name}-vsphere-cpi-addon in tkg-system namespace in the management cluster context, then wait for Kapp reconciliation. Once reconciled, vsphere-cloud-config configmap in kube-system namespace will be updated.
   

   • kubectl patch secret {mgmt-cluster-name}-vsphere-cpi-addon -n tkg-system -p '{"data": {"vsphereconf-custom.lib.txt":"'${encoded_vsphereconf_content}'"}}'

1. Verify the configmap is updated using the below command on the management cluster context:
   

   • kubectl -n kube-system get cm vsphere-cloud-config -o yaml

1. Restart the vsphere-cloud-controller-manager pod so that the new configmap is mounted
   

   • kubectl rollout restart ds/vsphere-cloud-controller-manager -n kube-system