Symptoms-
In Telco Cloud Automation (TCA) 2.X, all cluster, nodepool, and Network Function (NF) operations require a secure trust to vCenter via a vSphere certificate thumbprint.
If a vCenter's certificate has been updated, the following steps must be followed to restore functionality to TCA 2.X
This Impacts Telco Cloud Automation 2.X
Resolution-
Update the vCenter thumbprint in TCA and TKG after a vCenter certificate has been updated.
Obtain the vSphere certificate thumbprint.
SSH into each TCA-CP that is attached to the vCenter as the admin user (for minikube).
Run the following command and take note of the Custom Resource (CR) name and namespace: kubectl get VCenterPrime -A
Run the following command to edit the CR of the target vCenter environment: kubectl edit VCenterPrime -n <namespace> <cr_name>
Change the thumbprint field to match the vSphere certificate thumbprint from step 1.
Synchronize the Cell Site Domain Data. Note: Wait until the Parent domain's status changes to PROVISIONED before proceeding.
Synchronize the Cell Site Host Data. Note: Only resync the host(s) that have failed to provision as resyncing all hosts is a time consuming process. Note: Ensure the partial resync checkbox is NOT checked to enable a full resync of the cell site host.
For each management cluster:
SSH into the management cluster control plane VIP with capv
Run the following command and note the CR name and namespace:
kubectl get VCenterPrime –A
Edit the CR for the target vCenter enviornment with the following command:
kubectl edit VCenterPrime -n <namespace> <cr_name>
Change the "thumbprint" field to match the correct thumbprint from step 1.
If the thumbprint of a market VC changes:
Run the following command and note the CR name and namespace.
kubectl get VCenterSub -A
Edit the CR for the target vCenter enviornment with the following command:
kubectl edit VCenterSub -n <namespace> <cr_name>
Change the "thumbprint" field to match the correct thumbprint from step 1.
Ensure the correct vCenter is being updated by verifying the IP address field in the CR.
apiVersion: telco.vmware.com/v1alpha1
kind: VCenterPrime
metadata:
name: vcprime-mgmt-cluster07
namespace: tca-system
spec:
server:
address: 10.208.70.139 ===> verify the correct vCenter
credentialRef:
kind: Secret
name: vcprime-mgmt-cluster07-secret
namespace: tca-system
subConfig:
datacenter: tcpscale-VMCCloudDC
thumbprint: FB:3A:8E:E1:B3:23:DD:FF:F3:6E:19:BE:FE:01:E1:18:E8:24:88:F3 ===> change here
To re-import the TLS thumbprint for TCA-M/TCA-CP (vCenter self-signed certificates only)
Login to TCA-M/TCA-CP manage page with tca-m/tca-cp:9443
Re-import the new vCenter TLS thumbprint by “Administration” -> Certificate -> Trusted CA Certificate -> Click the “Import” button
To update the TLS thumbprint for TCA TKG Management Clusters
If vCenter Certificate was changed for a stretch cluster environment:
SSH into management cluster control plane VIP with capv and update the {mgmt-cluster-name}-vsphere-cpi-addon secret in the management cluster context
kubectl get secret -A | grep cpi-addon
Save original CPI vSphere configure to temporary file
kubectl get secret -n tkg-system {mgmt-cluster-name}-vsphere-cpi-addon -o jsonpath='{.data.vsphereconf-custom\.lib\.txt}' |base64 -d >/tmp/vsphereconf.txt
Update the CPI vSphere config with new thumbprint in the temporary file
Sample as following of CPI vsphere config: [root@tca /home/admin]# vim /tmp/vsphereconf.txt ((@def vsphere_conf(): -@) [Global] user = "administrator@vsphere.local" password = "Admin!23" port = "443" datacenters = "os-test-dc, cellsite-dc" [VirtualCenter "10.185.11.97"] datacenters = "os-test-dc" thumbprint = "13:C1:98:D9:E2:DF:A9:5A:95:4C:6A:96:FA:8D:FE:CF:56:6C:D3:1C" ===> change to new thumbprint ip-family = "ipv4" [VirtualCenter "sc2-10-185-10-130.eng.vmware.com"] datacenters = "cellsite-dc" thumbprint = "FD:89:0D:8D:B6:A6:FA:EB:E2:B7:15:CF:D3:F0:57:EB:8C:E3:96:70" ip-family = "ipv4" [Workspace] server = "10.185.11.97" datacenter = "os-test-dc" thumbprint = "13:C1:98:D9:E2:DF:A9:5A:95:4C:6A:96:FA:8D:FE:CF:56:6C:D3:1C" ===> change to new thumbprint ip-family = "ipv4"
Encode the CPI vsphere configure with new thumbprint
export encoded_vsphereconf_content=`base64 -w 0 /tmp/vsphereconf.txt`
Update the secret {mgmt-cluster-name}-vsphere-cpi-addon in tkg-system namespace in the management cluster context, then wait for Kapp reconciliation. Once reconciled, vsphere-cloud-config configmap in kube-system namespace will be updated.
kubectl patch secret {mgmt-cluster-name}-vsphere-cpi-addon -n tkg-system -p '{"data": {"vsphereconf-custom.lib.txt":"'${encoded_vsphereconf_content}'"}}'
Verify the configmap is updated using the below command on the management cluster context:
kubectl -n kube-system get cm vsphere-cloud-config -o yaml
Restart the vsphere-cloud-controller-manager pod so that the new configmap is mounted
kubectl rollout restart ds/vsphere-cloud-controller-manager -n kube-system
Comments