tseed
/
openstack_rhosp16.2_nvidia_cumulus_ceph_erasure_coding
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
openstack_rhosp16.2_nvidia_.../9) Updating the external HT...

330 lines
15 KiB
Markdown
Raw Permalink Normal View History

initial commit
2022-10-26 16:58:48 +00:00
# check certificate for the Openstack Horizon dashboard
```sh
openssl s_client -showcerts -connect stack.university.ac.uk:443
Certificate chain
0 s:C = GB, ST = England, L = University, CN = stack.university.ac.uk
i:C = GB, ST = England, L = University, O = UOE, OU = Cloud, CN = University Openstack CA
```
We see the certificate is signed by the CA "University Openstack CA" created in the build guide, this is not quite a self signed certificate but has broadly the same level of security unless the CA cert is not installed on the client machines.
# Check the certificate bundle recieved from an external signing authority
## Unpack and inspect
```sh
sudo dnf install unzip -y
unzip stack.university.ac.uk.zip
tree .
├── stack.university.ac.uk.cer full certificate chain, order: service certificate, intermediate CA, intermediate CA, top level CA
├── stack.university.ac.uk.cert.cer service certificate for stack.university.ac.uk
├── stack.university.ac.uk.csr certificate signing request (sent to public CA)
├── stack.university.ac.uk.interm.cer chain of intermediate and top level CA certificates, order: intermedia CA (Extended CA), intermediate CA, top level CA 321
└── stack.university.ac.uk.key certificate private key
```
## Check each certificate to determine what has been included in the bundle
Some signing authorities will not include all CA certificates in the bundle, it is up to you to inspect the service certificate and trace back through the certificate chain to obtain the various CA certificates.
### certificate information
Inspect service certificate.
```sh
#openssl x509 -in stack.university.ac.uk.cert.cer -text -noout
cfssl-certinfo -cert stack.university.ac.uk.cert.cer
```
Service certificate attributes.
```
"common_name": "stack.university.ac.uk"
"sans": [
"stack.university.ac.uk",
"www.stack.university.ac.uk"
],
"not_before": "2022-03-16T00:00:00Z",
"not_after": "2023-03-16T23:59:59Z",
```
### full certificate chain content
Copy out each certificate from the full chain file `stack.university.ac.uk.cer` to its own temp file, run the openssl text query command `openssl x509 -in <cert.N> -text -noout` to inspect each certificate.
The full chain certificate file is listed in following order. From the service certificate `stack.university.ac.uk` each certificate is signed by the preceding CA.
| Certificate context name | purpose | capability |
| --- | --- | --- |
| CN = AAA Certificate Services | top level CA | CA capability |
| CN = USERTrust RSA Certification Authority | intermediate CA | CA capability |
| CN = GEANT OV RSA CA 4 | intermediate CA | CA capability<br>extended validation capability |
| CN = stack.university.ac.uk | the service certificate | stack.university.ac.uk certificate |
## Check that the certificate chain is present by default in the trust store on the clients
Open certmgr in windows, check in "Trusted Root Authorities/Certificates" for each CA/Intermediate-CA certificate, all certificated will likely be present.
- look for the context name (CN)
- check the "X509v3 Subject Key Identifier" matches the "subject key identifier" from the `openssl x509 -in stack.university.ac.uk.cert.cer -text -noout` output
Windows includes certificates for "AAA Certificate Services" and "USERTrust RSA Certification Authority", the extended validation Intermediate CA "GEANT OV RSA CA 4" maybe missing, this is not an issue as the client has the top level CAs so can validate and follow the signing chain.
For modern Linux distros we find only one intermediate CA, this should be sufficient as any handshake using certificates signed from this will be able to validate. If the undercloud can find a CA in its trust store the deployed cluster nodes will most likely have it.
```sh
trust list | grep -i label | grep -i "USERTrust RSA Certification Authority"
# generally all certificates imported into the trust store get rendered into this global file
/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
# search the trust store for "USERTrust RSA Certification Authority", copy the content of the certificate field into a temp file for the following stanza
nano -cw /usr/share/pki/ca-trust-source/ca-bundle.trust.p11-kit
[p11-kit-object-v1]
label: "USERTrust RSA Certification Authority"
trusted: true
nss-mozilla-ca-policy: true
modifiable: false
# check the "X509v3 Subject Key Identifier" matches the CA in the certificate chain you recieved from the signing authority.
openssl x509 -in <temp file> -text -noout | grep "53:79:BF:5A:AA:2B:4A:CF:54:80:E1:D8:9B:C0:9D:F2:B2:03:66:CB"
```
Browsers such as Edge and Chrome will use the OS trust store, Firefox distributes its own trust store.
- 3 bar burger -> settings -> security -> view certificates -> authorities -> The UserTrust Network -> USERTrust RSA Certification Authority
We find the fingerprint from the openssl command "X509v3 Subject Key Identifier" matches the certificate field "subject key identifier" in Firefox.
## Configure the undercloud to use the CAs
```sh
trust list | grep label | wc -l
148
sudo cp /home/stack/CERT/stack.university.ac.uk/stack.university.ac.uk.interm.cer /etc/pki/ca-trust/source/anchors/public_ca_chain.pem
sudo update-ca-trust extract
# although the certificate chain includes 3 certificates only 1 is imported, this is the imtermediate CA "CN = GEANT OV RSA CA 4" that is not part of a default trust store
trust list | grep label | wc -l
149
# check CA/trusted certificates available to the OS
trust list | grep label | grep -i "AAA Certificate Services"
label: AAA Certificate Services
trust list | grep label | grep -i "USERTrust RSA Certification Authority"
label: USERTrust RSA Certification Authority
label: USERTrust RSA Certification Authority
trust list | grep label | grep -i "GEANT OV RSA CA 4"
label: GEANT OV RSA CA 4
```
## Configure the controller nodes to use the publicly signed certificate
NOTE: "PublicTLSCAFile" is used both by the overcloud HAProxy configuration and the undercloud installer to contact https://stack.university.ac.uk:13000
- The documentation presents the "PublicTLSCAFile" configuration item as the root CA certificate.
- When the undercloud runs various custom Openstack ansible modules, the python libraries run have a completely empty trust store that do not reference the undercloud OS trust store and do not ingest shell variables to set trust store sources.
- For the python to validate the overcloud public API endpoint, the full trust chain must be present. Python is not fussy about the order of certificates in this file, the vendor CA trust chain file in this case was ordered starting with the root CA.
Backup /home/stack/templates/enable-tls.yaml `mv /home/stack/templates/enable-tls.yaml /home/stack/templates/enable-tls.yaml.internal_ca`
Create new `/home/stack/templates/enable-tls.yaml`, the content for each field is source as follows:
```
PublicTLSCAFile: '/etc/pki/ca-trust/source/anchors/public_ca_chain.pem'
SSLCertificate: content from stack.university.ac.uk.cer
SSLIntermediateCertificate: use both intermediate certificates, in the order intermediate-2, intermediate-1 (RFC5426)
SSLKey: content from stack.university.ac.uk.key
```
The fully populated /home/stack/templates/enable-tls.yaml:
NOTE: the intermediate certificates configuration item contains both intermediate certificates
Luckliy Openstack does not validate this field and pushes it directly into the HAProxy pem file, the order of the pem is as NGINX preferes (RFC5426), service certificate, intermediate CA2, intermediate CA1, root CA.
During the SSL handshake the client will check the intermediate certificates in the response, if they are not present in the local trust store signing will be checked up to the root CA which will be in the client trust store.
```yaml
parameter_defaults:
# Set CSRF_COOKIE_SECURE / SESSION_COOKIE_SECURE in Horizon
# Type: boolean
HorizonSecureCookies: True
# Specifies the default CA cert to use if TLS is used for services in the public network.
# Type: string
# PublicTLSCAFile: '/etc/pki/ca-trust/source/anchors/public_ca.pem'
PublicTLSCAFile: '/home/stack/templates/stack.university.ac.uk.interm.cer'
# The content of the SSL certificate (without Key) in PEM format.
# Type: string
SSLCertificate: |
-----BEGIN CERTIFICATE-----
MIIHYDCCBUigAwIBAgIRAK55qnAAkkQKzs6cusLn+0IwDQYJKoZIhvcNAQEMBQAw
.....
+vXuwEyJ5ULoW0TO6CuQvAvJsVM=
-----END CERTIFICATE-----
# The content of an SSL intermediate CA certificate in PEM format.
# Type: string
SSLIntermediateCertificate: |
-----BEGIN CERTIFICATE-----
MIIG5TCCBM2gAwIBAgIRANpDvROb0li7TdYcrMTz2+AwDQYJKoZIhvcNAQEMBQAw
.....
Ipwgu2L/WJclvd6g+ZA/iWkLSMcpnFb+uX6QBqvD6+RNxul1FaB5iHY=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFgTCCBGmgAwIBAgIQOXJEOvkit1HX02wQ3TE1lTANBgkqhkiG9w0BAQwFADB7
.....
vGp4z7h/jnZymQyd/teRCBaho1+V
-----END CERTIFICATE-----
# The content of the SSL Key in PEM format.
# Type: string
SSLKey: |
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAqXvJwxSDfxjapmRMqFlchTPPpGUi6n0lFbJ7G2YQ+HUBwaEZ
.....
PcVhU+Ybi7ABCOyRUzZWXDlf6DxF4Kgoe/Ak99nM7v0MIndlbgZBYA==
-----END RSA PRIVATE KEY-----
# ******************************************************
# Static parameters - these are values that must be
# included in the environment but should not be changed.
# ******************************************************
# The filepath of the certificate as it will be stored in the controller.
# Type: string
DeployedSSLCertificatePath: /etc/pki/tls/private/overcloud_endpoint.pem
```
## Update the overcloud nodes to have all of the CA + Intermediate CA certificates imported into their trust stores
Whilst the overcloud nodes shouldn't use the public certificate for inter-service API communication (this is not a TLS everywhere installation), include this CA chain as a caution.
Backup /home/stack/templates/inject-trust-anchor-hiera.yaml `mv /home/stack/templates/inject-trust-anchor-hiera.yaml /home/stack/templates/inject-trust-anchor-hiera.yaml.internal_ca`
Create new `/home/stack/templates/inject-trust-anchor-hiera.yaml`, the content for each field is source as follows:
```yaml
CAMap:
root-ca:
content: |
"CN = AAA Certificate Services" certificate content here
intermediate-ca-1:
content: |
"CN = USERTrust RSA Certification Authority" certificate content here
intermediate-ca-2:
content: |
"CN = GEANT OV RSA CA 4" certificate content here
```
The fully populated /home/stack/templates/inject-trust-anchor-hiera.yaml.
```sh
parameter_defaults:
# Map containing the CA certs and information needed for deploying them.
# Type: json
CAMap:
root-ca:
content: |
-----BEGIN CERTIFICATE-----
MIIEMjCCAxqgAwIBAgIBATANBgkqhkiG9w0BAQUFADB7MQswCQYDVQQGEwJHQjEb
.....
smPi9WIsgtRqAEFQ8TmDn5XpNpaYbg==
-----END CERTIFICATE-----
intermediate-ca-1:
content: |
-----BEGIN CERTIFICATE-----
MIIFgTCCBGmgAwIBAgIQOXJEOvkit1HX02wQ3TE1lTANBgkqhkiG9w0BAQwFADB7
.....
vGp4z7h/jnZymQyd/teRCBaho1+V
-----END CERTIFICATE-----
intermediate-ca-2:
content: |
-----BEGIN CERTIFICATE-----
MIIG5TCCBM2gAwIBAgIRANpDvROb0li7TdYcrMTz2+AwDQYJKoZIhvcNAQEMBQAw
.....
Ipwgu2L/WJclvd6g+ZA/iWkLSMcpnFb+uX6QBqvD6+RNxul1FaB5iHY=
-----END CERTIFICATE-----
```
## Deploy the overcloud
The FQDN of the floating IP served by the HAProxy containers on the controller nodes must have an upstream DNS A record, this should be present as the `CloudName:` parameter.
The DNS hosts should return the A record, for University - the internal DNS server and a publically published record resolve stack.university.ac.uk.
```sh
grep CloudName: /home/stack/templates/custom-domain.yaml
CloudName: stack.university.ac.uk
grep DnsServers: /home/stack/templates/custom-domain.yaml
DnsServers: ["144.173.6.71", "1.1.1.1"]
[stack@undercloud templates]$ grep 10.121.4.14 vips.yaml
PublicVirtualFixedIPs: [{'ip_address':'10.121.4.14'}]
dig stack.university.ac.uk @144.173.6.71
dig stack.university.ac.uk @1.1.1.1
;; ANSWER SECTION:
stack.university.ac.uk. 86400 IN A 10.121.4.14
```
Use the exact same arguments as the previous deployment to mitigate any unwanted changes to the cluster, for this build the script `overcloud-deploy.sh` should be up to date with this record.
```sh
./overcloud-deploy.sh
```
The update will complete for any overcloud nodes, however the undercloud may time out contacting the external API endpoint with the new SSL certificate changed.
The HAProxy containers on the controller nodes need to be restarted to pick up the new certificates.
If you were to run the deployment again (with no changes and restarted HAProxy containers) it should complete without issue and set the deployment with status 'UPDATE COMPLETE' when checking `openstack stack list`.
## Restart HAProxy containers on the controller nodes
Follow the instructions to restart the HAProxy containers on the overcloud controller nodes once the deployment has finished updating the SSL certificate.
> [https://access.redhat.com/documentation/en-us/red\_hat\_openstack\_platform/16.2/html/advanced\_overcloud\_customization/assembly\_enabling-ssl-tls-on-overcloud-public-endpoints#proc\_manually-updating-ssl-tls-certificates\_enabling-ssl-tls-on-overcloud-public-endpoints](https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html/advanced_overcloud_customization/assembly_enabling-ssl-tls-on-overcloud-public-endpoints#proc_manually-updating-ssl-tls-certificates_enabling-ssl-tls-on-overcloud-public-endpoints)
```sh
grep control /etc/hosts | grep ctlplane
10.122.0.30 overcloud-controller-0.ctlplane.university.ac.uk overcloud-controller-0.ctlplane
10.122.0.31 overcloud-controller-1.ctlplane.university.ac.uk overcloud-controller-1.ctlplane
10.122.0.32 overcloud-controller-2.ctlplane.university.ac.uk overcloud-controller-2.ctlplane
# for each controller node
ssh heat-admin@overcloud-controller-0.ctlplane.university.ac.uk
sudo su -
podman restart $(podman ps --format="{{.Names}}" | grep -w -E 'haproxy(-bundle-.*-[0-9]+)?')
```
# SSL notes
Verify a full chain of certificates easily.
```sh
openssl verify -verbose -CAfile <(cat CERT/stack.university.ac.uk/intermediate_ca_2.pem CERT/stack.university.ac.uk/intermediate_ca_1.pem CERT/stack.university.ac.uk/root_ca.pem) CERT/stack.university.ac.uk/service_cert.pe
```
Check a certificate key is valid for a certificate.
```sh
openssl x509 -noout -modulus -in CERT/stack.university.ac.uk/stack.university.ac.uk.cert.cer | openssl md5
(stdin)= 60a5df743ac212edb2b28bf315bce828
openssl rsa -noout -modulus -in CERT/stack.university.ac.uk/stack.university.ac.uk.key | openssl md5
(stdin)= 60a5df743ac212edb2b28bf315bce828
```
Format of nginx type cert, haproxy (built on nginx) uses this format. When populating Openstack configuration files with multiple intermediate certs in a single field order multiple intermediate certs as so.
```
# create chain bundle, order as per RFC5426 (IETF's RFC 5246 Section 7.4.2) search google for nginx cert chain order.
cat ../out/service.pem ../out/ca.pem > ../out/reg-chain.pem
# the order with multiple intermediate certs would resemble
cert
int 2
int 1
root
```