19 KiB
Executable File
19 KiB
Executable File
Proxmox installation
Proxmox hosts the undercloud node, this enables snapshots to assist in Update/DR/Rebuild scenarios, primarily this will allow a point in time capture of working heat-templates and containers.
| setting | value |
|---|---|
| filesystem | xfs |
| swapsize | 8GB |
| maxroot | 50GB |
| country | United Kingdom |
| time zone | Europe/London |
| keyboard layout | United Kingdom |
| password | Password0 |
| user@university.ac.uk (this can be changed in the web console @ datacenter/users/root) | |
| management interface | eno1 |
| hostname | pve.local |
| ip address | 10.122.0.5/24 |
| gateway | 10.122.0.1 (placeholder, there is no gateway on this range) |
| dns | 144.173.6.71 |
- Install from a standard version 7.2 ISO, use settings listed as above.
- Create a bridge on the 1G management interface, this is VLAN 1 native on the 'ctlplane' network with VLAN 2 tagged for IPMI traffic.
- Ensure the 25G interfaces are setup as an LACP bond, create a bridge on the bond with the 'tenant', 'storage', 'internal-api' and 'external' VLANs as tagged (the 'external' range has the default gateway).
- Proxmox host has VLAN interfaces into each openstack network for introspection/debug, nmap is installed.
cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!
auto lo
iface lo inet loopback
iface eno1 inet manual
iface eno2 inet manual
iface eno3 inet manual
iface eno4 inet manual
iface enx3a68dd4a4c5f inet manual
auto ens2f0np0
iface ens2f0np0 inet manual
auto ens2f1np1
iface ens2f1np1 inet manual
auto bond0
iface bond0 inet manual
bond-slaves ens2f0np0 ens2f1np1
bond-miimon 100
bond-mode 802.3ad
auto vmbr0
iface vmbr0 inet static
address 10.122.0.5/24
bridge-ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#vlan 1(native) 2 (tagged) ControlPlane
auto vmbr1
iface vmbr1 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#vlan 1(native) 11 12 13 1214 (tagged)
auto vlan2
iface vlan2 inet static
address 10.122.1.5/24
vlan-raw-device vmbr0
#IPMI
auto vlan13
iface vlan13 inet static
address 10.122.10.5/24
vlan-raw-device vmbr1
#Storage
auto vlan1214
iface vlan1214 inet static
address 10.121.4.5/24
gateway 10.121.4.1
vlan-raw-device vmbr1
#External
auto vlan12
iface vlan12 inet static
address 10.122.6.5/24
vlan-raw-device vmbr1
#InternalApi
auto vlan11
iface vlan11 inet static
address 10.122.8.5/24
vlan-raw-device vmbr1
#Tenant
Setup the no-subscription repository.
# comment/disable enterprise repo
nano -cw /etc/apt/sources.list.d/pve-enterprise.list
#deb https://enterprise.proxmox.com/debian/pve bullseye pve-enterprise
# insert pve-no-subscription repo
nano -cw /etc/apt/sources.list
deb http://ftp.uk.debian.org/debian bullseye main contrib
deb http://ftp.uk.debian.org/debian bullseye-updates main contrib
# security updates
deb http://security.debian.org bullseye-security main contrib
# pve-no-subscription
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
# update
apt-get update
apt-get upgrade -y
reboot
Download some LXC containers.
- LXC is not used in production, but during build LXC containers with network interfaces in all ranges (last octet suffix .6) was used to debug IP connectivity, switch configuration and serve linux boot images over NFS for XClarity.
pveam update
pveam available --section system
pveam download local almalinux-8-default_20210928_amd64.tar.xz
pveam download local rockylinux-8-default_20210929_amd64.tar.xz
pveam download local ubuntu-18.04-standard_18.04.1-1_amd64.tar.gz
pveam download local ubuntu-22.04-standard_22.04-1_amd64.tar.zst
pveam list local
NAME SIZE
local:vztmpl/almalinux-8-default_20210928_amd64.tar.xz 109.08MB
local:vztmpl/rockylinux-8-default_20210929_amd64.tar.xz 107.34MB
local:vztmpl/ubuntu-18.04-standard_18.04.1-1_amd64.tar.gz 203.54MB
local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst 123.81MB
Undercloud VM instance
Download RHEL 8.4 full DVD image
Select the RHEL8.4 image, choose the full image rather than the boot image, this will allow installation without registering the system during the installer, you may then attach the system to a license via the subscription-manager tool after the host is built.
Install spec
- RHEL8 (RHEL 8.4 specifically)
- 1 socket, 16 core (must use HOST cpu type for nested virtualization)
- 24GB ram
- 100GB disk (/root 89GiB lvm, /boot 1024MiB, swap 10GiB lvm)
- ControlPlane network interface on vmbr0, no/native vlan, 10.122.0.25/24, ens18
- IPMI network interface on vmbr0, vlan2 (vlan assigned in proxmox not OS), 10.122.1.25/24, ens19
- External/Routable network interface on vmbr1, vlan 1214 (vlan assigned in proxmox not OS), 10.121.4.25/24, gateway 10.121.4.1, dns 144.173.6.7,1 1.1.1.1, ens20
- ensure all network interfaces do not have the firewall enabled in proxmox or OS (mac spoofing will be required and should be allowed in the firewall if used)
- root:Password0
- undercloud.local
- minimal install with QEMU guest agents
- will require registering with redhat subscription service
OCF partner subscription entitlement
Register for a partner product entitlement.
https://partnercenter.redhat.com/NFRPageLayout Product: Red Hat OpenStack Platform, Standard Support (4 Sockets, NFR, Partner Only) - 25.0 Units
Once the customer has purchased the entitlement, this should be present in their own RedHat portal to consume on the production nodes.
Register undercloud node with the require software repositories
Browse to:
Create a new system with the following attributes.
- Virtual System
- Name: university_test
- Architecture: x86_64
- Number of vCPUs: 16
- Red Hat Enterprise Linux Version: 8
- Create
Attach the following initial subscription: 'Red Hat Enterprise Linux, Self-Support (128 Sockets, NFR, Partner Only)' Note the name and UUID of the system.
Register the system.
sudo su -
[root@undercloud ~]# subscription-manager register --name=university_test --consumerid=f870ae18-6664-4206-9a89-21f24f312866 --username=tseed@ocf.co.uk
Registering to: subscription.rhsm.redhat.com:443/subscription
Password:
The system has been registered with ID: a1b24b8a-933b-4ce8-8244-1a7e16ff51a3
The registered system name is: university_test
#[root@undercloud ~]# subscription-manager refresh
[root@undercloud ~]# subscription-manager list
+-------------------------------------------+
Installed Product Status
+-------------------------------------------+
Product Name: Red Hat Enterprise Linux for x86_64
Product ID: 479
Version: 8.4
Arch: x86_64
Status: Subscribed
Status Details:
Starts: 06/13/2022
Ends: 06/13/2023
[root@undercloud ~]# subscription-manager list
+-------------------------------------------+
Installed Product Status
+-------------------------------------------+
Product Name: Red Hat Enterprise Linux for x86_64
Product ID: 479
Version: 8.4
Arch: x86_64
Status: Subscribed
Status Details:
Starts: 06/13/2022
Ends: 06/13/2023
[root@undercloud ~]# subscription-manager identity
system identity: f870ae18-6664-4206-9a89-21f24f312866
name: university_test
org name: 4110881
org ID: 4110881
Add an entitlement to the license system.
# Check the entitlement/purchased-products portal
# you will find the SKU under a contract - this will help to identify the openstack entitlement if you have multiple
# find a suitable entitlement pool ID for Red Hat OpenStack Director Deployment Tools
subscription-manager list --available --all
subscription-manager list --available --all --matches="*OpenStack*"
Subscription Name: Red Hat OpenStack Platform, Standard Support (4 Sockets, NFR, Partner Only)
SKU: SER0505
Contract: 13256907
Pool ID: 8a82c68d812ba3c301815c6f842f5ecf
# attach to the entitlement pool ID
subscription-manager attach --pool=8a82c68d812ba3c301815c6f842f5ecf
Successfully attached a subscription for: Red Hat OpenStack Platform, Standard Support (4 Sockets, NFR, Partner Only)
1 local certificate has been deleted.
# set release version statically
subscription-manager release --set=8.4
Enable repositories, set version of container-tools, update packages.
subscription-manager repos --disable=* ;\
subscription-manager repos \
--enable=rhel-8-for-x86_64-baseos-eus-rpms \
--enable=rhel-8-for-x86_64-appstream-eus-rpms \
--enable=rhel-8-for-x86_64-highavailability-eus-rpms \
--enable=ansible-2.9-for-rhel-8-x86_64-rpms \
--enable=openstack-16.2-for-rhel-8-x86_64-rpms \
--enable=fast-datapath-for-rhel-8-x86_64-rpms ;\
dnf module disable -y container-tools:rhel8 ;\
dnf module enable -y container-tools:3.0 ;\
dnf update -y
reboot
Install Tripleo client
# install tripleoclient for install of the undercloud
dnf install -y python3-tripleoclient
# these packages are advised for the TLS everywhere functionality, probably not required for external TLS endpoint but wont hurt
dnf install -y python3-ipalib python3-ipaclient krb5-devel python3-novajoin
Install Ceph-Ansible packages, even if you are not initially using Ceph it cannot hurt to have an undercloud capable of deploying Ceph, to use external Ceph (as in not deployed by tripleo) you will need the following package.
There are different packages for different versions of Ceph, this is especially relevant when using external Ceph.
- Redhat Ceph 4.1 = Nautilus release
- Redhat Ceph 5.1 = Pacific release
subscription-manager repos | grep -i ceph
# Nautilus (default version in use with Tripleo deployed Ceph)
#subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms
# Pacific (if you are using external Ceph from the opensource repos you will likely be using this version)
#dnf remove -y ceph-ansible
#subscription-manager repos --disable=rhceph-4-tools-for-rhel-8-x86_64-rpms
subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms
# install
dnf info ceph-ansible
dnf install -y ceph-ansible
Configure and deploy the Tripleo undercloud
Prepare host
Disable firewalld.
systemctl disable firewalld
systemctl stop firewalld
Create user/sudoers, push ssh key. Sudoers required for the tripleo installer.
groupadd -r -g 1001 stack && useradd -r -u 1001 -g 1001 -m -s /bin/bash stack
echo "%stack ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/stack
chmod 0440 /etc/sudoers.d/stack
passwd stack # password is Password0
exit
ssh-copy-id -i ~/.ssh/id_rsa.pub stack@university-new-undercloud
Local ssh config setup.
nano -cw ~/.ssh/config
Host undercloud
Hostname 10.121.4.25
User stack
IdentityFile ~/.ssh/id_rsa
Set hostname, disable firewall (leave SElinux enabled, RHOSP tripleo requires it), install packages.
ssh undercloud
sudo su -
timedatectl set-timezone Europe/London
dnf install chrony nano -y
# replace server/pool entries with PHC (high precision clock device) entry to use the hypervisors hardware clock (which in turn is sync'd from online ntp pool), this should be the most acurate time for a VM
# the LXC container running ntp (192.168.101.43) does actually use the hypervisor hardware clock, the LXC container and VM should be on the same hypervisor if this is used
nano -cw /etc/chrony.conf
#server 192.168.101.43 iburst
#pool 2.centos.pool.ntp.org iburst
refclock PHC /dev/ptp0 poll 2
systemctl enable chronyd
echo ptp_kvm > /etc/modules-load.d/ptp_kvm.conf
# the undercloud installer should set the hostname based on the 'undercloud_hostname' entry in the undercloud.conf config file
# you can set it before deployment with the following, the Opensource tripleo documentation advises to allow the undercloud installer to set it
hostnamectl set-hostname undercloud.local
hostnamectl set-hostname --transient undercloud.local
# RHOSP hosts entry
nano -cw /etc/hosts
10.121.4.25 undercloud.local undercloud
reboot
hostname -A
hostname -s
# install some useful tools
sudo su -
dnf update -y
dnf install qemu-guest-agent nano tree lvm2 chrony telnet traceroute net-tools bind-utils python3 yum-utils mlocate ipmitool tmux wget -y
# need to shutdown for qemu-guest tools to function, ensure the VM profile on the hypervisor has guest agents enabled
shutdown -h now
Build the undercloud config file
The first interface (enp6s18 on the proxmox VM instance) will be on the ControlPlane range.
- Controller nodes are in all networks but cannot install nmap, can find hosts in ranges with
for ip in 10.122.6.{1..254}; do ping -c 1 -t 1 $ip > /dev/null && echo "${ip} is up"; done. - Proxmox has interfaces in every network and nmap installed
nmap -sn 10.122.6.0/24to assist with debug.
| Node | IPMI VLAN2 | Ctrl_plane VLAN1 | External VLAN1214 | Internal_api VLAN12 | Storage VLAN13 | Tenant VLAN11 |
|---|---|---|---|---|---|---|
| Proxmox | 10.122.1.54 (IPMI) (Proxmox interface 10.122.1.5) | 10.122.0.5 | 10.121.4.5 | 10.122.6.5 | 10.122.10.5 | 10.122.8.5 |
| Undercloud | 10.122.1.25 | 10.121.0.25-27 (br-ctlplane) | 10.121.4.25 (Undercloud VM) | NA | NA | NA |
| Temporary Storage Nodes | 10.122.1.55-57 | NA | 10.121.4.7-9 | NA | 10.122.10.7-9 | NA |
| Overcloud Controllers | 10.122.1.10-12 (Instance-HA 10.122.1.80-82 | 10.122.0.30-32 | 10.121.4.30-32 | 10.122.6.30-32 | 10.122.10.30-32 | 10.122.8.30-32 |
| Overcloud Networkers | 10.122.1.20-21 | 10.122.0.40-41 | NA (reserved 10.121.4.23-24) | 10.122.6.40-41 | NA | 10.122.8.40-41 |
| Overcloud Compute | 10.122.1.30-53/54,58-77 | 10.122.0.50-103 | NA | 10.122.6.50-103 | 10.122.10.50-103 | 10.122.8.50-103 |
sudo su - stack
nano -cw /home/stack/undercloud.conf
[DEFAULT]
certificate_generation_ca = local
clean_nodes = true
cleanup = true
container_cli = podman
container_images_file = containers-prepare-parameter.yaml
discovery_default_driver = ipmi
enable_ironic = true
enable_ironic_inspector = true
enable_nova = true
enabled_hardware_types = ipmi
generate_service_certificate = true
inspection_extras = true
inspection_interface = br-ctlplane
ipxe_enabled = true
ironic_default_network_interface = flat
ironic_enabled_network_interfaces = flat
local_interface = enp6s18
local_ip = 10.122.0.25/24
local_mtu = 1500
local_subnet = ctlplane-subnet
overcloud_domain_name = university.ac.uk
subnets = ctlplane-subnet
undercloud_admin_host = 10.122.0.27
undercloud_debug = true
undercloud_hostname = undercloud.local
undercloud_nameservers = 144.173.6.71,1.1.1.1
undercloud_ntp_servers = ntp.university.ac.uk,0.pool.ntp.org
undercloud_public_host = 10.122.0.26
[ctlplane-subnet]
cidr = 10.122.0.0/24
#dhcp_end = 10.122.0.140
#dhcp_start = 10.122.0.80
dhcp_end = 10.122.0.194
dhcp_start = 10.122.0.140
#dns_nameservers =
gateway = 10.122.0.25
#inspection_iprange = 10.122.0.141,10.122.0.201
inspection_iprange = 10.122.0.195,10.122.0.249
masquerade = true
RHEL Tripleo container preparation
Generate the /home/stack/containers-prepare-parameter.yaml config file using the default method for a local registry on the undercloud.
sudo su - stack
openstack tripleo container image prepare default \
--local-push-destination \
--output-env-file containers-prepare-parameter.yaml
Add the API key to download containers from RHEL Quay public registry.
RHEL requires containers to be pulled from Quay.io using a valid API token (unique to your RHEL account), containers-prepare-parameters.yaml must be modified to include the API key. The following opensource tripleo sections explain the containers-prepare-parameters.yaml in more detail, for a quick deployment use the following instructions.
Edit containers-prepare-parameter.yaml to include the Redhat Quay bearer token.
nano -cw /home/stack/containers-prepare-parameter.yaml
parameter_defaults:
ContainerImagePrepare:
- push_destination: true
set:
<....settings....>
tag_from_label: '{version}-{release}'
ContainerImageRegistryLogin: true
ContainerImageRegistryCredentials:
registry.redhat.io:
4110881|osp16-undercloud: long-bearer-token-here
Deploy the undercloud
Shutdown the Undercloud VM instance and take a snapshot in Proxmox, call it 'pre_undercloud_deploy'.
openstack undercloud install --dry-run
time openstack undercloud install
#time openstack undercloud install --verbose # if there are failing tasks
##########################################################
The Undercloud has been successfully installed.
Useful files:
Password file is at /home/stack/undercloud-passwords.conf
The stackrc file is at ~/stackrc
Use these files to interact with OpenStack services, and
ensure they are secured.
##########################################################
real 31m11.191s
user 13m28.211s
sys 3m15.817s
If you need to change any configuration in the undercloud.conf you can rerun the install over the top and the node should reconfigure itself (network changes likely necessitate redeployment, changinf ipxe/inspection ranges seems to require redeployment of VM).
# update undercloud configuration, forcing regeneration of passwords 'undercloud-passwords.conf'
openstack undercloud install --force-stack-update
Output
- undercloud-passwords.conf - A list of all passwords for the director services.
- stackrc - A set of initialisation variables to help you access the director command line tools.
Load env vars specific to the undercloud for the openstack cli tool.
source ~/stackrc
Check openstack undercloud endpoints, after a reboot always check the endpoints are up before performing actions.
openstack endpoint list