tseed
/
openstack_rhosp16.2_nvidia_cumulus_ceph_erasure_coding
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
openstack_rhosp16.2_nvidia_.../7) Example Project.md

23 KiB
Executable File
Raw Blame History

Example of a new project

The following example exclusively uses the CLI administration, this helps clarity the componets in play and their interdependencies. All steps can be be performed in the web console.

Load environment variables to use the Overcloud CLI

[stack@undercloud ~]$ source ~/stackrc
(undercloud) [stack@undercloud ~]$ source ~/overcloudrc
(overcloud) [stack@undercloud ~]$

Create project

# create project
openstack project create --domain 'ldap' --description "Bioinformatics Project" bioinformatics

Create an internal Openstack network/subnet for the project

openstack network create bioinformatics-network --internal --no-share --project bioinformatics
openstack subnet create bioinformatics-subnet --project bioinformatics --network bioinformatics-network --gateway 172.16.1.1 --subnet-range 172.16.1.0/16 --dhcp

Create a router for the project

openstack router create bioinformatics-router --project bioinformatics
openstack router set bioinformatics-router --external-gateway provider

Add an interface to the provider network to the project network

openstack router add subnet bioinformatics-router bioinformatics-subnet

Create a security group named 'linux-default' to allow inbound ssh for VM instances

  • a new security group injects rules on creation to allow outbound traffic by default, where multiple security groups are attached these default rules may be removed
openstack security group create --project bioinformatics linux-default
openstack security group rule create \
--ingress \
--protocol tcp \
--ethertype IPv4 \
--remote-ip '0.0.0.0/0' \
--dst-port 22 \
$(openstack security group list --project bioinformatics -f json | jq -r '.[] | select(.Name == "linux-default").ID')

# list security group rules
openstack security group rule list $(openstack security group list --project bioinformatics -f json | jq -r '.[] | select(."Name" == "default") | .ID')
openstack security group rule list $(openstack security group list --project bioinformatics -f json | jq -r '.[] | select(."Name" == "linux-default") | .ID') --long
+--------------------------------------+-------------+-----------+-----------+------------+-----------+-----------------------+
| ID                                   | IP Protocol | Ethertype | IP Range  | Port Range | Direction | Remote Security Group |
+--------------------------------------+-------------+-----------+-----------+------------+-----------+-----------------------+
| 99210e25-4b7f-4125-93bb-7abea3eddf07 | None        | IPv4      | 0.0.0.0/0 |            | egress    | None                  |
| adc21371-52bc-4c63-8e23-8e55a119407c | None        | IPv6      | ::/0      |            | egress    | None                  |
| d327baac-bdaa-437c-b506-b90659e92833 | tcp         | IPv4      | 0.0.0.0/0 | 22:22      | ingress   | None                  |
+--------------------------------------+-------------+-----------+-----------+------------+-----------+-----------------------+

Set quotas for the scope of the entire project

openstack quota set --instances 50 bioinformatics ;\
openstack quota set --cores 300 bioinformatics ;\
openstack quota set --ram 204800 bioinformatics ;\
openstack quota set --gigabytes 5000 bioinformatics ;\
openstack quota set --volumes 500 bioinformatics ;\
openstack quota set --key-pairs 50 bioinformatics ;\
openstack quota set --floating-ips 50 bioinformatics ;\
openstack quota set --networks 10 bioinformatics ;\
openstack quota set --routers 5 bioinformatics ;\
openstack quota set --subnets 10 bioinformatics ;\
openstack quota set --secgroups 100 bioinformatics ;\
openstack quota set --secgroup-rules 1000 bioinformatics

Create flavours for the project

  • flavours are pre-scoped specs of the instances
openstack flavor create small --ram 2048 --disk 10 --vcpus 2 --private --project bioinformatics ;\
openstack flavor create medium --ram 3072 --disk 10 --vcpus 4 --private --project bioinformatics ;\
openstack flavor create large --ram 8192 --disk 10 --vcpus 8 --private --project bioinformatics ;\
openstack flavor create xlarge --ram 16384 --disk 10 --vcpus 16 --private --project bioinformatics ;\
openstack flavor create xxlarge --ram 65536 --disk 10 --vcpus 48 --private --project bioinformatics

End-user access using Active Directory groups

  • In the Univerity Prod environment you would typically create an AD group with nested AD users
  • To illustrate the method, chose the pre-existing group 'ISCA-Admins'
openstack user list --group 'ISCA-Admins' --domain ldap
+------------------------------------------------------------------+--------+
| ID                                                               | Name   |
+------------------------------------------------------------------+--------+
| c633f80625e587bc3bbe492af57cb99cec59201b16cc06f614e36a6b767d6b29 | mtw212 |
| 0c4e3bdacda6c9b8abcd61de94deb47ff236cec3581fbbacf2d9daa1c584a44d | mmb204 |
| 2d4338bc2ba649ff15111519e535d0fc6c65cbb7e5275772b4e0c675af09002b | rr274  |
| b9461f113d208b54a37862ca363ddf37da68cf00ec06d67ecc62bb1e5caf06d4 | dma204 |
| 0fb8469b2d7e297151102b0119a4b08f6b26113ad8401b6cb79936adf946ba19 | ac278  |
+------------------------------------------------------------------+--------+

# bind member role to users in the access group for the project
openstack role add --group-domain 'ldap' --group 'ISCA-Admins' --project-domain 'ldap' --project bioinformatics member

# bind admin role to a specific user for the project
openstack role add --user-domain 'ldap' --user mtw212 --project-domain 'ldap' --project bioinformatics admin
openstack role assignment list --user $(openstack user show --domain 'ldap' mtw212 -f json | jq -r .id) --names
+-------+-------------+-------+---------------------+--------+--------+-----------+
| Role  | User        | Group | Project             | Domain | System | Inherited |
+-------+-------------+-------+---------------------+--------+--------+-----------+
| admin | mtw212@ldap |       | bioinformatics@ldap |        |        | False     |
+-------+-------------+-------+---------------------+--------+--------+-----------+

# bind member role for local user 'tseed' for the project
openstack role add --user-domain 'Default' --user tseed --project-domain 'ldap' --project bioinformatics member

# bind admin role for the (default) local user 'admin' for the project - we want the admin user to have full access to the project
openstack role add --user-domain 'Default' --user admin --project-domain 'ldap' --project bioinformatics admin

Import a disk image to be used specifically for the project

  • This can be custom image pre-baked with specific software or any vendor OS install image
  • Images should support cloud-init to support initial user login, generic distro images with cloud-init enabled should work
wget https://repo.almalinux.org/almalinux/8/cloud/x86_64/images/AlmaLinux-8-GenericCloud-8.6-20220513.x86_64.qcow2
openstack image create --disk-format qcow2 --container-format bare --private --project bioinformatics --property os_type=linux --file ./AlmaLinux-8-GenericCloud-8.6-20220513.x86_64.qcow2 alma_8.6

SSH keypairs

Generate an ssh key pair, this will be used for initial login to a VM instance.

  • the keypair in this example is owned by the admin user, other users will not see the ssh keypair in the web console and will need a copy of the ssh private key (unless a password is set in cloud-init userdata)
  • each user will have their own keypair that will be selected when provisioning a VM instance in the web console
  • once instantiated, additional users can import ssh keys to the authorized_keys file as per typical linux host
  • when generating ssh public keys Openstack requires a comment at the end of the key, when importing a keypair (even via the web console) the public key needs a comment

Generic distro (cloud-init) images generally have their own default user, typically these image specific such as 'almalinux' or 'ubuntu', this user will login with this user using the ssh private key counterpart to the specified public ssh key with the '--key-name' parameter. Some cloud-init images use the user in the comment of the ssh key as the default user (or as an additional user). Convention is that you provision instances with cloud-init userdata with the expectation you will provide your own user + credentials.

ssh-keygen -t rsa -b 4096 -C "bioinformatics@university.ac.uk" -f ~/bioinformatics_cloud
openstack keypair create --public-key ~/bioinformatics_cloud.pub bioinformatics

Cloud-init userdata

This OPTIONAL step is very useful, typically cloud providers utilise userdata to setup initial login, however userdata is much more powerful and often used to register the instance with a configuration management tool to install a suite of software (chef/puppet/ansible(in pull mode)) or even embed a shell script for direct software provision (pull+start containers), beware userdata is limited to 64KB.

NOTE: OCF have built cloud-init userdata for Linux (and Windows in Azure) to configure SSSD to join cloud instances to Microsoft Active Directory to enable multi-user access, this is highly environment/customer specific.

  • Openstack is kind, you dont have to base64 encode the userdata like some public cloud providers, it is automatic
  • generally each cloud-init image will have its own default user, typically these image specific such as 'almalinux' or 'ubuntu'
  • the following config will replace this default user with your own bioinformatics user, password and ssh key (It also adds the universityops user to ensure an admin can get into the system)
  • NOTE the ssh key entry below has had the trailing comment removed
  • passwords can be in cleartext but Instance users will be able to see the password in the userdata, create a hash with the command openssl passwd -6 -salt xyz Password0
  • userdata can be added to the instance when provisioning in the web console @ Customisation Script, it is always a good idea to provide a userdata template to the end user where they self provision
nano -cw userdata.txt # yaml format

#cloud-config
ssh_pwauth: true
groups:
  - admingroup: [root,sys]
  - bioinformatics
  - universityops
users:
  - name: bioinformatics
    primary_group: bioinformatics
    lock_passwd: false
    passwd: $6$xyz$4tTWyuHIT6gXRuzotBZn/9xZBikUp0O2X6rOZ7MDJo26aax.Ok5P4rWYyzdgFkjArIIyB8z8LKVW1wARbcBzn/
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD4Yh0OuBTvyXObUcJKLDNjIhmSkf/RiSPhPYzNECwC7hlIms/fYcbODPQmboo8pgtnlDK0aElWr16n1z+Yb/3btzmO/G8pZEUR607VmWINuYzSJyAieL6zNPn0XC2eP9mqWJJP44SjroVKLjnhajy761FaGxXJyXr3RXmIb4xc+qW8ETJQh98ucZZZQ3X8MernjIOO+VGVObDDDTZXsaL1wih0+v/R9gMJP8AgSCpi539o0A6RgFzMqFfroUKe6uYa1ohBrjii+teKETEb7isNOZFPx459zhqRPVjFlzVXNpDBPVjz32uuUyBRW4jMlwQ/GIrhT7+fNjpxG0CrVe0c3F+BoBnqfdrsLFCJ3dg+z19lBLnC2ulp511kqEVctjG96l9DeEPtab28p22aV3fuzdnx24y3BJi8Wea79U8+RTy0fYCM0Sm8rwREUHD2bAgjtIUU8gTKnQLyeUAc5+qJCFqa3H9/DJZ44MQzk/rC0shBUU7z+IwWhftU1P9GWURko11Bmg6pq+/fdGVm/eqilDabirbZxjqnxXCBGcOM6QsPoooJ9cgCU34k9KhUxPJ34frYfwHaWkDYxe+7VBrrzPWpOnOGt04eegwdNBDMnl703wfXqobnyy8nMmzH04j2PThJ7ZrRnA6bo/dYtVZXHocfq76yPxSsmYClebJBSQ==
  - name: universityops
    primary_group: bioinformatics
    lock_passwd: false
    passwd: $6$xyz$4tTWyuHIT6gXRuzotBZn/9xZBikUp0O2X6rOZ7MDJo26aax.Ok5P4rWYyzdgFkjArIIyB8z8LKVW1wARbcBzn/
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash

Create a floating ip

With the network design up to this point you can have a routable IP capable of accepting ingress traffic from the wider University estate by two methods:

  1. floating IP, a '1:1 NAT' of a provider network IP mapped to the VM interface IP in the private Openstack 'bioinformatics' network
  2. interface IP directly in the provider network

Floating IPs are more versatile as they can be moved between instances for all manner of blue-green scenarios, typically a VM instance does not have to be multihomed between networks either. Floating IPs in Openstack private networks are possible can be just useful in a multi-tiered application stack - think DR strategy, scripting the Openstack API to move the floating IP between instances. However end users may want a VM instance with only a provider network IP, this would only be able to communicate with other Openstack VM instances with a provider IP.

# create a floating IP in the 'provider' network on the 'provider-subnet' subnet range
openstack floating ip create --project bioinformatics --description 'bioinformatics01' --subnet provider-subnet provider
openstack floating ip list --project bioinformatics --long -c 'ID' -c 'Floating IP Address' -c 'Description'
+--------------------------------------+---------------------+------------------+
| ID                                   | Floating IP Address | Description      |
+--------------------------------------+---------------------+------------------+
| 0eb3f78d-d59d-4ec6-b725-d2c1f45c9a77 | 10.121.4.246        | bioinformatics01 |
+--------------------------------------+---------------------+------------------+

Check allocated 'ports', think of this as IP endpoints for objects known by openstack.

  • VM Instance = compute:nova
  • Floating IP = network:floatingip
  • DHCP service = network:dhcp (most networks will have one)
  • Primary router interface = network:router_gateway (usually in the provider network, for egress/SNAT access to external networks)
  • Secondary router interface = network:router_interface (router interface on a private Openstack network)
openstack port list --long -c 'ID' -c 'Fixed IP Addresses' -c 'Device Owner'
+--------------------------------------+-----------------------------------------------------------------------------+--------------------------+
| ID                                   | Fixed IP Addresses                                                          | Device Owner             |
+--------------------------------------+-----------------------------------------------------------------------------+--------------------------+
| 108171d9-cd76-49ab-944e-751f8257c8d1 | ip_address='10.121.4.150', subnet_id='92361cfd-f348-48a2-b264-7845a3a3d592' | compute:nova             |
| 3d86a21a-f187-47e0-8204-464adf334fb0 | ip_address='172.16.0.2', subnet_id='a92d2ac0-8b60-4329-986d-ade078e75f45'   | network:dhcp             |
| 3db3fe34-85a8-4028-b670-7f9aa5c86c1a | ip_address='10.121.4.148', subnet_id='92361cfd-f348-48a2-b264-7845a3a3d592' | network:floatingip       |
| 400cb067-2302-4f8e-bc1a-e187929afbbc | ip_address='10.121.4.205', subnet_id='92361cfd-f348-48a2-b264-7845a3a3d592' | network:router_gateway   |
| 5c93d336-05b5-49f0-8ad4-9de9c2ccf216 | ip_address='172.16.2.239', subnet_id='ab658788-0c5f-4d22-8786-aa7256db66b6' | compute:nova             |
| 62afa3de-5316-4eb6-88ca-4830c141c898 | ip_address='172.16.1.1', subnet_id='ab658788-0c5f-4d22-8786-aa7256db66b6'   | network:router_interface |
| 7c8b58c0-3ff7-44f6-9eb3-a601a139aab9 | ip_address='172.16.0.1', subnet_id='a92d2ac0-8b60-4329-986d-ade078e75f45'   | network:router_interface |
| 9f41db95-8333-4f6d-88e0-c0e3f7d4b7f0 | ip_address='172.16.1.2', subnet_id='ab658788-0c5f-4d22-8786-aa7256db66b6'   | network:dhcp             |
| c9591f1b-8d43-4322-acd6-75cd4cce04e3 | ip_address='10.121.4.239', subnet_id='92361cfd-f348-48a2-b264-7845a3a3d592' | network:router_gateway   |
| e3f35c0a-6543-4508-8d17-96de69f85a1c | ip_address='10.121.4.130', subnet_id='92361cfd-f348-48a2-b264-7845a3a3d592' | network:dhcp             |
+--------------------------------------+-----------------------------------------------------------------------------+--------------------------+

Create disk volumes

Create volumes that will be attached on VM instantiation (bioinformatics02).

# find the image to use on the boot disk
openstack image list -c 'ID' -c 'Name' -c 'Project' --long -f json | jq -r '.[] | select(.Name == "alma_8.6").ID'
0a0d99c1-4bce-4e74-9df8-f9cf5666aa98

# create a bootable disk
openstack volume create --bootable --size 50 --image $(openstack image list -c 'ID' -c 'Name' -c 'Project' --long -f json | jq -r '.[] | select(.Name == "alma_8.6").ID') --description "bioinformatics02 boot"  --os-project-domain-name='ldap' --os-project-name 'bioinformatics' bioinformatics02boot

# create a data disk
openstack volume create --non-bootable --size 100 --description "bioinformatics02 data"  --os-project-domain-name='ldap' --os-project-name 'bioinformatics' bioinformatics02data

Create VM instances

Creating instances via the CLI can save a lot of time VS the web console if the environment is not to be initially self provisioned by the end user, allowing you to template a bunch of machines quickly.

VM instances are not technically 'owned' by a user, they reside in a domain/project, they are provisioned by a user (initially with a user specific SSH key) and can be administered by users in same the project via the CLI/web-console. SSH access to the VM will be user specific unless the provisioning user adds access for other users (via password or SSH private key distribution at the operating system level). Userdata is the key to true multitenancy.

Instance from flavour with larger disk and floating IP

The following command illustrates:

  • create VM Instance in the Openstack 'bioinformatics' network with an additional floating IP
  • override the instance flavour 10GB disk with a 100GB disk, the disk is not removed when the instance is deleted
  • add multiple security groups, these apply to all interfaces by default, allowing specific ingress for only the floating IP would be achieved with a rule matching the destination of floating IP
# create VM instance
openstack server create \
--image alma_8.6 \
--flavor large \
--boot-from-volume 100 \
--network bioinformatics-network \
--security-group $(openstack security group list --project bioinformatics -f json | jq -r '.[] | select(.Name == "default").ID') \
--security-group $(openstack security group list --project bioinformatics -f json | jq -r '.[] | select(.Name == "linux-default").ID') \
--key-name bioinformatics \
--user-data userdata.txt \
--os-project-domain-name='ldap' \
--os-project-name 'bioinformatics' \
bioinformatics01

Attach the floating IP:

  • this command relies on the unique uuid ID of both the server and floating IP objects as the command doesn't support the --project parameter
  • we named both our floating IP and VM instance 'bioinformatics01', really this is where tags start to become useful
# attach floating IP
openstack server add floating ip $(openstack server list --project bioinformatics -f json | jq -r '.[] | select(.Name == "bioinformatics01").ID') $(openstack floating ip list --project bioinformatics --long -c 'ID' -c 'Floating IP Address' -c 'Description' -f json | jq -r '.[] | select(.Description == "bioinformatics01") | ."Floating IP Address"')

# check the IP addresses allocated to the VM instance, we see the floating IP 10.121.4.246 directly on the routable provider network
openstack server list --project bioinformatics
+--------------------------------------+------------------+--------+--------------------------------------------------+-------+--------+
| ID                                   | Name             | Status | Networks                                         | Image | Flavor |
+--------------------------------------+------------------+--------+--------------------------------------------------+-------+--------+
| ca402aed-84dd-47ad-b5ba-5fc74978f66b | bioinformatics01 | ACTIVE | bioinformatics-network=172.16.3.74, 10.121.4.246 |       | large  |
+--------------------------------------+------------------+--------+--------------------------------------------------+-------+--------+

'multi-homed' Instance from flavour with manually specified disk

Create the VM instance with the disk volumes attached and network interfaces in both the project's Openstack private network and the provider network.

# create a VM instance
## -v is a debug parameter, -vv for more
openstack server create \
--volume $(openstack volume list --name bioinformatics02boot --project bioinformatics -f json | jq -r .[].ID) \
--block-device-mapping vdb=$(openstack volume list --name bioinformatics02data --project bioinformatics -f json | jq -r .[].ID):volume::true \
--flavor large \
--nic net-id=provider \
--nic net-id=bioinformatics-network \
--security-group $(openstack security group list --project bioinformatics -f json | jq -r '.[] | select(.Name == "default").ID') \
--security-group $(openstack security group list --project bioinformatics -f json | jq -r '.[] | select(.Name == "linux-default").ID') \
--key-name bioinformatics \
--user-data userdata.txt \
--os-project-domain-name='ldap' \
--os-project-name 'bioinformatics' \
bioinformatics02 -v

# remove the server
## note that the data volume has been deleted, it was attached with the 'delete-on-terminate' flag set true in the '--block-device-mapping' parameter
## the main volume has not been removed, we see that 'delete-on-terminate' is set false in 'openstack server show'
## the web console will allow the boot volume to be delete-on-terminate, the CLI lacks this capability yet REST API clearly supports the functionality
openstack server delete $(openstack server show bioinformatics02 --os-project-domain-name='ldap' --os-project-name 'bioinformatics' -f json | jq -r .id)
openstack volume list --project bioinformatics
+--------------------------------------+----------------------+-----------+------+---------------------------------------------------------------+
| ID                                   | Name                 | Status    | Size | Attached to                                                   |
+--------------------------------------+----------------------+-----------+------+---------------------------------------------------------------+
| db137b16-67ed-4ade-8d89-fd57d463f573 |                      | in-use    |  100 | Attached to ca402aed-84dd-47ad-b5ba-5fc74978f66b on /dev/vda  |
| 1ff863bb-6cb3-4d40-8d25-06b61e974e38 | bioinformatics02boot | available |   50 |                                                               |
+--------------------------------------+----------------------+-----------+------+---------------------------------------------------------------+

Test access to VM instances

# check the IP addresses allocated to the VM instance
openstack server list --project bioinformatics -c 'Name' -c 'Networks' --long --fit-width
+------------------+-----------------------------------------------------------+
| Name             | Networks                                                  |
+------------------+-----------------------------------------------------------+
| bioinformatics02 | bioinformatics-network=172.16.3.254; provider=10.121.4.92 |
| bioinformatics01 | bioinformatics-network=172.16.3.74, 10.121.4.246          |
+------------------+-----------------------------------------------------------+

# gain access to the instances via native provider network ip and the floating ip respectively
ssh -i ~/bioinformatics_cloud bioinformatics@10.121.4.92
ssh -i ~/bioinformatics_cloud bioinformatics@10.121.4.246