redhat_cloudforms_azure_arm_ansible

16 KiB

Executable File

Raw Permalink Blame History

Azure Information

tenant

67bda7ee-fd80-41ef-ac91-358418290a1e # nottingham

subscriptions

GBUoN-uks-Dev 8a6722e9-035b-4b46-9408-ff040ff063e2 # nottingham dev

Production 1a3c8479-0046-4a6f-a2ad-397cb9a6f931 # limited access to use / list vnet+subnet and obtain ip (prod network)

Research Managed 5d0ffc51-cdb8-4de5-a76c-fca19f5b300c # nottingham research (prod)

research (prod) resource groups

rg-svc-rem-we-spp-1 # where the azure instances, vnic and resources reside rg-vn-rem-we-1 # where the vnet+subnet reside for the vnic

dev resource group

UI-SPP-DEV-001 # where the azure instances, vnic and resources reside

When provisioning resources in the GBUoN-uks-Dev subscription ensure your account has been delegated the contributor role to resource group UI-SPP-DEV-001. This method of authorization will give a URL that accepts the displayed device code then ask to authenticate (in this case with your nottingham.ac.uk account)

az login --tenant 67bda7ee-fd80-41ef-ac91-358418290a1e --use-device-code

SPN id: f45a0e1e-3f7f-44e8-971d-a56b563ef589 SPN secret: I5aAwcc9bIE]]kmhPIQpp5JCK1BiTg?]

az login --service-principal --username f45a0e1e-3f7f-44e8-971d-a56b563ef589 --password "I5aAwcc9bIE]]kmhPIQpp5JCK1BiTg?]" --tenant 67bda7ee-fd80-41ef-ac91-358418290a1e

SPN id: ccb8dfce-f33d-4b09-9ab3-5acc1c43f368 SPN secret: 622zlS45N.6_q.f6hy~5zgL.F53M6we7Qg

az login --service-principal --username ccb8dfce-f33d-4b09-9ab3-5acc1c43f368 --password "622zlS45N.6_q.f6hy~5zgL.F53M6we7Qg" --tenant 67bda7ee-fd80-41ef-ac91-358418290a1e

switch default subscription

az account set --subscription 8a6722e9-035b-4b46-9408-ff040ff063e2 # nottingham dev az account set --subscription 5d0ffc51-cdb8-4de5-a76c-fca19f5b300c # nottingham research (prod)

list available locations for subscription

Prod will use default location of "westeurope", ARM templates expect the display name "West Europe". Dev will use default location of "uksouth", ARM templates expect the display name "UK South".

az account list-locations

find resources in subscriptions

az group list # whichever subscription is set az group list --subscription 8a6722e9-035b-4b46-9408-ff040ff063e2 # nottingham dev az group list --subscription 5d0ffc51-cdb8-4de5-a76c-fca19f5b300c # nottingham research (prod)

find vnet in resource group

Prod vnet vn-rem-we-1 in resource group rg-vn-rem-we-1, this is used for ip address on vnics, no public ip's maybe allocated. Research (prod) rg-svc-rem-we-spp-1 should not contain any vnet or accociated subnet, if one is created public ip's can be allocated but ensure an nsg is not shared for hosts with nics in different resource group vnets.

az network vnet list | jq . # we see vnet in vn-rem-we-1 # we see subnets rg-vn-rem-we-1 | sn-vn-rem-we-1-frontend-1 | AzureBastionSubnet | sn-vn-rem-we-1-midtier-1 | vn-man-we-1, SPN may only use sn-vn-rem-we-1-midtier-1 az network vnet list --resource-group rg-svc-rem-we-spp-1

Create Azure storage blob to host installer scripts for prod 1.1 ARM templates

This is not required for the dev / prod 1.0 ARM templates, this change was to facilitate the carbon black install in the latest prod 1.1 ARM templates for only Windows Server and Ubuntu Server.

Create storage account

az storage account create --name extensionartefact --resource-group rg-svc-rem-we-spp-1 --location "West Europe" --sku Standard_ZRS --https-only true --min-tls-version TLS1_2 --publish-internet-endpoints false --publish-microsoft-endpoints true --routing-choice MicrosoftRouting --encryption-services blob --tags 'CFManaged=false'

Create blob container specifying auth mode

az storage container create --resource-group rg-svc-rem-we-spp-1 --public-access off --account-name extensionartefact --name extensionartefact --auth-mode key
az storage container list --account-name extensionartefact

Upload contents of the extensionartefact directory in this repo to the container

Retrieve a storage account key: az storage account keys list --resource-group rg-svc-rem-we-spp-1 --account-name extensionartefact

az storage blob upload-batch --destination extensionartefact --source /\<local-path-to-artefacts\>/extensionartefacts --account-name extensionartefact --account-key "\<place-key-here\>"

Setup managed identity

az identity create --name extensionartefact --resource-group rg-svc-rem-we-spp-1 --location "West Europe"
az identity list
az identity show --resource-group rg-svc-rem-we-spp-1 --name extensionartefact

Assign managed identity to container or storage account

https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-cli

This managed identity will be used with the ARM templates to download the custom script extension script and also download the carbon black package from the container during script runtime.

Retrieve the clientId of the managed identity for the assignee parameter: az identity show --resource-group rg-svc-rem-we-spp-1 --name extensionartefact
Check name of desired role definition, only "Storage Blob Data Reader" should be required in this scenario: az role definition list
Retrieve the id of the managed identity for the scope paramter: az storage account list

az role assignment create --assignee "87360148-9f10-45a2-a5ce-32d7c1134bd8" --role "Storage Blob Data Reader" --scope "/subscriptions/5d0ffc51-cdb8-4de5-a76c-fca19f5b300c/resourceGroups/rg-svc-rem-we-spp-1/providers/Microsoft.Storage/storageAccounts/extensionartefact"

The SPN will have insufficient privilidges to assign the rights, use your UoN account to obtain a bearer token to authenticate to complete the command.

Deploy ARM templates from command line

Ensure you are logged into the correct tenant and using the desired subscription.

To test prod subscription functionality run the templates with the following minimal parameters;

To test prod subscription functionality run the templates with all parameters to replicate CloudForms invocation;

az deployment group create --resource-group rg-svc-rem-we-spp-1 --subscription 5d0ffc51-cdb8-4de5-a76c-fca19f5b300c --parameters "{"adminUsername": {"value": "uonadmin"}, "adminPassword": {"value": "Z3qiaFJcbH2EC5DletL9"},"location": {"value": "West Europe"}, "email": {"value": "ucats@exmail.nottingham.ac.uk"}, "networkResourceGroup": {"value": "rg-vn-rem-we-1"}, "networkSecurityGroupName": {"value": "CFLinux"}, "virtualNetworkName": {"value": "vn-rem-we-1"}, "subnetName": {"value": "sn-vn-rem-we-1-midtier-1"},"imageUrn": {"value": "RedHat:RHEL:7.8:latest"},"projectCode": {"value": "UoN1234"}, "toggleShutdownSchedule": {"value": "t"}, "vmSize": {"value": "Standard_B4s"}, "location": {"value": "West Europe"}, "dataDiskType": {"value": "StandardSSD_LRS"}, "dataDiskSizeGB": {"value": "512"}}" --template-file Azure_RHEL_instance.json

az deployment group create --resource-group rg-svc-rem-we-spp-1 --subscription 5d0ffc51-cdb8-4de5-a76c-fca19f5b300c --parameters "{"adminUsername": {"value": "uonadmin"}, "adminPassword": {"value": "Z3qiaFJcbH2EC5DletL9"},"location": {"value": "West Europe"}, "email": {"value": "ucats@exmail.nottingham.ac.uk"}, "networkResourceGroup": {"value": "rg-vn-rem-we-1"}, "networkSecurityGroupName": {"value": "CFLinux"}, "virtualNetworkName": {"value": "vn-rem-we-1"}, "subnetName": {"value": "sn-vn-rem-we-1-midtier-1"},"imageUrn": {"value": "Canonical:UbuntuServer:18.04-LTS:latest"},"projectCode": {"value": "UoN1234"}, "toggleShutdownSchedule": {"value": "t"}, "vmSize": {"value": "Standard_D8s_v3"}, "location": {"value": "West Europe"}, "dataDiskType": {"value": "StandardSSD_LRS"}, "dataDiskSizeGB": {"value": "512"}}" --template-file Azure_UbuntuServer_instance.json

az deployment group create --resource-group rg-svc-rem-we-spp-1 --subscription 5d0ffc51-cdb8-4de5-a76c-fca19f5b300c --parameters "{"adminUsername": {"value": "uonadmin"}, "adminPassword": {"value": "Z3qiaFJcbH2EC5DletL9"},"location": {"value": "West Europe"}, "email": {"value": "ucats@exmail.nottingham.ac.uk"}, "networkResourceGroup": {"value": "rg-vn-rem-we-1"}, "networkSecurityGroupName": {"value": "CFWindows"}, "virtualNetworkName": {"value": "vn-rem-we-1"}, "subnetName": {"value": "sn-vn-rem-we-1-midtier-1"},"imageUrn": {"value": "MicrosoftWindowsServer:WindowsServer:2019-Datacenter:latest"},"projectCode": {"value": "UoN1234"}, "toggleShutdownSchedule": {"value": "t"}, "vmSize": {"value": "Standard_NC12"}, "location": {"value": "West Europe"}, "dataDiskType": {"value": "StandardSSD_LRS"}, "dataDiskSizeGB": {"value": "512"}}" --template-file Azure_WindowsServer_instance.json

To test prod subscription functionality of a security group template to replicate CloudForms invocation;

az deployment group create --resource-group rg-svc-rem-we-spp-1 --subscription 5d0ffc51-cdb8-4de5-a76c-fca19f5b300c --parameters "{"location": {"value": "West Europe"}, "networkSecurityGroupName": {"value": "rg-vn-rem-we-1"}}" --template-file CFLinux_Azure_network_security_group.json

To test dev subscription functionality run the templates with all parameters to replicate CloudForms invocation;

az deployment group create --resource-group UI-SPP-DEV-001 --subscription 8a6722e9-035b-4b46-9408-ff040ff063e2 --parameters "{"location": {"value": "UK South"}, "virtualNetworkName": {"value": "UI-SPP-DEV-001-vnet"}, "subnetName": {"value": "default"}, "networkSecurityGroupName": {"value": "CFLinux"}, "adminUsername": {"value": "uonadmin"}, "adminPassword": {"value": "Z3qiaFJcbH2EC5DletL9"}, "email": {"value": "ucats@exmail.nottingham.ac.uk"}, "projectCode": {"value": "1234"}, "toggleShutdownSchedule": {"value": "f"}}" --template-file Azure_UbuntuServer_instance_noscript_pubip.json

parameter behaviour

If networkResourceGroup is ommited the template will assume the vnet+subnet reside within the same resource group as the instance. If email is ommited the windows ARM template will not make group membership changes or initialize the data disk, the linux ARM template will populate the sssd acl and sudoers entries with the user 'unused'. If projectCode is ommited the Cost Code tag will display 'not classified'. If toggleShutdownSchedule is ommited it assumes default value of false 'f' and no shutdown policy will be applied. Other parameters have default values that maybe overriden to suit the desired instance spec, however the custom script extensions are likely to fail as they are very specific to the UoN research managed environment.

Template Information

Prod 1.0 Templates

Azure_RHEL_instance.json Azure_UbuntuServer_instance.json Azure_WindowsServer_instance.json CFLinux_Azure_network_security_group.json linux network security group, allows ingress SSH tcp 22 CFWindows_Azure_network_security_group.json linux network security group, allows ingress RDP tcp 3389 Azure_UbuntuServer_customscript.sh Ubuntu prep disk, join domain, install MATE desktop, shutdown timer, nvidia drivers extension Azure_RHEL_customscript.sh Redhat prep disk, join domain, install MATE desktop, shutdown timer, nvidia drivers extension

Prod 1.1 Templates

Azure_UbuntuServer_instance.json updated ARM template to use custom script extension using blob hosted setup script Azure_UbuntuServer_customscript.sh Azure_WindowsServer_instance.json updated ARM template to use custom sctipt extension using blob hosted setup script Azure_WindowsServer_customscript.ps1 extensionartefacts/ content of the blob container, setup scripts and UoN private packages such as Carbon Black and license key

Dev Templates

*The dev rev1/rev2 or prod network security group templates are suitable for use in the UI-SPP-DEV-001 resource group* *The rev2 templates are equivalent to the prod templates wihtout the additional disk or shutdown timer extension they are suitable for the rg-svc-rem-we-spp-1 resource group* *The rev3 Azure_UbuntuServer_instance_noscript_pubip.json template is a backported prod template for use on the CloudForms dev appliance using the UI-SPP-DEV-001 resource group, boot scripts disabled due to not domain connectivity this is intended as a starting point for future tempate changes that need to be tested in dev first*

rev1/*templates* templates for instance and network security group in the UI-SPP-DEV-001 resource group rev2/*templates* templates for instance and network security group in the rg-svc-rem-we-spp-1 resource group rev3/*templates* derived from prod with the addition of a public ip but scripts disabled, in the UI-SPP-DEV-001 resource group

json vs yaml

The templates are written in yaml and converted to json with yarn. Conversion operates both ways, it is helpful to take example json arm templates and convert to yaml.

Building templates in json is tedious and error prone, if a yarn sucessfully converts a template to json but does not run on the cli try an online json lint website to find syntax errors. If the template fails on the cli with no useful information run the template in Azure custom deployment for more debug, https://portal.azure.com/#create/Microsoft.Template.

usage;

https://github.com/Azure/azure-quickstart-templates Yaml allows comments and is much easier to read. https://github.com/TeamYARM/YARM-CLI

./Yarm.ConsoleApp.exe -i CFInstance_win.yaml
CFInstance_win.yaml => CFInstance_win.json

Prod 1.0 customscript extensions

These scripts prepare the data disk, join the domain and install MATE desktop / browsers for Linux or install browsers and change key settings for Windows, they are specific to the UoN research managed environment. For the Prod 1.0 templates these scripts were base64 encoded and included inline within the respective templates.

Azure_RHEL_customscript.sh Azure_UbuntuServer_customscript.sh

Prod 1.1 customscript extensions

These scripts prepare the data disk, join the domain, install the carbon black service and install MATE desktop / browsers for Linux or install browsers and change key settings for Windows, they are specific to the UoN research managed environment. The templates no longer use inline scripts or commands, instead electing to use an Azure managed identity to pull the installer scripts and carbon black packages for an Azure storage blob, this was necessary as the carbon black package is not publicly avaialable and the license key file was required to be kept private.

extensionartefacts/Azure_UbuntuServer_customscript.sh extensionartefacts/Azure_WindowsServer_customscript.ps1

create customscript extension script property

This is only used with the Prod 1.0 Linux ARM templates, the script is base64 encoded and seeded into the template.

https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-linux

cat rhel_customscript_extension.sh | gzip -9 | base64 -w 0

16 KiB Executable File Raw Permalink Blame History