RDIS ESXi7 v2

# Supporting Information

## Jump host profiles

> local ssh config file

```
#Initial account setup
        Host robots
        Hostname login.robots.university.ac.uk
        #user ocf@robots.university.ac.uk
        User ocf
        #run custom script /usr/local/bin/changemypw

        #External jump host
        Host rdis
        HostName login.robots.university.ac.uk
        ProxyJump nemesis
        #full path for key when using sudo ssh -F
        IdentityFile  /home/tseed/.ssh/id_rsa
        #IdentityFile  ~/.ssh/id_rsa
        Port 22
        User ocf

        #OCF setup jump host
        Host rdis-20
        Hostname 129.67.94.40
        ProxyJump rdis
        User ocfuser
        IdentityFile  ~/.ssh/id_rsa
        Port 22
        #browser socks5 proxy @127.0.0.1:8080
        DynamicForward 8080

        #OCF setup jump host profile for VCSA install
        Host rdis-vsphere
        Hostname 129.67.94.40
        ProxyJump rdis
        User ocfuser
        #full path for key when using sudo ssh -F
        IdentityFile  /home/tseed/.ssh/id_rsa
        #IdentityFile  ~/.ssh/id_rsa
        Port 22
        #port forward to ESXi host API endpoint
        LocalForward 443 10.0.1.21:443
```

This config files was used with WSL, enabling the local forward port to be presented on the loopback of the windows laptop for the VCSA installer tool target, an equivalent putty config could be used.

## Server ilom/ipmi

> user: USERID
> pass: Password0

| Host | TSM Console |
| --- | --- |
| rdis-vmserver01 | https://10.0.1.11/#login |
| rdis-vmserver02 | https://10.0.1.12/#login |
| rdis-vmserver03 | https://10.0.1.13/#login |
| rdis-vmserver04 | https://10.0.1.41/#login |

## ESXi Web console

> user: root
> pass: Password0

| Host | IP  | vSphere Console |
| --- | --- | --- |
| rdis-vmserver01 | 10.0.1.21/24 | https://10.0.1.21/ui/#/login |
| rdis-vmserver02 | 10.0.1.22/24 | https://10.0.1.22/ui/#/login |
| rdis-vmserver03 | 10.0.1.23/24 | https://10.0.1.23/ui/#/login |
| rdis-vmserver04 | 10.0.1.51/24 | https://10.0.1.51/ui/#/login |

## Network interfaces

To get to the ESXi shell from the lenovo TSM lom console:

> https://kb.vmware.com/s/article/2004746

Use the following command to get the associations between mac address, physical interface and vmnic interface:

> esxcli network nic list

|     |     |     |     |     |     |     |     |
| --- | --- | --- | --- | --- | --- | --- | --- |
| **Server** | **NIC** | **MAC** | **vSwitch** | **Portgroup**<br>**VMkernel** | **Physical Switch Port** | **VLAN** | **Class** |
| rdis-vmserver01 | ilom | 3ce1.a1c8.108b | N/A | N/A IP = 10.0.1.11/24 | (Primary)<br>Ethernet1/5 | access vlan 2001 | out of band management |
| rdis-vmserver01 | vmnic0 | 0c:42:a1:d2:fb:72<br>0c42.a1d2.fb72 | Storage | DVP\_iSCSI\_A (VLAN 0 = native VLAN 2011)<br>vmk2 10.0.3.11/27<br>DVP\_iSCSI\_B (VLAN 0 = native VLAN 2011)<br>vmk3 10.0.3.43/27<br>DVP\_iSCSI\_C (VLAN 0 = native VLAN 2011)<br>vmk4 10.0.3.75/27<br>DVP\_iSCSI\_D (VLAN 0 = native VLAN 2011)<br>vmk4 10.0.3.107/27 | (Primary)<br>Ethernet1/49/1 | access vlan 2011 | iSCSI |
| rdis-vmserver01 | vmnic1 | 0c:42:a1:d2:fb:73<br>0c42.a1d2.fb73 | Guest | Guest (VLAN 0 = native 2012)<br>Additional port groups on *any VLAN for guest networks to be configured | (Primary)<br>Ethernet1/49/3 | trunk native vlan 2012 | VM-Guest |
| rdis-vmserver01 | vmnic2 | 0c:42:a1:d2:fa:ea<br>0c42.a1d2.faea | Storage | DVP\_iSCSI\_A (VLAN 0 = native VLAN 2011)<br>vmk2 10.0.3.11/27<br>DVP\_iSCSI\_B (VLAN 0 = native VLAN 2011)<br>vmk3 10.0.3.43/27<br>DVP\_iSCSI\_C (VLAN 0 = native VLAN 2011)<br>vmk4 10.0.3.75/27<br>DVP\_iSCSI\_D (VLAN 0 = native VLAN 2011)<br>vmk4 10.0.3.107/27 | (Primary)<br>Ethernet1/49/2 | access vlan 2011 | iSCSI |
| rdis-vmserver01 | vmnic3 | 0c:42:a1:d2:fa:eb<br>0c42.a1d2.faeb | Guest | Guest (VLAN 0 = native 2012)<br>Additional port groups on *any VLAN for guest networks to be configured | (Primary)<br>Ethernet1/49/4 | trunk native vlan 2012 | VM-Guest |
| rdis-vmserver01 | vmnic4 | 34:80:0d:2d:85:c8<br>3480.0d2d.85c8 | vSwitch0 / DVS_Management | Management Network (VLAN 0 = native VLAN 2001)<br>vmk0 10.0.1.21/24<br>External Management Network (VLAN 20)<br>vmk1 129.67.94.20/23 | (Primary)<br>Ethernet1/16 | trunk native vlan 2001 | ESX Mgmnt / HA/DRS |
| rdis-vmserver01 | vmnic5 | 34:80:0d:2d:85:d0<br>3480.0d2d.85d0 | vSwitch0 / DVS_Management | Management Network (VLAN 0 = native VLAN 2001)<br>vmk0 10.0.1.21/24<br>External Management Network (VLAN 20)<br>vmk1 129.67.94.20/23 | (Primary)<br>Ethernet1/17 | trunk native vlan 2001 | ESX Mgmnt / HA/DRS |
| rdis-vmserver02 | ilom | 3ce1.a1c8.1121 | N/A | N/A IP = 10.0.1.12/24 | (Primary)<br>Ethernet1/6 | access vlan 2001 | out of band management |
| rdis-vmserver02 | vmnic0 | 0c:42:a1:d2:fb:6a<br>0c42.a1d2.fb6a | Storage | DVP\_iSCSI\_A (VLAN 0 = native VLAN 2011)<br>vmk2 10.0.3.13/27<br>DVP\_iSCSI\_B (VLAN 0 = native VLAN 2011)<br>vmk3 10.0.3.45/27<br>DVP\_iSCSI\_C (VLAN 0 = native VLAN 2011)<br>vmk4 10.0.3.77/27<br>DVP\_iSCSI\_D (VLAN 0 = native VLAN 2011)<br>vmk4 10.0.3.109/27 | (Primary)<br>Ethernet1/50/1 | access vlan 2011 | iSCSI |
| rdis-vmserver02 | vmnic1 | 0c:42:a1:d2:fb:6b<br>0c42.a1d2.fb6b | Guest | Guest (VLAN 0 = native 2012)<br>Additional port groups on *any VLAN for guest networks to be configured | (Primary)<br>Ethernet1/50/3 | trunk native vlan 2012 | VM-Guest |
| rdis-vmserver02 | vmnic2 | 0c:42:a1:d2:fb:5e<br>0c42.a1d2.fb5e | Storage | DVP\_iSCSI\_A (VLAN 0 = native VLAN 2011)<br>vmk2 10.0.3.13/27<br>DVP\_iSCSI\_B (VLAN 0 = native VLAN 2011)<br>vmk3 10.0.3.45/27<br>DVP\_iSCSI\_C (VLAN 0 = native VLAN 2011)<br>vmk4 10.0.3.77/27<br>DVP\_iSCSI\_D (VLAN 0 = native VLAN 2011)<br>vmk4 10.0.3.109/27 | (Primary)<br>Ethernet1/50/2 | access vlan 2011 | iSCSI |
| rdis-vmserver02 | vmnic3 | 0c:42:a1:d2:fb:5f<br>0c42.a1d2.fb5f | Guest | Guest (VLAN 0 = native 2012)<br>Additional port groups on *any VLAN for guest networks to be configured | (Primary)<br>Ethernet1/50/4 | trunk native vlan 2012 | VM-Guest |
| rdis-vmserver02 | vmnic4 | 34:80:0d:2d:8e:80<br>3480.0d2d.8e80 | vSwitch0 / DVS_Management | Management Network (VLAN 0 = native VLAN 2001)<br>vmk0 10.0.1.22/24<br>External Management Network (VLAN 20)<br>vmk1 129.67.94.21/23 | (Primary)<br>Ethernet1/18 | trunk native vlan 2001 | ESX Mgmnt / HA/DRS |
| rdis-vmserver02 | vmnic5 | 34:80:0d:2d:8e:88<br>3480.0d2d.8e88 | vSwitch0 / DVS_Management | Management Network (VLAN 0 = native VLAN 2001)<br>vmk0 10.0.1.22/24<br>External Management Network (VLAN 20)<br>vmk1 129.67.94.21/23 | (Primary)<br>Ethernet1/19 | trunk native vlan 2001 | ESX Mgmnt / HA/DRS |
| rdis-vmserver03 | ilom | 3ce1.a1c8.109b | N/A | N/A IP = 10.0.1.13/24 | (Primary)<br>Ethernet1/7 | access vlan 2001 | out of band management |
| rdis-vmserver03 | vmnic0 | 0c:42:a1:d2:fb:7e<br>0c42.a1d2.fb7e | Storage | DVP\_iSCSI\_A (VLAN 0 = native VLAN 2011)<br>vmk2 10.0.3.15/27<br>DVP\_iSCSI\_B (VLAN 0 = native VLAN 2011)<br>vmk3 10.0.3.47/27<br>DVP\_iSCSI\_C (VLAN 0 = native VLAN 2011)<br>vmk4 10.0.3.79/27<br>DVP\_iSCSI\_D (VLAN 0 = native VLAN 2011)<br>vmk4 10.0.3.111/27 | (Primary)<br>Ethernet1/51/1 | access vlan 2011 | iSCSI |
| rdis-vmserver03 | vmnic1 | 0c:42:a1:d2:fb:7f<br>0c42.a1d2.fb7f | Guest | Guest (VLAN 0 = native 2012)<br>Additional port groups on *any VLAN for guest networks to be configured | (Primary)<br>Ethernet1/51/3 | trunk native vlan 2012 | VM-Guest |
| rdis-vmserver03 | vmnic2 | 0c:42:a1:d2:fb:7a<br>0c42.a1d2.fb7a | Storage | DVP\_iSCSI\_A (VLAN 0 = native VLAN 2011)<br>vmk2 10.0.3.15/27<br>DVP\_iSCSI\_B (VLAN 0 = native VLAN 2011)<br>vmk3 10.0.3.47/27<br>DVP\_iSCSI\_C (VLAN 0 = native VLAN 2011)<br>vmk4 10.0.3.79/27<br>DVP\_iSCSI\_D (VLAN 0 = native VLAN 2011)<br>vmk4 10.0.3.111/27 | (Primary)<br>Ethernet1/51/2 | access vlan 2011 | iSCSI |
| rdis-vmserver03 | vmnic3 | 0c:42:a1:d2:fb:7b<br>0c42.a1d2.fb7b | Guest | Guest (VLAN 0 = native 2012)<br>Additional port groups on *any VLAN for guest networks to be configured | (Primary)<br>Ethernet1/51/4 | trunk native vlan 2012 | VM-Guest |
| rdis-vmserver03 | vmnic4 | 34:80:0d:2d:8b:50<br>3480.0d2d.8b50 | vSwitch0 / DVS_Management | Management Network (VLAN 0 = native VLAN 2001)<br>vmk0 10.0.1.23/24<br>External Management Network (VLAN 20)<br>vmk1 129.67.94.22/23 | (Primary)<br>Ethernet1/20 | trunk native vlan 2001 | ESX Mgmnt / HA/DRS |
| rdis-vmserver03 | vmnic5 | 34:80:0d:2d:8b:58<br>3480.0d2d.8b58 | vSwitch0 / DVS_Management | Management Network (VLAN 0 = native VLAN 2001)<br>vmk0 10.0.1.23/24<br>External Management Network (VLAN 20)<br>vmk1 129.67.94.22/23 | (Primary)<br>Ethernet1/21 | trunk native vlan 2001 | ESX Mgmnt / HA/DRS |
| rdis-vmserver04 (DR) | ilom | 3ce1 a1c8 165b | N/A | N/A IP = 10.0.1.41/24 | (DR)<br>Ethernet1/3 | access vlan 2001 | out of band management |
| rdis-vmserver04 (DR) | vmnic0 | 0c:42:a1:d2:fb:66 | Storage | DVP\_iSCSI\_A (VLAN 0 = native VLAN 2011)<br>vmk2 10.0.3.17/27<br>DVP\_iSCSI\_B (VLAN 0 = native VLAN 2011)<br>vmk3 10.0.3.49/27<br>DVP\_iSCSI\_C (VLAN 0 = native VLAN 2011)<br>vmk4 10.0.3.81/27<br>DVP\_iSCSI\_D (VLAN 0 = native VLAN 2011)<br>vmk4 10.0.3.113/27 | (DR)<br>Ethernet1/49/3 | access vlan 2011 | iSCSI |
| rdis-vmserver04 (DR) | vmnic1 | 0c:42:a1:d2:fb:67 | Guest | Guest (VLAN 0 = native 2012)<br>Additional port groups on *any VLAN for guest networks to be configured | (DR)<br>Ethernet1/49/1 | trunk native vlan 2012 | VM-Guest |
| rdis-vmserver04 (DR) | vmnic2 | 0c:42:a1:d2:fb:62 | Storage | DVP\_iSCSI\_A (VLAN 0 = native VLAN 2011)<br>vmk2 10.0.3.17/27<br>DVP\_iSCSI\_B (VLAN 0 = native VLAN 2011)<br>vmk3 10.0.3.49/27<br>DVP\_iSCSI\_C (VLAN 0 = native VLAN 2011)<br>vmk4 10.0.3.81/27<br>DVP\_iSCSI\_D (VLAN 0 = native VLAN 2011)<br>vmk4 10.0.3.113/27 | (DR)<br>Ethernet1/49/2 | access vlan 2011 | iSCSI |
| rdis-vmserver04 (DR) | vmnic3 | 0c:42:a1:d2:fb:63 | Guest | Guest (VLAN 0 = native 2012)<br>Additional port groups on *any VLAN for guest networks to be configured | (DR)<br>Ethernet1/49/4 | trunk native vlan 2012 | VM-Guest |
| rdis-vmserver04 (DR) | vmnic4 | 34:80:0d:2d:8b:c8 | vSwitch0 / DVS_Management | Management Network (VLAN 0 = native VLAN 2001)<br>vmk0 10.0.1.51/24<br>External Management Network (VLAN 20)<br>vmk1 129.67.94.29/23 | (DR)<br>Ethernet1/8 | trunk native vlan 2001 | ESX Mgmnt / HA/DRS |
| rdis-vmserver04 (DR) | vmnic5 | 34:80:0d:2d:8b:d0 | vSwitch0 / DVS_Management | Management Network (VLAN 0 = native VLAN 2001)<br>vmk0 10.0.1.51/24<br>External Management Network (VLAN 20)<br>vmk1 129.67.94.29/23 | (DR)<br>Ethernet1/9 | trunk native vlan 2001 | ESX Mgmnt / HA/DRS |

# Vmware hypervisor install

> Use the lenovo ESXi image to ensure all network card drivers are included
> 
> [https://vmware.lenovo.com/content/2020\_08/Lenovo\_Custom_ISO/7.0/](https://vmware.lenovo.com/content/2020_08/Lenovo_Custom_ISO/6.7u3/)

Mount the ISO from the ilom KVM viewer, run through the ESXi installer using the RAID1 SSD.

After install unmount the install media and enter the bios.

## Enable TPM module from the bios and set boot from the UEFI partition

![18199247e9dfa4058d2f36d9ed110ac5.png](_resources/8c22b2b544734b7a8c2f204a944887a2.png)

Check TPM is set to version 2.0, if the option to switch to 1.2 is present the host is already running with TPM version 2.0 and does not need to be updated.

![aa6a3c0784bad01ad9d86dc7768e6dbf.png](_resources/7dd61b90021547a984850d481765c16e.png)

Change the first boot option to UEFI OS on M.2 RAID virtual disk.

![f651a7140d45764b132f11d1aed5928a.png](_resources/676365faba3c4659b28cd12d11d10d42.png)

![78fff1e872b833a53eb3b608213c3544.png](_resources/e144fba8e1ab4a40a68d6c73e7094d51.png)

## Ensure the OCP adapter interfaces are assigned to the management vmkernel interface

Configure Management Network:

![4fc92cc4cc1ddd39577c7c2e4e8791e6.png](_resources/fc8dbdc2707641c591e3dd8e89d48d4a.png)

Assign Network Adapters to the the Management Network:

## ![cc9eabb84afce5e6370be212c38ece80.png](_resources/c58e9161c2b14fdabc65ec2520df7e90.png)

![de39f76b1ecd7785e83d9b56aea3172b.png](_resources/41c26ff2a41e4ea2a7331606a19b541c.png)

Configure IPv4 address:

![fc5ae6e8e4c4e7e781551c211a43d236.png](_resources/d185f915d58a4db08e91defe9665e79a.png)

| ESXI host | Management IP |
| --- | --- |
| rdis-vmserver01 | 10.0.1.21/24 |
| rdis-vmserver02 | 10.0.1.22/24 |
| rdis-vmserver03 | 10.0.1.23/24 |
| rdis-vmserver04 | 10.0.1.51/24 |

Ensure the gateway is set to 0.0.0.0, this internal network is isolated.

## ![7817bf895077a701dd7a043481372d6f.png](_resources/acf77b15bfe440aab3a023b9cf37ad5d.png)

## ![ec92e5ff95a838fd9801194df33b36db.png](_resources/b7171ddc06bf4a2a978f9541bc0f130e.png)

This default configuration creates an active-standby pair for vmnic4 and vmnic5 respectively with the default NIC teaming policy Routed based on originating port ID.

# Setup switch ports

| Switch | SSH IP |
| --- | --- |
| Primary site switch | 129.67.94.5 |
| DR site switch | 129.67.94.7 |

- Ensure any physical switch trunk ports have are configured with a native VLAN configuration.
- Ensure management trunks allows VLAN 20 (Ext management), has native access to VLAN 2001 (Int Management) and allows VLAN 2002 (vMotion).
- Ensure storage trunks have native VLAN access to VLAN 2011.
- Ensure guest trunks allow all VLANs to ensure customer can extend networks over the passed over the "University RDIS Trunk" uplink.
- LACP configuration is applied to VMware guest ports.

## Primary site switch

> ssh admin@129.67.94.5
> password: Password0

```
#find switch port by mac address, this only works where the interface is configured with an ip
show mac address-table
show mac address-table address 3480.0d2d.85c8

NE1072T>show mac address-table address 3480.0d2d.85c8
 VLAN     MAC Address     Type      Ports
  -----+----------------+---------+-----------------------+
  2001   3480.0d2d.85c8   dynamic   Ethernet1/17

#where the interface is not yet configured in vmware, change the interface link speed in vmware console from auto/manual/auto and watch the switch logs to correlate the vmic label to the switch port.

#setup ports to trunk mode or LACP trunk mode and assign vlan

## VM management/vSphere traffic

enable
configure

# configure ports for management
interface Ethernet1/16-21
no channel-group
switchport mode trunk
switchport trunk native vlan 2001
switchport trunk allowed vlan 2001,2002,20

## SR635 #1 VM iSCSI

# configure ports for VM iSCSI
interface Ethernet1/49/1
no channel-group
no description
switchport mode trunk
switchport trunk native vlan 2011
switchport trunk allowed vlan 2011
description "SR635 #1 S2P0 iSCSI"
exit

interface Ethernet1/49/2
no channel-group
no description
switchport mode trunk
switchport trunk native vlan 2011
switchport trunk allowed vlan 2011
description "SR635 #1 S3P0 iSCSI"
exit

## SR635 #2 VM iSCSI

# configure ports for VM iSCSI
interface Ethernet1/50/1
no channel-group
no description
switchport mode trunk
switchport trunk native vlan 2011
switchport trunk allowed vlan 2011
description "SR635 #2 S2P0 iSCSI"
exit

interface Ethernet1/50/2
no channel-group
no description
switchport mode trunk
switchport trunk native vlan 2011
switchport trunk allowed vlan 2011
description "SR635 #2 S3P0 iSCSI"
exit

## SR635 #3 VM iSCSI

# configure ports for VM iSCSI
interface Ethernet1/51/1
no channel-group
no description
switchport mode trunk
switchport trunk native vlan 2011
switchport trunk allowed vlan 2011
description "SR635 #3 S2P0 iSCSI"
exit

interface Ethernet1/51/2
no channel-group
no description
switchport mode trunk
switchport trunk native vlan 2011
switchport trunk allowed vlan 2011
description "SR635 #3 S3P0 iSCSI"
exit

## SR635 #1 VM Guest

# configure ports for VM Guest
interface Ethernet1/49/3
no channel-group
no description
switchport mode trunk
switchport trunk native vlan 2012
switchport trunk allowed vlan all
description "SR635 #1 S2P1 VM-Guest"
exit

interface Ethernet1/49/4
no channel-group
no description
switchport mode trunk
switchport trunk native vlan 2012
switchport trunk allowed vlan all
description "SR635 #1 S3P1 VM-Guest"
exit

# remove existing lag
no interface port-channel 11

# create lag for VM Guest
interface port-channel 11
switchport mode trunk
switchport trunk native vlan 2012
switchport trunk allowed vlan all
description "SR635 #1 VM Guest LACP"
exit

# add ports to VM Guest lag in LACP mode
interface Ethernet1/49/3-4
channel-group 11 mode active
exit

## SR635 #2 VM Guest

# configure ports for VM Guest
interface Ethernet1/50/3
no channel-group
no description
switchport mode trunk
switchport trunk native vlan 2012
switchport trunk allowed vlan all
description "SR635 #2 S2P1 VM-Guest"
exit

interface Ethernet1/50/4
no channel-group
no description
switchport mode trunk
switchport trunk native vlan 2012
switchport trunk allowed vlan all
description "SR635 #2 S3P1 VM-Guest"
exit

# remove existing lag
no interface port-channel 12

# create lag for VM Guest
interface port-channel 12
switchport mode trunk
switchport trunk native vlan 2012
switchport trunk allowed vlan all
description "SR635 #2 VM Guest LACP"
exit

# add ports to VM Guest lag in LACP mode
interface Ethernet1/50/3-4
channel-group 12 mode active
exit

## SR635 #3 VM Guest

# configure ports for VM Guest
interface Ethernet1/51/3
no channel-group
no description
switchport mode trunk
switchport trunk native vlan 2012
switchport trunk allowed vlan all
description "SR635 #3 S2P1 VM-Guest"
exit

interface Ethernet1/51/4
no channel-group
no description
switchport mode trunk
switchport trunk native vlan 2012
switchport trunk allowed vlan all
description "SR635 #3 S3P1 VM-Guest"
exit

# remove existing lag
no interface port-channel 13

# create lag for VM Guest
interface port-channel 13
switchport mode trunk
switchport trunk native vlan 2012
switchport trunk allowed vlan all
description "SR635 #3 VM Guest LACP"
exit

# add ports to VM Guest lag in LACP mode
interface Ethernet1/50/3-4
channel-group 13 mode active
exit

#save
show running
exit
copy running-config startup-config
write
```

Note: the LACP (channel-group config) setup is being disabled above, only vCentre configured DVswitches may leverage this type of bond.

## DR site switch

> ssh admin@129.67.94.7
> password: Password0

```
enable
configure

## VM management/vSphere traffic

# configure ports for management
interface Ethernet1/8-9
no channel-group
switchport mode trunk
switchport trunk native vlan 2001
switchport trunk allowed vlan 20,2001,2002

## VM iSCSI config

# configure ports for VM iSCSI
interface Ethernet1/49/1-2
no channel-group
no description
switchport mode trunk
switchport trunk native vlan 2011
switchport trunk allowed vlan 2011
description "SR635 #4 S3P0 iSCSI"
exit

# client-switch LACP for iSCSI not required 

# remove existing lag
#no interface port-channel 4

# create lag for VM iSCSI
#interface port-channel 4
#switchport mode trunk
#switchport trunk native vlan 2011
#switchport trunk allowed vlan 2011
#description "SR635 #4 VM iSCSI LACP"
#exit

# add ports to VM iSCSI lag in LACP mode
#interface Ethernet1/49/1-2
#channel-group 4 mode active
#exit

## VM Guest config

# configure ports for VM Guest
interface Ethernet1/49/3-4
no channel-group
no description
switchport mode trunk
switchport trunk native vlan 2012
switchport trunk allowed vlan all
description "SR635 #4 S3P1 VM-Guest"
exit

# remove existing lag
no interface port-channel 14

# create lag for VM Guest
interface port-channel 14
switchport mode trunk
switchport trunk native vlan 2012
switchport trunk allowed vlan all
description "SR635 #4 VM Guest LACP"
exit

# add ports to VM Guest lag in LACP mode
interface Ethernet1/49/3-4
channel-group 14 mode active
exit

# commit
show running
exit
copy running-config startup-config
write
```

# Configure ESXi Management networking

## Configure local web browser

To configure ESXi hosts via their web interfaces ensure the rdis-20 ssh profile is started and your web browser is configured for a socks5 proxy.

![4f5062bf662df41e7e48f42cb9675d0b.png](_resources/f9f741fa8af843639cfa429ed52f5695.png)

## Hypervisor web console login

> user: root
> pass: Password0

|     |     |
| --- | --- |
| rdis-vmserver01 | https://10.0.1.21 |
| rdis-vmserver02 | https://10.0.1.22 |
| rdis-vmserver03 | https://10.0.1.23 |
| rdis-vmserver04 | https://10.0.1.51 |

## Setup vSwitch0 nic teaming

During install the OCP ports are teamed as Active/Failover sets, change these to Active/Active with policy Routed based on originating port ID. Ensure this change is reflected on each ESXi host.

**![5645875f06763adcb88244c9bb57603f.png](_resources/d8521d3e4c2d4d72bfef9dbb89e9ca12.png)**

## External management network

> For vSphere integration a full FQDN / PTR pair are required.

|     |     |     |
| --- | --- | --- |
| Host | External IP | FQDN / PTR |
| rdis-vmserver01 | 129.67.94.20 | rdis-vmserver01.robots.university.ac.uk |
| rdis-vmserver02 | 129.67.94.21 | rdis-vmserver02.robots.university.ac.uk |
| rdis-vmserver03 | 129.67.94.22 | rdis-vmserver03.robots.university.ac.uk |
| rdis-vmserver04 | 129.67.94.29 | rdis-vmserver04.robots.university.ac.uk |

|     |     |
| --- | --- |
| Netmask | 255.255.254.0 /23 |
| Gateway | 129.67.95.254 |
| DNS server | 129.67.1.180 129.67.1.1 |
| DNS suffix | robots.university.ac.uk |
| NTP server | ntp0.robots.university.ac.uk,ntp1.robots.university.ac.uk,ntp2.robots.university.ac.uk,ntp3.robots.university.ac.uk |
| SMTP server | mail.robots.university.ac.uk |

## Setup port group for external management on VLAN 20

![42ed23010245508ce4cc9aee0512a9dd.png](_resources/c2f7ebc0ad2e4d11845fc6b5ec264189.png)

## Setup VMKernel external interface

Create a new VMKernel NIC on the External management network port group for each hypervisor. This will likely not be used but assists with initial vCenter installation.

![49d1560dae5f41b8a7d0d15ccf40a9b3.png](_resources/8ca7fea1dcd24fbda05aea32b3ccb8d9.png)

## Setup a vMotion portgroup (vlan 2002)

![54c42d6850afd94f5c709782d17fdb48.png](_resources/527a9ef2d1914702ba5bc13087a9d523.png)

## Setup VMKernel vMotion interface

| ESXi host | vMotion IP |
| --- | --- |
| rdis-vmserver01 | 10.0.2.1/24 |
| rdis-vmserver02 | 10.0.2.2/24 |
| rdis-vmserver03 | 10.0.2.3/24 |
| rdis-vmserver04 | Not Required |

## ![0983031984f26fb1ce5d5190c52f4f7a.png](_resources/8c0878f7a0c74ce7bebd7f11952a7298.png)

## Rename the default Port Group VM Network

Port group VM Network is automatically created with vSwitch0 which is in-turn is an uplink trunk to the physical switch with native VLAN 2001.

This port group is not intended for segregated guest traffic and is in on a non routable IP range utilised for private management interfaces.

Management VM guests such as XClarity and Think System Manager will have interfaces in this port group (or equivalent DVSwitch port group).

Rename the port group to VLAN2001_native and set the VLAN ID to 0 to denote native/untagged traffic.

![29777d87336316a1206e9fb8a6691623.png](_resources/c0387cff72cf45cfa3b822f96dfdbb18.png)

## Setup hostname, dns and gateway

![5d2a639ac9b5f9f07eaa2a3f039b999c.png](_resources/478763c554fd4b4ab6ff18c3a79bfc09.png)

## Setup NTP

![13fe3554d83a6f38d88b276d2609df2e.png](_resources/321a9036664e48948a1a36be18d3e03f.png)

Reboot the esxi host, restarting various network services may suffice.

# License the ESXi hosts

Whilst this step is duplicated within the vCentre configuration the VCSA appliance installer is buggy on an unlicensed ESXi host.

| ESXi host | License Key - vSphere v7 (upgraded) |
| --- | --- |
| rdis-vmserver01 | license-key-here |
| rdis-vmserver02 | license-key-here |
| rdis-vmserver03 | license-key-here |
| rdis-vmserver04 | license-key-here |

## ![f74047a508c64f54945df92803e65559.png](_resources/e9cf53169171448db8297a327bb56055.png)

# Setup vCenter appliance

vCenter configuration only applies to rdis-vmserver01-03, rdis-vmserver04 could be managed by vCenter as a stand alone host or a cluster consisting of one machine, however as there is not site based vCentre redundancy and data replication is offloaded between storage arrays I would be a more flexible configuration to manage the host directly. In a scenario where the primary site is offline recovered virtual machines will need to point at differently labelled network and storage to be temporarily brought up on the DR site.

## Create the VCSA port group

- This port group is required as the VCSA interface cannot share a management network.
- Create VCSA port group on vlan20.
- For this installation the vCentre is not in HA mode, however to enable recovery/migration of the VCSA appliance ensure the VCSA port group is created on rdis-vmserver01-03.

![b004e91e9e82833e9cbccc0f63e85388.png](_resources/9aefb3d2311a47588288689373257c2d.png)

## Install the VCSA appliance

### Prerequisites

The VCSA appliance installer will get stuck installing over an SSH forwarded port, call backs from the appliance to itself to install RPM packages hosted on port 443 will use the endpoint address used in the VCSA installer, a spoof local hosts entry is temporarily required be placed on your local laptop and the ssh tunnel run as root to ensure ports <1024 can be mapped locally.

This is a messy workaround resultant from installation over a chain of ssh tunnels, you could build a virtual machine on the hypervisor for a cleaner installation.

```
C:\\Windows\\System32\\drivers\\etc\\hosts
127.0.0.1  rdis-vmserver01.robots.university.ac.uk
```

The VCSA appliance has a 365day root password timeout, you may only use ISO files within this release period, download the latest release of the version of VCSA as required.

The VCSA appliance requires full forward and reverse DNS records.

### VCSA installer parameters

| Attribute | Value |
| --- | --- |
| ISO | VMware-VCSA-all-6.7.0-17713310.iso |
| VM name | rdis-vcsa01 |
| Root password | Password0 |
| Deployment size / Storage size | small / default |
| Network | VCSA |
| IP version | IPv4 |
| FQDN | rdis-vcsa01.robots.university.ac.uk |
| IP address | 129.67.94.11 |
| Subnet prefix | 23  |
| Default gateway | 129.67.95.254 |
| DNS servers | 129.67.1.180, 129.67.1.1 |

### Install the appliance

- Ensure the local hosts spoof entry is in place for the duration of the installation.
- Run the rdis-vsphere ssh profile as the root user to start a port forward to the ESXi host API endpoint.

> sudo ssh -F /home/tseed/.ssh/config rdis-vsphere

- Run the vsphere appliance installer from your local desktop ensuring the target ESXi host is the localhost ssh forwarded port.

> Download and mount the vsphere VCSA appliance installer ISO
> 
> VMware-VCSA-all-6.7.0-17713310.iso
> 
> E:\\vcsa-ui-installer\\win32\\installer.exe

- Select install from the installer, accept any end user agreements, select the embedded controller (applications and databases reside within a single appliance).

![83ce8e37a833e8819565614903c2165d.png](_resources/ef543104c7e24a5b873c68fd6a053a9f.png)

Use the ssh port forward endpoint with the spoof hosts entry.

![4a176fc674b8d681b75632d2e42e10ee.png](_resources/2af89ded3c184ba793a8b6fd93c38648.png)

Accept the SSL certificate.

![6703c9ca6bd914f8d52a4244bc139b70.png](_resources/b1b7b264183946ba98a1906ed38172e4.png)

Configure the name of the virtual machine and VCSA root credentials.

![400649b3f74f9cac49613bcbebf062f6.png](_resources/3052e6be9e2c4824adccf3180fd629a7.png)

Select deployment size, the local host datastore is not large enough to thick provision the appliance, ensure thin provisioned disk.

![a08b640b464cac48885d2b48bb110643.png](_resources/7d59afd025924e96b5f70109c4006e55.png)

![be62b9d9765c7c1174d16d18c995a202.png](_resources/28e1992d26894a30a05ac41fa436608d.png)

Configure appliance networking, ensure that the VCSA port group is selected for Network.

![dfd411751e5e6fc7d9b8aa02340e4422.png](_resources/11701df4a1454b23a672c7d7fb50990c.png)

The installer will deploy the appliance ready for configuration, make note of the location of the installer log.

![7b0efac2be3466ee0dc6fa60d0db4659.png](_resources/0da8a4d031bc44009a949b6979a90603.png)

### Configure the appliance

- Ensure the rdis-vsphere ssh profile is closed and the temporary hosts file entry is removed.
- Open the rdis-20 ssh profile and continue the installation in a browser using the socks5 profile.

> https://129.67.94.11:5480

Select Set up.

![f5c6c2f0e6ed63a3f4df6fb36e3575f3.png](_resources/f35c81c82eb04b25904fa15aa4514387.png)

![5bca8e508ce93afb31d420471fd49fb9.png](_resources/edd2a0f209f6448491c2de239110a049.png)

Ensure network configuration is correct, enable ssh access and synchronise time with the ESXi host.

![8f0d42ecd0d23701797b604c7ffa3ed8.png](_resources/6054a228be5e4b68a94f7fa7a1da7ab6.png)

Configure the mandatory SSO domain.

|     |     |
| --- | --- |
| SSO Domain | vcsa.robots.university.ac.uk |
| SSO Admin User | administrator |
| SSO Admin Pass | Password0 |

![7f5fcb35506ac2098ac3612bc2e53348.png](_resources/57b696f569ec465f872660e21c0e2ee7.png)

![e32b8e0aa505352a0483354ad02fe05c.png](_resources/e60d37ed62594f3eabe7f8ff7efd5f1a.png)

Completed.

![16ad17f8337e4ed5677f348e50149eb4.png](_resources/1cc7db67b44e4b2081ab854878fea7a7.png)

## vCentre Web consoles

| Web Console | SSH rdis-20 socks5 address | Internal FQDN | User | Pass |
| --- | --- | --- | --- | --- |
| Appliance administration | https://129.67.94.11:443 | https://rdis-vcsa01.robots.university.ac.uk | administrator@vcsa.robots.university.ac.uk | Password0 |
| Vsphere  administration | https://129.67.94.11:5480 | https://rdis-vcsa01.robots.university.ac.uk:5480 | root | Password0 |

## vCentre assign licenses

Ensure the rdis-20 ssh profile is started and your browser is using the local socks5 ssh proxy on 127.0.0.1:8080, edit the local hosts on your laptop to include the following entry:

> 129.67.94.11 rdis-vcsa01.robots.university.ac.uk

Login to the appliance administration console with the FQDN rather than the appliance IP and a use compatible browser (google Chrome seems to render the application better than Firefox).

launch HTML5 portal navigate to menu -> administration -> licensing -> and add new licenses.

Enter license keys, ensure each key is named with the ESXi host it is being used against.

| ESXi host | License Key - vSphere v7 |
| --- | --- |
| rdis-vmserver01 | license-key-here |
| rdis-vmserver02 | license-key-here |
| rdis-vmserver03 | license-key-here |
| rdis-vmserver04 | license-key-here |
| rdis-vcsa01 (vCentre license) | license-key-here |

![9bb6608e27efab2b0c4364e308aade0d.png](_resources/cac0368153944bbf93f2b2211366ecdb.png)

Navigate to menu -> hosts and clusters then check actions and assign license to vsphere.

![6b30ae9643a3984a2a1573df6f7af7ec.png](_resources/ce6599d4fa8843e58281772db506f7a2.png)

## ![95a6f5ed9047db3fda8f9a4c14dfaca6.png](_resources/658d70f211ea4a3f8d10c35fd05fd032.png)

## Create Datacenter and Cluster

Navigate to menu -> hosts and clusters, right click on the vcsa object and create a new datacenter named 'Robots'.

![862e6085f2486537b99df2e7226812f8.png](_resources/bed23312a9984467b998634fe8ac8002.png)

Right click the Robots datacenter and create a new cluster named 'Primary' with DRS and HA enabled.

![9627a7278b07d256b6933450d8049689.png](_resources/e225399bab9747a2bfdee180d5cb8840.png)

![f075f32cdaf353e40e8bfdd56eedbd8d.png](_resources/17d00a30a6364c40b583f05bd06b55d0.png)

Select the 'Primary' cluster object, navigate to Quickstart, add hosts:

| Host | External IP | FQDN / PTR |
| --- | --- | --- |
| rdis-vmserver01 | 129.67.94.20 | rdis-vmserver01.robots.university.ac.uk |
| rdis-vmserver02 | 129.67.94.21 | rdis-vmserver02.robots.university.ac.uk |
| rdis-vmserver03 | 129.67.94.22 | rdis-vmserver03.robots.university.ac.uk |
| rdis-vmserver04 (don't add to cluster) | 129.67.94.29 | rdis-vmserver04.robots.university.ac.uk |

- Ensure each ESXi host is added
- Use credentials root : Password0
- If asked accept certificate thumbprint and assign the corresponding vSphere license
- If prompted to disable lockdown mode check this option, this can be re-enabled after the cluster is operational and if there is no requirement for direct access to the ESXi host admin console.
- Once the hosts are attached to the Primary cluster object right click each host and check if it has been assigned the corresponding licence, if not assign the correct license.

![106c3bf8847dcbfd0e65d4af7480e0ce.png](_resources/dacc7e8a722041a1b1a0c620efa0ac48.png)

Each host will now display multiple errors:

- Host TPM attestation alarm
- vSphere HA agent on this host could not reach isolation address: 129.67.95.254
- The number of vSphere HA heartbeat datastores for this host is 0, which is less than required: 2

The vSphere HA agent cannot ping the gateway in the RDIS environment, ESXi hosts cannot tell if they are isolated from the network thus HA is disabled, to remedy this we select the switch admin IP as the source of truth for this condition and disable the default gateway check, if a host cannot ping the switch management IP there is a good chance it is isolated.

> https://kb.vmware.com/s/article/1002478

Select 'Primary' cluster, navigate to vSphere Availability -> configure -> vSphere HA is Turned ON -> edit -> advanced options -> add

![8b3f35cc218ba86fdf809004a32d8e42.png](_resources/2418b30f6166401781c521301330c46b.png)

To check if this has corrected the fault, disable vSphere HA is Turned ON then re-enable. There should now be a new fault displayed for each hosts summary stating the heartbeat datastore cannot be reached, this will will be cleared once an iSCSI datastore dedicated to heartbeat data is created.

# Create Management DVSwitch and LACP LAG

## Remove vmnic5 from vswitch0 on each host in preparation of the DVSwitch creation

- Select each host in the cluster, navigate to configure -> networking -> virtual switches -> vswitch0 -> manage physical adapters
- Select vmnic5 and remove it from the vswitch

![a01a2fe783e89752d624a0cd4ef6748d.png](_resources/babba410c963400fa25f61e9744b16e5.png)

## Create DVSwitch for Management(+vMotion) traffic

- The DVSwitch will have only 2 uplinks, the ESXi hosts have a total of 6 physical adapters.
- No default port group will be created.
- Network I/O control will be required specifically on the Management DVSwitch for vMotion traffic.
- Ensure the DVSwitch is version 7.0.0 to ensure compatibility with the Lenovo specific ESXi image.

### Management DVSwitch

Navigate to networking, right click the Robots datacenter object and add new distributed switch.

![39815c51d90aac3677cdbac2c854b7c6.png](_resources/e41205193a084698ad3fc28bed7931ff.png)

![eecce6fa071aacbb372a80819038f8dc.png](_resources/51e71a82980b45f19731a9abb6a0146f.png)

Select version 7.0.0 to ensure compatibility with the Lenovo specific ESXi image version.

![660c4cdda55de6ba919af642286d12e8.png](_resources/34965da92ed24ab99aa1472f0ec9b970.png)

![917b91b661f4e0ad3d1940051bf5a37f.png](_resources/9ad4b039afd047898d6bb94a677a7696.png)

### Create Management DVSwitch distributed port groups

All distributed port groups are created with static binding with elastic port allocation.

Create the following distributed port groups:

| Distribute port group name | VLAN |
| --- | --- |
| DVP\_VLAN2001\_native | 0 (none) |
| DVP_VCSA | 20  |
| DVP_vMotion | 2002 |
| DVP\_External\_Management_Network | 20  |
| DVP\_Management\_Network | 0 (none) |

![dd62ae3cefa87bbc03bf3c9028de37c1.png](_resources/69724731f7f14657916850986dfdc4f9.png)

![e8d086942858c7deb875e0f40dae9de6.png](_resources/022d984329e64e8da11e2385fcae2fd8.png)

![74045ca682fe2fc63fa04525db8fb96c.png](_resources/a0546f88c8e44d62a1a7196fcc00ebf8.png)

### Assign the DVSwitch to hosts

Right click the 'Management' switch, select add and manage hosts.

![69436fd7382d132f3bef6f0cc661ded5.png](_resources/215fd990336b4659993ec39e09df65ce.png)

![70e67d938df3498073081607597d80eb.png](_resources/03cf75cfb52a428ab0633a412fd23e0d.png)

Select physical adapter vmnic5 on each host and assign as uplinks, this is an OCP adapter removed from vSwitch0.

Assign vmnic5 to uplink2, tick the checkbox to apply the uplink assignment to all of the hosts.

![044545f5d053734e1137d7e79c23c48e.png](_resources/4915eb00bb6d41958510ecbb0aeb6525.png)

![64c13a4464aa75335fb26966cf6bae88.png](_resources/e270c7d1342b49a68827e9608fe61268.png)

The various VMkernel virtual interfaces will need to be migrated to the matching named distributed port groups we have created.

![c3fb9cf201ed02409032215e8785a4a1.png](_resources/bd5360cbb19b4082a1635ce94c69ba80.png)

Select each VMkernel vmk0-2, check assign port group, select the matching distributed port group, select apply port assignment for all hosts.

![9a81439bdb9c38cef9ea67519401eb4a.png](_resources/20ff58b793a94c81ba82ecf9d5928497.png)

![69b404e039ebb85378b5d53ab7fa980f.png](_resources/f2cc23913119460bb6508c189c288d06.png)

Existing virtual machines can also be migrated to the distributed port groups, the VCSA appliance must be migrated to maintain connectivity, select the virtual machine and check assign port group, selecting the VCSA distributed port group. The reassigned destination port group will confirm the selection.

![3cf2648e8988ece42521caa3f5d62540.png](_resources/06dcc26e86ef4660b4f09b9ddb6e8f72.png)

![1923e9379364a846e0efca79aa224d1f.png](_resources/73013be72b3f458ebeea7d5bc5ab9b43.png)

You may lose connectivity for a minute or two.

Remove unused default vSwitch0.

Navigate to hosts and clusters, select each host, navigate to configure -> networking -> virtual switches and remove vSwitch0.

![a5cb38537be6b7081fe09fec86d0121f.png](_resources/8bbd442bccb140468b1e3743567b4498.png)

Add the newly released vmnic4 to the Management DVSwitch on each ESXi host, navigate to the the virtual switches and select manage physical adapters for the DVS_Management switch.

![91959d696611954815ad1d4c16d05000.png](_resources/7ab1f45dee47408f965d336d5c790c13.png)

Select Uplink1 and add vmnic4.

![62e56df3954624b60bf71c7bf769cfdf.png](_resources/46a773bb5e0741b6973fa46af85fe38f.png)

![19ec43cb027c517f37ff51b605602814.png](_resources/f7725b463d3f4799b4bca44e20fcecf2.png)

## Create LACP port channel LAG for Management DVSwitch

The ESXi hosts have a total of 6 physical network adapters, these are equally divided into traffic classes for Management, Storage and Guest for resilience. In this configuration the vMotion traffic will traverse the Management network adapters, to maximise single socket bandwidth these ports will be moved to a LACP LAG.

Generally it is preferable to have a dedicated standard vSwitch physical adapter not in a LAG and not on a DVSwitch for easy access when troubleshooting. Physical adapter restraints and the fact that this cluster is unlikely to have many network topology changes over its lifecycle dictate a preference for maximum bandwidth specifically for the vMotion traffic thus setting the topology of the Management physical adapters for LACP.

At this stage the Management DVSwitch has vmnic4-5 set as trunk ports, vmnic4 will be moved to a LAG with complimentary physical switch configurations updated and a migration of DVSwitch traffic to this LAG, once this is complete vmnic5 will be moved to the LAG to complete the operation.

### Remove vmnic4 from the DVS_Management virtual switch

From the vCentre web client navigate to hosts and clusters, select each ESXi host -> configure -> networking -> virtual switches, select DVS_Management switch -> manage physical adapters.

Select Uplink1/vmnic4 and remove.

![44dd06388f001cee5ef67c17cd52f467.png](_resources/23d3e7d8b3da41e6a984a75f76603066.png)

### Create the Management LAG

From the vCentre web client navigate to networking, select the DVS_Management switch, select configure -> LACP -> NEW.

Create a new LAG named 'LAG_Management' with 2 ports in passive mode with load balancing profile 'source and destination IP address, TCP/UDP port and VLAN' this will loadbalance and aggregate across layer 3 traffic, generally layer 2 MAC address policies aren't required over virtual networks.

![a74cafc819f9e9106e369bd7be7ebb91.png](_resources/be1faac136bb4960a932838a602e4619.png)

Select migrating network traffic to LAGs then manage distributed port groups in the popup.

## ![7095b4e733456644c2b516df84922e6d.png](_resources/35fa79a1019c41e791d621804eebb6fd.png)

Check teaming and failover.

![1fd5cc6eae24899f0f61eb89a0a71aed.png](_resources/255bfbb9bf2a4ddcb5dc2edae6f9483c.png)

Select all of the management distribute port groups.

| Distributed port groups |
| --- |
| DVP\_External\_Management_Network |
| DVP\_Management\_Network |
| DVP_VCSA |
| DVP\_VLAN2001\_native |
| DVP_vMotion |

![11dda204c8ee5938844009ec383c7afc.png](_resources/54c0314cb4374aac8196ea0b74d48813.png)

Move the LAG 'LAG_Management' into the standby uplink list and accept all teaming defaults, the LAG cannot become an active uplink until the physical switch is configured for LACP port channel, this will show a warning stating this is an 'invalid' intermediate configuration intended for migration to the LAG, this is as expected.

![e2e9fda093d374310839b213bedaaa43.png](_resources/968b35a674c84fa89992b141ef598eb1.png)

![1e04656c8f984bfc91776ca758fec9b4.png](_resources/930a6d6646814e279ce52b3350caf94d.png)

Select migrating network traffic to LAGs then add and manage hosts in the popup.

![bb65b181dab19dda139c308aed452c3c.png](_resources/8a6a3e4af5fb44208f53c8db64c1a79b.png)

Select manage host networking

## ![cb3eca5c471bad16bf9053ff3623fe08.png](_resources/81e80982157f41a896d5ae0ba194c304.png)

Select all hosts in the 'Primary' cluster.

![5ebbdb356598f31e530ad14cae903d30.png](_resources/12b5a9b56e9e4e299bccd8e137f2bc25.png)

Assign vmnic4 to LAG_Management-0 port, apply this uplink to all hosts.

![47161cd6ad3b61d178aba0ead4c727fd.png](_resources/d8873683b955480cbcdaa445b5f2e0e4.png)

Do not migrate any VMKernel adapters or virtual machines to any port groups at this stage. The topology will now show the management lag containing vmnic4 for each ESXi host.

![e45132c2aea9ce365b776bbbb0e1047f.png](_resources/295c9b73b80c4680814d416034174036.png)

## Configure the LAG port channels on the primary site switch.

> ssh admin@129.67.94.5
> password: Password0

```
## VM Management traffic LACP for vmnic4

enable
configure

## SR635 #1 VM Management

# remove existing lag
no interface port-channel1

# create lag for VM Management
interface port-channel1
switchport mode trunk
switchport trunk native vlan 2001
switchport trunk allowed vlan 2001,2002,20
description "SR635 #1 VM Management LACP"
exit

#configure vmnic4 port for VM Management
interface Ethernet1/16
channel-group 1 mode active
exit

## SR635 #2 VM Management

# remove existing lag
no interface port-channel2

# create lag for VM Management
interface port-channel2
switchport mode trunk
switchport trunk native vlan 2001
switchport trunk allowed vlan 2001,2002,20
description "SR635 #2 VM Management LACP"
exit

#configure vmnic4 port for VM Management
interface Ethernet1/18
channel-group 2 mode active
exit

## SR635 #3 VM Management

# remove existing lag
no interface port-channel3

# create lag for VM Management
interface port-channel3
switchport mode trunk
switchport trunk native vlan 2001
switchport trunk allowed vlan 2001,2002,20
description "SR635 #3 VM Management LACP"
exit

#configure vmnic4 port for VM Management
interface Ethernet1/20
channel-group 3 mode active
exit

show running
exit
copy running-config startup-config
write
```

### Set the now active LAGs to be active for the distributed port groups and remove the native DVswitch uplink (vmnic5 adapter)

Select migrating network traffic to LAGs then manage distributed port groups in the popup.

![0efb7bca68218394f032bb9361cf0cfb.png](_resources/62cf7bde6921423b841f2ac6f64f76e6.png)

Select traffic teaming and failover.

![1fd5cc6eae24899f0f61eb89a0a71aed.png](_resources/255bfbb9bf2a4ddcb5dc2edae6f9483c.png)

Select all of the management distributed port groups.

|     |
| --- |
| DVP\_External\_Management_Network |
| DVP\_Management\_Network |
| DVP_VCSA |
| DVP\_VLAN2001\_native |
| DVP_vMotion |

![11dda204c8ee5938844009ec383c7afc.png](_resources/54c0314cb4374aac8196ea0b74d48813.png)

Move Uplink1-2 to unused uplink list and move LAG_Management to active uplinks.

![ea8533f529f95fee0ce43667d61d0386.png](_resources/6224bcfaac054e01b1987447cd9d3684.png)

Refresh the topolgy page and check that traffic is now only active over the LAG Management uplink.

![4657ff5b8d0ab007849bf7de2d78ed3a.png](_resources/0c969f08ddfa4d2a9f9eb73b74f1db7f.png)

## Remove vmnic5 from DVS\_Management switch on each host in preparation to add to the LAG LAG\_Management

- Select each host in the cluster, navigate to configure -> networking -> virtual switches -> DVS_Management -> manage physical adapters
- Select Uplink2 vmnic5 and remove

![1595153710c82655028484e9bca6ba9f.png](_resources/51e469ed017140a2b3f3df2e84ba2396.png)

## Configure the LAG port channels on the primary site switch to include the now released vmnic5

> ssh admin@129.67.94.5
> password: Password0

```
## VM Management traffic LACP for vmnic5

enable
configure

## SR635 #1 VM Management

#configure vmnic5 port for VM Management
interface Ethernet1/17
channel-group 1 mode active
exit

## SR635 #2 VM Management

#configure vmnic5 port for VM Management
interface Ethernet1/19
channel-group 2 mode active
exit

## SR635 #3 VM Management

#configure vmnic5 port for VM Management
interface Ethernet1/21
channel-group 3 mode active
exit

show running
exit
copy running-config startup-config
write
```

## Add vmnic5 to the DVS\_Management switch uplink LAG LAG\_Management

From the vCentre web client navigate to networking, select the DVS_Management switch, select configure -> LACP -> migrating network traffic to LAGs -> add and manage hosts.

![0efb7bca68218394f032bb9361cf0cfb.png](_resources/62cf7bde6921423b841f2ac6f64f76e6.png)

Select manage host networking.

![102759762e8244e1f1004dd27d0c4072.png](_resources/fa03562970de47b1a1dc55296243293c.png)

Select ESXi hosts.

![8fa6a84459bdbcc88a0c12bb9c2f54b0.png](_resources/f48ec38a008c42c78d775a2e5b90e163.png)

Select vmnic5 and assign uplink to LAG_Management-1 and apply to all hosts.

![cb5f6321e9c4f39b067fa21655dfb7a4.png](_resources/af7371d134bb4ba490308c984bc96754.png)

Do not migrate any VMKernel adapters or virtual machines to any port groups at this stage.

The topology will now show active traffic on the vmnic5 interface.

![e7b22a966ece0bbc4db4c01b3653b993.png](_resources/da77efd4b6904afeb143a98bcfd4ee25.png)

The switch console should show the LACP port channel up and the vmnic5 adapters leave the suspended state.

![6d69b16868bcf9a461c593bd864337bd.png](_resources/10b094b022664912b2bd775628b7f06f.png)

# Create Guest LACP LAG and DVSwitch

## Create DVSwitch for Guest traffic

- The DVSwitch will have only 2 uplinks, the ESXi hosts have a total of 6 physical adapters.
- No default port group will be created.
- Ensure the DVSwitch is version 7.0.0 to ensure compatibility with the Lenovo specific ESXi image.

### Guest DVSwitch

Navigate to networking, right click the Robots datacenter object and add new distributed switch.

![39815c51d90aac3677cdbac2c854b7c6.png](_resources/e41205193a084698ad3fc28bed7931ff.png)

![f6b7e9a990d3ed42d91d968ca6045e9d.png](_resources/dd25e93ac646497b80518e34af7227d1.png)

Select version 7.0.0 to ensure compatibility with the Lenovo specific ESXi image version.

![660c4cdda55de6ba919af642286d12e8.png](_resources/34965da92ed24ab99aa1472f0ec9b970.png)

![917b91b661f4e0ad3d1940051bf5a37f.png](_resources/9ad4b039afd047898d6bb94a677a7696.png)

## Create Guest LAG

From the vCentre web client navigate to networking, select the DVS_Guest switch, select configure -> LACP -> NEW.

Create a new LAG named 'LAG_Guest' with 2 ports in passive mode with load balancing profile 'source and destination IP address, TCP/UDP port and VLAN' this will loadbalance and aggregate across layer 3 traffic, generally layer 2 MAC address policies aren't required over virtual networks.

Select the new LAG then migrating network traffic to LAGs ->  add and manage hosts -> add hosts.

![844e7140ce3a9ef7bca7c4ad73ca82b7.png](_resources/b0943b689efe4084bca3621451f300e2.png)

Select all hosts in the 'Primary' cluster.

## ![148345388ae9880778be9ff90a5edc74.png](_resources/c724a19cda234daeafa842ea89c59824.png)

Select the vmnic adapters and assign as uplinks for all ESXi hosts.

- The initial physical switch configuration has the VM Guest LACP port channel configured.
- The supporting info section of this document lists the vmnic adapters and their corresponding physical switch port ID.
- For quick reference use the vmnic1 and vmnic3 adapters as LAG\_Guest-0 and LAG\_guest-1 for the DVS_Guest LACP configuration.

![f62b568f5a0f87406520223f1c81f360.png](_resources/3bf8cc2363214be3941bc16883aba061.png)

## ![dbb2f59416ca1208ddd58525267fc59d.png](_resources/fe41ea383f3749d08eee3202814774bc.png)

Do not migrate any VMKernel adapters or virtual machines to any port groups at this stage.

The switch console will show messages relating to port channel 11-13 negotiating and being pinned up, check the switch output to confirm:

![eae656ba335e726c10e804b886ba1805.png](_resources/30de07198d74431089f1212812d1e289.png)

The topology should show LAG_guest with two active vmnics per ESXi host.

![3ceafb30fa12817e5aa27a8ee116b0af.png](_resources/38caa0982c2546baaeec64220fecefd5.png)

- To home virtual machines on this guest DVSwitch, distributed port groups must be created with an appropriate VLAN tag presented across the uplinks from the customer core switches.
- The physical trunk ports on the switch are set with a native VLAN of 2001 with no VLAN mask, 0-4096 allowed.

# Create Storage DVSwitch

## Set the physical switch port to access mode on VLAN 2011

### Primary site switch

> ssh admin@129.67.94.5
> password: Password0

```sh
enable
configure

interface Ethernet1/49/1
description "SR635 #1 S2P0 iSCSI"
switchport mode access
switchport access vlan 2011
exit

interface Ethernet1/49/2
description "SR635 #1 S3P0 iSCSI"
switchport mode access
switchport access vlan 2011
exit

interface Ethernet1/50/1
description "SR635 #2 S2P0 iSCSI"
switchport mode access
switchport access vlan 2011
exit

interface Ethernet1/50/2
description "SR635 #2 S3P0 iSCSI"
switchport mode access
switchport access vlan 2011
exit 

interface Ethernet1/51/1
description "SR635 #3 S2P0 iSCSI"
switchport mode access
switchport access vlan 2011
exit

interface Ethernet1/51/2
description "SR635 #3 S3P0 iSCSI"
switchport mode access
switchport access vlan 2011
exit

show running
exit
copy running-config startup-config
write
```

### DR site switch

> ssh admin@129.67.94.7
> password: Password0

```sh
enable
configure

interface Ethernet1/49/2
description "SR635 #4 S3P0 iSCSI"
switchport mode access
switchport access vlan 2011
exit

interface Ethernet1/49/3
description "SR635 #4 S3P1 iSCSI"
switchport mode access
switchport access vlan 2011

show running
exit
copy running-config startup-config
write
```

## Create DVSwitch for Storage traffic

- The DVSwitch will have only 2 uplinks, the ESXi hosts have a total of 6 physical adapters.
- No default port group will be created.
- Ensure the DVSwitch is version 7.0.0 to ensure compatibility with the Lenovo specific ESXi image.
- No port/channel bonding is in use, the storage vendor recommends iSCSI connectivity is multipath'd without LACP at both the storage and client endpoints.

### Storage DVSwitch

Navigate to networking, right click the Robots datacenter object and add new distributed switch.

![39815c51d90aac3677cdbac2c854b7c6.png](_resources/e41205193a084698ad3fc28bed7931ff.png)

![23e65d16a6e80395d1f37a67b4fb5121.png](_resources/518bb25d1df54c5f81d8ad1ef3996acd.png)

Add hosts to the switch.

![d136d5d14ece63462206eff4eb7f7cf2.png](_resources/f62b897723f24f6483f92c145f266f56.png)

![aa894f24ca4535c31cdb55e14b68a735.png](_resources/8bacf6fbee4546dc848ef9554e089bff.png)

Select all hosts in the 'Primary' cluster.

![c9b6f987abd6e2c54f63ac8976053dbf.png](_resources/b38e5e7b436f454b9fc6c55f649658a0.png)

Select the vmnic adapters and assign as uplinks for all ESXi hosts.

- The initial physical switch configuration has the VM Storage ports configured as trunks.
- The supporting info section of this document lists the vmnic adapters and their corresponding physical switch port ID.
- For quick reference use the vmnic0 and vmnic2 adapters as LAG\_Guest-0 and LAG\_guest-1 for the DVS_Guest LACP configuration.

![aa7c2b1686a6706a4af3349bf9e13b1f.png](_resources/a590de9e6f3f4894bd9f7b945fefe415.png)

![1f8b3e9b590621392c5855df9a0b3481.png](_resources/fa2fc8009ec9431da2c3d41097467540.png)

![ab580e1526dbe301c709223b3a9a6408.png](_resources/1c68eaf4fef34f698abaf1e696199bb0.png)

## Create distributed port groups for the iSCSI targets.

| Port group | vSwitch |
| --- | --- |
| DVP\_ISCSI\_A | DVS_Storage |
| DVP\_ISCSI\_B | DVS_Storage |
| DVP\_ISCSI\_C | DVS_Storage |
| DVP\_ISCSI\_D | DVS_Storage |

Navigate to networking, expand the Robots datacenter object, open the context menu for switch DVS_Storage and add the listed distributed port groups.

Select defaults for the port group and ensure VLAN is set to none, the trunk native VLAN is 2011, the storage subnet is supernetted containing /27 ranges.

![4bac0fda5ff7dc0f5767cd97693ce43b.png](_resources/a0b9cc8ba9b34a3495b5645edc931fca.png)

![a700d34b27d0ed8014ed0768f50c1f12.png](_resources/7481d5c09d27462ba7fc877d27839270.png)

![8a63adbe92da40b5e0f8d02e98c4b771.png](_resources/80cdc5332e52444e849da2831e9e1584.png)

![be62967071cfb66321fa3d04cb44ad6c.png](_resources/ded1674d6c3e43cf98e4ac057c779181.png)

# Configure the DR cluster

- The DR cluster will consist of a single ESXI host, rdis-vmserver04.
- A cluster is used in preference to a standalone host to enable vmotion between clusters and keep the distributed switch architecture.
- An emergency / full DR is a manual process requiring a the replicated volumes to be promoted from (read-only) secondary status to (read-write) primary status via the storage array controller admin interfaces.
- A planned DR / maintenance event - where essential virtual machines can vmotion from one cluster to another without a requirement to promote storage, can occur as a live migration with this configuration.

## Create new cluster

From the vCentre admin console navigate to hosts and clusters, select the Robots datacenter and create new cluster.

![268f1df34707ba0f6b2d7aa53158abe3.png](_resources/d4445738410045ab86a6637f949f1a3a.png)

Uncheck DRS, HA and vSAN, this cluster will consist of a single host.

![df9a1f3e4412ff7a4c3041c7487f6211.png](_resources/c62d133dbe7442a2878ce634876c8658.png)

Select the 'DR' cluster and open the context menu, add hosts, add rdis-vmserver04.

![69e438c9078966730ca9f5507d2202ab.png](_resources/deec045da79e4f6ca7f4e55b71ddd808.png)

## Configure the Management network for rdis-vmserver04

### Release vmnic5 from vswitch0

Select rdis-vmserver04, navigate to configure -> networking -> virtual switches -> vswitch0 -> manage physical adapters

Select vmnic5 and remove it from the vswitch0.

![ca97879c7e3c429989e512ea6340df4f.png](_resources/aa71851f6bb0447db8fe8aef71fe42f6.png)

![6094ea5679742ed698e702f136fcb085.png](_resources/42eff0119677431c95cf8a1a9283ca05.png)

Assign rdis-vmserver04 to the DVS_Management switch.

![f0789c6f7566ef7f781c551b802a8094.png](_resources/7156faca53764780a2fe914c7668534a.png)

![24b66b6677a43375a2c43234baaa5ea7.png](_resources/5605fa906855401e833663f2d05d669f.png)

Do not assign any vmnic adapters from the host to the distributed switch at this stage. The DVP_Management switch is configured as a LACP lag, vmnic adapters must be assigned to the lag not the switch.

Navigate to hosts and clusters, select the rdis-vmserver04 host in the 'DR' cluster, select configure -> virtual switches -> DVS_Management -> manage physical adapters.

Select the uplink port named LAG_Management-1 and attach vmnic5.

### Configure the LAG port channel on the DR site switch to include vmnic5

Login to the DR site switch and configure the vmnic5 physical switch port to be in a LACP port channel.

ssh admin@129.67.94.7

password: Password0

Move vmnic5 to the port channel lag.

```sh
## VM Management traffic LACP for vmnic5

enable
configure

## SR635 #4 VM Management

# remove existing lag
no interface port-channel4

# create lag for VM Management
interface port-channel4
switchport mode trunk
switchport trunk native vlan 2001
switchport trunk allowed vlan 2001,2002,20
description "SR635 #4 VM Management LACP"
exit

#configure vmnic5 port for VM Management
interface Ethernet1/9
channel-group 4 mode active
exit

show running
exit
copy running-config startup-config
write
```

Navigate to hosts and clusters, select the rdis-vmserver04 host in the 'DR' cluster, select configure -> virtual switches -> DVS_Management -> migrate networking.

Do not select any vmnic adapters to be assigned to an uplink.

Select VMkernel0 and VMkernel1 and assign to distributed port groups, DVP\_Management\_Network and DVP\_External\_Management_Network respectively.

![d703e8759470a152dfb553aefc5b7fa2.png](_resources/339d1d64f5aa437aa31a11ee78dcfdb1.png)

Wait for the ESXi host to be available, this should be no longer than a minute, check you can connect to the hosts admin console to confirm changes https://129.67.94.29/ui.

### Remove the standard vSwitch0 virtual switch on rdis-vmserver04

Navigate to hosts and clusters, select the rdis-vmserver04 host in the 'DR' cluster, select configure -> virtual switches -> vSwitch0 -> remove.

the vmnic4 adapter will now be released.

### Add vmnic4 to the DVS_Management distributed virtual switch LAG uplink

Select the DVS\_management switch, check manage physical adapters, select LAG\_Management-0 and assign vmnic4

### Configure the LAG port channel on the DR site switch to include vmnic4

Login to the DR site switch and configure the vmnic5 physical switch port to be in a LACP port channel.

ssh admin@129.67.94.7

password: Password0

Move vmnic4 to the port channel lag.

```sh
## VM Management traffic LACP for vmnic4

enable
configure

## SR635 #4 VM Management

#configure vmnic4 port for VM Management
interface Ethernet1/8
channel-group 4 mode active
exit

show running
exit
copy running-config startup-config
write
```

Check port channel 4 is up. Channel-group 14 has an issue in this example.

![370e2e9b611fe8120f9e6339be7d2e49.png](_resources/2669a3466b30494a827b3b12fa222f12.png)

### Create vMotion VM kernel interface for rdis-vmserver04

Navigate to hosts and clusters, select the rdis-vmserver04 ESXi host in the 'DR' cluster, select configure -> VMkernel adapters -> add networking.

Select VMkernel, select existing network 'DVP_vMotion', check enabled services 'vMotion'. This should be VMkernel device vmk2 at this stage of the build.

Set a static IPv4 address 10.0.2.21/24.

![89cb4db87ccaf90e79e37f9db2d2e66e.png](_resources/5a9e800f004542c1976f94e87250b550.png)

## Configure the Guest network for rdis-vmserver04

Navigate to networking, select the DVS_Guest distributed virtual switch from the Robots datacenter, open the context menu -> add and manage hosts, select rdis-vmserver04.

Do not add any vmnic adapters, these will be added to the LACP uplinks, do not migrate any VMkernels or VM networking (distributed port groups).

Navigate to hosts and clusters, select the rdis-vmserver04 ESXi host in the 'DR' cluster, select configure -> virtual switches -> DVS_guest -> manage physical adapters.

Select LAG\_Guest-0 add adapter vmnic1, select LAG\_Guest-1 add adapter vmnic3

![7b9eef36f5bf540e23ee78470a53dc40.png](_resources/fd37e729ea804d939c4d56e099675433.png)

## Configure the Storage network for rdis-vmserver04

Navigate to networking, select the DVS_Storage distributed virtual switch from the Robots datacenter, open the context menu -> add and manage hosts, select rdis-vmserver04.

Add vmnic adapters vmnic0 and vmnic2 to uplinks 1 and 2 respectively. There are no LACP uplinks used for storage.

Do not migrate any VMkernels or VM networking (distributed port groups).

# Configure Storage

## Create the VMkernel interfaces for each server to access the iSCSI multipath target ports.

| Server | Portgroup | VMkernel | IP  |
| --- | --- | --- | --- |
| rdis-vmserver01 | DVP\_iSCSI\_A | vmk3 | 10.0.3.11/27 |
| rdis-vmserver01 | DVP\_iSCSI\_B | vmk4 | 10.0.3.43/27 |
| rdis-vmserver01 | DVP\_iSCSI\_C | vmk5 | 10.0.3.75/27 |
| rdis-vmserver01 | DVP\_iSCSI\_D | vmk6 | 10.0.3.107/27 |
| rdis-vmserver02 | DVP\_iSCSI\_A | vmk3 | 10.0.3.13/27 |
| rdis-vmserver02 | DVP\_iSCSI\_B | vmk4 | 10.0.3.45/27 |
| rdis-vmserver02 | DVP\_iSCSI\_C | vmk5 | 10.0.3.77/27 |
| rdis-vmserver02 | DVP\_iSCSI\_D | vmk6 | 10.0.3.109/27 |
| rdis-vmserver03 | DVP\_iSCSI\_A | vmk3 | 10.0.3.15/27 |
| rdis-vmserver03 | DVP\_iSCSI\_B | vmk4 | 10.0.3.47/27 |
| rdis-vmserver03 | DVP\_iSCSI\_C | vmk5 | 10.0.3.79/27 |
| rdis-vmserver03 | DVP\_iSCSI\_D | vmk6 | 10.0.3.111/27 |
| rdis-vmserver04 | DVP\_iSCSI\_A | vmk3 | 10.0.3.17/27 |
| rdis-vmserver04 | DVP\_iSCSI\_B | vmk4 | 10.0.3.49/27 |
| rdis-vmserver04 | DVP\_iSCSI\_C | vmk5 | 10.0.3.81/27 |
| rdis-vmserver04 | DVP\_iSCSI\_D | vmk6 | 10.0.3.113/27 |

Navigate to hosts and clusters in the vCentre admin console, for each ESXi host select configure -> networking -> vmkernel adapters -> add networking -> vmkernel network adapter

![c0ba430c285ebe0727c3c21aa9583b0c.png](_resources/36e4bf32ab6545a38b9a93d25cc76665.png)

Browse and select the distributed port group, accept defaults and do not enable specific services (management, vmotion etc).

![82769350c7978c4217fca3a36081c170.png](_resources/ae4ae87c717342d7af0ecd2852f17aeb.png)

Configure the IP address for the VMkernel interface.

![5d0a74d3e884ee82c2dd0a502fcbb5f2.png](_resources/fbc952d2e7ca4ad48eb3cbeed53a038b.png)

A host with a full compliment of iSCSI VMKernel adapters will resemble.

![884dac0546dd91adaa838bfd7f6a43eb.png](_resources/f8ad1ce9677b420ba21268f53423dd4c.png)

To continue iSCSI target configuration, the storage array networking requires adjustment for multipath.

# Configure Storage Array

Configure the Primary and DR site storage arrays.

### Links

https://thinksystem.lenovofiles.com/storage/help/index.jsp

Management Software:

- ThinkSystem System Manager - embedded web-based management console on each controller
- Secure Command Line Interface - CLI
- ThinkSystem SAN Manager - Web application installed on a server instance configured with a web server with proxy configuration

## Array web console login

> user: admin
> pass: Password0

| Site | Array | Controller consoles |
| --- | --- | --- |
| Primary | DE2000H | https://10.0.1.1:8443<br>https://10.0.1.2:8443 |
| Primary | DE4000H | https://10.0.1.3:8443<br>https://10.0.1.4:8443 |
| DR  | DE4000H | https://10.0.1.31:8443<br>[https://10.0.1.32:8443](https://10.0.1.31:8443) |

### Array Storage Management Interface Configuration

Ensure interfaces are configured with the following IP attributes.

| Site | Array | Controller | Management Port | Label | IP  | VLAN<br>(native/untagged) |
| --- | --- | --- | --- | --- | --- | --- |
| Primary | DE2000H | A   | P1 (P2 reserved by vendor) | rdis-vmshelf01a | 10.0.1.1/24 | 2001 native |
| Primary | DE2000H | B   | P1 (P2 reserved by vendor) | rdis-vmshelf01b | 10.0.1.2/24 | 2001 native |
| Primary | DE4000H | A   | P1 (P2 reserved by vendor) | rdis-vmshelf02a | 10.0.1.3/24 | 2001 native |
| Primary | DE4000H | B   | P1 (P2 reserved by vendor) | rdis-vmshelf02b | 10.0.1.4/24 | 2001 native |
| DR  | DE4000H | A   | P1 (P2 reserved by vendor) | rdis-vmshelf03a | 10.0.1.31/24 | 2001 native |
| DR  | DE4000H | B   | P1 (P2 reserved by vendor) | rdis-vmshelf03b | 10.0.1.32/24 | 2001 native |

### Array Storage iSCSI Interface Configuration

Multipath IP configuration scheme example:

[https://thinksystem.lenovofiles.com/storage/help/index.jsp?topic=%2Fthinksystem\_storage\_de\_himg\_11.60.2%2FConfigure\_hosts\_Windows.html](https://thinksystem.lenovofiles.com/storage/help/index.jsp?topic=%2Fthinksystem_storage_de_himg_11.60.2%2FConfigure_hosts_Windows.html)

Ensure interfaces are configured with the following IP scheme.

| Site | Array | Controller | Host Interface | Label | IP  | VLAN | Subnet |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Primary | DE2000H | A   | e0c | rdis-vmshelf01a | 10.0.3.1/27 | 2011 native | A   |
| Primary | DE2000H | A   | e0d | rdis-vmshelf01a | 10.0.3.33/27 | 2011 native | B   |
| Primary | DE2000H | B   | e0c | rdis-vmshelf01b | 10.0.3.65/27 | 2011 native | C   |
| Primary | DE2000H | B   | e0d | rdis-vmshelf01b | 10.0.3.97/27 | 2011 native | D   |
| Primary | DE4000H | A   | e0c | rdis-vmshelf02a | 10.0.3.2/27 | 2011 native | A   |
| DR  | DE4000H | A   | e0d | rdis-vmshelf02a | 10.0.3.34/27 | 2011 native | B   |
| DR  | DE4000H | B   | e0c | rdis-vmshelf02b | 10.0.3.66/27 | 2011 native | C   |
| DR  | DE4000H | B   | e0d | rdis-vmshelf02b | 10.0.3.98/27 | 2011 native | D   |
| DR  | DE4000H | A   | e0c | rdis-vmshelf02a | 10.0.3.3/27 | 2011 native | A   |
| DR  | DE4000H | A   | e0d | rdis-vmshelf02a | 10.0.3.35/27 | 2011 native | B   |
| DR  | DE4000H | B   | e0c | rdis-vmshelf02b | 10.0.3.67/27 | 2011 native | C   |
| DR  | DE4000H | B   | e0d | rdis-vmshelf02b | 10.0.3.99/27 | 2011 native | D   |

## Disable unused network adapters in the storage array heads

The purpose of this is to ensure the non routable address ranges are not displayed in iSCSI discovery with VMWare.

Login to each array and disable the unused interfaces each controller.

Navigate to settings -> system -> configure iSCSI ports -> Controller A -> Port 0a -> disable IPv4 / IPv6

Navigate to settings -> system -> configure iSCSI ports -> Controller A -> Port 0b -> disable IPv4 / IPv6

Navigate to settings -> system -> configure iSCSI ports -> Controller B -> Port 0a -> disable IPv4 / IPv6

Navigate to settings -> system -> configure iSCSI ports -> Controller B -> Port 0b -> disable IPv4 / IPv6

## Array Disk Pool Configuration

All disk in the DE2000H and DE4000H arrays are configured in a single DDP (dynamic disk pool) pool, Pool0.

# ESXi Create iSCSI storage adapters

Navigate to hosts and clusters and select each ESXi host in the 'Primary' cluster and perform the following task.

Select configure -> storage -> storage adapters -> add storage adapter -> add software iSCSI adapter

![824cdd06ea827c58470806cd901f289e.png](_resources/a90a5a623f2e43a8855329b4cb768c25.png)

![aa7bbb9f8875e0d0b995e447e201bb31.png](_resources/4dd2b7273b3b490aa2b99ad464ee844f.png)

The iSCSI adapter will not use port bindings, chap authentication (isolated storage network) or any non default configuration.

For the ESXi hosts to discover iSCSI exports an ACL must be populated on each storage array.

## Create a host object for each ESXi host on each storage array

- This acts an ACL to allow granular access for sets of ESXi hosts.
- The host IQN identifier can be found in the vCenter console at the iSCSI adapter for each host.
- Add each ESXi host rdis-vmserver01-04 as a host profile on each storage array.
    ![692f472c2ead4a4f3c94643a7926d7d4.png](_resources/1644bc1070a4482795866192c3126190.png)

![73fc64844c1dbc4dace426d7d33512b8.png](_resources/bc8367c6731e4dd1b7acaf703361a4f9.png)

![fcd7fae2d39ff1fed6a64a3804419c2a.png](_resources/2d505be2d62d4b2e98795d4103fa6239.png)

### Create a host cluster

A volume can only be assigned to one host object, to allow the DR host rdis-vmserver04 to access volumes on the primary site storage arrays add all hosts to the rdis-vmserver cluster object.

![4238cabf31d7afb1b47b8bccd7feba07.png](_resources/c70235ca5a094c249ad28b7c33c4d960.png)

## Dynamic discovery for iSCSI endpoints

Once the iSCSI adapter is present on each ESXi host and the each storage array has a populated iSCSI ACL, perform a dynamic discovery

For each ESXi host navigate to hosts and clusters -> configure -> storage adapters, select the iSCSI adapter -> dynamic discovery.

Enter an IP for each storage array, the discovery process will enumerate all other endpoints on the storage in the A/B/C/D subnets, enter addresses 10.0.3.1, 10.0.3.2 and 1.0.3.3.

![66c280548d88420b9d4519866ab217df.png](_resources/8c8561c02bd044eaba9136fa966dab4e.png)

Select rescan adapter.

![2daa066695286d466d071793491f3851.png](_resources/c1d5eeb76e0140c4b8c4d401b4cff13c.png)

- The Static targets tab will now be populated. Even though only 1 IP of the 4 used on each storage array was used for discovery, all iSCSI IP endpoints are discovered for each array.
- The target IQN endpoint for an array remains the same regardless of the controller or physical adapter IP/endpoint, there is now a multipath circuit to each array, on each array controller and each array controllers configured physical network adapters.
- If IQNs are listed with 192.168.130/131 addresses (or any inaccessible IPs), these are likely the default IP's for unconfigured physical adapters on the array controllers (usually 1Gbps links), these can be removed from the static target list but it is preferable to disable the IP stack for this adapter on the storage array controllers.

![0d0250c09ce1295e0a64e2ef0a6881c1.png](_resources/651bc4bac2ef4ad19d2647174dff9747.png)

Rescan the iSCSI adapter to remove configuration change notifications.

# Create test volumes on each storage array

Refer to this table to name storage volumes and vCenter datastores.

| Storage Array | Volume Name (VMware datastore name) | Workload Name | Assign Host Cluster | Note |
| --- | --- | --- | --- | --- |
| rdis-vmshelf01 | PRI\_DE2000H\_vmcluster_admin | PRI\_DE2000H\_vmfs | rdis-vmserver | admin volume mounted as datastore on Primary and DR clusters, this should house essential virtual machines such as the VCSA appliance. |
| rdis-vmshelf01 | PRI\_DE2000H\_vol | PRI\_DE2000H\_vmfs | rdis-vmserver | example volume to be mounted as datastore and replicated to DR array |
| rdis-vmshelf02 | PRI\_DE4000H\_vol | PRI\_DE4000H\_vmfs | rdis-vmserver | example volume to be mounted as datastore and replicated to DR array |
| rdis-vmshelf03 | DR\_DE4000H\_vmcluster_admin | DR\_DE4000H\_vmfs | rdis-vmserver | admin volume mounted as datastore on Primary and DR clusters, for a planned DR / failover the essential virtual machines can be moved via vMotion here. |
| rdis-vmshelf03 | PRI\_DE2000H\_vol_repl | DR\_DE4000H\_vmfs | rdis-vmserver | In full DR outage this will be promoted within the mirro from secondary to primary then mounted in vCentre on host rdis-vmserver04. Not mounted as a datastore at present. |
| rdis-vmshelf03 | PRI\_DE4000H\_vol_repl | DR\_DE4000H\_vmfs | rdis-vmserver | In full DR outage this will be promoted within the mirro from secondary to primary then mounted in vCentre on host rdis-vmserver04. Not mounted as a datastore at present. |

> The replication volumes must be equal or larger in size to the source volume for a mirror.

## Create Volumes on each array

- Create new Volume, assign host later
- Create new workload or select existing workload (where previous created for another volume), workloads should be type 'VMware VMFS', the name is unimportant the respective names have been chosen to be easily identified, PRI\_DE2000H\_vmfs / PRI\_DE4000H\_vmfs / DR\_DE4000H\_vmfs
- Add new Volume, set size = 1000GiB, this can be expanded as required

![50beabd20c5bf99279584a065680f624.png](_resources/582ffabd015f44eeb29293bcdd3d1df3.png)

![234e0a07219da3558110d84204a8dc97.png](_resources/fec653b6ae7f49ed8931e195972f7711.png)

## Assign Volumes to the new Host ACL

Use the cluster object as the for the 'assigned to' host ACL.

![f0665b7033045d9b9fe4322af491c1cc.png](_resources/8781d2121d9645f6bfcf6fd4caae503a.png)

# Create VMware Datastores

Create datastores on the on the new volumes presented over iSCSI.

To help identify the Volume check the WWID of the volume in the storage array console.

![cb116993a92e04a73c4bf68c7109c7fa.png](_resources/90228244dfeb462daa0891af0ab4f50e.png)

Navigate to hosts and clusters and select any ESXi host (the new datastore will be present for all members of the cluster), select actions -> storage -> new datastore

![5d2dbf3b8ceffde0980012b4bb76858b.png](_resources/cc1f21241bab402d83253e56f4664190.png)

Identifying the storage device by the WWID set the name according to the above table.

![1e4ae4212132173c9af9f187ad1f277f.png](_resources/48fa3f3bd2a244839e93fa718ee5ad79.png)

Format as VMFS version 6, use all available partions and available space.

Navigate to storage and ensure any local datastores are renamed to denote the host to which they are attached.

The storage array datastores should be now listed.

![3ddbac0cab030bb3d404ed350cbb3458.png](_resources/14aab6a6265541dca130d7311d014e32.png)

## Migrate VCSA to shared storage

The VCSA appliance is bound to the first ESXi host in the cluster residing upon a local datastore from the initial build..

Ensure the PRI\_DE2000H\_vmcluster_admin datastore has been created.

Navigate to hosts and clusters, find the rdis-vcsa01 virtual machine, open context menu and select migrate.

![444fb8549a24de38a106d19f7c9d3e7f.png](_resources/9b28829fa10d49848201ffff45ce5352.png)

Select change both compute resource and storage.

![3f54049ff3f1552aae91dd0dc5c05c1f.png](_resources/34f4c72b637c46818e8e0ddf677844ee.png)

Select another host and the new admin datastore. The initial appliance was thin provisioned, no disk format change should be required but can be changed. Retain the destination port group.

![f33b74e3147396d003ff1b00df7f4908.png](_resources/5747284b086c4c42bb899d27cb36279e.png)

### Test HA failover of the VCSA appliance

To verify the VCSA appliance is now decoupled from local storage set the ESXi host currently hosting the VCSA appliance into maintenance mode and migrate hosts to other ESXi cluster hosts.

![8de13fedfe11a76887a0c6b0acc2f285.png](_resources/3d2b576c1e844a12b5ed8decaab9d9d8.png)

## ![1d665a449c1e9ef89d32012fa1ef4f4e.png](_resources/b179b7bbe234465d90083994cb382c00.png)

### Migrate a virtual machine to the DR cluster and DR storage

Select the virtual machine context menu and migrate.

![5876a1f658733f267e7136c65bb9d8a4.png](_resources/4c4cdc4174704dd3a78b1bf86dcd1a5d.png)

Select both compute and storage resource change.

![74e0acf5a1135380f1a20f87d86d2eb3.png](_resources/e86426870f9943e087956eaff35877e5.png)

Select the DR host.

![0d8229ffc1707ac2301b0e7fce290369.png](_resources/24037c5d0b264e408644431e24442c63.png)

Select the DR storage.

![72a85a381e6ba9cdf1349b2c4d5b9239.png](_resources/d77d7efdf8bb4041b0971f9d556f9643.png)

The networks should not require changing as they have been extended to both clusters, the uplinks to customer core switches pass the storage VLAN.

This operation is slower than a local cluster migration owing to the traversal across the core switches, although still acceptably fast.

The vMotion task can (automatically) traverse only the vMotion network where the target + destination storage is not presented to both clusters if there is a requirement for this architecture.

# Update cluster software

| Web Console | SSH rdis-20 socks5 address | Internal FQDN | User | Pass |
| --- | --- | --- | --- | --- |
| Appliance administration | https://129.67.94.11:443 | https://rdis-vcsa01.robots.university.ac.uk | administrator@vcsa.robots.university.ac.uk | Password0 |
| Vsphere  administration | https://129.67.94.11:5480 | https://rdis-vcsa01.robots.university.ac.uk:5480 | root | Password0 |

Ensure the rdis-20 ssh profile is started and your browser is using the local socks5 ssh proxy on 127.0.0.1:8080, edit the local hosts on your laptop to include the following entry:

> 129.67.94.11 rdis-vcsa01.robots.university.ac.uk

## vCenter update

Login to the appliance administration console with the FQDN rather than the appliance IP and a use compatible browser (google Chrome seems to render the application better than Firefox).

Navigate to update, select check updates -> check CDROM + URL. Relevant patches will be listed.

![fd36e493ed7530d728837259f502f7f5.png](_resources/8ca55336cea04657a66914af97501bf6.png)

Select the latest update then select run pre-update checks, this should pass validation, select stage and install, follow all prompts.

![f75f21bbc7d781a98cef2bd2f1666861.png](_resources/525b0f8fcccf44cfb888cd072b93db3c.png)

As this is a fresh install there is no backup of the vCenter server.

The appliance will reboot, reconnect to the appliance administration console to follow the post install actions progress.

## Virtual machine update

Login to the vCenter administration console with the FQDN.

Navigate to hosts and clusters -> Robots datacenter -> Primary cluster -> updates -> VM hardware.

Select all virtual machines (currently VCSA appliance and HA state appliances), select upgrade to match host.

![cecf47252a4bbdf21839fa151f9d4f0f.png](_resources/4f763cf38f2d4a1f9e5ae94178277d59.png)

Virtual machines will be updated to compatibility version 11.

The vcsa appliance will be unable to update itself to a later virtual machine type.

### ![c7f1bc5f25777097c6d8a50fc25a5480.png](_resources/4f2fa8c3d7de4aca95005544c6bc6265.png)

![a49394ff9705693e57f2fcbb382bb8c9.png](_resources/3087ee6291d5447888572d7a0343c46a.png)

Select Esxi 7.0 profile for a version 17 virtual machine version.

![615e2703a87ca05247d9ea5c9d3692e4.png](_resources/5e5edc3ad4764523848ae478cf8cd00f.png)

Restart guest OS, vCentre will be unavailable until reboot has completed. Login directly to the ESXi host where the VCSA appliance is located to check progress if required.

![d0826e3eb4ea1155d1ac068108760874.png](_resources/2b1f65ac7d914258922a6e9324a0f5f0.png)

After reboot check the VCSA appliance compatability version in the vCentre console.

![87fe5966cf25255db3c0ca3b8348511b.png](_resources/dc1b953d21b94bea91d831cbe7f8c905.png)

## Esxi host update

To assist in this update, HA should be enabled and the VCSA appliance should reside upon shared storage and tested to migrate successfully between ESXi hosts as they patch and reboot.

Navigate to hosts and clusters, select the Robots datacenter, select updates -> baselines -> check compliance -> check pre-check remediation

![678c65c212a968cab798b88840ca8a94.png](_resources/c2f867118c364cec89a50cfb4746f826.png)

Apply patches for ESXi hosts, check all attached baseline profiles -> stage.

![14ebc0530793b2c386898a5ccec0dbf8.png](_resources/2b9b501cc847447dbfe8872b670f7465.png)

The stage task should complete without error, check remediate. ESXi hosts will enter maintenance mode and each host will begin update, this process is applied in series, expect ESXi hosts to become unavailable and virtual machines to migrate.

![3164fc2b653ca53298d70f7debcf208b.png](_resources/c1f56a514a2b45b8a7dd1b28c8bc9328.png)

Once complete all hosts should be compliant.

![b0a0865fc6e5c59333be57044721e1d5.png](_resources/911274a1fc8f4b7994a61e67f50a6f7e.png)

Note: With the DR single host cluster you may have a remediation warning that DRS/HA are disabled.

# Build OCF Admin Virtual Machine

Owing limited connectivity to the environment and the need to run some admin applications in local proximity to the cluster, a small server is required.

Upload a guest operating system ISO to one of the ESXi hosts via ssh/scp, ideally to the DE4000H\_vmcluster\_admin datastore.

## Create virtual machine

Navigate to hosts and clusters, select the 'Primary' cluster, open the context menu and select new virtual machine.

Name the machine 'OCF-admin' and place on the Robots datacenter and 'Primary' cluster.

Select storage DE4000H\_vmcluster\_admin, select version 18 hardware compatibility, ESXi 7.0.1.

Select the correct guest and guest OS version, example; Linux and Centos 7 (64-bit) / Linux and Ubuntu Linux (64-bit).

Select compute requirements and ensure adequate adequate disk size is selected ~ 64GB should suffice for a desktop and admin applications.

Configure the virtual hardware with appropriate compute resource and, select the DVP\_External\_Management_Network port group and mount the OS install ISO.

![1a0e53383b435c128567c3c496a18bb7.png](_resources/9cf205014fe74671a94eba971775990c.png)

Run through the guest installer, with the following attributes.

| Attribute | Option |
| --- | --- |
| IP address | 129.67.94.25 |
| Netmask | 255.255.254.0 |
| Gateway | 129.67.95.254 |
| Name server address | 129.67.1.180 129.67.1.1 |
| Hostname | OCF-admin |
| Domain name | robots.university.ac.uk |
| User name | ocf |
| Password | Password0 |
| Enable OpenSSH server | yes |

Edit virtual machine settings to remove/disconnect the boot ISO, reboot.

SSH to the virtual machine ssh `ocf@129.67.94.25` ensure all upgrade packages are applied for kernel enhancements running on top of hypervisors.

Edit virtual machine settings and add an additional network adapter to the DVP\_Management\_Network.

![a0026ca22b51719eb1a3477ec3c578ac.png](_resources/980c94451b2743e3b4317dea2a81facc.png)

SSH to the virtual machine `ssh ocf@129.67.94.25` and configure the secondary adapter for the isolated management network.

Centos; edit /etc/sysconfig/network-scripts/ifcfg-ens192 and apply changed with `systemctl restart network`

```sh
TYPE=Ethernet
BOOTPROTO=none
NAME=ens192
DEVICE=ens192
ONBOOT=yes
IPADDR=10.0.1.60
PREFIX=24
```

Ubuntu; edit /etc/netplan/01-netcfg.yaml and apply changes with `sudo netplan apply`

```sh
network:
  version: 2
  renderer: networkd
  ethernets:
    ens160:
      addresses: [ 129.67.94.25/23 ]
      gateway4: 129.67.95.254
      nameservers:
          search: [ robots.university.ac.uk ]
          addresses:
              - "129.67.1.180"
              - "129.67.1.1"
    ens192:
      addresses: [ 10.0.1.60/24 ]
```

Traceroute/ping and check ARP cache to ensure layer 2 connectivity to adjacent management interfaces of the VM servers and storage admin interfaces.

## Install VMware tools (optional - not applied in this instance)

This may overwrite open vmware-tools where auto configured by the operating system, many operating systems do not come out of the box with these packages and you may install open vmware-tools (Linux) or opt to install the proprietary VMware tools package as per this example (Windows/Linux).

Propriety vmware-tools compile kernel modules, every kernel update requires a tools update.

### Open vmware tools

Centos; `yum install open-vm-tools`

Ubuntu; `sudo apt-get install open-vm-tools`

### Proprietary vmware tools

Reboot the virtual machine, before the OS is loaded select to install VMware tools, this will mount an ISO containing the installer packages.

![dc46e6be8812f032aee129dbf5cc8638.png](_resources/90631f0304a342c2819402633589b01c.png)

Mount the ISO once the OS has started, extract the VMware tools installer, run the installer accepting all defaults and overwrite open vmware-tools packages.

```sh
ocf@OCF-admin:~$ lsblk
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                         8:0    0   16G  0 disk
└─sda1                      8:1    0   16G  0 part
  ├─OCF--admin--vg-root   253:0    0   15G  0 lvm  /
  └─OCF--admin--vg-swap_1 253:1    0  980M  0 lvm  [SWAP]
sr0                        11:0    1 55.9M  0 rom
ocf@OCF-admin:~$ sudo mkdir /mnt/cdrom;sudo mount /dev/sr0 /mnt/cdrom
[sudo] password for ocf:
mount: /mnt/cdrom: WARNING: device write-protected, mounted read-only.
ocf@OCF-admin:~$ ll /mnt/cdrom/
total 56855
dr-xr-xr-x 2 root root     2048 Mar 25  2020 ./
drwxr-xr-x 3 root root     4096 Jun 21 10:57 ../
-r-xr-xr-x 1 root root     1976 Mar 25  2020 manifest.txt*
-r-xr-xr-x 1 root root     4943 Mar 25  2020 run_upgrader.sh*
-r--r--r-- 1 root root 56414224 Mar 25  2020 VMwareTools-10.3.22-15902021.tar.gz
-r-xr-xr-x 1 root root   872044 Mar 25  2020 vmware-tools-upgrader-32*
-r-xr-xr-x 1 root root   918184 Mar 25  2020 vmware-tools-upgrader-64*
ocf@OCF-admin:~$ sudo cp -a /mnt/cdrom/VMwareTools-10.3.22-15902021.tar.gz /tmp/
ocf@OCF-admin:~$ cd /tmp
ocf@OCF-admin:/tmp$ tar -xvzf VMwareTools-10.3.22-15902021.tar.gz
ocf@OCF-admin:/tmp$ cd vmware-tools-distrib
ocf@OCF-admin:/tmp/vmware-tools-distrib$ sudo ./vmware-install.pl
```

## Install desktop

See remote admin desktop.

### Install Thinksystem SAN manager

Obtain the following installer from the Lenovo website, this is a download for the DE storage series and will require a chassis serial number found in the array web admin console for download authorisation.

Redhat/Centos based systems require `yum install -y redhat-lsb`

> lnvgy\_utl\_sanmanager\_05.00.54.9013\_linux_x86-64.bin

Run the installer from the Guacamole desktop for the graphical installer `sudo ./lnvgy_utl_sanmanager_05.00.54.9013_linux_x86-64.bin`

![b196ff68cafd8de6f26f5ae69eb70a4d.png](_resources/972cf44e5ea74bfcb0e15f4eb11d827c.png)

Follow the prompts accepting defaults unless listed below.

- Disable autosupport, this will be enabled after configuration.
- Allow storage array connection without validating certificates, the arrays do not have CA validated certificates, they reside upon an isolated networks in way of compensating control.

The application installs a systemctl unit file and listens on tcp 8080 / 8443.

```sh
ocf@OCF-admin:~$ systemctl list-unit-files | grep -i think
ThinkSystemSANManager.service                        enabled
```

> https://129.67.94.25:8443/

Open a web browser on the Guacamole desktop and navigate to the application URL. The OCF-admin host does not have a full matching FQDN, the application does not readily accept external connections from a host that cannot be validated (this is likely configurable), launching from the Guacamole desktop allows the browsers to run on the same host as the application with both server and client validating hostname from the common OS.

Set the password to `Password0`

![592152599fdaebd62e93f88b2d6eb1b1.png](_resources/4cd3b4af5db64542b3867b3dff2cfccf.png)

Discover all storage arrays on the management network.

![6aaf54977af592f5ffd31399a12d346c.png](_resources/d442f6b634de46649c0b5732d1f7df6e.png)

Select the discovered storage arrays, the certificates will be automatically trusted.

![b27dfabd29fb4d0e91a42f67e0b49332.png](_resources/c074c43e79094b77b9c150ccaf2c7114.png)

Enter storage array admin passwords.

![9db7ddf14838b2345389e35bbccbb27a.png](_resources/ab1d60c3dd694a52bfd3836a35d7793e.png)

Mirroring options are now available.

![fb171b5ffed1271264b142db2f2c5c71.png](_resources/ca9da25935db4adf8efa210576b73086.png)

### Install Thinksystem storage manager

> There is no need to install this package in a production environment
> 
> lnvgy\_utl\_storagemanager\_11.60.0A54.0007\_linux_x86-64.bin
> 
> This will only install correctly on an rpm based system owing to rpm pre/post script actions
> 
> This is only run to setup the new array with no configuration in conjunction with a local dhcp service, the utility will discover a new unconfigured array and push a minimal configuration intended to allow access to the the system manager web application running on the array controllers.
> 
> [https://thinksystem.lenovofiles.com/storage/help/topic/thinksystem\_storage\_de\_himg\_11.60.2/ThinkSystem\_Storage\_DE_HIMG.pdf](https://thinksystem.lenovofiles.com/storage/help/topic/thinksystem_storage_de_himg_11.60.2/ThinkSystem_Storage_DE_HIMG.pdf)
> 
> Thinksystem host utilities are a subset of this installer without the agent.

Run the installer `chmod +x lnvgy_utl_storagemanager_11.60.0A54.0007_linux_x86-64.bin;sudo ./lnvgy_utl_storagemanager_11.60.0A54.0007_linux_x86-64.bin`
Disable the service once array is built.
systemctl stop smagent
systemctl disable smagent

## Setup NTP daemon

The storage replication mechanisms require synchronised time sources.

TSMs require time sync with XClarity Administrator, the TSM will be populated with the same time source as XClarity, as they are on an isolated network they cannot use the University time servers so must target the OCF admin virtual machine which is multi-homed and times from the University ntp server (this VM is not timed from the hypervisor).

Edit the chronyd config to include the RDIS ntp server and ensure the daemon listens on the isolated management network for the storage to sync against.

```sh
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
server ntp0.robots.university.ac.uk
server ntp1.robots.university.ac.uk
server ntp2.robots.university.ac.uk
server ntp3.robots.university.ac.uk

# Record the rate at which the system clock gains/losses time.
driftfile /var/lib/chrony/drift

# Allow the system clock to be stepped in the first three updates
# if its offset is larger than 1 second.
makestep 1.0 3

# Enable kernel synchronization of the real-time clock (RTC).
rtcsync

# Enable hardware timestamping on all interfaces that support it.
#hwtimestamp *

# Increase the minimum number of selectable sources required to adjust
# the system clock.
#minsources 2

# Allow NTP client access from local network.
#allow 192.168.0.0/16
allow 10.0.1.0/24

# Serve time even if not synchronized to a time source.
#local stratum 10

# Specify file containing keys for NTP authentication.
#keyfile /etc/chrony.keys

# Specify directory for log files.
logdir /var/log/chrony

# Select which information is logged.
#log measurements statistics tracking
```

Restart chronyd `systemctl enable chronyd;systemctl restart chronyd`

## Setup mail relay

The storage arrays are on an isolated network, in order to send alerts the OCF-admin host must run a simple relay.

```sh
apt-get update && apt-get install postfix mailutils libsasl2-2 libsasl2-modules curl

nano -cw /etc/postfix/main.cf

inet_interfaces = 10.0.1.60, 127.0.0.1
mynetworks = 127.0.0.0/8 10.0.1.0/24
relayhost = mail.robots.university.ac.uk:25
smtpd_relay_restrictions =
    permit_mynetworks
smtpd_recipient_restrictions =
    permit_mynetworks
    reject_unauth_destination
smtpd_sender_restrictions =
    check_sender_access hash:/etc/postfix/allowed_senders
    warn_if_reject reject_unverified_sender
    reject

nano -cw /etc/postfix/allowed_senders

root@robots.university.ac.uk permit
*                                  reject

postmap /etc/postfix/allowed_senders
systemctl enable postfix
systemctl restart postfix
```

# Create a content library

## Upload OVA templates, browse for address for library import

If you have local connectivity to the cluster, templates/ISOs maybe simply uploaded via the vCenter web console using 'local file' as the source, this step assumes you do not have local connectivity.

Upload files to one of the ESXi hosts via ssh/scp, ideally to the DE4000H\_vmcluster\_admin datastore, enable SSH on the ESXi host as appropriate.

Browse to the ESXi host at the 'folder' endpoint, you will be prompted for a local account credentials, root may only be available.

The vCentre server must be able to route to the ESXi management interface, in this example the 129.67.94/23 external management address of the ESXi host is used.

![5a0a8df55f38d82d0167b97f8d84dffc.png](_resources/12e8aa7447744e60b7c3c419eb00c008.png)

Navigate to the datastore/folder containing the target file.

![061d3f4d650ba33935f003afabe2836b.png](_resources/dc5de8894afb4db5a7d2c33919f302b8.png)

Copy the link of the target file intended to be imported to content library, edit the link to include ESXi host credentials, the link will be used to import the file to the content library.

| URL | URL modified with credential (special characters in the password entry should not be URL encoded) |
| --- | --- |
| [https://10.0.1.21/folder/ISO%2FXClarity\_administrator\_vmware\_appliance/lnvgy\_sw\_lxca\_83-3.3.0\_vmware\_x86-64.ova?dcPath=ha-datacenter&dsName=DE4000H\_vmcluster\_admin](https://10.0.1.21/folder/ISO%2fXClarity_administrator_vmware_appliance/lnvgy_sw_lxca_83-3.3.0_vmware_x86-64.ova?dcPath=ha-datacenter&dsName=DE4000H_vmcluster_admin) | [https://root:Password0@10.0.1.21/folder/ISO%2FXClarity\_administrator\_vmware\_appliance/lnvgy\_sw\_lxca\_83-3.3.0\_vmware\_x86-64.ova?dcPath=ha-datacenter&dsName=DE4000H\_vmcluster\_admin](https://root:Password0@10.0.1.21/folder/ISO%2fXClarity_administrator_vmware_appliance/lnvgy_sw_lxca_83-3.3.0_vmware_x86-64.ova?dcPath=ha-datacenter&dsName=DE4000H_vmcluster_admin) |

## Add admin library

Navigate to menu -> content libraries -> create.

Name the content library 'Admin Library', create this on the vCentre server rdis-vcsa01.robots.ac.uk.

Check local content library and accept all default.

Select DE2000H\_vmcluster\_admin for storage.

## Import OVA templates and ISOs

Navigate to menu -> content libraries -> admin library -> actions -> import item

![fcd4f93fad4111305b7ddcbea0605c6e.png](_resources/7a1e23e46d93468ea0b8f4b855e9f185.png)

Select the templates/ISOs from URL.

| Item name | Notes | Source File URL |
| --- | --- | --- |
| lnvgy\_sw\_lxca\_83-3.3.0\_vmware_x86-64.ova | XClarity Administrator | [https://root:Password0@129.67.94.20/folder/ISO%2FXClarity\_administrator\_vmware\_appliance/lnvgy\_sw\_lxca\_83-3.3.0\_vmware\_x86-64.ova?dcPath=ha-datacenter&dsName=DE4000H\_vmcluster\_admin](https://root:Password0@10.0.1.21/folder/ISO%2fXClarity_administrator_vmware_appliance/lnvgy_sw_lxca_83-3.3.0_vmware_x86-64.ova?dcPath=ha-datacenter&dsName=DE4000H_vmcluster_admin) |
| lnvgy\_sw\_vmuim\_520-7.2.0\_vmware_x86-64.ova | XClarity VMware Integrator | [https://root:Password0@129.67.94.20/folder/ISO%2FXClarity\_integrator\_vmware\_appliance/lnvgy\_sw\_vmuim\_520-7.2.0\_vmware\_x86-64.ova?dcPath=ha-datacenter&dsName=DE4000H\_vmcluster\_admin](https://root:Password0@10.0.1.21/folder/ISO%2fXClarity_integrator_vmware_appliance/lnvgy_sw_vmuim_520-7.2.0_vmware_x86-64.ova?dcPath=ha-datacenter&dsName=DE4000H_vmcluster_admin) |
| ubuntu-18.04.4-server-amd64.iso | Ubuntu 18.04 | https://releases.ubuntu.com/18.04/ubuntu-18.04.5-live-server-amd64.iso |
| CentOS-7-x86_64-Minimal-2009.iso | Centos 7 | [http://mirror.vorboss.net/centos/7.9.2009/isos/x86\_64/CentOS-7-x86\_64-Minimal-2009.iso](http://mirror.vorboss.net/centos/7.9.2009/isos/x86_64/CentOS-7-x86_64-Minimal-2009.iso) |

## ![c56cef84a73d6099295e4f81454e405b.png](_resources/3b9949f9e89a44c38ce7b6fd9d91a901.png)

![e4d4b3dc03e80ab3f1b7427dfc5513db.png](_resources/e3c4d68eabdb4049b8b105a80ab052f4.png)

Now that the content library is populated you may delete the files from the datastore that you uploaded to the content library. Note the new content library folder on the datastore.

## ![5ec1f6ec61641464c0e28f2c8a1a4b81.png](_resources/c0b7b94da55448568a45ea3729b315a8.png)

# Configure Storage Alerts

As the storage management interfaces are on an isolated network ensure the OCF-admin host is running a mail relay.

For each storage array, navigate to settings -> alerts and add the mail server and recipient email address.

Mail server: 10.0.1.60

Sender: root@robots.university.ac.uk

Recipient: root@robots.university.ac.uk

![d5aa15feb03d7afc5c99415530de6777.png](_resources/f9eb3290d3a646f89cc6885d0f73a713.png)

# Configure vCenter Alerts

From vCenter navigate to menu -> administration -> single sign on -> users and groups -> domain: vcsa.robots.university.ac.uk.

Select the administrator user, check edit and populate the email field with: root@robots.university.ac.uk.

![7e726b48552079eff98e2ae2d9f6954a.png](_resources/695c12ca7632429fb8ccdb38ce97953b.png)

Navigate to menu -> hosts and cluster -> select the VCSA instance -> configure -> general -> mail -> edit.

Enter the mail server: mail.robots.university.ac.uk and the mail sender: root@robots.university.ac.uk.

![20915a7c8d64905af137849d7471d03e.png](_resources/9e65127f522d4844924839baa4a2fe19.png)

# Configure ThinkSystem SAN Manager asynchronous replication

Open the ThinkSystem SAN manager web application, the chromium bookmark is on the OCF jump host.

|     |     |
| --- | --- |
| URL | https://129.67.94.25:8443 |
| User | admin |
| Password | Password0 |

Discover hosts, trust certificates when asked.

![3db881dcafe495356e6ed2aea22dcc92.png](_resources/837c805f91cb4c72ab7cf2ad04891889.png)

![eb74bf464fa8143eb5a5ccdc9d0a5bd7.png](_resources/4f5698ead986466e982f7a2f550c291d.png)

Enter the password for each array.

![9db7ddf14838b2345389e35bbccbb27a.png](_resources/ab1d60c3dd694a52bfd3836a35d7793e.png)

The inventory will now list the storage arrays.

![7dfe7bbc80991168047418b09274e673.png](_resources/9c3002cab4c546ce8c39c236a470dffa.png)

Select all storage arrays and check Launch.

NOTE: Aspects of the mirror creation are performed on array system manager web consoles, when launched from SAN manager web tokens are passed to enable/authorise the action.

![e9dd273f2b72ff51972a34407111220f.png](_resources/980534d1a0ec490ea9ffa07591519bff.png)

## Create storage replication mirrors

PUT DOCUMENTATION HERE - JOHN HAS PDF! - where can i find the link to this?

- The following replication profiles are for illustration, the destination volumes must be at least the equivalent size as the source volume, in production it would be safest to ensure the destination volume is a fraction larger.
- There is no tuning involved in this example, default replication frequency if quite aggressive at 10 minutes, the default source volume reservation is 20%, the default destination volume reservation is 20%, these reservations exist directly on the pool thus taking space from the pool.
- With the DE storage series the target replication volume is in read only mode, for a DR cluster to use the volume it must be promoted to primary status and remounted manually, this procedure is intended for a full primary site outage and inherently requires failback planning.

| Consistency group name | Source shelf | Source volume | Destination shelf | Destination volume |
| --- | --- | --- | --- | --- |
| PRI\_DE2000H\_vol_repl | rdis-vmshelf01 | PRI\_DE2000H\_vol | rdis-vmshelf03 | PRI\_DE2000H\_vol_repl |
| PRI\_DE4000H\_vol_repl | rdis-vmshelf01 | PRI\_DE4000H\_vol | rdis-vmshelf03 | PRI\_DE4000H\_vol_repl |
| PRI\_DE2000H\_vmcluster\_adm\_repl | rdis-vmshelf01 | PRI\_DE2000H\_vmcluster_admin | rdis_vmshelf02 | PRI\_DE2000H\_vmcluster\_adm\_repl |

Create and asynchronous mirrored pair, synchronous mirrored pairs are only compatible with FC networks.

![b3ef6a105b91b27bfec1c29ee036259f.png](_resources/cf0e014b287d49bca86089e1adbaa939.png)

Check new mirror consistency group.

![1fdbb72a9dccc3f36b2eec5224e801fd.png](_resources/f194e80d24bf48ceb8ae318ce5b6915f.png)

Select a unique name for the consistency group and the remote target array to replicate to.

![347f8dac287fde0920f884feb2fb2a62.png](_resources/d81612aca2d64df188fb9b147bfa5184.png)

Select the source volume to replicate from.

![0d780d478f0c53983a586d67717eef09.png](_resources/519bee72320747dfb6f60d0fedfde07c.png)

Select the destination volume to replicate to.

![620f33e4b2f79f443c7d0d8a86d02178.png](_resources/a47355b9b0e5438d8cbceae98ed98d61.png)

Check operations in progress on the source array. Launch the system manager web console for the source array from SAN manager to see the progress.

![408033cda2619702482bb8c465a80a69.png](_resources/04c7a1d8d0654a6095e0141ae3868f7b.png)

Check operations in process on the destination array. Launch the system manager web console for the destination array from SAN manager to see the progress.

![5a3d06f950090496c1604cf9cf76ee01.png](_resources/ec5afe57441d413a8a5ccec0a8c03d79.png)

When replication between two primary switches we observe 10Gbps speeds, when replicating to the DR array we see 1GBps speeds.

![6d21797442c27ae6340191f55cb5d3ea.png](_resources/0dac207ce1ed45cbaf010244aac62ba1.png)

Replication occurs from controller to controller over a single channel (uplink), this is illustrated in the performance metrics.

To make best use of bandwidth (for local switch/local site replication) ensure the source and target volumes are located on the same controllers on the arrays, multiple concurrent replication jobs between the same source and destination arrays should be balanced between array controllers.

![071b79447697f03d3d3e4d1869ccaf91.png](_resources/0a4225c5837545bc85b6491804b0170e.png)

Once volumes are synchronised and in an active replication schedule, you can see the amount of data to be replicated in the next replication event, this is a helpful metric when tuning the replication frequency and keeping your network administrators informed of the expected traffic profile.

![cefdf07589fec1c3f3a34efcfd2cc60d.png](_resources/326e962aa9824b409b5be0c6700b4ee1.png)

# XClarity VMware integration

| Appliance | Template | Source |
| --- | --- | --- |
| XClarity Administrator | lnvgy\_sw\_lxca\_83-3.3.0\_vmware_x86-64.ova | https://datacentersupport.lenovo.com/us/en/solutions/lnvo-lxcaupd |
| XClarity VMware Integrator | lnvgy\_sw\_vmuim\_520-7.2.0\_vmware_x86-64.ova | https://datacentersupport.lenovo.com/gb/en/documents/lnvo-vmware |

## Install XClarity administrator

Navigate to menu -> content libraries -> admin library -> ovf & ova templates, select the XClarity Administrator template, select the context menu and check new VM from this template.

![180e27dc39c125d84294468d4ed50b86.png](_resources/8a3fc84146eb46c88c81097390138ccc.png)

Name the virtual machine 'XClarity-Administrator', select the Robots datacenter, select the 'Primary' cluster.

Select deployment size small, select the DE4000H\_vmcluster\_admin datastore and select virtual disk format as thin.

Select the DVP\_Management\_Network and set static IPv4 address allocation, the appliance does not have vSphere template integration with the IP address attributes and requires configuration from a DHCP service, there is no need to populate any static IP attributes in this step.

![f43797e00b0ef9a0f85d48324d203259.png](_resources/fccc82ecea6f4134aa10b80199558320.png)

### Create a temporary DHCP service on DVP\_Management\_Network

SSH to the OCF-admin host, install some packages and build a tiny DHCP service.

Firewalld or equivalent service should be stopped or a rule to allow DHCP present. `systemctl stop firewalld`

> The docker daemon will lose the IPtables rules with this action, restart the docker service to remedy.

```sh
ssh ocf@129.67.94.25
su -
yum install dnsmasq
systemctl stop dnsmasq
systemctl disable dnsmasq
#start DHCP service
dnsmasq --no-daemon --port=0 --interface=ens192 --conf-file='' --dhcp-range=10.0.1.70,10.0.1.80,255.255.255.0,12h
```

Start the appliance from the vCenter console, watch the DHCP service output to find the appliance IP.

![6e060e9e65ca2094710db77a4f5cc5c4.png](_resources/847111a4ba6e4ef9b72d397ac19ea683.png)

Open the admin console and follow the setup wizard.

> https://10.0.1.72/

- Accept the license.
- Create user account.

> username: administrator
> password: Password0

- Return to initial setup (checkbox at bottom left of page).
- Configure network access, set IPv4 to static assignment and IPv6 to auto configuration.

> IP address: 129.67.94.17
> Network Mask: 255.255.254.0
> Gateway: 129.67.95.254

- Save IP settings and restart networking.
- In the vCenter console move the primary network adapter to DVS\_External\_Management_Network.
- Open the Xclarity Administrator admin console and login with the new administrator account.

> https://129.67.94.17

- The page will return to initial setup, check configure network access.
- Select the DNS tab, populate DNS attributes

> Host name: XClarity
> Domain name: robots.university.ac.uk
> DNS operating mode: static
> DNS server: 129.67.1.180
> DNS server: 129.67.1.1

- Save DNS settings.
- Return to initial setup.
- Configure date and time preferences.

> UTC -00:00, Greenwich Mean Time Great Britain
> 24h clock format
> NTP servers ntp0.robots.university.ac.uk ntp1.robots.university.ac.uk ntp2.robots.university.ac.uk ntp3.robots.university.ac.uk

- Return to initial setup.
- Service and support.

> periodic upload data - no thanks
> Call home configuration - revist this with customer number in hand (skip step).
> Lenovo upload facility, set prefix to robots.university.ac.uk, set email to root@robots.university.ac.uk, apply and enable.
> Lenovo warrenty web service, enabled, apply.
> Service recovery password (if this management server cannot be recovered, use this password to collect logs and service data): Password0

- Return to initial setup.
- Start managing systems.

> The SR635 and NE1072T devices will be auto discovered.

### Licence the XClarity pro appliance

> license file recieved by customer: lnvgy\_fod\_0038\_1212230389\_anyos_noarch.key

[lnvgy\_fod\_0038\_1212230389\_anyos_noarch.key](_resources/475a9baca7df44b996e6ab9e7f950425.key)

From the XClarity web console dashboard select administration -> licenses -> select files -> import and apply.

### add secondary network adapter for BMC/storage communication

Navigate to the XClarity dashboard, select administration -> network access -> ip settings.

- Check two network devices detected, select Eth1 inteface.
- Add the following address information for Eth1.

> IP address: 10.0.1.61
> Network Mask: 255.255.255.0

- Change the NTP configuration to use the OCF-Admin virtual machine 10.0.1.60.
- Save IP settings.

### Create entries for hardware devices

| Site | Device | Model | Interface | Credentials |
| --- | --- | --- | --- | --- |
| Primary | Server | SR635 | TSM IPMI 10.0.1.11 | TSM user: USERID pass: Password0 |
| Primary | Server | SR635 | TSM IPMI 10.0.1.12 | TSM user: USERID pass: Password0 |
| Primary | Server | SR635 | TSM IPMI 10.0.1.13 | TSM user: USERID pass: Password0 |
| DR  | Server | SR635 | TSM IPMI 10.0.1.41 | TSM user: USERID pass: Password0 |
| Primary | Switch | NE1072T | SSH 129.67.94.5 | user: admin pass: Password0 |
| DR  | Switch | NE1072T | SSH 129.67.94.7 | user: admin pass: Password0 |
| Primary | Storage | DE2000H controller A | HTTPS API 10.0.1.1 | user: admin pass: Password0 |
| Primary | Storage | DE2000H controller B | HTTPS API 10.0.1.2 | user: admin pass: Password0 |
| Primary | Storage | DE4000H controller A | HTTPS API 10.0.1.3 | user: admin pass: Password0 |
| Primary | Storage | DE4000H controller B | HTTPS API 10.0.1.4 | user: admin pass: Password0 |
| DR  | Storage | DE4000H controller A | HTTPS API 10.0.1.31 | user: admin pass: Password0 |
| DR  | Storage | DE4000H controller B | HTTPS API 10.0.1.32 | user: admin pass: Password0 |

Naviagate to hardware -> discover and manage new devices -> manual input.

Enter the IP of one of the devices in the table, check ok.

Check use stored credentials, select manage stored credentials.

Create a new stored credential for each of the users admin / USERID in the above table.

![e941fe6d012e1121cb531f022ba1a6fd.png](_resources/5b55e422a591422494797d9979924d77.png)

Complete the entry by checking the manage button.

Enter all Server IP addresses, select the appropriate credential.

![3643c066cae341a9a4ed6699ac024d2e.png](_resources/dc78b5b55b684a2cb04eee39da59c078.png)

![c8b360f9d170632e9a4d8b419bffba7e.png](_resources/b696c83cce8f422e9b3215d0b1632c60.png)

Use the same procedure to add the remaining switches / storage hardware.

Adding hardware devices can often be troublesome, generally regenerate the SSL certificates in the storage and TSM consoles and remove devices from the inventory with the force option, often restarting the appliance will clear devices that 'seem' to be unable to be removed/fail-removed from the inventory.

### Configure Lenovo call home

![c4817bbfafd8ae7386fb962ba161b524.png](_resources/335c1f944d834609b66627e4e6a139eb.png)

### Configure customer alerts

To be configured by the customer.

![f6ac7d0efa9a5126fe72db9d0e6973b0.png](_resources/a7c9a05c8fd944278aaa3dacb49b4205.png)

## Configure event forwarding

Setup email alerts, navigate to monitoring -> event forwarding -> new item.

Input the mail relay, port and recipient.

![ab37d10fa7980b30b6d3943fa789f726.png](_resources/6147ad6b43374c92ac6a17ca4a432a43.png)

Select all devices.

![debb54ef45be8b663a2328f8c8e53bb7.png](_resources/295f358143584654a7464c27d2d6bcd8.png)

Tune the level and class of alerts, there can be a lot of alert noise with informational classes of data.

Warranty events have actually been disabled whilst server host licenses have not been applied in the license portal, this may not be a functional issue according to Lenovo pre-sales.

![23828a727ada07a005c0bd05024a2c80.png](_resources/a8f3b2a8dc9145569a080a703b0cfb9f.png)

Dont select a specific schedule to ensure alerts are always sent to the root@robots mailbox.

![3ef409ca87a84ca9daafb44738c08482.png](_resources/9b3b23ffd4524804923526dff17f55f0.png)

Once the profile is created the new profile will be shown and editable.

![2f79a63539141f3cd0fcc491b2c3ed2f.png](_resources/3106eae6453d47c6ac62c795f1083bb1.png)

## Install XClarity VMware integrator

Navigate to menu -> content libraries -> admin library -> ovf & ova templates, select the XClarity Administrator template, select the context menu and check new VM from this template.

![9de59c1fe1533c0b00f964f45ff92bb4.png](_resources/642165d2dc38450bb242427ef342ba29.png)

Name the virtual machine 'XClarity-Integrator', select the Robots datacenter, select the 'Primary' cluster.

Select the DE4000H\_vmcluster\_admin datastore and select virtual disk format as thin.

Select the DVP\_Management\_Network and set dhcp/default IPv4 address allocation, the appliance does not have vSphere template integration with the IP address attributes and requires configuration from a DHCP service, there is no need to populate any static IP attributes in this step.

### Create a temporary DHCP service on DVP\_Management\_Network

SSH to the OCF-admin host, install some packages and build a tiny DHCP service.

Firewalld or equivalent service should be stopped or a rule to allow DHCP present. `systemctl stop firewalld`

> The docker daemon will lose the IPtables rules with this action, restart the docker service to remedy.

```sh
ssh ocf@129.67.94.25
su -
yum install dnsmasq
systemctl stop dnsmasq
systemctl disable dnsmasq
#start DHCP service
dnsmasq --no-daemon --port=0 --interface=ens192 --conf-file='' --dhcp-range=10.0.1.70,10.0.1.80,255.255.255.0,12h
```

Start the appliance from the vCenter console, watch the DHCP service output to find the appliance IP.

![c4cc583b9763eb038013970907d87472.png](_resources/d2c9dccc20fc4e739a5fe684f62c6762.png)

Open the admin console and follow the setup wizard.

> [https://10.0.1.74/](https://10.0.1.72/)

- Accept the license.
- Set host name and IP adress attributes

> hostname: XClarity-lxci
> Domain name: do not configure a domain, without a FQDN resolvable via the dns servers PTR lookups in the com.lenovo.lxci-7.2.0.0 package install will fail
> DNS: 129.67.1.180,129.67.1.1
> Eth0 IP: 129.67.94.18
> Eth0 Netmask: 255.255.254.0
> Default Gateway: 192.67.95.254
> Eth1 IP: 10.0.1.62
> 
> Eth1 Netmask: 255.255.255.0

- In the vCenter console move the primary network adapter to DVS\_External\_Management_Network (leave the secondary adapter, this is not used).
- You may need to restart the guest OS in vCenter if networking does not automatically reconfigure.
- Open the Xclarity Integrator admin console to continue setup, the 10.0.1/24 range is used for BMC integration, the 129.67.94/23 range is used for vCenter integration.

> Used for initial configuration
> 
> [https://10.0.1.62](https://10.0.1.62 "https://10.0.1.62/")
> 
> Used for subsequent configuration
> 
> https://129.67.94.18

- Setup user account.

> Username: Administrator
> Password: Password0

- Register the integrator appliance with vCenter, select vCenter connection -> register.

> vCenter server: 129.67.94.11 (you may not use the FQDN here unless the integrator appliance IP is also has an FQDN)
> Username: administrator@vcsa.robots.university.ac.uk
> Password: Password0

- Configure date and time preferences.

> Region: Europe
> Time Zone: London
> NTP server: ntp0.robots.university.ac.uk

- Restart the guest OS in vCenter.
- Apply the latest patch.

> https://support.lenovo.com/us/en/solutions/LNVO-VMWARE

![ee239f272013540a25adc7519a43f91d.png](_resources/79dcadd59db6495e823520eef2559696.png)

![01677aa3763da7705b67b02a7535739d.png](_resources/d897fc5946534a689ebe4c91c6dad65d.png)

![37b5b9278b399920b6e67c7e632b3c61.png](_resources/37839c95f83242f2b216c1f37ab7ba3d.png)

- Regenerate the server certificate, security settings -> server certificate.
    
- Enter the primary IP of the appliance that will be used for vCenter registration as the common name and regenerate the server certificate.
    ![1458b4731007a064efbd1b8a658a4e8e.png](_resources/44970045689c44ca9ca1939570c4a1f7.png)
    
- Register the Integrator appliance with the vCenter VCSA server, use the Administrator@vcsa.robots.university.ac.uk root account.
    ![ad8f1bf5d4e436fb31c9565e8f9ce7a2.png](_resources/fc28e367a1624ed58cb7ecdc20fb19e0.png)
    
- Disable Lifecycle manager, this is not compatible with the SR635 server models.
    

![c1557ae37017387006700a890aecc670.png](_resources/cf7aead6369340fca4eda38ec2d2c82d.png)

- Log in to the vCenter server over SSH, check the UI extensions are in place, if the file `.com.lenovo.lxci-7.2.0.0.incomplete` is present you likely have FQDN related issues in the integrator appliance networking.

> SSH user: root
> SSH password: Password0

![63e236e2b648e8e195c457ae751e6c53.png](_resources/88baf5e9bf644c089cccc312171888b9.png)

If there is no menu option for Xclarity Integrator restart the vCenter appliance to ensure the web service picks up the new extensions and reloads the additional web modules on boot.

![651d3b1e0ae6e4c8dc79d21e9293e808.png](_resources/6d1652f7d80e4de6a6e30ae0a895e073.png)

From the vCenter menu open Lenovo XClarity Integrator, select the XClarity integrator service and check edit.

![67639d364fc758967c2a2cf560aea93e.png](_resources/ed31ccda0608432ba3a7c71141606afa.png)

Populate the host username and password.

> Host: 129.67.94.18

> User: Administrator

> Password: Password0

![2c305c55d2b1827111fcccbb8c8ef869.png](_resources/ef90166c5f104a66ab74127ae6b25862.png)

Select refresh, both the Xclarity Integrator and the vCenter servers should now have an online status.

![ef83d24b1bbfd467cdf410b6e02411ba.png](_resources/701a552ac2b24c3d97ae87f6079ce0bb.png)

On the XClarity Administrator appliance ensure the server certificate is regenerated with a common name matching the primary IP address of 129.67.94.17.

![655250d2412b8dea1382ffcb03e09d63.png](_resources/83017c189e784994897e11da975f9242.png)

Connect to the XClarity Integrator to the XClarity Administrator.

This task is performed in vCenter, an embedded frame is opened to an administration console on the XClarity integrator in this step.

![dd98d043bd94aa92d0b0bbe13fa52220.png](_resources/6e31341c98e54211a92a87abeed937ef.png)

Enter the XClarity integrator IP address and root credentials, the registration process will ask to trust the XClarity Administration certificate, if the certificate is empty the XClarity Administrator certificate was not regenerated with its common name as the primary IP (or FQDN).

![8f8ab36d7db2eb7d2e0adf81d3c9977d.png](_resources/65a17ace222747568403d3aaaed7651b.png)

Close the registration console, the vCenter Lenovo XClarity Integrator pane will now show the status of all services.

![4286406231e77f52fd0973ac8107c28d.png](_resources/ca1a0341f1cc4fb890b12e3684870228.png)

The vCenter Xclarity integrator extension should now show 4 managed ESXi servers (there maybe a short wait whilst the integrator appliance relays the information to vCenter), where ESXi servers are not yet shown as managed servers, manually select them in the discover new servers pane, to manually add an ESXi host use the private management IP such as 10.0.1.11.

![ec95bbf57cf09694cff6a4d256b9ce10.png](_resources/4124d6c59b68478e8db0b741f5bcac43.png)

The vCenter XClarity extension will now show relevant information for the chassis.

![3f3b8e348118eff74c4f5d9dd4059326.png](_resources/4bfa2f8956544c26a2742a8f0307bff8.png)

# vCenter alert clearing

## Host attestation alert

The host attestation alarm requires the TPM mode set to 2.0 as per the installation instructions. This alarm can be addresses by enabling secure boot in the host bios, however this can only be done with a local monitor and keyboard not via the TPM KVM. We assume that the host is in a secured datacenter and disable the alarm from within vCenter.

Navigate to hosts and clusters, select the VCSA domain, check configure -> alarm definitions -> host TPM attestation alarm, select disable.

![76886c87236057a71db5be0a9761af1d.png](_resources/2051dda05d8e4e09999fa93deb0d0b3d.png)

## Status of other host hardware objects

Once vmware integration with the Lenovo agents are configured, you may see an cryptic error following multiple firmware updates and OS installs.

![3ce89663c7e8c65bb87487a2287a43de.png](_resources/ec97fcf6bb284aeda0109a33dd7a0969.png)

Navigate to hardware health to get more information.

![53f7f07e634e55fd4d8510b7a760bf7a.png](_resources/e0f14b8e23d64b3abbb5e38f9a227be9.png)

Log into each hosts TPM adapter to clear the event logs, it is likely full.

Watch for non informational warnings such as failed hardware.

![0881255e5e41b269dfc52bf96ba8e6ec.png](_resources/4ccfd1858ecd48da81b39de6d49e394b.png)

# Storage array ASUP alerts

The storage array management interfaces sit in isolated management network, snmp trap and API communication is facilitated between the storage array and the XClarity appliance which also has a listening interface in this network.

Auto support is not enabled by default on the storage arrays whereby the primary vendor Lenovo/Netapp would receive storage array reports, currently critical alerts are picked up through XClarity and emails sent to the designated mailbox.

Even with auto support disabled the auto support report is collected daily, this is useful for OCF engineers to collect and send to Lenovo in the event of hardware issues.

The OCF jump box acts as an email relay for any external email alerts for the storage arrays. The ASUP alert messages are delivered over HTTPS by default, they can be reconfigured for SMTP, this should stop critical alert emails stating there is no path to Netapp.

![8ecdfef3281742d9fde5369f9c979aba.png](_resources/b12b6357673143b1be8445d87db32015.png)

> Set mail relay as 10.0.1.60
> Set sender as root@robots.university.ac.uk, this is the only white listed sender for this relay

Enable Auto support for the array.

Configure the auto support delivery method.

![a3a1b398d188f7038e9ca986caca6e33.png](_resources/f6b2c1bf4618469dbb4a2f77c532f73c.png)

You may now disable auto support.

This configuration can be also changed via the cli via the SMcli tool, enabling auto support and checking status seemed to work correctly however setting the delivery method via the cli failed - I suspect the command reference documentation does not match the syntax expected by the updated version of the tool.

> [https://thinksystem.lenovofiles.com/storage/help/index.jsp?topic=%2Fthinksystem\_storage\_command\_line\_interface_11.60.3%2Foverview.html&lang=en/US](https://thinksystem.lenovofiles.com/storage/help/index.jsp?topic=%2Fthinksystem_storage_command_line_interface_11.60.3%2Foverview.html&lang=en/US)
> Navigate to the Auto support commands section

Download from the SAN Manager web console and unzip.

![b6e47b29ff2cd83a8eb1334ce5662191.png](_resources/1cb9d9d10c3544da9f221c3bcfbf0543.png)

To check the auto support settings (useful when auto support is disabled in the web interface) run the following SMCli command.

```
[root@OCF-admin bin]# ./SMcli 10.0.1.1 -u admin@local -p Password0 -k -c "show storageArray autoSupport;" 
Performing syntax check...

Syntax check complete.

Executing script...

The AutoSupport feature is disabled on this storage array.
The AutoSupport OnDemand feature is disabled on this storage array.
The AutoSupport Remote Diagnostics feature is disabled on this storage array.

Delivery Method: Email (SMTP)
  Destination email address: autosupport@netapp.com
  Mail relay server: 10.0.1.60
  Sender email address: root@robots.university.ac.uk

The AutoSupport daily schedule preferred time range is from 12:00 AM to 11:59 PM.
The AutoSupport weekly schedule preferred time range is from 12:00 AM to 11:59 PM on any day of the week.

AutoSupport Capable  AutoSupport OnDemand Capable  Chassis Serial Number  Daily Schedule  Weekly Schedule      
Yes (disabled)       Yes                           S4BXG548               09:18 AM        03:53 AM on Saturdays

Script execution complete.

SMcli completed successfully.
```