What is this?

Gather (non-native supported) PDU metrics via SNMP for Bright Computing, scrape SNMP to present metrics in compatible json format using a Python helper script and a custom SNMP MIB.

Bright reference

https://support.brightcomputing.com/manuals/8.0/admin-manual.pdf https://support.brightcomputing.com/manuals/8.1/developer-manual.pdf

Dependencies

install system dependencies to run easysnmp

NOTE: Loughborough Bright master node already includes packages "net-snmp-devel, gcc python-devel"

To install easysnmp with python pip some system dependencies are required, easysnmp uses net-snmp for enhanced performance and compatibility.

https://easysnmp.readthedocs.io/en/latest/

Install easysnp via pip in the python virtual env

Install MIB for the ipower PDU used

sudo cp PDUSNMP_V1.05.00.mib /usr/share/snmp/mibs/PDUSNMP_V1.05.00.txt
sudo chmod 644 /usr/share/snmp/mibs/PDUSNMP_V1.05.00.txt
sudo sed -i 's/mibs :/# mibs :/g' /etc/snmp/snmp.conf # the Bright master servers did not need this step, your local host might

test snmpwalk with the new MIB

snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Meters

OID's to walk

snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP         #all fields returned, will likely hit a socket timeout before you reach the end of the walk
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Configuration
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Information
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Configuration
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::slave-SiteTotals
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::slave-Server
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Configuration
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Information
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Meters
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-sockets
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Environmental
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Security
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Logs
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::slave-pdus
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::slave-Information
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP | grep -i uptime
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP | grep -i up
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::slave-Status
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP | grep -i days
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP | grep -i d
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-sockets
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP | more

Key metrics are in milliamps / millvolts / watts, I have no idea how the PDU's calculate KWh I believe there is a snmp field to set for the cost of a KWh unit, but i didnt see a resultant price field?

snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Meters

PDUSNMP::pdu-meter1-VRMS.0 = INTEGER: 2371		Volts RMS 237.1
PDUSNMP::pdu-meter1-IRMS.0 = INTEGER: 866		Amps RMS 8.66
PDUSNMP::pdu-meter1-KW.0 = INTEGER: 1964		KWs 1.964

Local environment setup to work on the script

This environment is built to replicate the Loughborough Bright master node.

setup local python environment to replicate bright master node

cd ~/WORK/OCF_GIT/python-snmp
mkdir src
cd src
wget https://www.python.org/ftp/python/2.7.5/Python-2.7.5.tgz
tar -xvzf Python-2.7.5.tgz
mkdir localpython
cd Python-2.7.5
./configure --prefix ~/WORK/OCF_GIT/python-snmp/src/localpython
make
make -n install # check youre not about to spam installer files to system directories
make install

create virtual environment with local python and activate

cd ~/WORK/OCF_GIT/python-snmp
virtualenv venv -p ~/WORK/OCF_GIT/python-snmp/src/localpython/bin/python

activate the local python virtual environment

cd ~/WORK/OCF_GIT/python-snmp
source venv/bin/activate
python --version
which pip
pip install easysnmp
deactivate # exit venv when you're done

Run the script

There are two scripts:

• ipower-pdu.py this is the script used, it sets standalone Bright 'entities' to be monitored
• ipower-pdu-concept.py this was the first iteration of the script that did not work, the aim was to add metrics to existing pdu entities, Bright did not allow this

To run on the command line:

Edit the targets dictionary in the script, in the format 'bright entity' : 'target address/hostname'. There is an ipower pdu in the OCF office to test against targets = {'ocf-pdu1': '192.168.10.161'}.

# this will output the json document for bright to set monitoring fields per entity (entity, metric, value), used when the collection script is initialized by Bright
python ipower-pdy.py --initialize

# this will output the stats for each bright entity listed in the targets dict
python ipower-pdy.py

JSON output looks like:

[
    {
        "metric": "VRMS",
        "value": 460.8,
        "entity": "10.0.255.104"
    },
    {
        "metric": "IPK",
        "value": 5.15,
        "entity": "10.0.255.104"
    },
    {
        "metric": "KW",
        "value": 0.721,
        "entity": "10.0.255.104"
    },
    {
        "metric": "IRMS",
        "value": 3.24,
        "entity": "10.0.255.104"
    },
    {
        "metric": "VPK",
        "value": 326.5,
        "entity": "10.0.255.104"
    }
]

Setup Bright to use the script

As we are monitoring a pdu device that is already used/uptime-monitored by Bright we have to create some standalone 'entities' to monitor, crucially these must have different names than existing pdus, unfortunatly unlike server entities we cannot collect additional stats for 'powerdistributionunits' type entities thus this work around.

https://support.brightcomputing.com/manuals/8.1/developer-manual.pdf

• page 17 chapter 2.5 - collection data producers
• page 22 chapter 2.11 - Collection Monitoring Data Producers With Filter And Multiplexer
• page 22 chapter 2.12 - Collection Monitoring Data Producers For Standalone Entities

cp ipower-pdu.py /cm/local/apps/cmd/scripts/powerscripts/ipower-pdu.py
chmod 700 /cm/local/apps/cmd/scripts/powerscripts/ipower-pdu.py

cmsh commands

• add monitoring script (ensure this is on both master01 + master02)
• set node execution filter (which node the script will run from, the active head node such as master001)
• check metrics are being collected
• change metric collection retension
• add a sample graph in bright view for each standalone entity (ipower-pdu1-4) for the KW metric

cmsh

# Add standalone targets
monitoring standalone
add ipower-pdu1
set type ipower-pdu
commit
exit
add ipower-pdu2
set type ipower-pdu
commit
exit
add ipower-pdu3
set type ipower-pdu
commit
exit
add ipower-pdu4
set type ipower-pdu
commit
exit

# Create the collection, assign the monitoring script
monitoring setup
add collection ipower-pdu
set script /cm/local/apps/cmd/scripts/powerscripts/ipower-pdu.py
set format JSON
set interval 1m
commit

# Create an execution filter to run the collection only on the active head node (master01)
nodeexecutionfilters
active
commit
exit

# Check monitoring is being collected
monitoring standalone
list

Name (key)               Type
------------------------ ------------------------
ipower-pdu1              ipower-pdu
ipower-pdu2              ipower-pdu
ipower-pdu3              ipower-pdu
ipower-pdu4              ipower-pdu

use ipower-pdu1
latestmetricdata

Measurable   Parameter    Type         Value      Age        State      Info
------------ ------------ ------------ ---------- ---------- ---------- ----------
IPK                       ipower-pdu   25.65 A    7.86s      filtered
IRMS                      ipower-pdu   17.05 A    7.86s      filtered
KW                        ipower-pdu   3.794 KW   7.86s      filtered
VPK                       ipower-pdu   648.4 V    7.86s      filtered
VRMS                      ipower-pdu   901 V      7.86s      filtered

# Set retension and use default consolidator policy

exit
measurable
list
set VPK consolidator default # we set the consolidator to consolidate RLE data (samples) to be consolidated with the bright default profile at frequencies of hours/days/weeks
set VPK maximalsamples 11520 # cannot set to 0 'infinite', this value amounts to a sample every minute for 8 days
set VPK maximalage 604800 # this value represents seconds in a week, all RLE data should be discarded after this time and only a subset available in the consolidator data
show VPK

set IPK consolidator default
set IPK maximalsamples 11520
set IPK maximalage 604800
show IPK

set KW consolidator default
set KW maximalsamples 11520
set KW maximalage 604800
show KW

set IRMS consolidator default
set IRMS maximalsamples 11520
set IRMS maximalage 604800
show IRMS

set VRMS consolidator default
set VRMS maximalsamples 11520
set VRMS maximalage 604800
show VRMS

commit