# What is this? Gather (non-native supported) PDU metrics via SNMP for Bright Computing, scrape SNMP to present metrics in compatible json format using a Python helper script and a custom SNMP MIB. # Bright reference https://support.brightcomputing.com/manuals/8.0/admin-manual.pdf https://support.brightcomputing.com/manuals/8.1/developer-manual.pdf # Dependencies ## install system dependencies to run easysnmp NOTE: Loughborough Bright master node already includes packages "net-snmp-devel, gcc python-devel" To install easysnmp with python pip some system dependencies are required, easysnmp uses net-snmp for enhanced performance and compatibility. https://easysnmp.readthedocs.io/en/latest/ Install easysnp via pip in the python virtual env ## Install MIB for the ipower PDU used ``` sudo cp PDUSNMP_V1.05.00.mib /usr/share/snmp/mibs/PDUSNMP_V1.05.00.txt sudo chmod 644 /usr/share/snmp/mibs/PDUSNMP_V1.05.00.txt sudo sed -i 's/mibs :/# mibs :/g' /etc/snmp/snmp.conf # the Bright master servers did not need this step, your local host might ``` ## test snmpwalk with the new MIB ``` snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Meters ``` ### OID's to walk ``` snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP #all fields returned, will likely hit a socket timeout before you reach the end of the walk snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Configuration snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Information snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Configuration snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::slave-SiteTotals snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::slave-Server snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Configuration snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Information snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Meters snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-sockets snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Environmental snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Security snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Logs snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::slave-pdus snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::slave-Information snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP | grep -i uptime snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP | grep -i up snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::slave-Status snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP | grep -i days snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP | grep -i d snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-sockets snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP | more ``` Key metrics are in milliamps / millvolts / watts, I have no idea how the PDU's calculate KWh I believe there is a snmp field to set for the cost of a KWh unit, but i didnt see a resultant price field? ``` snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Meters PDUSNMP::pdu-meter1-VRMS.0 = INTEGER: 2371 Volts RMS 237.1 PDUSNMP::pdu-meter1-IRMS.0 = INTEGER: 866 Amps RMS 8.66 PDUSNMP::pdu-meter1-KW.0 = INTEGER: 1964 KWs 1.964 ``` # Local environment setup to work on the script This environment is built to replicate the Loughborough Bright master node. ## setup local python environment to replicate bright master node ``` cd ~/WORK/OCF_GIT/python-snmp mkdir src cd src wget https://www.python.org/ftp/python/2.7.5/Python-2.7.5.tgz tar -xvzf Python-2.7.5.tgz mkdir localpython cd Python-2.7.5 ./configure --prefix ~/WORK/OCF_GIT/python-snmp/src/localpython make make -n install # check youre not about to spam installer files to system directories make install ``` ## create virtual environment with local python and activate ``` cd ~/WORK/OCF_GIT/python-snmp virtualenv venv -p ~/WORK/OCF_GIT/python-snmp/src/localpython/bin/python ``` ## activate the local python virtual environment ``` cd ~/WORK/OCF_GIT/python-snmp source venv/bin/activate python --version which pip pip install easysnmp deactivate # exit venv when you're done ``` # Run the script There are two scripts: * ipower-pdu.py this is the script used, it sets standalone Bright 'entities' to be monitored * ipower-pdu-concept.py this was the first iteration of the script that did not work, the aim was to add metrics to existing pdu entities, Bright did not allow this To run on the command line: Edit the targets dictionary in the script, in the format `'bright entity' : 'target address/hostname'`. There is an ipower pdu in the OCF office to test against `targets = {'ocf-pdu1': '192.168.10.161'}`. ``` # this will output the json document for bright to set monitoring fields per entity (entity, metric, value), used when the collection script is initialized by Bright python ipower-pdy.py --initialize # this will output the stats for each bright entity listed in the targets dict python ipower-pdy.py ``` JSON output looks like: ``` [ { "metric": "VRMS", "value": 460.8, "entity": "10.0.255.104" }, { "metric": "IPK", "value": 5.15, "entity": "10.0.255.104" }, { "metric": "KW", "value": 0.721, "entity": "10.0.255.104" }, { "metric": "IRMS", "value": 3.24, "entity": "10.0.255.104" }, { "metric": "VPK", "value": 326.5, "entity": "10.0.255.104" } ] ``` # Setup Bright to use the script As we are monitoring a pdu device that is already used/uptime-monitored by Bright we have to create some standalone 'entities' to monitor, crucially these must have different names than existing pdus, unfortunatly unlike server entities we cannot collect additional stats for 'powerdistributionunits' type entities thus this work around. https://support.brightcomputing.com/manuals/8.1/developer-manual.pdf * page 17 chapter 2.5 - collection data producers * page 22 chapter 2.11 - Collection Monitoring Data Producers With Filter And Multiplexer * page 22 chapter 2.12 - Collection Monitoring Data Producers For Standalone Entities ``` cp ipower-pdu.py /cm/local/apps/cmd/scripts/powerscripts/ipower-pdu.py chmod 700 /cm/local/apps/cmd/scripts/powerscripts/ipower-pdu.py ``` ## cmsh commands * add monitoring script (ensure this is on both master01 + master02) * set node execution filter (which node the script will run from, the active head node such as master001) * check metrics are being collected * change metric collection retension * add a sample graph in bright view for each standalone entity (ipower-pdu1-4) for the KW metric ``` cmsh # Add standalone targets monitoring standalone add ipower-pdu1 set type ipower-pdu commit exit add ipower-pdu2 set type ipower-pdu commit exit add ipower-pdu3 set type ipower-pdu commit exit add ipower-pdu4 set type ipower-pdu commit exit # Create the collection, assign the monitoring script monitoring setup add collection ipower-pdu set script /cm/local/apps/cmd/scripts/powerscripts/ipower-pdu.py set format JSON set interval 1m commit # Create an execution filter to run the collection only on the active head node (master01) nodeexecutionfilters active commit exit # Check monitoring is being collected monitoring standalone list Name (key) Type ------------------------ ------------------------ ipower-pdu1 ipower-pdu ipower-pdu2 ipower-pdu ipower-pdu3 ipower-pdu ipower-pdu4 ipower-pdu use ipower-pdu1 latestmetricdata Measurable Parameter Type Value Age State Info ------------ ------------ ------------ ---------- ---------- ---------- ---------- IPK ipower-pdu 25.65 A 7.86s filtered IRMS ipower-pdu 17.05 A 7.86s filtered KW ipower-pdu 3.794 KW 7.86s filtered VPK ipower-pdu 648.4 V 7.86s filtered VRMS ipower-pdu 901 V 7.86s filtered # Set retension and use default consolidator policy exit measurable list set VPK consolidator default # we set the consolidator to consolidate RLE data (samples) to be consolidated with the bright default profile at frequencies of hours/days/weeks set VPK maximalsamples 11520 # cannot set to 0 'infinite', this value amounts to a sample every minute for 8 days set VPK maximalage 604800 # this value represents seconds in a week, all RLE data should be discarded after this time and only a subset available in the consolidator data show VPK set IPK consolidator default set IPK maximalsamples 11520 set IPK maximalage 604800 show IPK set KW consolidator default set KW maximalsamples 11520 set KW maximalage 604800 show KW set IRMS consolidator default set IRMS maximalsamples 11520 set IRMS maximalage 604800 show IRMS set VRMS consolidator default set VRMS maximalsamples 11520 set VRMS maximalage 604800 show VRMS commit ```