initial commit

main
tseed 2022-10-26 17:43:16 +01:00
commit 0937faf73e
5 changed files with 5611 additions and 0 deletions

5078
PDUSNMP_V1.05.00.mib Executable file

File diff suppressed because it is too large Load Diff

281
README.md Executable file
View File

@ -0,0 +1,281 @@
# What is this?
Gather (non-native supported) PDU metrics via SNMP for Bright Computing, scrape SNMP to present metrics in compatible json format using a Python helper script and a custom SNMP MIB.
# Bright reference
https://support.brightcomputing.com/manuals/8.0/admin-manual.pdf
https://support.brightcomputing.com/manuals/8.1/developer-manual.pdf
# Dependencies
## install system dependencies to run easysnmp
NOTE: Loughborough Bright master node already includes packages "net-snmp-devel, gcc python-devel"
To install easysnmp with python pip some system dependencies are required, easysnmp uses net-snmp for enhanced performance and compatibility.
https://easysnmp.readthedocs.io/en/latest/
Install easysnp via pip in the python virtual env
## Install MIB for the ipower PDU used
```
sudo cp PDUSNMP_V1.05.00.mib /usr/share/snmp/mibs/PDUSNMP_V1.05.00.txt
sudo chmod 644 /usr/share/snmp/mibs/PDUSNMP_V1.05.00.txt
sudo sed -i 's/mibs :/# mibs :/g' /etc/snmp/snmp.conf # the Bright master servers did not need this step, your local host might
```
## test snmpwalk with the new MIB
```
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Meters
```
### OID's to walk
```
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP #all fields returned, will likely hit a socket timeout before you reach the end of the walk
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Configuration
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Information
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Configuration
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::slave-SiteTotals
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::slave-Server
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Configuration
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Information
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Meters
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-sockets
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Environmental
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Security
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Logs
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::slave-pdus
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::slave-Information
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP | grep -i uptime
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP | grep -i up
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::slave-Status
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP | grep -i days
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP | grep -i d
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-sockets
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::ipt-pdu-SNMP | more
```
Key metrics are in milliamps / millvolts / watts, I have no idea how the PDU's calculate KWh
I believe there is a snmp field to set for the cost of a KWh unit, but i didnt see a resultant price field?
```
snmpwalk -c public -r 5 -t 10 -v 1 192.168.10.161 PDUSNMP::pdu-Meters
PDUSNMP::pdu-meter1-VRMS.0 = INTEGER: 2371 Volts RMS 237.1
PDUSNMP::pdu-meter1-IRMS.0 = INTEGER: 866 Amps RMS 8.66
PDUSNMP::pdu-meter1-KW.0 = INTEGER: 1964 KWs 1.964
```
# Local environment setup to work on the script
This environment is built to replicate the Loughborough Bright master node.
## setup local python environment to replicate bright master node
```
cd ~/WORK/OCF_GIT/python-snmp
mkdir src
cd src
wget https://www.python.org/ftp/python/2.7.5/Python-2.7.5.tgz
tar -xvzf Python-2.7.5.tgz
mkdir localpython
cd Python-2.7.5
./configure --prefix ~/WORK/OCF_GIT/python-snmp/src/localpython
make
make -n install # check youre not about to spam installer files to system directories
make install
```
## create virtual environment with local python and activate
```
cd ~/WORK/OCF_GIT/python-snmp
virtualenv venv -p ~/WORK/OCF_GIT/python-snmp/src/localpython/bin/python
```
## activate the local python virtual environment
```
cd ~/WORK/OCF_GIT/python-snmp
source venv/bin/activate
python --version
which pip
pip install easysnmp
deactivate # exit venv when you're done
```
# Run the script
There are two scripts:
* ipower-pdu.py this is the script used, it sets standalone Bright 'entities' to be monitored
* ipower-pdu-concept.py this was the first iteration of the script that did not work, the aim was to add metrics to existing pdu entities, Bright did not allow this
To run on the command line:
Edit the targets dictionary in the script, in the format `'bright entity' : 'target address/hostname'`.
There is an ipower pdu in the OCF office to test against `targets = {'ocf-pdu1': '192.168.10.161'}`.
```
# this will output the json document for bright to set monitoring fields per entity (entity, metric, value), used when the collection script is initialized by Bright
python ipower-pdy.py --initialize
# this will output the stats for each bright entity listed in the targets dict
python ipower-pdy.py
```
JSON output looks like:
```
[
{
"metric": "VRMS",
"value": 460.8,
"entity": "10.0.255.104"
},
{
"metric": "IPK",
"value": 5.15,
"entity": "10.0.255.104"
},
{
"metric": "KW",
"value": 0.721,
"entity": "10.0.255.104"
},
{
"metric": "IRMS",
"value": 3.24,
"entity": "10.0.255.104"
},
{
"metric": "VPK",
"value": 326.5,
"entity": "10.0.255.104"
}
]
```
# Setup Bright to use the script
As we are monitoring a pdu device that is already used/uptime-monitored by Bright we have to create some standalone 'entities' to monitor, crucially these must have different names than existing pdus, unfortunatly unlike server entities we cannot collect additional stats for 'powerdistributionunits' type entities thus this work around.
https://support.brightcomputing.com/manuals/8.1/developer-manual.pdf
* page 17 chapter 2.5 - collection data producers
* page 22 chapter 2.11 - Collection Monitoring Data Producers With Filter And Multiplexer
* page 22 chapter 2.12 - Collection Monitoring Data Producers For Standalone Entities
```
cp ipower-pdu.py /cm/local/apps/cmd/scripts/powerscripts/ipower-pdu.py
chmod 700 /cm/local/apps/cmd/scripts/powerscripts/ipower-pdu.py
```
## cmsh commands
* add monitoring script (ensure this is on both master01 + master02)
* set node execution filter (which node the script will run from, the active head node such as master001)
* check metrics are being collected
* change metric collection retension
* add a sample graph in bright view for each standalone entity (ipower-pdu1-4) for the KW metric
```
cmsh
# Add standalone targets
monitoring standalone
add ipower-pdu1
set type ipower-pdu
commit
exit
add ipower-pdu2
set type ipower-pdu
commit
exit
add ipower-pdu3
set type ipower-pdu
commit
exit
add ipower-pdu4
set type ipower-pdu
commit
exit
# Create the collection, assign the monitoring script
monitoring setup
add collection ipower-pdu
set script /cm/local/apps/cmd/scripts/powerscripts/ipower-pdu.py
set format JSON
set interval 1m
commit
# Create an execution filter to run the collection only on the active head node (master01)
nodeexecutionfilters
active
commit
exit
# Check monitoring is being collected
monitoring standalone
list
Name (key) Type
------------------------ ------------------------
ipower-pdu1 ipower-pdu
ipower-pdu2 ipower-pdu
ipower-pdu3 ipower-pdu
ipower-pdu4 ipower-pdu
use ipower-pdu1
latestmetricdata
Measurable Parameter Type Value Age State Info
------------ ------------ ------------ ---------- ---------- ---------- ----------
IPK ipower-pdu 25.65 A 7.86s filtered
IRMS ipower-pdu 17.05 A 7.86s filtered
KW ipower-pdu 3.794 KW 7.86s filtered
VPK ipower-pdu 648.4 V 7.86s filtered
VRMS ipower-pdu 901 V 7.86s filtered
# Set retension and use default consolidator policy
exit
measurable
list
set VPK consolidator default # we set the consolidator to consolidate RLE data (samples) to be consolidated with the bright default profile at frequencies of hours/days/weeks
set VPK maximalsamples 11520 # cannot set to 0 'infinite', this value amounts to a sample every minute for 8 days
set VPK maximalage 604800 # this value represents seconds in a week, all RLE data should be discarded after this time and only a subset available in the consolidator data
show VPK
set IPK consolidator default
set IPK maximalsamples 11520
set IPK maximalage 604800
show IPK
set KW consolidator default
set KW maximalsamples 11520
set KW maximalage 604800
show KW
set IRMS consolidator default
set IRMS maximalsamples 11520
set IRMS maximalage 604800
show IRMS
set VRMS consolidator default
set VRMS maximalsamples 11520
set VRMS maximalage 604800
show VRMS
commit
```

118
ipower-pdu-concept.py Executable file
View File

@ -0,0 +1,118 @@
#!/usr/bin/python
# # cmsh commands to use this script
# #
# # note this did not work, we were trying to add new metrics to the PowerDistributionUnit class of entities
# # Bright doesnt seem to treat PDU's like servers where you can add custom metrics, instead the solution is to standalone entities
# # create the collection, assign the monitoring script
# monitoring setup
# add collection ipower-pdu
# set script /cm/local/apps/cmd/scripts/powerscripts/ipower-pdu.py
# set format JSON
# set interval 1m
# commit
# # create an execution filter to run the collection only on the active head node
# nodeexecutionfilters
# active
# commit
# exit
# # create a multiplexer config to run the script only against 'entities' of type PowerDistributionUnit
# executionmultiplexers
# add type PowerDistributionUnit
# commit
# remove this configuration
# monitoring setup
# use ipower-pdu
# executionmultiplexers
# list
# show PowerDistributionUnit
# remove PowerDistributionUnit
# exit
# nodeexecutionfilters
# list
# show "Active head node"
# remove "Active head node"
# exit
# # we back at 'monitoring setup'
# list
# show ipower-pdu
# remove ipower-pdu
import os
import sys
import json
from easysnmp import Session
def initialize(entity, meter_values):
metric = []
for i in meter_values:
metric_name = i
unit = meter_values[i]['unit']
metric_class = meter_values[i]['class']
entry = {"metric": metric_name, "entity": entity, "unit": unit, "class": metric_class}
dict_copy = entry.copy()
metric.append(dict_copy)
return metric
def sample(entity, meter_values):
# this only returns values which are set, snmpwalk will return zero values or no values at all (if powerbanks or phases arent in use the data is discarded and not returned it seems)
# to force return of empty fields you may need to use get instead of get_bulk
# get_bulk is like snmpwalk, get is equivalent to snmpget
session = Session(hostname=entity, community='public', version=2, remote_port=161)
system_items = session.get_bulk('PDUSNMP::pdu-Meters')
for i in system_items:
for j in meter_values:
if i.oid.endswith(j):
#print (i.oid + "." + i.oid_index + " = " + i.snmp_type + ": " + i.value)
divide_value = int(meter_values[j]['divide'])
meter_values[j]['value'] = float(meter_values[j]['value']) + (float(i.value) / divide_value)
metric = []
for i in meter_values:
metric_name = i
value = meter_values[i]['value']
entry = {"metric": metric_name, "entity": entity, "value": value}
dict_copy = entry.copy()
metric.append(dict_copy)
return metric
def main():
# KWh removed, believe there is a snmp setting to adjust what you pay per hour that must be set for this to be meaningful, hopefully can be calculated in the bright gui?
# meter_values = {'VRMS': {'unit': 'V', 'class': "ipower-pdu", 'value': 0, 'divide': 10}, 'IRMS': {'unit': 'A', 'class': "ipower-pdu", 'value': 0, 'divide': 100}, 'KWH': {'unit': 'KWH', 'class': "ipower-pdu", 'value': 0, 'divide': 1000000}, 'KW': {'unit': 'KW', 'class': "ipower-pdu", 'value': 0, 'divide': 1000}, 'IPK': {'unit': 'A', 'class': "ipower-pdu", 'value': 0, 'divide': 100}, 'VPK': {'unit': 'V', 'class': "ipower-pdu", 'value': 0, 'divide': 10}}
meter_values = {'VRMS': {'unit': 'V', 'class': "ipower-pdu", 'value': 0, 'divide': 10}, 'IRMS': {'unit': 'A', 'class': "ipower-pdu", 'value': 0, 'divide': 100}, 'KW': {'unit': 'KW', 'class': "ipower-pdu", 'value': 0, 'divide': 1000}, 'IPK': {'unit': 'A', 'class': "ipower-pdu", 'value': 0, 'divide': 100}, 'VPK': {'unit': 'V', 'class': "ipower-pdu", 'value': 0, 'divide': 10}}
# shell invoked by bright with one argument --initialize when first classifying the json metric fields
# subsequent invocations usually pass no arguments and the script uses a shell env variable (entity) to determine the script target host
# the bright cli console indicated additional arguments can be passed
# uncomment to test on command line - this relates to the execution multiplexers
os.environ["CMD_HOSTNAME"] = "192.168.10.161"
#os.environ["CMD_HOSTNAME"] = "10.0.255.101"
#os.environ["CMD_HOSTNAME"] = "10.0.255.102"
#os.environ["CMD_HOSTNAME"] = "10.0.255.103"
#os.environ["CMD_HOSTNAME"] = "10.0.255.104"
try:
entity = os.environ["CMD_HOSTNAME"]
except:
sys.stderr.write('Target device not specified in environment\n')
return
if len(sys.argv) > 1 and sys.argv[1] == "--initialize":
data = initialize(entity, meter_values)
else:
data = sample(entity, meter_values)
print json.dumps(data, indent=4)
if __name__ == '__main__':
main()

134
ipower-pdu.py Executable file
View File

@ -0,0 +1,134 @@
#!/usr/bin/python
# Bright would not allow us to add additional metrics to existing pdu entities (as you could with regular servers) so we create new standalone entities that we add via cmsh
# the /etc/hosts has the ip of pdu001 (we could use the device ip), our entity is ipower-pdu01, the targets dict provides the mapping so easysnmp knows what to target
#
# cmsh commands to add this monitoring script
#
# # Add standalone targets
# monitoring standalone
# add ipower-pdu1
# set type ipower-pdu
# commit
# exit
# add ipower-pdu2
# set type ipower-pdu
# commit
# exit
# add ipower-pdu3
# set type ipower-pdu
# commit
# exit
# add ipower-pdu4
# set type ipower-pdu
# commit
# exit
#
# # Create the collection, assign the monitoring script
# monitoring setup
# add collection ipower-pdu
# set script /cm/local/apps/cmd/scripts/powerscripts/ipower-pdu.py
# set format JSON
# set interval 1m
# commit
#
# # Create an execution filter to run the collection only on the active head node
# nodeexecutionfilters
# active
# commit
# exit
#
# # Check monitoring is being collected
# monitoring standalone
# list
#
# Name (key) Type
# ------------------------ ------------------------
# ipower-pdu1 ipower-pdu
# ipower-pdu2 ipower-pdu
# ipower-pdu3 ipower-pdu
# ipower-pdu4 ipower-pdu
#
# use ipower-pdu1
# latestmetricdata
#
# Measurable Parameter Type Value Age State Info
# ------------ ------------ ------------ ---------- ---------- ---------- ----------
# IPK ipower-pdu 25.65 A 7.86s filtered
# IRMS ipower-pdu 17.05 A 7.86s filtered
# KW ipower-pdu 3.794 KW 7.86s filtered
# VPK ipower-pdu 648.4 V 7.86s filtered
# VRMS ipower-pdu 901 V 7.86s filtered
import os
import sys
import json
from easysnmp import Session
def initialize(entity, meter_values):
metric = []
for i in meter_values:
metric_name = i
unit = meter_values[i]['unit']
metric_class = meter_values[i]['class']
entry = {"metric": metric_name, "entity": entity, "unit": unit, "class": metric_class}
dict_copy = entry.copy()
metric.append(dict_copy)
return metric
def sample(entity, hostname, meter_values):
# this only returns values which are set, snmpwalk will return zero values or no values at all (if powerbanks or phases arent in use the data is discarded and not returned it seems)
# to force return of empty fields you may need to use get instead of get_bulk
# get_bulk is like snmpwalk, get is equivalent to snmpget
#
# the endwith filter allows us to itterate over similarly named fields and aggregate them, for example you get multiple current readings for multiple phases that we want to aggregate for the pdu
# however this also happens for voltage so you may get 230vX3 if all phases are in use
session = Session(hostname=hostname, community='public', version=2, remote_port=161)
system_items = session.get_bulk('PDUSNMP::pdu-Meters')
for i in system_items:
for j in meter_values:
if i.oid.endswith(j):
#print (i.oid + "." + i.oid_index + " = " + i.snmp_type + ": " + i.value)
divide_value = int(meter_values[j]['divide'])
meter_values[j]['value'] = float(meter_values[j]['value']) + (float(i.value) / divide_value)
metric = []
for i in meter_values:
metric_name = i
value = meter_values[i]['value']
entry = {"metric": metric_name, "entity": entity, "value": value}
dict_copy = entry.copy()
metric.append(dict_copy)
return metric
def main():
# template for metric ititialization, also used for metric collection
meter_values = {'VRMS': {'unit': 'V', 'class': "ipower-pdu", 'value': 0, 'divide': 10}, 'IRMS': {'unit': 'A', 'class': "ipower-pdu", 'value': 0, 'divide': 100}, 'KW': {'unit': 'KW', 'class': "ipower-pdu", 'value': 0, 'divide': 1000}, 'IPK': {'unit': 'A', 'class': "ipower-pdu", 'value': 0, 'divide': 100}, 'VPK': {'unit': 'V', 'class': "ipower-pdu", 'value': 0, 'divide': 10}}
#targets = {'ocf-pdu1': '192.168.10.161'} # format = 'bright entity' : 'target address/hostname'
targets = {'ipower-pdu1': 'pdu001', 'ipower-pdu2': 'pdu002', 'ipower-pdu3': 'pdu003', 'ipower-pdu4': 'pdu004'}
data = []
if len(sys.argv) > 1 and sys.argv[1] == "--initialize":
for i in targets:
entity = i
entry = initialize(entity, meter_values)
for j in entry:
data.append(j)
else:
for i in targets:
entity = i
hostname = targets[i]
entry = sample(entity, hostname, meter_values)
for j in entry:
data.append(j)
print json.dumps(data, indent=4)
if __name__ == '__main__':
main()