# What? Script to query IPSEC VPN tunnels on device type IP-VPNHUB / IP-P2PAGG. - Finds operational VPN type devices tickets in NMS MongoDB. - Finds FQDN and IP. - Gets OS over SSH, finds Scrapli platform type. - Gets phase1/phase2 IPSEC tunnel details, correlates entries to complete tunnels in hub-spoke format. - Writes tunnel records to DB, if a record pre-exists the date stamp will be updated, new records will be appended. - Writes from DB to Excel spreadsheet and generates stats summary. Script shortcomings: - Cycles only one set of device SSH credentials, does not yet use team-pass as a 'secrets' vault. - Cisco devices audited, no other vendor scrapes written, spoke investigation may require Sarian/Digi/VA scraping (very manual output parsing - VA maybe ok as they are Openwrt). - Append/Update only operation for tunnel capture, timestamps used to indicate record relevance, tunnels only captured at time of scrape (the more the scrape is run the more active tunnels will be recorded). Learned/Investigated: - Cisco commands to capture phase1/2 tunnel information and how to correlate multiple command outputs to a single tunnel record. - Netmiko/textfsm+n2c-templates, getting screen scrapes into ordered data from legacy devices without API. - Scrapli/genie(OR Scrapli/textfsm+n2c-templates), getting screen scrapes into ordered data from legacy devices without API. - Scraping logic, most libraries do not have inbuilt text processor templates for the commands in use. - MongoDB/Pandas/Xlsxwriter. # Status - POC - WIP - Code Reference # Run Script ## Python venv Create venv example, add dependencies. ```sh $HOME/WORK/python/3.10.6/bin/python3 -m venv 3.10.6_vpn $HOME/WORK/python/vpn_venv source $HOME/WORK/python/vpn_venv/bin/activate python --version which python pip install --upgrade pip python -m pip install -r pip_requirements.txt deactivate # enter/exit source $HOME/WORK/python/vpn_venv/bin/activate deactivate ``` Pip modules installed: - pip install python-dotenv - pip install scrapli - pip install scrapli[genie] - pip install pandas - pip install xlsxwriter - pip install fabric invoke==2.0.0 - pip install pymongo==3.13.0 ## Tacacs credential Create credentials file `.env`, populate: ```sh SSH_USER= SSH_PASSWORD= ``` ## Local ssh config Netmiko/Scrapli offer ways to select various encryption settings, for ease the following routines can be set in home ssh config or system wide. ```sh # append to the bottom of the file cat /etc/ssh/ssh_config Ciphers 3des-cbc,aes128-cbc,aes192-cbc,aes256-cbc,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com MACs hmac-sha1,hmac-sha1-96,hmac-sha2-256,hmac-sha2-512,hmac-md5,hmac-md5-96,umac-64@openssh.com,umac-128@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-md5-etm@openssh.com,hmac-md5-96-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com HostKeyAlgorithms ssh-ed25519,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256 KexAlgorithms diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group-exchange-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,curve25519-sha256,curve25519-sha256@libssh.org,sntrup761x25519-sha512@openssh.com ``` ## Input list Devices are listed in ipdesk. - http://ipdesk.corp.tnsi.com/newipdesk/dmvpnhubscan.php - http://ipdesk.corp.tnsi.com/newipdesk/report_p2p_aggs.php The script pulls operational vpn devices from the NMS mongoDB with the following query. ```py query = { "raw.DeviceType": {"$in": ["IP-VPNHUB", "IP-VPNAGG", "IP-P2PAGG", "IP-VCSR-HUB"]}, "raw.Environment_Usage": {"$nin": ["QA", "UAT"]}, "raw.DeviceStatus": {"$nin": ["Order Cancelled", "De-Installed", "Installed", "New", "Hold", "Pend De-Install"]}, "raw.DeviceName": {"$nin": [re.compile('.*_old$'), re.compile('.*_ol$')]} } ``` The following devices were used to catch command parsing issues, there are a few 'easy/normal' parse devices in the list. There are devices with slightly non-standard output, devices that aren't in dns, devices where tunnels have same source/dest but different ports (RRI), devices with SSH issues (large scrape over slow links) and devices with no tunnels or incomplete tunnels. ```sh asb-cofs-dmvpn-hub04 mep-shared-rri-agg09 syd-rri-agg02 asb-test-p2p-agg01 lon-dmvpn-hub05 vxmy-dmvpn-hub01 lon-bml-rri-agg03 sdy-rri-agg04 wcd-bml-rri-agg03 ``` ## Operating modes ```sh (3.10.6_vpn_venv) [tseed@asblpnxpdev01 vpn_discovery]$ ./main.py -h usage: main.py [-h] {inventory,audit,report,list} ... Collect VPN tunnel info positional arguments: {inventory,audit,report,list} inventory Query NMS MongoDB to generate VPN device table audit Collect tunnel info for target devices report Generate VPN XLSX report list Return all target devices in VPN device table options: -h, --help show this help message and exit ``` - inventory - reads the NMS MongoDB and find all VPN devices, then run commands against each device to determine connectivity and version for the scrapli_platform type - list - returns the names of all devices in the device table - audit - runs tunnel audit against devices, accepts arguments -a for all devices and -d for a selection of comma delimited device short names (as seen with the list argument) - report - builds XLSX report of captured tunnels for all devices, requires argument -e for email The script captures tunnels seen during the scrape, this will include records with UP-ACTIVE / UP-IDLE / DOWN-NEGOTIATING session states. Owing to phase1/2 timeouts you will not capture every tunnel, often the tunnel will not have all the fields to qualify as a full record at the time of scrape (p1 only / p2 only), rerun the audit task against a device to capture more tunnels and also update tunnel records that have already been captured. The script can take a while to complete all scrapes (up to 1 hour), some devices fail (usually on connection quality) and the script tries to handle this gracefully, for speed - you may want to only scrape a new device or re-scrape an interesting device. ## Run Start venv: `source $HOME/WORK/python/vpn_venv/bin/activate` Run script: `python main.py -h` or `./main.py -h` # Outstanding - implement a 'current_config' key for tunnels indicating changed configuration (useful for 3des remediation activity). - remedy/dns lookup of spokes - use `peer_id` field for lookup and possibly some `nhrp` commands. - thread pool queue count to log, 5/10 devices remaining - email wrapper # Content ```sh . ├── main.py ├── pip_requirements.txt ├── README.md ├── vpn_audit │ ├── config.py │ ├── __init__.py │ ├── vpn_cisco.py │ ├── vpn_inventory.py │ ├── vpn_mongo.py │ ├── vpn_scrapli.py │ └── vpn_spreadsheet.py └── vpn_discovery.code-workspace ```