irods_testbench_training_python_bash/covid_csv_upload/README.md

PHE python script example

See Run phe_upload.py.

• Designed to run on a windows host with Python 3
• TOML format config file initially acts as a rudimentary ruleset for generating lists of files with metadata to upload to iRODS
• Windows filesystem creation date is provided as an optional metadata key value pair
• Add custom key value pairs to the ruleset
• Generated upload_files.toml list is used for subsequent run of the script, this maybe used as a template to target any files with any key value pairs
• Where upload_files.tml exists the config file will only be used iRODS connectivity

Python iRODS client

https://github.com/irods/python-irodsclient

Windows 10 vitual env setup

# setup python venv
cd C:\PHE_iRODS
mkdir venv
python -m venv C:\PHE_iRODS\venv

# enter virtual env
C:\PHE_iRODS\venv\Scripts\Activate.ps1

# install pip packages
pip install --upgrade pip
pip install python-irodsclient
pip install toml

# to save all pip module requirements
pip3 freeze > venv-requirements.txt

# to install requirements on a new venv
pip3 install -r venv-requirements.txt

# exit venv
deactivate

Required PIP packages, should already be installed from venv-requirements

pip install python-irodsclient
pip install toml

Run phe_upload.py

• run once to create sample config file
• modify config for your environment
• run again to produce a sample file list based on your modified config
• modify the file list - this maybe wiped and used as a template to include files from any local/mounted path, each entry must have a unique numerical top level key, this is arbritrary and no sequence is required
• run again to upload files
• a log file is produced for each run of the script, containing skipped files that did not meet validation and failed/sucessful uploads
• to run the script without confirmation, pass the argument --confirm