58 lines
1.9 KiB
Markdown
58 lines
1.9 KiB
Markdown
|
# PHE python script example
|
||
|
|
|
||
|
|
See Run phe_upload.py.
|
||
|
|
|
||
|
|
- Designed to run on a windows host with Python 3
|
||
|
|
- TOML format config file initially acts as a rudimentary ruleset for generating lists of files with metadata to upload to iRODS
|
||
|
|
- Windows filesystem creation date is provided as an optional metadata key value pair
|
||
|
|
- Add custom key value pairs to the ruleset
|
||
|
|
- Generated upload_files.toml list is used for subsequent run of the script, this maybe used as a template to target any files with any key value pairs
|
||
|
|
- Where upload_files.tml exists the config file will only be used iRODS connectivity
|
||
|
|
|
||
|
|
## Python iRODS client
|
||
|
|
|
||
|
|
> https://github.com/irods/python-irodsclient
|
||
|
|
|
||
|
|
## Windows 10 vitual env setup
|
||
|
|
|
||
|
|
```
|
||
|
|
# setup python venv
|
||
|
|
cd C:\PHE_iRODS
|
||
|
|
mkdir venv
|
||
|
|
python -m venv C:\PHE_iRODS\venv
|
||
|
|
|
||
|
|
# enter virtual env
|
||
|
|
C:\PHE_iRODS\venv\Scripts\Activate.ps1
|
||
|
|
|
||
|
|
# install pip packages
|
||
|
|
pip install --upgrade pip
|
||
|
|
pip install python-irodsclient
|
||
|
|
pip install toml
|
||
|
|
|
||
|
|
# to save all pip module requirements
|
||
|
|
pip3 freeze > venv-requirements.txt
|
||
|
|
|
||
|
|
# to install requirements on a new venv
|
||
|
|
pip3 install -r venv-requirements.txt
|
||
|
|
|
||
|
|
# exit venv
|
||
|
|
deactivate
|
||
|
|
```
|
||
|
|
|
||
|
|
## Required PIP packages, should already be installed from venv-requirements
|
||
|
|
|
||
|
|
```
|
||
|
|
pip install python-irodsclient
|
||
|
|
pip install toml
|
||
|
|
```
|
||
|
|
|
||
|
|
## Run phe_upload.py
|
||
|
|
|
||
|
|
* run once to create sample config file
|
||
|
|
* modify config for your environment
|
||
|
|
* run again to produce a sample file list based on your modified config
|
||
|
|
* modify the file list - this maybe wiped and used as a template to include files from any local/mounted path, each entry must have a unique numerical top level key, this is arbritrary and no sequence is required
|
||
|
|
* run again to upload files
|
||
|
|
* a log file is produced for each run of the script, containing skipped files that did not meet validation and failed/sucessful uploads
|
||
|
|
* to run the script without confirmation, pass the argument --confirm
|