iRODS server setup # iRODS test server This is a sample virtual machine with an iRODS server instance and S3 compatible storage. ## VM test server 4 cpu 4GB RAM 40GB disk hostname irods.local user irods pass password ip 192.168.150.56 (should be a static ip to avoid hostname/domain/ssl type issues, millage may vary, virtualbox/hyperv NAT network maybe help) os Ubuntu 18.04 ## iRODS server installation > https://docs.irods.org/4.2.8/getting_started/installation/ ### Server prep ``` sudo apt-get install wget ca-certificates ``` ### Netplan Static ip required for minio s3 connectivity from irods. ``` sudo nano -cw /etc/netplan/01-netcfg.yaml network: version: 2 renderer: networkd ethernets: eth0: addresses: - 192.168.150.56/24 gateway4: 192.168.150.1 nameservers: search: [local] addresses: [192.168.150.1] sudo netplan apply ``` ### Sudoers Setup /etc/sudoers to include: ``` irods ALL=(ALL) NOPASSWD:ALL ``` ### Ubuntu software repo ``` wget -qO - https://packages.irods.org/irods-signing-key.asc | sudo apt-key add - echo "deb [arch=amd64] https://packages.irods.org/apt/ $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/renci-irods.list sudo apt-get update ``` ### Setup Postgresql database, install iRODS server Install postgress packages. ``` sudo apt-get install postgresql-10 postgresql-client-10 postgresql-client-common postgresql-common postgresql-doc-10 postgresql-contrib ``` Create the iRODS user and database. ``` irods@iRODS:~$ sudo su - postgres postgres@iRODS:~$ psql psql (10.16 (Ubuntu 10.16-0ubuntu0.18.04.1)) Type "help" for help. postgres=# CREATE USER irods WITH PASSWORD 'password'; CREATE ROLE postgres=# CREATE DATABASE "ICAT"; CREATE DATABASE postgres=# GRANT ALL PRIVILEGES ON DATABASE "ICAT" TO irods; GRANT postgres=# \l List of databases Name | Owner | Encoding | Collate | Ctype | Access privileges -----------+----------+----------+-------------+-------------+----------------------- ICAT | postgres | UTF8 | en_GB.UTF-8 | en_GB.UTF-8 | =Tc/postgres + | | | | | postgres=CTc/postgres+ | | | | | irods=CTc/postgres postgres | postgres | UTF8 | en_GB.UTF-8 | en_GB.UTF-8 | template0 | postgres | UTF8 | en_GB.UTF-8 | en_GB.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres template1 | postgres | UTF8 | en_GB.UTF-8 | en_GB.UTF-8 | =c/postgres + | | | | | postgres=CTc/postgres (4 rows) postgres=# \q postgres@iRODS:~$ exit logout ``` Install irods server, irods-dev is required for any msi* microservice (the built-in functions called by rules) ``` irods@iRODS:~$ sudo apt-get install irods-server irods-database-plugin-postgres irods-dev ``` Run the iRODS installer script, this will be a provider setup to ensure all roles and storage are provided locally. Most defaults are fine for this installation, enter the following at the installer prompts. | | | | --- | --- | | ODBC driver | Unicode | | postgres user | irods | | postgres user password | password | | salt | randomsalt | | Zone name | OCF | | iRODS server's zone key | randomzonekey | | iRODS server's negotiation key | Is8710XeBKO26xlhCN5TcPfkE9qHJMDu | | Control Plane key | ohyHLaej2s9CWMTUq5AiQ37DBYdIE6wG | | iRODS server's administrator password | password | ``` irods@iRODS:~$ sudo python /var/lib/irods/scripts/setup_irods.py ``` Some of the script output that maybe useful. ``` iRODS server's role: 1. provider Database Type: postgres ODBC Driver: PostgreSQL Unicode Database Host: localhost Database Port: 5432 Database Name: ICAT Database User: irods Zone name: OCF iRODS server port: 1247 iRODS port range (begin): 20000 iRODS port range (end): 20199 Control plane port: 1248 Schema validation base URI: file:///var/lib/irods/configuration_schemas iRODS server administrator: rods ``` ``` irods@iRODS:~$ ienv irods_version - 4.2.8 irods_client_server_negotiation - request_server_negotiation irods_server_control_plane_key - ohyHLaej2s9CWMTUq5AiQ37DBYdIE6wG irods_server_control_plane_port - 1248 irods_client_server_policy - CS_NEG_REFUSE irods_cwd - /tempZone/home/rods irods_connection_pool_refresh_time_in_seconds - 300 irods_encryption_key_size - 32 irods_default_hash_scheme - SHA256 irods_environment_file - /home/irods/.irods/irods_environment.json irods_default_number_of_transfer_threads - 4 irods_encryption_algorithm - AES-256-CBC irods_encryption_salt_size - 8 schema_version - v3 irods_home - /tempZone/home/rods irods_encryption_num_hash_rounds - 16 irods_default_resource - demoResc irods_match_hash_policy - compatible irods_maximum_size_for_single_buffer_in_megabytes - 32 irods_session_environment_file - /home/irods/.irods/irods_environment.json.1692 irods_port - 1247 irods_server_control_plane_encryption_algorithm - AES-256-CBC schema_name - irods_environment irods_server_control_plane_encryption_num_hash_rounds - 16 irods_user_name - rods irods_zone_name - tempZone irods_transfer_buffer_size_for_parallel_transfer_in_megabytes - 4 irods_host - iRODS ``` Change the iRODS administrator account password, the account 'rods' with password 'rods' with permission 'rodsadmin'. After the password change re-run iinit to login. ``` irods@iRODS:~$ iadmin moduser rods password password irods@iRODS:~$ iinit Enter your current iRODS password: irods@iRODS:~$ ils irods@iRODS:~$ ilsresc ``` ### Edit irods_environment.json Whilst almost all commands will function correctly with the default irods_host target being the localhost name, change this to the FQDN/IP of the host to emulate external administration/configuration to avoid weird gotchas. ``` nano -cw ~/.irods/irods_environment.json irods_host - 192.168.150.56 iinit ``` ### Change the Zone name This doesnt work with the default zone only ones you subsequently create it woudl seem? irods@iRODS:~$ iadmin lz tempZone irods@iRODS:~$ iadmin modzone tempZone name OCF ### Add a new user Add a new user (tseed) with password (password). ``` irods@iRODS:~$ iadmin mkuser tseed rodsuser irods@irods:~$ iadmin lu rods#OCF tseed#OCF irods@iRODS:~$ iadmin help moduser irods@iRODS:~$ iadmin moduser tseed#OCF password password ``` ## Minio Install ### Install docker ``` sudo apt-get remove docker docker-engine docker.io containerd runc sudo apt-get update sudo apt-get install \ apt-transport-https \ ca-certificates \ curl \ gnupg-agent \ software-properties-common curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo apt-key fingerprint 0EBFCD88 sudo add-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable" sudo apt-get update sudo apt-get install docker-ce docker-ce-cli containerd.io sudo systemctl start docker sudo systemctl enable docker sudo usermod -G sudo,docker irods #logout/login to update access to docker socket docker run hello-world ``` ### Install docker compose ``` sudo curl -L "https://github.com/docker/compose/releases/download/1.29.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose sudo chmod +x /usr/local/bin/docker-compose ``` ### Setup Minio S3 service Create directories. ``` mkdir ~/minio sudo mkdir /opt/minio_data ``` Create minio compose file and run. ``` cd ~/minio sudo nano -cw ~/minio/docker-compose.yml version: "3" services: minio: image: minio/minio:latest container_name: irods-minio environment: - MINIO_ACCESS_KEY=minio - MINIO_SECRET_KEY=password command: server /data1 volumes: - /opt/minio_data:/data1 ports: - "9000:9000/tcp" restart: unless-stopped healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] interval: 30s timeout: 20s retries: 3 miniomc: links: - minio:minio-container image: minio/mc:latest container_name: irods-miniomc volumes: - ./createbucket.sh:/tmp/createbucket.sh entrypoint: /bin/sh /tmp/createbucket.sh minio-container irods minio password ``` Create miniomc bucket create script. ``` nano -cw ~/minio/createbucket.sh #!/bin/bash # params target=$1 directory=$2 directory_user=$2 user=$3 password=$4 # vars mc=/usr/bin/mc # now rhel8 container with new package manager, install netcat microdnf install nc # wait for minio container until nc -z $target 9000 do echo "waiting for minio" sleep 1 done sleep 2 # auth $mc config host add minioinstance http://$target:9000 $user $password sleep 2 setup () { # create bucket $mc mb minioinstance/$directory # allow public read $mc policy set download minioinstance/$directory # add user $mc admin user add minioinstance $directory_user $password # define policy cat > /tmp/bucketallaction.json << EOF { "Version": "2012-10-17", "Statement": [ { "Action": ["s3:*"], "Effect": "Allow", "Resource": ["arn:aws:s3:::$directory_user/*"] } ] } EOF # create policy $mc admin policy add minioinstance bucketallaction /tmp/bucketallaction.json # attach policy to user $mc admin policy set minioinstance bucketallaction user=$directory_user } # create bucket if $mc ls minioinstance/$directory 2>/dev/null then echo "minioinstance/"$directory" exists" else setup echo "minioinstance/"$directory" created" fi # destroy bucket #/usr/bin/mc rm -r --force minioinstance/$directory exit 0 ``` Start the minio service, once the compose file is run the minio bucket and user has been created. ``` cd ~/minio docker-compose up docker-compose down ``` ### Create systemd unit files to start minio container on boot. Find the docker run command from a running container started by compose. ``` docker run --rm -v /var/run/docker.sock:/var/run/docker.sock assaflavie/runlike irods-minio ``` Take any parameters that have been specified in the compose file, effectively any parameter that isnt auto populated, this will look like the following: ``` docker stop irods-minio docker rm irods-minio # if the following command runs correctly populate the systemd unit file accordingly. docker run --name=irods-minio --env=MINIO_ACCESS_KEY=minio --env=MINIO_SECRET_KEY=password --volume=/opt/OCF_s3:/data1:rw -p 9000:9000 --restart=unless-stopped --detach=true minio/minio:latest server /data1 ``` NOTE: if the /opt/minio_data directory is wiped the minio config will also be wiped and the compose file will need to be rerun to create the irods bucket and user. To ensure systemd can start the container ensure the --detach parameter is not used, the container restarts must be controlled by systemd and not the docker daemon. ``` sudo nano -cw /etc/systemd/system/minio.service [Unit] Description=minio Wants=network-online.target After=network.target network-online.target [Service] TimeoutStartSec=30 Restart=always ExecStartPre=-/usr/bin/docker kill irods-minio ExecStartPre=-/usr/bin/docker rm irods-minio ExecStartPre=/usr/bin/docker pull docker.io/minio/minio:latest ExecStart=/usr/bin/docker run \ --name=irods-minio \ --env=MINIO_ACCESS_KEY=minio \ --env=MINIO_SECRET_KEY=password \ --volume=/opt/minio_data:/data1:rw \ -p 9000:9000 \ minio/minio:latest server /data1 ExecStop=/usr/bin/docker stop irods-minio [Install] WantedBy=multi-user.target sudo systemctl daemon-reload systemctl list-unit-files | grep -i minio sudo systemctl start minio systemctl status minio sudo systemctl enable minio ``` ## iRODS Resources ### Log location > /var/lib/irods/log/ ### Add local filesystem Resource There needs to be a default resource on the service, it shouldnt be the default demo resource. NOTE: dont set the host as localhost, external users will have an issue resolving a remote host. ``` sudo mkdir /opt/OCFResc sudo chown -R irods.irods /opt/OCFResc iadmin mkresc OCFResc unixfilesystem $(hostname):/opt/OCFResc ilsresc #iadmin rmresc OCFResc nano -cw ~/.irods/irods_environment.json # change "irods_default_resource": "demoResc", to "irods_default_resource": "OCFResc", iinit ``` ### Add new S3 Resource, this is a cached setup where files a local directory is sync'd with s3 > https://docs.irods.org/4.2.0/plugins/composable_resources/ > [https://github.com/irods/irods\_resource\_plugin_s3](https://github.com/irods/irods_resource_plugin_s3) ``` irods@iRODS:~$ sudo apt-get install irods-resource-plugin-s3 ``` This is odd, unsure why this worked and previous didnt, maybe its because of using locally resolved hostname instead of ip, maybe it is due to /irods/Vault being one level deep vs two? S3\_DEFAULT\_HOSTNAME=$(hostname):9000 will work for a cached compound s3 resource but not for a cacheless resource?, S3\_DEFAULT\_HOSTNAME is probably best as the ip of the s3 compatible storage. ``` sudo mkdir /opt/iRODS_s3_cache /opt/iRODS_s3_cred sudo nano -cw /opt/iRODS_s3_cred/minio.keypair irods password iadmin mkresc compResc compound iadmin mkresc cacheResc unixfilesystem $(hostname):/opt/iRODS_s3_cache iadmin mkresc archiveResc s3 $(hostname):/irods/Vault "S3_DEFAULT_HOSTNAME=$(hostname):9000;S3_AUTH_FILE=/opt/iRODS_s3_cred/minio.keypair;S3_RETRY_COUNT=1;S3_WAIT_TIME_SEC=3;S3_PROTO=HTTP" iadmin addchildtoresc compResc cacheResc cache iadmin addchildtoresc compResc archiveResc archive iput -R compResc docker-compose.yml ``` To remove this config: ``` irods@irods:~/minio$ iadmin rmchildfromresc compResc archiveResc irods@irods:~/minio$ iadmin rmresc archiveResc irods@irods:~/minio$ iadmin rmchildfromresc compResc cacheResc irods@irods:~/minio$ iadmin rmresc cacheResc irods@irods:~/minio$ iadmin rmresc compResc irods@irods:~/minio$ ilsresc demoResc:unixfilesystem ``` If there are files on the compound resource children you need to a combination of commands to clear it for deletion: ``` iadmin rmchildfromresc compResc archiveResc iadmin rmresc archiveResc error -835000 CAT_RESOURCE_NOT_EMPTY iadmin rmchildfromresc compResc cacheResc iadmin rmresc cacheResc error -835000 CAT_RESOURCE_NOT_EMPTY # re-add both children back to compound resource and clear out iadmin addchildtoresc compResc cacheResc cache iadmin addchildtoresc compResc archiveResc archive nano -cw ~/.irods/irods_environment.json # change "irods_default_resource": "demoResc", to "irods_default_resource": "compResc", iinit icd ils irm -r -f * ils itrim -vrMN 1 -S cacheResc /OCF irmtrash -f nano -cw ~/.irods/irods_environment.json # change "irods_default_resource": "compResc", to "irods_default_resource": "demoResc", iinit iadmin rmchildfromresc compResc archiveResc iadmin rmresc archiveResc iadmin rmchildfromresc compResc cacheResc iadmin rmresc cacheResc iadmin rmresc compResc ``` ### Add new S3 Resource in cacheless mode, write direct to the bucket (great for local s3 service) Create a path under the bucket to replicate a sub folder. ``` docker run -it --entrypoint=/bin/sh minio/mc mc config host add minioinstance http://192.168.150.56:9000 irods password mc mb minioinstance/irods/Data exit ``` Create a resource, this could be created as a compound resource as in the previous example, or standalone and set a replica ruleset to duplictae files to it or even use the s3 resource as the sole resource. A cache directory is still required however this only holds transient data. An s3 resource cannot be mounted/bound at a certain location/collection in a directory structure. ``` sudo mkdir /opt/iRODS_s3_cred /opt/iRODS_s3_cache sudo nano -cw /opt/iRODS_s3_cred/minio.keypair irods password iadmin mkresc OCFs3Resc s3 $(hostname):/irods/Data "S3_DEFAULT_HOSTNAME=192.168.150.56:9000;S3_AUTH_FILE=/opt/iRODS_s3_cred/minio.keypair;S3_RETRY_COUNT=1;S3_WAIT_TIME_SEC=3;S3_PROTO=HTTP;ARCHIVE_NAMING_POLICY=consistent;HOST_MODE=cacheless_attached;S3_SIGNATURE_VERSION=4;S3_CACHE_DIR=/opt/iRODS_s3_cache" ilsresc -l ilsresc -l OCFs3Resc iput -R OCFs3Resc docker-compose.yml irm docker-compose.yml # remove, might want to remove and recreate before any iput commands to ensure there is no directory structure #irmtrash #iadmin rmresc OCFs3Resc ``` ## Removing resources #### list resources irods@irods:~/minio$ ilsresc compResc:compound ├── archiveResc:s3 └── cacheResc:unixfilesystem demoResc:unixfilesystem irods@irods:~/minio$ ilsresc compResc compResc:compound ├── archiveResc:s3 └── cacheResc:unixfilesystem #### iadmin help iadmin help #### admin list resources irods@irods:~/minio$ iadmin lr bundleResc demoResc compResc cacheResc archiveResc irods@irods:~/minio$ iadmin lr demoResc resc_id: 10014 resc_name: demoResc zone_name: OCF resc\_type\_name: unixfilesystem resc_net: irods.local resc\_def\_path: /var/lib/irods/Vault free_space: free\_space\_ts: Never resc_info: r_comment: resc_status: create_ts: 2021-05-11.13:11:07 modify_ts: 2021-05-11.13:11:07 resc_children: resc_context: resc_parent: resc_objcount: 0 resc\_parent\_context: irods@irods:~/minio$ iadmin lr compResc resc_id: 10019 resc_name: compResc zone_name: OCF resc\_type\_name: compound resc\_net: EMPTY\_RESC_HOST resc\_def\_path: EMPTY\_RESC\_PATH free_space: free\_space\_ts: Never resc_info: r_comment: resc_status: create_ts: 2021-05-11.13:49:51 modify_ts: 2021-05-11.13:49:51 resc_children: resc_context: resc_parent: resc_objcount: 0 resc\_parent\_context: #### admin remove resource (from a compound) Find any child resource. irods@irods:~/minio$ ilsresc compResc compResc:compound ├── archiveResc:s3 └── cacheResc:unixfilesystem Remove child resources. iadmin rmchildfromresc compResc archiveResc iadmin rmchildfromresc compResc cacheResc Remove resources that were used to create the compound resource. iadmin rmresc archiveResc iadmin rmresc cacheResc # this threw an error iadmin rmresc compResc to remove cacheResc, i had to switch the env file to use default resource cacheResc: nano -cw ~/.irods/irods\_environment.json # "irods\_default\_resource": "demoResc", to "irods\_default_resource": "cacheResc", iinit run a selection of commands to check if the resource is empty and clean it up if not, ready for deletion: iquest "SELECT DATA\_NAME WHERE DATA\_RESC_NAME like 'cache%'" iquest "SELECT COLL\_NAME WHERE DATA\_RESC_NAME = 'cacheResc'" irmtrash -f itrim -vrMN 1 -S cacheResc /OCF irm -r -f /OCF After this switched back to "irods\_default\_resource": "demoResc", and removed the resource: iadmin rmresc cacheResc