irods_testbench_training_python_bash/documentation/3) Using the native rule engine.md

Using the native rule engine

Using the native rule engine

https://docs.irods.org/4.2.8/plugins/pluggable_rule_engine/ https://docs.irods.org/4.1.5/manual/rule_language/ https://docs.irods.org/4.2.8/doxygen/ # list of microservices https://groups.google.com/g/irod-chat # user group with many examples https://groups.google.com/g/irod-chat/c/ECt5oWSA978/m/hCtoKUjrBQAJ # dynamic PEP - new way of invoking rules (not tried)

List rule engines installed:

tseed@tseed-irods:~$ irule -a
Level 0: Available rule engine plugin instances:
        irods_rule_engine_plugin-python-instance
        irods_rule_engine_plugin-irods_rule_language-instance
        irods_rule_engine_plugin-cpp_default_policy-instance

NOTE: To test these examples it will be neccessary to copy the scripts to a text editor that handles unix end of line characters, then copy/paste to the script file on the linux host.

simple client side rule, run from desktop

NOTE: if you have installed the python rule engine as the first engine in the server.json schema you must select the rule engine to use in irule (not sure if there is a client side way to set the which rule engine to use as default?)

tseed@tseed-irods:~$ cat test.r
main() {
  writeLine("stdout", "Success!");
}
OUTPUT ruleExecOut

tseed@tseed-irods:~$ irule -r irods_rule_engine_plugin-irods_rule_language-instance -F test.r
Success!

sample client side rule to illustrate metadata manipulation

This example shows most of the methods to manipulate metadata in the native rule engine, it is designed to run on the client but most of the inbuilt microservices (functions) will run server side, there are some exceptions listed in the doxygen documentation.

tseed@tseed-irods:~$ cat test-meta.r

main() {
  #### metadata examples ####

  # before version 4.2.7 several microservices(functions) are required achieve metadata manipulation
  # generally data structures of key pairs are created, applied or removed from the data-object/collection/resource
  # from version 4.2.7 the msiModAVUMetadata microservice greatly simplifies metadata transformation
  # for any version - retrieve metadata with sql like queries against the iRODS database, this is equivalent to the 'imeta ls' command

  ### metadata structures

  # sample metadata in the % delimted format for msiString2KeyValPair
  *str = "a=10%zebra=horse%hula=dance"

  # load kvp structure from string
  msiString2KeyValPair(*str, *kvp);

  # add some keys with values
  msiAddKeyVal(*kvp, "art", "crayon");
  msiGetSystemTime(*Time, "human");
  msiAddKeyVal(*kvp, "timestamp", *Time);
  
  # print all keys and values
  msiPrintKeyValPair("stdout", *kvp);
  writeKeyValPairs("stdout", *kvp, " || ")

  # print all keys
  foreach(*kvp) {
    writeLine("stdout", *kvp);
  }

  # print all keys with values
  foreach(*i in *kvp) {
    msiGetValByKey(*kvp,*i,*result);
    writeLine("stdout", "1: *i = *result"); 
  }

  # print key=value where key like zeb*
  # print key=value where value matches *Time (set with msiGetSystemTime above)   
  # print key=value where string value matches 10, convert to int and divide
  foreach(*i in *kvp) {
    msiGetValByKey(*kvp,*i,*result);
    if (*i like "zeb*") then {
      writeLine("stdout","2: *i = *result")
    } else if (*result == *Time) then {
      writeLine("stdout","2: *i = *result")
    } else if (*result == "10") then {
      *number=int("*result")/2
      writeLine("stdout","2: *i = *number")
    } else {
      writeLine("stdout","2: no match")
    }
  }

  # more conditional behaviour
  foreach(*i in *kvp) {
    msiGetValByKey(*kvp,*i,*result);
    #if (*result == "dance" || *i == "art" || *i == "zebra") then { # this is a valid condition, however multiple OR with != / "not like" operators are not pre-evaluated correctly
    if (*i == "a" && *result == str(10)) then { # must preceed the else statement or will not be matched
      writeLine("stdout","3: AND *i = *result")
    } else if (*result not like "dance") then {
      writeLine("stdout","3: *i = *result")
    }
  } 

  ## Add/Remove metadata for files(data objects) or directories(collections) ##
 
  # print all session variables (rei) with msiGetSessionVarValue, when running the rule locally without script parameters the only useful variables are userNameClient= / rodsZoneClient=
  # when run from the server rules engine, there are many more useful session variables
  #msiGetSessionVarValue("all", "client");

  # access the variables as $<variable name>
  *testpath = "/$rodsZoneClient/home/$userNameClient"
  *newfile = "/$rodsZoneClient/home/$userNameClient/test.txt"
  
  # test for valid path, errorcode microservice is used to ensure the script does not exit on failure instead return boolean
  *a = errorcode(msiObjStat(*testpath,*status)) >=0
  writeLine("stdout","4: collection exists: *testpath *a")

  # if path exists add/remove metadata
  if (errorcode(msiObjStat(*testpath,*status)) >=0) then {
    # remove file without sending to trash and unregister from database
    if (errorcode(msiObjStat(*newfile,*status)) >=0) then {
      msiDataObjUnlink("objPath=*newfile++++forceFlag=++++unreg=",*status)
      writeLine("stdout","4: file removed: *newfile")
    }

    # create a file, forceFlag attribute required to overwrite file, resource can be specified here, note the field delimiter "destRescName=demoResc++++forceFlag="
    *content = "test.txt content"
    msiDataObjCreate(*newfile,"forceFlag=",*file_descriptor)
    msiDataObjWrite(*file_descriptor,*content,*write_length)
    msiDataObjClose(*file_descriptor,*status)
    writeLine("stdout","4: file created: *newfile")

    # apply metadata to object from kvp structure
    msiAssociateKeyValuePairsToObj(*kvp,*newfile,"-d")

    # get data object and collection from a full path string
    #*filepath_element = ( size( (split(*newfile,"/")) ) )
    #*file = (elem((split(*newfile,"/")), (*filepath_element - 1) ))
    #*data_object = (elem( (split(*newfile,"/")), ( (size((split(*newfile,"/")))) - 1) ))
    msiSplitPath(*newfile,*collection,*file)
    
    # query iRODS db for metadata of file, load into a new key pair structure
    *query = SELECT META_DATA_ATTR_NAME,META_DATA_ATTR_VALUE WHERE DATA_NAME = '*file' AND COLL_NAME = '*collection'
    foreach(*row in *query) {
      #msiPrintKeyValPair("stdout",*row)
      #writeLine("stdout","next row")
      msiGetValByKey(*row,"META_DATA_ATTR_NAME",*key);
      msiGetValByKey(*row,"META_DATA_ATTR_VALUE",*value);
      msiAddKeyVal(*query_kvp, *key, *value);
    }

    # create a new 'trimmed' metadata structure including the key pairs to be removed
    foreach(*i in *query_kvp) {
      #writeLine("stdout", "key is *i")
      if (*i == "a" || *i == "art") then {
        msiGetValByKey(*query_kvp,*i,*result)
        writeLine("stdout","4: metadata to keep on *newfile, *i=*result")
      } else {
        msiGetValByKey(*query_kvp,*i,*result)
        writeLine("stdout","4: metadata to remove from *newfile, *i=*result")
        msiAddKeyVal(*new_kvp, *i, *result)        
      }
    }
    # remove key pairs listed in the new metadata structure from the data object 
    msiRemoveKeyValuePairsFromObj(*new_kvp,*newfile,"-d")

    # create a new kvp structure, add key pairs
    msiAddKeyVal(*kvp2, "company", "OCF");
    msiAddKeyVal(*kvp2, "department", "Cloud");
    msiGetSystemTime(*created_epoc, "unix")
    msiAddKeyVal(*kvp2, "create_date_epoc", *created_epoc );

    # get system time, load into list and grab elements based on position
    msiGetFormattedSystemTime(*created,"human","%d-%02d-%02d-%02d-%02d-%02d")
    writeLine("stdout", "4: year:" ++ " " ++ (elem((split(*created,"-")),0)) )
    *year = elem((split(*created,"-")),0)
    *month = elem((split(*created,"-")),1)
    *day = elem((split(*created,"-")),2)
    msiAddKeyVal(*kvp2, "create_year", *year );
    msiAddKeyVal(*kvp2, "create_month", *month );
    msiAddKeyVal(*kvp2, "create_day", *day );

    # add meta data to the data object; -d file(data object), -C directory(collection)
    msiAssociateKeyValuePairsToObj(*kvp2,*newfile,"-d");

    # find files with metadata between an epoc date range
    # supported operators
    #>=
    #<=
    #=
    #<
    #>
    #'1' '100'
    #
    # 2020(1575072000) - 2030(1890691200)
    *query = SELECT DATA_NAME WHERE COLL_NAME = '*collection' AND META_DATA_ATTR_NAME = 'create_date_epoc' AND META_DATA_ATTR_VALUE BETWEEN '01575072000' '01890691200'
    foreach(*row in *query) {
      msiGetValByKey(*row,"DATA_NAME",*data_name)
      writeLine("stdout", "4: file: " ++ "*data_name" ++ " created between 2020 - 2030" )
    }

  }

  ### msiModAVUMetadata - change metadata directly on the object/collection/resource ###

  # this is new microservice as of version 4.2.7 and easy to use
  # msiModAVUMetadata allows key, value and unit (AVU) manipulation, much like the imeta icommand
  # assigning an additional attribute 'unit' to the key pair is useful and can be treated as a secondary value or left empty ""
  
  # remove all key pairs directly from the data object
  msiModAVUMetadata("-d","*newfile","rmw", "%", "%", "%")

  # add new key pair directly to the data object
  msiModAVUMetadata("-d","*newfile","add", "car", "ford", "string")

  # change value for key directly on the data object
  msiModAVUMetadata("-d","*newfile","set", "car", "toyoda", "string")

  # remove key pair directly on the data object
  msiModAVUMetadata("-d","*newfile","rm", "car", "toyoda", "string")

  # wildcard remove key pairs directly on the data object
  msiModAVUMetadata("-d","*newfile","add", "car", "subaru", "string")
  msiModAVUMetadata("-d","*newfile","add", "car", "suzuki", "string")
  msiModAVUMetadata("-d","*newfile","add", "car", "saab", "string")
  msiModAVUMetadata("-d","*newfile","rmw", "car", "su%", "%")
  #msiModAVUMetadata("-d","*newfile","rmw", "ca%", "%", "%")

  # add some meta data with arbitrary unit types
  msiModAVUMetadata("-d","*newfile","add", "catC", "yes", "damage")
  msiModAVUMetadata("-d","*newfile","add", "price", "1200", "sterling")

  ## searching with metadata

  # search for files in a collection where the key unit matches damage and the key value matches yes, return the filename key value key name with value yes and unit damage 
  *query = SELECT DATA_NAME,META_DATA_ATTR_NAME WHERE COLL_NAME = '*collection' AND META_DATA_ATTR_UNITS = 'damage' AND META_DATA_ATTR_VALUE like 'y%'
  foreach(*row in *query) {
    msiGetValByKey(*row,"DATA_NAME",*target_file)
    msiGetValByKey(*row,"META_DATA_ATTR_NAME",*damage_type)
    # search for car key value using the file name 
    *sub_query = SELECT META_DATA_ATTR_VALUE WHERE COLL_NAME = '*collection' AND DATA_NAME = '*target_file' AND META_DATA_ATTR_NAME = 'car'
    foreach(*sub_row in *sub_query) {
      msiGetValByKey(*sub_row,"META_DATA_ATTR_VALUE",*car)
      # search for the price key value under threshold (string is dynamically evaluated as numeric)
      *sub_query = SELECT META_DATA_ATTR_VALUE WHERE COLL_NAME = '*collection' AND DATA_NAME = '*target_file' AND META_DATA_ATTR_NAME = 'price' AND META_DATA_ATTR_VALUE < '1201'
      foreach(*sub_row in *sub_query) {
        msiGetValByKey(*sub_row,"META_DATA_ATTR_VALUE",*price)
        #writeLine("stdout", *price)
      }
      # if price variable set, its value below 1201
      if (errorcode(*price) >=0) then {
        writeLine("stdout","5: writeoff: *damage_type *car £*price")
      }
    }
  }

}
INPUT null
OUTPUT ruleExecOut

tseed@tseed-irods:~$ irule -h # display command parameters
tseed@tseed-irods:~$ irule -t -r irods_rule_engine_plugin-irods_rule_language-instance -F test-meta.r
a = 10
zebra = horse
hula = dance
art = crayon
timestamp = 2021-05-24.09:46:23
a || 10
zebra || horse
hula || dance
art || crayon
timestamp || 2021-05-24.09:46:23
a
zebra
hula
art
timestamp
1: a = 10
1: zebra = horse
1: hula = dance
1: art = crayon
1: timestamp = 2021-05-24.09:46:23
2: a = 5
2: zebra = horse
2: no match
2: no match
2: timestamp = 2021-05-24.09:46:23
3: AND a = 10
3: zebra = horse
3: art = crayon
3: timestamp = 2021-05-24.09:46:23
4: collection exists: /OCF/home/rods true
4: file removed: /OCF/home/rods/test.txt
4: file created: /OCF/home/rods/test.txt
4: metadata to keep on /OCF/home/rods/test.txt, a=10
4: metadata to keep on /OCF/home/rods/test.txt, art=crayon
4: metadata to remove from /OCF/home/rods/test.txt, hula=dance
4: metadata to remove from /OCF/home/rods/test.txt, timestamp=2021-05-24.09:46:23
4: metadata to remove from /OCF/home/rods/test.txt, zebra=horse
4: year: 2021
4: file: test.txt created between 2020 - 2030
5: writeoff: catC saab £1200

TODO developing server side rules

Two server side "native language" rules examples in "python server side rule engine", need one more with some replication to s3 resource for reference.

https://groups.google.com/g/irod-chat/c/ABED29dReBs/m/5fWo87WYCAAJ https://groups.google.com/g/irod-chat/c/ObjcBN7W1j0/m/LpQCzp-OAAAJ python rule instead of native - this needs to be put in the server side rul engine example. https://groups.google.com/g/irod-chat/c/evYIHiG0R60/m/QaaluRjpBwAJ print out https://groups.google.com/g/irod-chat/c/gZSB3Pzv8XM/m/t_eXf0LZAAAJ nice keyval stuff

https://slides.com/irods/ugm2019-administration-rule-engine-plugins#/5 might explain how the rule engine has changed with new dynamic PEPs (these will mature over time and hopefully be better documented for the transition)

ok what have we learned? - must have python rulesets ONLY for specific PEPS or rule engine breaks if the python rule engine is enabled any client irule activity must specify which rule engine to use (a bit of a pain - cant find a way to set client side defaults) why do they have it like this? - doesnt matter the python weighted matching engine is easier to configure in many ways - keep it but ensure the policy only overrides single PEPs - what if we want a mix of PEPS triggered from native ruleset vs python? - have to write additional python logic for the PEP to callback the r. rules engine.