Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research...
Transcript of Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research...
![Page 1: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/1.jpg)
06/09/2020
Lazlo [email protected]
Yoda and the iRODS Python rule engine plugin
Chris [email protected]
![Page 2: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/2.jpg)
Collaborate safely as a group
Maintain integrity, deposit a folder in the vault
Allow FAIR reuse, publish a data package
Research
Vault
Yoda: 'FAIR' Research Data Management
![Page 3: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/3.jpg)
2015
•Yoda Portal and Intake module for Youth project, iRODS 3.3 based•Groupmanager module
2016
•Yoda Disk (DavRods module)•Yoda Portal supports dynamic Plug-in modules, iRODS 4.0
2017
•Research Workspace, revisions, metadata form•Vault archive, deposit workflow, statistics, data publication workflow (DOI)
2018
•Vault metadata operations, EPIC PID, External user provisioning•(de/re)publication workflows, OAI-PMH harvestable, iRODS 4.1
2019
•Metadata-schema management, dynamic metadata forms rendering•Metadata form based on JSON schema, file up/download in Portal, iRODS 4.2.6
2020
•Metadata format changed from XML to JSON, with JSON-AVU•Python rules engine, iRODS 4.2.8
Yoda v0.4 – v0.9
Yoda v0.9.7
Yoda v1.0 – v1.3
Yoda v1.4
Yoda v1.5
Yoda v1.6
Yoda milestones
![Page 4: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/4.jpg)
iRODS implementation for RDM
Share Deposit Publish
GroupMgmnt
Davro
ds
Sud
o
Revis
ion
DO
I
EPIC
ExtU
ser
Json2Avu
Gro
ups
Asy
ncR
epl
Sw
ord
Sto
reTi
er
Preserve(EASY)
ReuseCreate
Account
OA
I-PM
H
sch
em
as
Noti
fy
RD
M F
un
ctio
nR
ule
s &
Serv
ices
https://github.com/UtrechtUniversity/yoda
![Page 5: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/5.jpg)
2015 2016 2017 2018 2019 2020Q10
500
1000
1500
2000
2500
3000
3500
4000
4500
Internal users External users 2015 2016 2017 2018 2019 2020Q10
40
80
120
160
200
240
280
320
360
400
440
480
520
560
600
640
680
Users Storage (TB)
Yoda in numbers
![Page 6: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/6.jpg)
Why switch to Python?
● Maintainability● Performance● Readability● Learning curve● Libraries and frameworks● Available tooling● Web development
![Page 7: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/7.jpg)
Tidying up our rules with Python
● iRODS rule language lines of code decreased with a third
● PHP lines of code cut in half
● More maintainable code● Improved readability• Easier and faster development• Better performance● Fewer lines of code
![Page 8: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/8.jpg)
![Page 9: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/9.jpg)
Modular approach
● Where do we start?● core.py (vs. core.re)
● Our RE ruleset was a set of concatenated rule files● How would a Python programmer approach this?
core.py
from rules_uu import *# ...
![Page 10: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/10.jpg)
Modular approach
● Where do we start?● core.py (vs. core.re)
● Our RE ruleset was a set of concatenated rule files● How would a Python programmer approach this?
● Modules and packages● core.py imports other packages as rulesets● Allows namespacing and private helper functions
/etc/irods│ (...)├── core.py imports * from rules_uu└── rules_uu/ imports * from modules ├── __init__.py ├── datacite.py exports via '__all__' │ (...)
![Page 11: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/11.jpg)
Converting a ruleto Python
● ‘rule_args’ calling convention● Boilerplate● Non-pythonic● Difficult to interface from Python functions
● Can we make this easier?
# iRODS rule language.concat(*x, *y, *foo) { *foo = *x ++ *y;}
# Equivalent Python rule.def concat(rule_args, callback, rei): x, y = rule_args[0:2] rule_args[2] = x + y
# Can we not simplify it like this?def concat(callback, x, y): return x + y
![Page 12: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/12.jpg)
Converting a ruleto Python
● ‘rule_args’ calling convention● Boilerplate● Non-pythonic● Difficult to interface from Python functions
● Can we make this easier?● Delegate argument and return value handling to a
decorator● Support any mix of in, out, in/out args● Support writing returned value to stdout
# Old Python rule.def concat(rule_args, callback, rei): x, y = rule_args[0:2] rule_args[2] = x + y
# … simplified:from rules_uu.util import rule
@rule.make(inputs=[0,1], outputs=[2])def concat(callback, x, y): return x + y
![Page 13: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/13.jpg)
Converting a ruleto Python
● ‘rule_args’ calling convention● Boilerplate● Non-pythonic● Difficult to interface from Python functions
● Can we make this easier?● Delegate argument and return value handling to a
decorator● Support any mix of in, out, in/out args● Support writing returned value to stdout
# Arguments are in/out by default:@rule.make()def uppercase2(callback, x, y): return x.upper(), y.upper()
![Page 14: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/14.jpg)
Interfacing with rules
● How can we call rules from our web portal?● With structured inputs and outputs?
● RE approach: JSON microservices● Creating structured data is cumbersome● Manually handle JSON in/out for each rule
● In Python, can we do better?
foo(*x, *result) { *x.y = "abc"; *x.z = "123"; uuKvpList2JSON(*x, *result);}
![Page 15: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/15.jpg)
Interfacing with rules
● How can we call rules from our web portal?● With structured inputs and outputs?
● RE approach: JSON microservices● Creating structured data is cumbersome● Manually handle JSON in/out for each rule
● In Python, can we do better?● @api decorator
● JSON input → Python args● Python return value → JSON output
● Checks required/optional function args● Supports dicts, lists...● Standardizes error handling● Abstract away!
from rules_uu.util import api
@api.make()def api_uu_concat(callback, foo, bar): return foo + bar
// Callable from frontend JavaScript:let str = await
Yoda.call('uu_concat', {'foo': 'test', 'bar': '123'});
![Page 16: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/16.jpg)
Additional challenges
● Genquery support limited● Improved and merged!
● Microservice error handling (no errorcode)● Wrapped microservices, custom exceptions
● Python2 EOL● Work around until it’s upgraded
![Page 17: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/17.jpg)
Future work
● Packaging (pip install irods_ruleset_uu)● Python3 support?
● Removes unicode cruft● Type hints & type checking● Modern libraries
![Page 18: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/18.jpg)
Side note
● Davrods, our WebDAV → iRODS bridge● 1.5 released for iRODS 4.2.8● Ticket support● Apache conditional configuration support
● All mentioned code available at● https://github.com/UtrechtUniversity/irods-ruleset-uu/● https://github.com/UtrechtUniversity/davrods/
![Page 19: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/19.jpg)
$ iexit
![Page 20: Yoda and the iRODS · •Yoda Portal supports dynamic Plug-in modules, iRODS 4.0 2017 •Research Workspace, revisions, metadata form •Vault archive, deposit workflow, statistics,](https://reader033.fdocuments.us/reader033/viewer/2022042709/5f5462194b820812574733ac/html5/thumbnails/20.jpg)
© Utrecht University
The information in this presentation has been compiled with the utmost care, but no rights can be derived from its contents.
DISCLAIMER