Slide 1

USING DATA MANAGEMENT PLANS as a RESEARCH TOOL for IMPROVING DATA SERVICES in ACADEMIC LIBRARIES Jake Carlson, Patricia Hswe & Susan Wells Parham Amanda Whitmire, Lizzy Rolando & Brian Westra IASSIST 2015 Minneapolis, MN 2-6 June 2015

Slide 2

Amanda Whitmire | @AWhitTwit Jake Carlson | @jrcarlso Patricia M. Hswe | @pmhswe Susan Wells Parham | Lizzy Rolando | Brian Westra | @bdwestra D A R T Team 5 June 20152 DART Project | @DMPResearch http://bit.ly/dmpresearch

Slide 3

Acknowledgements Amanda Whitmire | Oregon State University Libraries Jake Carlson | University of Michigan Library Patricia M. Hswe | Pennsylvania State University Libraries Susan Wells Parham | Georgia Institute of Technology Library Lizzy Rolando | Georgia Institute of Technology Library Brian Westra | University of Oregon Libraries This project was made possible in part by the Institute of Museum and Library Services grant number LG-07-13-0328. D A R T Team 5 June 20153

Slide 4

4

Slide 5

5

Slide 6

6

Slide 7

7

Slide 8

transition slide 5 June 20158

Slide 9

Levels of data services the basics DMP reviewworkshopswebsite mid-level dedicated research services metadata support facilitate deposit in DRs consults high level infrastructuredata curation From: Reznik-Zellen, Rebecca C.; Adamick, Jessica; and McGinty, Stephen. (2012). "Tiers of Research Data Support Services." Journal of eScience Librarianship 1(1): Article 5. http://dx.doi.org/10.7191/jeslib.2012.1002 5 June 20159

Slide 10

Informed data services development 5 June 201510 Survey

Slide 11

Informed data services development 5 June 201511 SurveyDCPs

Slide 12

Informed data services development 5 June 201512 SurveyDCPsDMPs DMP

Slide 13

5 June 201513 DART Premise DMP Research Data Management needs practices capabilities knowledge researcher

Slide 14

5 June 201514

Slide 15

5 June 201515 DART Premise Research Data Management needs practices capabilities knowledge Research Data Services

Slide 16

5 June 201516 DART Premise

Slide 17

5 June 201517 We need a tool

Slide 18

5 June 201518 Solution: an analytic rubric Performance Levels Performance Criteria HighMediumLow Thing 1 Thing 2 Thing 3

Slide 19

NSF Directorate or Division BIOBiological SciencesENGEngineering DBIBiological InfrastructureCBET Chemical, Bioengineering, Environmental, & Transport Systems DEBEnvironmental BiologyCMMICivil, Mechanical & Manufacturing Innovation EFEmerging Frontiers OfficeECCSElectrical, Communications & Cyber Systems IOSIntegrative Organismal SystemsEECEngineering Education & Centers MCBMolecular & Cellular BiosciencesEFRIEmerging Frontiers in Research & Innovation CISEComputer & Information Science & Engineering IIPIndustrial Innovation & Partnerships ACIAdvanced Cyberinfrastructure CCFComputing & Communication Foundations GEOGeosciences CNSComputer & Network SystemsAGSAtmospheric & Geospace Sciences IISInformation & Intelligent SystemsEAREarth Sciences EHREducation & Human Resources OCEOcean Sciences DGEDivision of Graduate EducationPLRPolar Programs DRL Research on Learning in Formal & Informal Settings MPSMathematical & Physical Sciences DUEUndergraduate EducationASTAstronomical Sciences HRDHuman Resources DevelopmentCHEChemistry DMRMaterials Research DMSMathematical Sciences PHYPhysics 5 June 201519 division-specific guidance

Slide 20

5 June 201520 SourceGuidance text NSF guidelines The standards to be used for data and metadata format and content (where existing standards are absent or deemed inadequate, this should be documented along with any proposed solutions or remedies) BIO Describe the data that will be collected, and the data and metadata formats and standards used. CSE The DMP should cover the following, as appropriate for the project:...other types of information that would be maintained and shared regarding data, e.g. the means by which it was generated, detailed analytical and procedural information required to reproduce experimental results, and other metadata ENG Data formats and dissemination. The DMP should describe the specific data formats, media, and dissemination approaches that will be used to make data available to others, including any metadata GEO AGS Data Format: Describe the format in which the data or products are stored (e.g. hardcopy logs and/or instrument outputs, ASCII, XML files, HDF5, CDF, etc).

Slide 21

5 June 201521 Advisory Board Project team testing & revisions Feedback & iteration Rubric

Slide 22

5 June 201522 Performance Level Performance CriteriaComplete / detailed Addressed issue, but incomplete Did not address issue Directorates General Assessment Criteria Describes what types of data will be captured, created or collected Clearly defines data type(s). E.g. text, spreadsheets, images, 3D models, software, audio files, video files, reports, surveys, patient records, samples, final or intermediate numerical results from theoretical calculations, etc. Also defines data as: observational, experimental, simulation, model output or assimilation Some details about data types are included, but DMP is missing details or wouldnt be well understood by someone outside of the project No details included, fails to adequately describe data types. All Directorate- or division- specific assessment criteria Describes how data will be collected, captured, or created (whether new observations, results from models, reuse of other data, etc.) Clearly defines how data will be captured or created, including methods, instruments, software, or infrastructure where relevant. Missing some details regarding how some of the data will be produced, makes assumptions about reviewer knowledge of methods or practices. Does not clearly address how data will be captured or created. GEO AGS, GEO EAR SGP, MPS AST Identifies how much data (volume) will be produced Amount of expected data (MB, GB, TB, etc.) is clearly specified. Amount of expected data (GB, TB, etc.) is vaguely specified. Amount of expected data (GB, TB, etc.) is NOT specified. GEO EAR SGP, GEO AGS

Slide 23

5 June 201523 Mini-reviews 1 & 2 23

Slide 24

5 June 201524

Slide 25

Inter-rater reliability 5 June 201525

Slide 26

5 June 201526 Wherein I try not to put you to sleep. Inter-rater reliability

Slide 27

A primer on scoring 5 June 201527 X = T + E Very helpful excerpts from: Hallgren, Kevin A. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutorials in Quantitative Methods for Psychology 8, no. 1 (2012): 2334. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3402032/.

Slide 28

A primer on scoring 5 June 201528 X = T + E Observed Score True Score Measurement Error

Slide 29

A primer on scoring 5 June 201529 X = T + E If there were no error noise Observed Score True Score Measurement Error

Slide 30

A primer on scoring 5 June 201530 X = T + E Could be issues of: internal consistency test-retest reliability inter-rater reliability Observed Score True Score Measurement Error

Slide 31

A primer on scoring 5 June 201531 Var(X) = Var(T) + Var(E) Variance in Observed Scores Variance in True Scores Variance in Errors

Slide 32

Inter-rater reliability 5 June 201532 IRR analysis aims to determine how much of the variance in the observed scores is due to variance in the true scores after the variance due to measurement error between coders has been removed. Hallgren, Kevin A. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutorials in Quantitative Methods for Psychology 8, no. 1 (2012): 2334. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3402032/.

Slide 33

Inter-rater reliability 5 June 201533 IRR analysis aims to determine how much of the variance in the observed scores is due to variance in the true scores after the variance due to measurement error between coders has been removed. Hallgren, Kevin A. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutorials in Quantitative Methods for Psychology 8, no. 1 (2012): 2334. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3402032/. If IRR = 0.80: 80% of Var(X) is due to Var(T) 20% of Var(X) is due to Var(E) Var(X) = Var(T) + Var(E)

Slide 34

Measures of IRR 5 June 201534 1.Percentage agreement | not for ordinal data; overestimates agreement 2.Cronbachs alpha | works for 2 raters only 3.Cohens kappa | used for nominal data; works for 2 raters only 4.Fleisss kappa | for nominal variables 5.Intra-class correlation (ICC) | perfect!

Slide 35

5 June 201535 Intra-class correlation (ICC) Variance due to rated subjects (DMPs) ICC = (Variance due to DMPs + Variance due to raters + Residual Variance) 6 variations of ICC must choose carefully based on study design Shrout PE, Fleiss JL. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin. 1979; 86(2):420428. McGraw KO, Wong SP. Forming inferences about some intraclass correlation coefficients. Psychological Methods. 1996; 1(1):3046.

Slide 36

Intra-class correlation (ICC) 5 June 201536 ICC_results

Slide 37

ICC: consistency vs. agreement 5 June 201537 Rater 2 = 1.5 x Rater 1Rater 2 always rates 4 points higher than Rater 1 Rater 2 = Rater 1

Slide 38

Intra-class correlation (ICC) 5 June 201538 ICC_results

Slide 39

Inter-rater reliability 5 June 201539 Mean = 0.731 | Median = 0.759 Standard Deviation = 0.146 Mean = 0.487 | Median = 0.464 Standard Deviation = 0.112 51631

Slide 40

Inter-rater reliability 5 June 201540 Mean = 0.731 | Median = 0.759 Standard Deviation = 0.146 Mean = 0.487 | Median = 0.464 Standard Deviation = 0.112 5 5 16 712 31

Slide 41

Inter-rater reliability 5 June 201541 Mean = 0.731 | Median = 0.759 Standard Deviation = 0.146 Mean = 0.487 | Median = 0.464 Standard Deviation = 0.112

Slide 42

5 June 201542 poor fair good excellent