SRDR Quarterly Training Brown Evidence-based Practice Center
Brown University
December 5th, 2014 1:00pm-2:00pm
Best Practices: Improving the efficiency and effectiveness, and consistency of data entry
SRDR™ was initially developed by the Tufts EPC and currently maintained by the Brown EPC under contract with AHRQ (Contract No. HHSA 290-2007-10055-I & HHSA 290-2012-1200012-I ).
12/5/2014 http://srdr.ahrq.gov 2
Presentation outline } Summarize common difficulties / challenges encountered by
users entering data into SRDR } Demonstrate best practices to mitigate these challenges
12/5/2014 http://srdr.ahrq.gov 3
} Inefficient design of extraction forms } Extraction forms with lots of empty space } Unintuitive question titles on extraction forms } Unrestrained extraction forms giving too much latitude to extractors
} Entering data prospectively into SRDR } Pitfalls and limitations for copying and pasting data into extraction forms } Losing track of data extractor progress, and questions they might have
} Entering data retrospectively into SRDR } Many different file formats (MS Word, MS Excel, PDFs) } Preparing results data for importing (coming feature)
Challenges entering data into SRDR
12/5/2014 http://srdr.ahrq.gov 4
} Exporting data from SRDR in a format usable to others (statisticians, analysts) or in a format ready to be used as evidence tables in appendix of an EPC report
} To be addressed in a future Webinar in January } new features added to SRDR since the last training Webinar held in
September
Challenges exporting data from SRDR
12/5/2014 http://srdr.ahrq.gov 5
} Compilations of strategies used by SRDR Users, EPCs, and the SRDR Team
} Recommendations that will enable you to overcome the challenges associated with entering data into SRDR
} Benefits of using best practices include: } Reduce time and effort required to enter data into SRDR } Improve accuracy and consistency of data collected } Increase accessibility of data for other SRDR contributors and public
users } Take advantage of SRDR Tools and features
Best Practices
12/5/2014 http://srdr.ahrq.gov 6
Best Practices for Designing Efficient Extraction
Forms
12/5/2014 http://srdr.ahrq.gov 7
} Background } Key Questions (KQ) entered into an SRDR project should parallel the
KQs addressed in your systematic review or EPC Report. } Each KQ can only be associated with a single (primary) extraction form } Several KQs may use the same extraction form
} Challenge } Since a KQ can only be associated with one (primary) extraction form,
collection of certain data across all eligible studies for the KQ (i.e. study quality information for multiple study designs) can result in extraction forms with long lists of mutually exclusive questions (i.e questions relevant to only one type of study design) that could be unwieldy to navigate
Create accessory forms to complement primary extraction form
12/5/2014 http://srdr.ahrq.gov 8
} Best Practice } Create “accessory” extraction forms to complement the data collected
on a primary extraction form (each accessory extraction form will require adding a new KQ to your project)
} Note: You will need to describe in the “Project notes” field of the Project Information page of the SRDR project, which KQs were added to the project to create accessory extraction forms and the KQs from the report they address.
} Benefit } Creating “accessory” extraction forms will shorten the number of data
fields that an extractor need enter data into for any one extraction form.
Create accessory forms to complement primary extraction form (Cont.)
12/5/2014 http://srdr.ahrq.gov 9
Example: Denoting addition of accessory extraction forms in “Project Notes” field.
12/5/2014 http://srdr.ahrq.gov 10
} Background } A question title (data header) is required for each question added to the
extraction form } Displayed as column (spreadsheet) headers in simple and advanced
export of data from SRDR
} Challenge } It is difficult to understand arbitrarily named or unintuitive question
titles (e.g., BMI_X instead of Average BMI). This difficulty adds time required by the data extractor to interpret the meaning of the question and presents confusion to other users.
Use self-explanatory question titles on extraction forms
12/5/2014 http://srdr.ahrq.gov 11
} Best Practice } Question titles should be self-explanatory and include: units
and/or specific data ranges / formats (i.e. only whole number values; or DD/MM/YYYY date format)
} Benefits } Minimize data extraction errors } Improve accessibility of data by users
Use self-explanatory question titles (Cont.)
12/5/2014 http://srdr.ahrq.gov 12
Example: Intuitive and unintuitive question titles
Self-explanatory:
Self-explanatory:
Not self-explanatory:
12/5/2014 http://srdr.ahrq.gov 13
} Background } Users can add a list of “suggested” arm and outcome titles to the Arm
and Outcome tabs of SRDR extraction forms. } During individual data abstraction, data extractors are given the option
to use these pre-specified arm and outcome titles or create their own.
} Challenge } When extractors have the latitude to name the arms and outcomes on
their own, they may come up with different names (e.g., Cardiovascular disease could be named as: CVD, CV disease, etc.. ). Variations in the naming of arms and outcomes is unwanted and require time to standardize. It will also confuse users of the data unfamiliar with the project.
Pre-specify arm and outcome titles for data extractors to use on the extraction form
12/5/2014 http://srdr.ahrq.gov 14
} Best Practice } Step 1: The project lead should pre-specify a list of the arm and outcome
titles and then add the titles from this list for other extractors to use as “suggested” arms and outcomes on the Arms and Outcomes tabs of the SRDR extraction form.
} Step 2: Train data extractors in the team to use suggested arms and outcomes, unless it is absolutely necessary to create a new name (and then the new name should be added to the suggested list).
} Benefits } Reduce data entry time } Reduce spelling errors, case errors and unwanted redundancy } Reduce data cleaning needs
Pre-specify arm and outcome titles for data extractors to use on the extraction form
12/5/2014 http://srdr.ahrq.gov 15
Example of the unwanted variation that can result in Outcome titles
12/5/2014 http://srdr.ahrq.gov 16
} Background } Users may choose from the following selection of question types to add
to an SRDR extraction form: } Text } Dropdown menu } Multiple choice (select one) aka. Radio button question } Multiple choice (select any) aka. Checkbox question } Matrix of choices (only one answer allowed) aka. Matrix of Radio buttons } Matrix of choices (select more than one answer) aka. Matrix of Checkboxes } Matrix of dropdowns (specify option in each cell)
} Challenge } SRDR extraction forms relying primarily on text, dropdown menu, and
multiple choice questions could result in very long forms. Long forms are a challenge to use because of excessive vertical scrolling which increases the time needed by data extractors to enter data.
Shortening extraction forms using dropdown and matrix-type questions
12/5/2014 http://srdr.ahrq.gov 17
} Best Practice } When possible, consolidate multiple questions into a single matrix
question. (e.g. when the same set of descriptive statistics and other information is asked for multiple risk factors on the Baseline tab of SRDR extraction forms, the individual questions can be consolidated into separate cells within a single matrix question).
} Benefits } Any number of columns or rows of cells may be added to matrix
questions on SRDR extraction forms enabling users to minimize the length of the form.
} Values entered into each cell of a matrix question are exported top separate columns of simple export spreadsheets allowing for easy analysis/isolation of data.
Using Dropdown and Matrix-type questions (Cont.)
12/5/2014 http://srdr.ahrq.gov 18
Example contrasting vertical space used by matrix and non-matrix questions Multiple Text Qs Single Matrix Q Multiple Matrix Qs
12/5/2014 http://srdr.ahrq.gov 19
} Background } Arm and outcome descriptions can be added to arms and outcomes
entered into the Arm and Outcomes tabs of SRDR extraction forms.
} Challenge } When exporting data from SRDR, Arm Title and Arm Description (same
for Outcome) are concatenated into a single character string (e.g. the title,” Vitamin D”, with description, “consumed”, will appear on export as Arm: “Vitamin D [consumed]”) to distinguish arms that may have the same title but different descriptions. This concatenated string may require additional processing steps before analysis.
Save time cleaning data by creating arm and outcome description questions on your forms
12/5/2014 http://srdr.ahrq.gov 20
} Best Practice } Instead of entering information into arm description and outcome
description fields on Arm and Outcome tabs, create a question on Arm Details, Outcome Details tabs to capture this information.
} Note: To use this best practice, Arm Details and Outcome Details tabs on extraction forms should be set to display questions for each arm or outcome, respectively.
} Benefit } Less time needed to clean arm and outcome titles exported from
SRDR
Avoid using arm, outcome description fields on Arm, Outcome tab (Cont.)
12/5/2014 http://srdr.ahrq.gov 21
Example: Adding arm description value to arm title on Arm tab
Arms tab of extraction
form
Simple export of Arms tab
SRDR Simple Export tool used to export data entered
into extraction form
12/5/2014 http://srdr.ahrq.gov 22
Example: Adding arm description value to arm title on Arm tab (Cont.)
Export of extraction form using Report Builder tool
12/5/2014 http://srdr.ahrq.gov 23
Example: Adding arm description value to question on Arm Details tab
Arm Details tab of
extraction form
Simple export of Arm
Details tab
SRDR Simple Export tool used to export data entered
into extraction form
12/5/2014 http://srdr.ahrq.gov 24
Example: Adding arm description value to question on Arm Details tab (Cont.)
Export of extraction form using Report Builder tool
12/5/2014 http://srdr.ahrq.gov 25
} Background } Questions asked on an extraction form (excluding questions concerning
study design or study quality), are typically asked for a specific arm or outcome defined in a study.
} This is typically accomplished by creating questions for specific arms or outcomes on the form, or instead creating blank data fields into which miscellaneous arms or outcomes may be entered (the same applies when creating extraction forms in SRDR).
} Challenge } An extraction form (e.g., paper, software) seeking to predefine all
possible arms/outcomes could be lengthy and unwieldy. } To accommodate unexpected arms/outcomes during data extraction, the
extraction form will need to be revised (this problem also applies to SRDR when features are not used).
Use SRDR’s feature to dynamically create arm/outcomes questions in tabs
12/5/2014 http://srdr.ahrq.gov 26
} Best Practice } Rather than enumerating each question explicitly on an SRDR extraction
form, use the “dynamic” arm/outcome question feature to ask a question on extraction form tabs ( Arm Details, Baseline, Outcome Details, Results, Adverse Events) for just the arms and outcome that are defined in a particular study.
} Pro-tip: After constructing an extraction form, always check to see how data fields are presented in simple and advanced data export. Also, pilot the extraction form.
} Benefits } Minimizes the number of times questions need to be asked on a tab,
resulting in a smaller extraction form and quicker navigation } Extraction form will readily accommodate unforeseen arm/outcomes
keeping the number of revisions to the extraction form at a minimum
Use SRDR’s feature to dynamically create arm/outcomes questions in tabs (Cont.)
12/5/2014 http://srdr.ahrq.gov 27
Example: Extraction form that collects data for a predefined number of arms
12/5/2014 http://srdr.ahrq.gov 28
Example: Replicating the extraction form in SRDR and exporting data
Simple export of Arm Details tab of SRDR extraction form
12/5/2014 http://srdr.ahrq.gov 29
Simple export of Arm Details tab of SRDR extraction form
Example: Incorporating feature to dynamically create arm questions in form and exporting data
1. 2.
3.
12/5/2014 http://srdr.ahrq.gov 30
Best Practices for Entering Data Prospectively into SRDR
12/5/2014 http://srdr.ahrq.gov 31
} Definition of prospective data entry } Copying data from the full text of studies included in a systematic review,
directly into data fields on SRDR extraction forms.
} Method } Manual entry: Requires typing or copying and pasting data values,
individually, into data fields on the extraction form.
Background
12/5/2014 http://srdr.ahrq.gov 32
} Background } Symbols, such as those listed below are commonly found in the full text
of studies and can be typed or copied and pasted into extraction forms in SRDR } Mathematical operations (÷, ×, ±, °, >, < , ≥, ≤, ≠, =, ≅,≈,…) } Greek Letters (α, β, γ, ε, ζ, η,θ, λ, μ, ξ, ω, …) } Other Symbols ( &, ®, ™, $, €, –, —, ^, ©, …)
} Challenge } Among the symbols that can be entered into extraction forms, only a
slim minority can be exported from SRDR. Among the symbols listed above as examples, only >, <, &, $, and ^ can be exported to a simple export spreadsheet. The remainder are substituted with a “blank” space on the simple export spreadsheet.
} Note: All symbols imported into SRDR using the import tool may be viewed upon export.
Use work-arounds when entering symbols unsupported by SRDR into extraction form data fields
12/5/2014 http://srdr.ahrq.gov 33
} Best Practice } Data extractors should be trained to use the following commonly
accepted work-arounds for symbols that can otherwise not be exported from SRDR } ≤, ≥, and ≠ should be written as “<=“, “>=“, “/=“ } “cm3” can instead be written as “cm^3”
} Otherwise, extractors should refrain from entering symbols into SRDR or when necessary, spell symbols out. (i.e. “α”entered as “alpha”, or “And” instead of “&”)
} Benefits } Use of work-arounds or spelling out symbols means that data from a
project can be exported by other users and the absence of specific symbols will not throw off the meaning of data values.
Use work-arounds when entering symbols unsupported by SRDR into extraction form data fields (Cont.)
12/5/2014 http://srdr.ahrq.gov 34
Example: How string of symbols is not exported completely from SRDR
Response as it appears on Simple Export spreadsheet
Response as it appears on
SRDR extraction form
SRDR Simple Export tool
used to export data entered into extraction form
12/5/2014 http://srdr.ahrq.gov 35
} Background } Superscript and subscript text are often used in full text studies for
footnotes (xi), mathematical operations (cm3) and citations (Smith, 201423).
} Challenge } When superscript or subscript text is copied and pasted into data fields
on an extraction form in SRDR, the super/subscripted text reverts to a regular sized font which can change the entire meaning of a particular response thereby reducing the accuracy of data collected (e.g. “Unadjusted RR = 1.34” appears as “Unadjusted RR = 1.34” on extraction form) or otherwise make the data entirely incomprehensible to other users (e.g. “xi” appears as “xi” on extraction form)
Copying and pasting super/subscripted text into extraction form data fields
12/5/2014 http://srdr.ahrq.gov 36
} Best Practice } Data extractors should either avoid copying and pasting super-
subscripted text into data fields on SRDR extraction forms or find an alternative way to enter these values (i.e. enter cm3 as cm^3) such that their meaning is not lost to other users.
} Benefits } More accurate data collection } Less confusion for other users
Copying and pasting super/subscripted text into extraction form data fields
12/5/2014 http://srdr.ahrq.gov 37
} Background } When a question on an SRDR extraction form is left with a blank
response it is recognized by SRDR as a null-value.
} Challenge } When reviewing the export of data from a project in SRDR, it is
therefore impossible to know whether the response to an extraction form question was 1) left blank because the full-text of the study simply did not contain the data requested by the question OR 2) left blank because the data extractor forgot to answer the question.
Use of “No Data” (ND) response to answer questions on form when no data is available
12/5/2014 http://srdr.ahrq.gov 38
} Best Practice } Data extractors should enter “ND” into all extraction form data fields
for which no data is given in the article to respond.
} Benefit } Project leads are able to more readily identify data fields that were
missed or forgotten by extractors and have them corrected, thus improving accuracy of data collected.
Use of “ND” for questions on form for which no data is available (Cont.)
12/5/2014 http://srdr.ahrq.gov 39
} Background } At the bottom of each tab on an SRDR extraction form is the button
“Mark Section Complete” which data extractors may use to keep track of the sections they have completed on the extraction form.
} Clicking the button twice will change the status of the tab from “Complete” to “Incomplete”
} The completion status of all tabs in the extraction form may be viewed as well as “toggled” by the extractor on the Finalize tab of the extraction form.
} Challenge } Keeping tabs on the progress of data extractors during data abstraction
can be a difficult and time-consuming process
Use of “Mark Section Complete” button during individual data extraction
12/5/2014 http://srdr.ahrq.gov 40
} Best Practice } Each time a data extractor completes entering data into data fields on an
extraction form in SRDR, they should mark the tab “Complete” using the Mark Section Complete button and remember to review the completion status of each tab from the Finalize tab of the extraction form before exiting.
} Benefits } Enables data extractors to remember whether there is a section they
have yet to complete, thereby improving accuracy of data collected. } Pro-tip: Allows the Project Lead to use the Progress Report tool
located under project Tools to view a report of each extractor’s overall progress.
Use of “Mark Section Complete” button during individual data extraction (Cont.)
12/5/2014 http://srdr.ahrq.gov 41
Example Report generated by the SRDR Progress Report Tool
12/5/2014 http://srdr.ahrq.gov 42
} Background } Notes can be left by data extractors on the Finalize tab of the extraction
form for a particular study, in the “Note” data field. } Notes may be viewed by project leads from the Complete Study List of
the project.
} Challenge } It is important that project leads are able to address questions
extractors may have while extracting data from studies.
} Best Practice } Data extractors should be trained to leave notes on the extraction
form for project lead, when needed.
} Benefit } Faster data extraction
Use of “Note” data field on extraction forms during individual data extraction
12/5/2014 http://srdr.ahrq.gov 43
Example of Notes left by extractor for a study on study list
12/5/2014 http://srdr.ahrq.gov 44
Best Practices for Entering Data Retrospectively into SRDR
12/5/2014 http://srdr.ahrq.gov 45
} Definition of retrospective data entry } Transferring data that has already been collected from the full text of multiple
studies and saved in an external file (MS Word, MS Excel, PDF file…) into data fields on SRDR extraction forms OR otherwise uploading the external files to a project for easy downloading by other SRDR contributors and public users
} Method(s) } Manual entry: Requires typing or copying and pasting data values,
individually, into data fields on the extraction form } Importing: Requires, first, formatting and saving the data as a Microsoft
Excel spreadsheet. Data from the spreadsheet is then read and transferred, in batch, by the SRDR Import tool into data fields on the extraction form
} Uploading: Requires using the SRDR Manage Reports tool to “attach” files (PDF, text , MS Word, MS Excel …) such as EPC report appendices to SRDR projects. These files will then be available to download by other users
Background
12/5/2014 http://srdr.ahrq.gov 46
1. Log into SRDR with your SRDR username and password.
2. Check that you are in the MySRDR tab.
3. Click the edit link under the project to which you would like to upload the PDF files.
4. In the Navigation menu, on the left side of your project, click the Data Export Tool button located under Project Tools.
5. Next, on the Export Data Tools page, click the Manage Reports link, located in the box labelled Project Reports to access the Manage Reports Tool.
6. On the following page, click the Add Report button.
7. Under Upload Report, in the Description field, enter a brief description of the PDF file you want to upload to your project.
8. Next to the Description field, click the Choose File button to select the PDF file from your computer.
9. Select the File.
10. Below the Description field, click the Upload button, to upload the file to your project.
11. Repeat steps 1 through 10, to add additional PDF files to your project,
12. At the top of the page, below the MySRDR tab, click the Edit Project tab to return to your project information page.
Steps for Uploading a file to SRDR
12/5/2014 http://srdr.ahrq.gov 47
} Manual Entry } Pro: Data can be entered into every tab of SRDR extraction form } Con: Data must be entered into tabs on extraction form individually
(time intensive process)
} Importing } Pro: Batch Process, can enter data into tab for any number of studies at
once } Con: Data on Excel spreadsheets has to be formatted to meet
requirements for importing (see prev webinar for requirements); Data cannot yet be imported into tabs supporting “dynamic” arm/outcome question feature, including the Results tab (feature coming soon)
} Uploading } Pro: Files are uploaded quickly; Files are viewable by other SRDR users } Con: Data values are not incorporated into the SRDR database
Methods for Entering data saved on a Microsoft Excel file into SRDR
12/5/2014 http://srdr.ahrq.gov 48
} Manual Entry } Same Pros/Cons listed for entering data saved on MS Excel files
} Importing } Pro: Batch Process, can enter data into tab for any number of studies at
once. } Con: Additional step required to transfer data to MS Excel files; After
which, same cons listed for importing data from MS Excel files apply. } Pro-tip: Watch for the inclusion of returns when copying and pasting data from
MS Word into excel, or else each new line in the copied text will be pasted into an individual cell.
} Uploading } Same Pros/Cons listed for entering data saved on MS Excel files
Entering data saved on a Microsoft Word file into SRDR
12/5/2014 http://srdr.ahrq.gov 49
} Manual Entry } Same Pros/Cons listed for entering data saved on MS Excel files
} Importing } Pro: Batch Process, can enter data into tab for any number of studies at once. } Con: Additional step required to transfer data from to MS Excel files (best
accomplished using PDF to Excel conversion software, i.e. Able2Extract Pro distributed by Investintech Inc.); After which, same cons listed for importing data from MS Excel files apply. } Pro-tips:
¨ Can print MS Word files to PDFs and then use PDF conversion software to convert PDFs to MS Excel files.
¨ Using Conversion software some symbols on text of PDF file may not convert correctly to the MS Excel file (i.e. ≥, ≤ symbols in particular)
} Uploading } Same Pros/Cons listed for entering data saved on MS Excel files
Entering data saved on a PDF file into SRDR
12/5/2014 http://srdr.ahrq.gov 50
} Manual Entry } Same Pros/Cons listed for entering data saved on MS Excel files
} Uploading } Same Pros/Cons listed for entering data saved on MS Excel files
} Importing – Not Yet Available } Requirements:
} Data must be saved in a Microsoft Excel spreadsheet } Data arranged in a vertical fashion with:
¨ Multiple rows per study record ¨ Column headers for Arm, Outcome, Time point, Time point units, Subgroup, and the
desired outcome measures
Methods for entering results data into SRDR
12/5/2014 http://srdr.ahrq.gov 51
Example: Importing into the Results tab of an SRDR Extraction form
PMID Title Author Year Arm Outcome Time Point Time Point Unit Mean SD
23812661
Compara?ve efficacy of LEAP, TEACCH and non-‐model-‐specific special
educa?on programs for preschoolers with au?sm spectrum disorders
Boyd 2013
LEAP IQ Baseline N/A 5 3
23812661
Compara?ve efficacy of LEAP, TEACCH and non-‐model-‐specific special
educa?on programs for preschoolers with au?sm spectrum disorders
Boyd 2013
LEAP IQ 1 years 10 4
23812661
Compara?ve efficacy of LEAP, TEACCH and non-‐model-‐specific special
educa?on programs for preschoolers with au?sm spectrum disorders
Boyd 2013
LEAP AUen?on Baseline N/A 6 2
23812661
Compara?ve efficacy of LEAP, TEACCH and non-‐model-‐specific special
educa?on programs for preschoolers with au?sm spectrum disorders
Boyd 2013
LEAP AUen?on 1 years 9 3
Template of Microsoft Excel spreadsheet to be imported:
12/5/2014 http://srdr.ahrq.gov 52
Example: Importing into the Results tab of an SRDR Extraction form (Cont.)
Result of Import as seen in the Results tab of the SRDR extraction from:
12/5/2014 http://srdr.ahrq.gov 53
Reminder These slides may be downloaded from the Quarterly Training tab
of the SRDR Help & Training page (http://srdr.ahrq.gov/help)
12/5/2014 http://srdr.ahrq.gov 54
Questions?
Top Related