Using Commcare Data in MVP Detailed Reference. OUTLINE BACKGROUND The Importance of Data (Slide 3) ...
-
Upload
stanley-gordon -
Category
Documents
-
view
221 -
download
6
Transcript of Using Commcare Data in MVP Detailed Reference. OUTLINE BACKGROUND The Importance of Data (Slide 3) ...
Using Commcare Data in MVPDetailed Reference
OUTLINEBACKGROUND The Importance of Data (Slide 3) The CommCare Software (Slide 4) Summary of Data Flow in the CommCare Platform (Slide 5) Key CommCareHQ Terminology (Slide 6)
ACCESSING COMMCARE DATA Permissions for Data Access (Slide 7) How to Manage User Access (Slide 8)
TYPES OF COMMCARE REPORTS Methods to Use CommCare Data (9) Examples of Custom Reports (Slides 11-15) Built-In Worker Monitoring Reports (Slide 16)
Worker Activity (Slides 17-18) Daily Form Activity (Slide 19) Submissions by Form (Slide 20) Form Completion (Slide 21) Case Activity (Slide 22) Form Completion vs. Submission Trends (Slide 23) Worker Activity Times (Slide 24)
Setting Up Email Delivery of CommCareHQ Reports (Slide 25) Additional Monitoring Tools (Slides 26-27)
RAW DATA Why Export Raw Data? (Slide 28) Exporting Raw Data: Cases (Slide 29) Exporting Raw Data: Forms (Slides 30-31) Why Set Up a Custom Export (Slide 32) How to Set Up a Custom Export (Slides 33-35) Where to Find Your Custom Export (Slide 36) Excel Tips for Calculating Indicators (Slide 37) Calculating Indicators from Raw Data (Slides 38-40)
CASE STUDY Using Maternal & Child Health Application to Identify Problem &
Inform Response (Slides 41-46) Using Morbidity Data to Identify Problem & Inform Response
(Slides 47-49)
SUMMARY (Slide 50)
BACKGROUND
The Importance of Data
Data collected through mobile health technologies such as CommCare can help improve health service delivery by: Providing real-time, immediate feedback on:
The performance of community health workers (CHWs) The health status of the community
Helping to identify challenges in service delivery and inform solutions Providing evidence-based support for operational decisions and health
sector programming
BACKGROUND
The CommCare Software Millennium Villages Project (MVP) uses CommCare as its primary mobile health
platform for monitoring its CHW program CommCare is a mobile phone application designed to serve as a decision guide
during household visits, a counseling aid, and a data collection tool CommCare’s web-based interface allows for real-time monitoring of CHWs’ activities
and community health status by health managers CommCare includes:
Easy to follow questionnaires with point-of-care decision making algorithms to ensure quality visits and reduce data entry errors
Longitudinal monitoring of thousands of children, mothers, and households over long periods of time SMS-based visit reminders for CHWs Mapping of cases via GPS Multimedia for strengthened counseling (video, audio, image) Capacity to produce real-time reports on a number of worker activity and health status reports
BACKGROUND
Summary of Data Flow in CommCare Platform
Cloud server
De-identified static reports
Web user
Web user
Mobile user
BACKGROUNDKey CommCareHQ TerminologyTerminology Description
Application The complete CommCare executable that gets downloaded and installed on a phone. This is the application that will be used by the mobile user to submit data to CommCareHQ. Example MVP applications include: Maternal and Child Health (MCH), Verbal Autopsy (VA), CHW Supervision
FormThe questionnaire that is filled out by the CHW. A form often corresponds to an action. In the MVP MCH application, there are 4 main form types – register a new case, visit a case, close a case, and edit the registration information of a case. In the MVP VA application, there are 3 main forms - Verbal Autopsy Neonatal, Verbal Autopsy Child, and Verbal Autopsy Adult
Case Anything that is being tracked by the CommCare system over time, often an individual or a household
Submission A form that has been sent to the CommCare server, which is dependent on internet access
Completion Time Time at which a form is actually completed, as indicated when reaching the end of the form questionnaire and pressing “save and exit”
Received Time Time at which a completed form has been received by CommCareHQ, which is dependent on internet access
Mobile User Someone who completes the forms in CommCare on a mobile device - typically a CHW, VA Specialist, or CHW Supervisor
Web User Someone who uses CommCareHQ to manage the project or view reports – typically a CHW Manager, Health Manager or eHealth Manager
Group A collection of users that can be manually classified into “groups” by geographic area (i.e. parish)
ACCESSING COMMCARE DATA
Permissions for Data Access Ask yourself: Should this user have access to
patient-level health data? If not, grant access only to aggregate-level or de-identified
reports Access to the various reports can and should be
customized based on whether this user should have access to patient-level health data
Appropriate permissions can be set when creating new web user accounts on CommCareHQ
ACCESSING COMMCARE DATA
How to Manage User Access Select “Users” in the CommCare toolbar
and “Web Users & Roles”
To assign permissions to a user: Add a user by clicking the “+Invite New
user” button Enter the e-mail address of the user you
want to invite Assign the permissions/level of access for
the user using the “Role” dropdown
To edit permissions of a role: Scroll down on the “Web Users & Roles”
page to the “Roles” section Click on “Edit Role.” You can rename the
project role if you wish Check off the permissions you want to
give to the role. Click “Save”https://confluence.dimagi.com/display/commcarepublic/CommCareHQ+Web+Users
TYPES OF COMMCARE REPORTS
Methods to Use CommCare Data CommCare mobile data submissions can be used to generate a variety
of accessible, user-friendly reports to monitor CHW performance and community health status, including: Worker Activity reports available on CommCareHQ for all CommCare projects Commissioned custom reports programmed to be available on CommCareHQ Other custom collections of indicators or visualizations generated from data
submissions in Excel or by uploading submissions to a third-party analytics tool
TYPES OF COMMCARE REPORTS
Data Tools Definition Pros Cons Examples
Custom Reports
(General)
Reports generated by manipulating CommCare raw data exports in Excel or by uploading CommCare submissions into a third-party analytics tool
• Provides the most flexibility for defining indicators and generating visualizations
• Can be easily integrated with qualitative observations• Can be e-mailed, which facilitates data consumption in
areas were internet connectivity is too unreliable to load CommCare reports regularly
• Static report• Aggregate level data only• Requires staff with strong
knowledge in Excel or other data analytics tool
• Can be time consuming to develop
• Quarterly Health Program Status Report (Microsoft Excel)
• Monthly CHW Report Card (Microsoft Word)
Custom Reports(CommCareHQ)
Data aggregated at the CHW-level for performance analysis and to protect patient data privacy. These were custom designed by the MVP team with Dimagi at the outset of the project specifically for the MVP Maternal & Child Health application
• Updated in real-time and generated automatically• Can be e-mailed, which facilitates data consumption in
areas where internet connectivity is too unreliable to load CommCare reports regularly
• Health Coordinator Report recommended for overall indicator tracking; CHW Manager Report recommended for individual performance tracking
• Can have data errors if there is an issue with how report is programmed
• Health Coordinator Report• CHW Manager Report• Verbal Autopsy Report
Worker Activity Reports
CommCareHQ report that comes with each CommCare project. This is useful for monitoring worker activity – i.e. number of CommCare submissions per day by CHW
• Updated in real-time and generated automatically• Can be e-mailed, which facilitates data consumption in
areas where internet connectivity is too unreliable to load CommCare reports regularly
• Limited to worker activity indicators only
• Worker Activity• Daily Form Activity• Submissions by Form• Form Completion• Case Activity• Form Completion vs.
Submission Trends• Worker Activity Times
Raw Data Exports
CSV or Excel files that can be exported from any CommCare project. For MVP, these raw exports contain all submissions entered by CHWs into CommCare
• Most flexibility for defining indicators and creating visualizations
• Useful for looking at case-specific information (i.e. Who are the specific U5s who did not receive a CHW visit last week?)
• Most accurate and detailed because viewing CHW submissions directly
• Large spreadsheets can be cumbersome to use and time-consuming to analyze
• Large files can take a long time to download
N/A
MVP Data Reports
TYPES OF COMMCARE REPORTS
Examples of Custom Reports in MVPHealth Coordinator Report on CommCareHQ
Aggregated health indicators. Available in real-time on CommCareHQ
Note: These reports are not automatically available on CommCareHQ. Dimagi, the company that owns CommCare, programmed them for MVP at additional cost.
TYPES OF COMMCARE REPORTS
Examples of Custom Reports in MVP CHW Manager Report on CommCareHQ
Health indicators organized by CHW. Available in real-time on CommCareHQ
Note: These reports are not automatically available on CommCareHQ. Dimagi, the company that owns CommCare, programmed them for MVP at additional cost.
TYPES OF COMMCARE REPORTS
Examples of Custom Reports in MVP CommCare Verbal Autopsy Report on CommCareHQ
Indicators on types of deaths that are occurring, including social and medical factors contributing to deaths. Available in real-time on CommCareHQ
Note: This report is not automatically available on CommCareHQ. It was developed by the MVP eHealth team.
TYPES OF COMMCARE REPORTS
Examples of Custom Reports in MVP Health Sector Report
Static cross-site health indicator performance, generated monthly in Microsoft Excel
Note: These reports are not automatically available on CommCareHQ. MVP staff designed these reports.
TYPES OF COMMCARE REPORTS
Examples of Custom Reports in MVP CHW Report Card
Selection of CHW performance indicators, generated monthly in Microsoft Word
Note: These reports are not automatically available on CommCareHQ. MVP staff designed these reports.
TYPES OF COMMCARE REPORTS
Built-In Worker Monitoring Reports on CommCareHQ 1. Select “Reports” in the toolbar
https://confluence.dimagi.com/display/commcarepublic/Worker+Monitoring+Reports
For quick performance assessment of CHWs:
• Worker Activity• Daily Form Activity• Submissions by Form• Form Completion Time• Case Activity• Form Completion vs. Submission Trends• Worker Activity Times
TYPES OF COMMCARE REPORTS
(1) Worker Activity, Pt 1Worker Activity Report shows a summary of form and case activity by user or group. Can be used to identify: Users not using the phone (no forms sent in) Amount of work done (number of forms sent in) Follow-up rates with beneficiaries
Number of forms sent in time period
Forms sent on average in time period of same length
Date of last form sent (or “None” if no forms sent)
TYPES OF COMMCARE REPORTS
(1) Worker Activity, Pt 2
New contacts registered
Child, pregnant woman, or household closed (child now over 5 yrs, woman no longer
pregnant, death, moved, etc.)
Number of contacts being visited in date range
Total number of cases per user
% of contacts visited*
*% of contacts visited could be greater than 100% if there were contacts who were closed, but also visited in the time frame
TYPES OF COMMCARE REPORTS
(2) Daily Form ActivityThis report displays daily form submissions organized by date and by user. This report allows the manager to see exactly when CHWs are working (submitting forms), and when they are able to submit the forms to CommCareHQ.
Filter by completion time or submission time
Daily number of forms completed or submitted per CHW
TYPES OF COMMCARE REPORTS
(3) Submissions by FormThis report displays daily form submissions organized by date and by user. This report allows the manager to see exactly when CHWs are working (based on when they are completing forms), and when they are able to submit the forms to CommCareHQ.
Daily number of forms submitted per CHW, per form
TYPES OF COMMCARE REPORTS
(4) Form CompletionThis report displays the average completion times for a specific form, per user, over a given time period. This allows an understanding of how long a CHW is taking to complete a form on their phone. Completion time can be an important red flag to investigate performance; if CHWs are completing a form very quickly, then they may not be using the form correctly.
No. of forms: the total number of forms submitted in the
selected time period
Average: the average completion time for the selected form
Standard Deviation: the standard deviation of this users' completion times
Shortest: the shortest completion time for the selected form
Longest: the longest completion time for the selected form
TYPES OF COMMCARE REPORTS
(5) Case ActivityThis report displays the number and percentage of open cases (i.e. child, household, pregnant women) that have been modified over a given time period. This helps identify whether the CHWs are routinely visiting their clients up to your expectation.
TYPES OF COMMCARE REPORTS
(6) Form Completion vs. Submission TrendsThis report displays a log of all submissions, comparing submission time to completion time. Long delays between completion and submission times could indicate connectivity issues or errors with a mobile user phone’s date configuration.
Filter by module and by the type of form
For each of the selected form that the CHW submitted, one can see the completion time, submission time, and time difference between the two
TYPES OF COMMCARE REPORTS
(7) Worker Activity TimesThis report is a graphical representation of the form completion or submission trends over the entire duration of the project. Y axis: displays each day of the week. X axis: displays the hours of the day. This can also be filtered by specific CHWs, as well as by a specific date range.
Filter by module and by the type of form
TYPES OF COMMCARE REPORTS
Setting Up Email Delivery of CommCareHQ Reports Click on “Reports” “My Scheduled Reports” Click the large green button “Create a New Scheduled Report” Select the saved report(s) you would like to schedule as e-mails by clicking on the button next to the report name Select the interval, day, time, recipient e-mail addresses, and whether you want an Excel report attached Click on “Submit”
https://confluence.dimagi.com/display/commcarepublic/Managing+Saved+and+Scheduled+Email+Reports
Tip: You can also generate an email report at any time by applying the desired filters in CommCareHQ and then selecting the “Email Report” button.
TYPES OF COMMCARE REPORTS
Additional Monitoring Tools “Manage Deployments” – Use these reports to support troubleshooting
Track which CHWs have updated to most recent version of CommCare Click on “Reports” in CommCare Header”Application Status” under “Manage Deployments”
Investigate if form submissions are not making it into the automated reports because of an error Click on “Reports” in CommCare Header ”Raw forms, errors, and duplicates” under “Manage
Deployments”
https://confluence.dimagi.com/display/commcarepublic/Manage+Deployment+Reports
TYPES OF COMMCARE REPORTS
Additional Monitoring Tools “Inspect Data” – Use these reports to access specific
details on CHW submissions without having to export the raw data in Excel View complete patient history
Click on “Reports” in CommCare Header”Case List” under “Inspect Data”
To find a specific user’s or group’s cases, type the name into the “Groups or Users” box in the filter
Review forms submitted by CHWs within a designated time period
Click on “Reports” in CommCare Header”Submit History” under “Inspect Data”
Review by group or individual CHW Worker Activity Reports will link to these tools too
https://confluence.dimagi.com/display/commcarepublic/Data+Inspection
RAW DATA
Why Export Raw Data? Raw data provides case-specific information, categorizing submitted
data by patient instead of aggregated information by CHW Raw data is the first place to look if any issues with custom or
CommCareHQ reports are suspected Remember raw data is organized in the export as follows:
For form exports: Each row is a visit/individual form submission by a CHW NOT unique because an export may include multiple visits/records for the same person
For case exports: Each row is a case (i.e. a unique household, child, or pregnant woman)
https://confluence.dimagi.com/display/commcarepublic/Data+Export+Overview
RAW DATA
Exporting Raw Data: Cases1. Select “Data” in the heading
3. If desired, filter for a specific group (i.e. geographic area, type of user)
4. Select “all open” (only active contacts or “all” (includes closed
contacts)
2. Select export cases 5. Download Export
RAW DATA
Exporting Raw Data: Forms1. Select “Data” in the heading
2. Select export forms
3. If desired, filter by group (i.e. geographic area, CHW supervision groups, type of user)
RAW DATA
Exporting Raw Data: Forms4. Download the form data you want
Tip: To avoid a separate export for each form type, on the right-hand side of the screen, check the box to include the desired forms in a bulk export. This is not recommend when network connectivity is poor.
RAW DATA
Why Set Up a Custom Export?Sometimes, there is too much data! Downloading full raw data exports can take longer than desired and can lead to possible errors when analyzing. If you know what information you need, you can specify for CommCare to only output those fields.
Forms: https://confluence.dimagi.com/display/commcarepublic/Form+Data+ExportCase: https://confluence.dimagi.com/display/commcarepublic/Case+Data+Export
RAW DATA
How to Set Up a Custom Export
1. Select “Data” in the heading
2. Select export forms
RAW DATA
How to Set Up a Custom Export
3. Create Custom Export
RAW DATA
How to Set Up a Custom Export4. Check which data fields you would like to include in your export. You can reorder fields by dragging and dropping fields. You can also rename fields
5. At bottom of page, press “Create”
RAW DATA
Where to Find Your Custom ExportDownload your custom export to Excel:
1. Return to “Data””Export Forms”
2. Find the name of the Custom Export
3. Download your Custom Export
1
2
3
RAW DATA
Excel Tips for Calculating Indicators from Raw Data ExportsHOW TO: Filter
When to use: to narrow down the data in your spreadsheet, allowing you to view only the information you need
Additional resources on filtering in Excel Pivot Table
When to use: to quickly summarize and analyze large amounts of data in lists and tables—independent of the original data layout in the spreadsheet—by dragging and dropping columns to different rows, columns, or summary positions
Additional resources on creating a pivot table in Excel V-lookup
When to use: to centrally locate data from multiple Excel sheets or hard-to-see location; need to match associated data (i.e. when comparing # of children visited by a CHW this month with the # children the CHW is responsible for, use the V-lookup function with CHW name as the “Lookup Value”)
Command: =VLOOKUP() Additional resources on using V-lookup in Excel
RAW DATA
Calculating Indicators from Raw Data ToolsExample 1: Number of Rapid Diagnostic Tests (RDTs) positive for malaria per CHW:
1. Export FormsChild List ModuleVisit: download as CSV file
2. In Excel, remove duplicates for form|case|@case_id This is using Excel 2008 for Mac; other Excel versions may slightly
differ. Column names may differ slightly if full export vs. custom export is used
3. Data tabPivot Table Report
This is what exports to Excel from CommCare.
Note: This example is based on the structure of the MVP MCH application.
RAW DATA
Calculating Indicators from Raw Data4. Creating an Excel Pivot Table, place form|meta|username in rows. Place form|case|@case_id in values and then place
form|patient_available|referral_follow_on|rdt_result in values as well5. Right click form|patient_available|referral_follow_on|rdt_result headerField SettingsSubtotals Custom: Count. Hide
all items but Positive6. Copy and paste the CHWs who have found +RDTs and their number of +RDTs in another Excel sheet (titled “Working”)
CHWs with 0 + RDTs will not be included in Pivot Table
RAW DATA
Calculating Indicators from Raw Data7. In another table with all CHW names, write a VLOOKUP formula to search in “Working” sheet how many +RDT cases
each CHW has. Replace N/A with 0
*Note: The Excel file pictured here is created from what CommCare exports. In a new sheet of the exported CommCare Excel file, create columns for the health indicators of interest for each CHW and use the Pivot Table function to find the corresponding values for each CHW.
CASE STUDY
Using Maternal & Child Health Application to Identify Problem & Inform Response Case Study – Appropriate Treatment of Malaria:
More severe malaria cases were coming to the health center than during similar times of the year. After
using the MCH application to access and data by CHW, it was noted that some CHWs found children with
positive Rapid Diagnostic Tests for malaria (+RDTs), but not all the CHWs were treating children with +RDTs
with coartem as per protocol. Upon further questioning of the CHWs, the local health team learned that
CHWs in several parishes had run out of coartem stock, prompting a replenishment of supplies.
Additionally, CHWs were given refresher training on the importance of timely referral of malaria cases to
health centers and reporting of stockouts.
CASE STUDY
Using Maternal & Child Health Application to Identify Problem & Inform Response Why the MCH Application was Important:
The MCH application documented that some CHWs were diagnosing malaria but not treating malaria, which
was leading to more severe cases of malaria coming to the health center. Data collected by the application
also allowed the health management team to see which CHWs were not providing appropriate treatment to
child malaria cases and to follow-up with these CHWs. Upon further investigation, the health management
team realized that the government stocks of coartem needed to be replenished. Although CHWs are
supposed to report low stock levels, they do not always do so in a timely manner. Without the MCH
application, the reason for an increase in severe child malaria cases may not have been well-understood
and the problem may not have been resolved so quickly.
CASE STUDY
Using Maternal & Child Health Application to Identify Problem & Inform Response How readily accessible reports can be used to quickly identify a problem:
Health Coordinator Report can be used to identify general issues (i.e. performance on malaria
treatment indicator is low)
CHW Manager Report can be used to identify which specific CHWs have not provided treatment for
their U-5 cases during the time period
Finally, if further investigation is necessary, raw data can be used to identify specific cases that did not
receive treatment.
CASE STUDY
Using Maternal & Child Health Application to Identify Problem & Inform Response (Continuing from Calculating Indicators from Raw Data example)
8) In Excel Pivot Table, remove form|patient_available|referral_follow-on|rdt_result and place form|
patient_available|medication_type in column
9) Right click form|patient_available|medication_type headerField SettingsSubtotals Custom: Count.
Hide all medications that do not include coartem
CASE STUDY
Using Maternal & Child Health Application to Identify Problem & Inform Response 10) Copy and paste the total count of coartem medications administered by each CHW in another Excel
sheet (titled “Working.” if CHW has not given any coartem, the CHW’s name will not be included in
the table produced by Pivot Table)
11) In the column next to the existing table with CHWs who have found +RDTs, write VLOOKUP formula
to search how much coartem each CHW has. Replace N/A with 0
12) Divide the # +RDTs by # coartem for each CHW
CASE STUDY
Using Maternal & Child Health Application to Identify Problem & Inform Response 13) In table with all CHW names, write VLOOKUP formula to search the percentage of +RDTs treated with
coartem. Replace N/A with --
14) Select this column’s cells, right click on cells, and “Format Cell”NumberCategory: Percentage to
show percentage
15) Highlight any CHW that did not treat 100% +RDT cases with coartem
CASE STUDY
Using Morbidity Data to Identify Problem & Inform ResponseCase Study – Midwife Performance:
In one MVP site, several neonatal deaths were occurring around one health center. After using the
CommCare Verbal Autopsy application to access and analyze Verbal Autopsy (VA) data on these deaths, it
was noted that the same midwife was responsible for these mothers. Upon interviewing these mothers
further, it was found that that the mothers were afraid of this particular midwife, and therefore avoiding the
health center until too late or opting to go to a traditional healer instead. The midwife was found to be
aggressive, unresponsive and intimidating to the mothers, leading mothers to delay seeking care and
ultimately leading to neonatal mortalities.
CASE STUDY
Using Morbidity Data to Identify Problem & Inform ResponseHow the VA Data Contributed to the Solution:
Trends are typically hard to notice through paper forms, which are tedious to aggregate. Electronic VA data
made it possible to see that the same midwife played a role in each of the neonatal deaths, permitting the
management team to take appropriate action. Without the VA data, factors contributing to the deaths
may have been incorrectly attributed to medical or other reasons rather than a preventable social cause.
CASE STUDY
Using Morbidity Data to Identify Problem & Inform Response1. “Data” in headingExport Forms
2. Export Full FormsVA Specialist ApplicationVerbal AutopsyDownload form of choice
If desired, filter by group
Basic mortality information can also be found by exporting the Child Close forms (Export VisitsChild ListClose Form, download CSV). This data is also included in a vital statistics section in the MVP custom reports created by Dimagi.
SUMMARY
By facilitating real-time data monitoring, CommCare can help management teams identify trends in CHW performance or community health status, and inform a response
CHWs enter data into the CommCare application on their mobile phones, which is uploaded to a cloud server. A CommCareHQ user can then access and download the data for analysis
It is important to consider what type of reporting tool will help your team reach their program goals, as well as who should have access to different levels of data
Data collected with CommCare can be accessed and analyzed through a variety of tools:
Raw data: full records of case-specific information in CSV or Excel CommCareHQ custom reports: automated reports with data aggregated by CHW or for the entire
project Other custom reports: manual, static reports generated from CommCare submissions CommCareHQ Worker Activity Reports: automated reports to monitor CHW frequency of CHW
submissions