Consistency and Repeatability in Analytics (Using M2) wfunk@kennellinc

Consistency and

Repeatability in Analytics

(Using M2)[email protected]

FOR OFFICIAL USE ONLY2

Consistency and Repeatability in Analytics

1) Context: The practice of analytics requires discipline to ensure results are repeatable, consistent and reliable.

2) Purpose: This presentation will describe best practices for reporting

3) Outcome: After attending this session, participants will meet the objectives described on the next slide.

Objectives

• Attendees can:1. List best practices for preparing reports in

M22. Utilize the M2 data dictionary.3. Retrieve a copy of the M2 Data Dictionary4. Describe an analysts reponsibilities with

respect to data protection.5. Describe a good M2 deliverable product.6. Track work on repeated projects for

anomalies7. Keep proper documentation associated with

the use of M2.3

Motivation

• Reliable analysts usually become very valuable to an organization.• Especially true when analysts use ad-hoc

systems to prepare studies.• Ad-hoc offers maximum flexibility for question

answering• But there is always the risk that the user will

not properly retrieve/handle data.• This presentation offers tips on the optimal

use of M2, to achieve reliability, consistency, and repeatability.


Preparing M2 Reports


Restricted vs. Unrestricted M2

There are two options for M2 access:•M2 Restricted:

• Contains direct identifiers for beneficiaries (i.e. name, SSN)

• Contains rank of sponsor• Most users do not need the restricted M2.• Appropriate for care managers or other people who need to

know who the person is on a record in M2• Can only retrieve <10K rows of data.

•M2:• Encrypted person identifiers (mostly), grouped rank only• Considered safer, but still contains protected health

information.• Can only retrieve up to 500K rows of data at a time.

6Use the Unrestricted M2 when you can!!!!!

Report Writing

• Check to see if there is a corporate report that meets your needs.• Handbook can be obtained from M2 Infoview


Report Writing

• If you find a candidate report, read the documentation carefully before using the report.• [Corporate Reports session on Tuesday at the

Symposium]

• If there is no available corporate report• Use available resources• M2 Data Dictionary• WISDOM notes• DCO Refresher Sessions• Navy Newsletters• Navy DCOs• Reports previously done


M2 Data Dictionary

9FOR OFFICIAL USE ONLY

M2 Data Dictionary

• M2 Data Dictionary is published once per month• http://www.tricare.mil/ocfo/bea/functional_specs.cfm• Or even easier, just Google “M2 Data Dictionary”!

• Excel workbook• Tabs for each type of data• Additional Reference tabs• Hyperlinked for easy navigation

• May have to click “enable content” for navigation to work.


Use the M2 Data Dictionary when you write reports!!!!!

M2 Data Dictionary

M2 Alerts are not all that useful.

11

• It says population, so this must be something about counting people??

• What does 0005 mean? What about age group ‘A” or Beneficiary Category ‘DA’?


M2 Data Dictionary

• Click on M2 Data Dictionary and open it.• Click on the Table of Contents tab.• Each file is listed with a hyperlink to the page that

describes the file.• Click on “Population Summary”.• Data elements are listed in the same order as in M2.• Review the header row.• Review the rows for “Age Group Code” and

“Beneficiary Category”.• Review the row for “Catchment Area ID”. Click on

the DMISID link.



• To use the DD: Click on the title that represents the data file you are looking at

• In this case, it’s population summary.


• Header Information Above.

• Detailed data descriptions in each row below

Using the DD


• Description of valid values for age group code


Using the DD

• Description of valid values for Beneficiary Category


Using the DD

• No values are listed for Catchment Area ID

• But if you click on the link…..


Using the DD

• DMISIDs that represent the area being reported on, and associated names

• Notice how the values you see in this report line up with what you just saw in the dictionary?

• 2,759 Active Duty Family members ages 0-4 live within 40 miles of Bassett ACH in Alaska


M2 Data Dictionary• As with Catchment Area DMISID, when a variable in

M2 has too many valid values to list, either:• Use the hyperlinks to click to an appendix with information, OR• Use reference tables within M2

• There are a few variables that do not have valid value lists in the DD or in an M2 reference table.

• Might have to resort to web searches or other means in these cases

• Sometimes the data dictionary tells you not to use a variable.

• Back to table of contents and click on “Purchased Care Non-Institutional Detail”

• Look at the comment for “Number of Visits, Raw”

• DD contains all variables, restricted or not, you just won’t be able to see restricted variables if you only have regular access. 21

FOR OFFICIAL USE ONLY


Using the DD

• After reading this note, users should not retrieve this element and called what they report a “visit”.

• Very important to always check the DD.

M2 Data Dictionary• The M2 Data Dictionary is an excellent

resource to use, but it is not always correct.


• This MTF Service Area definition is incorrect.

• MTF Service Areas apply to inpatient and ambulatory clinics.

• Has been reported and will be corrected.

• Be careful! Really evaluate your data!

Other Resources

• WISDOM CD:• Given to WISDOM Attendees• Detailed descriptions of data in M2 and how to use it.• Reports and more reports• Step by step instructions for Business Objects.• WISDOM instructors contact information.

• WISDOM DCOs:• Conducted monthly, for all M2 users.• Materials are posted on Infoview (like the corporate

reports)

• Navy Analytics Newsletters (also in Infoview)• Navy DCOs (coming soon)


Previous Reports

• Many questions are similar• Similar cohorts

• Active Duty? Prime? Diabetics?• Etc…

• Most users save their reports so they can be referred to again.

• Can be cumbersome to find things though• It’s also a very good idea to keep a

running list of criteria that you use to address questions for easy reference.


Example of a Running List of Query Criteria

Data Need RulePrime ACV Group = PrimePlus ACV Group = PlusTFL<65 MERHCF Flag = UTFL 65+ MERHCF Flag = TMedicare Eligibles Medicare Code is not NUnit Members UIC matches patten

Navy AfloatSponsor Svc Agg = V, use bencat to sift out dependents

Deployed Members OCO Deployed Flag = YGuard/Reserve Bencat (GRD, IDG)Active Duty w/o guard/res Bencat = ACTUnenrolled AD ACV = MNetwork Enrollees Enrollment Site Service = M



Data Status Table

• Run the Data Status table to check for freshness of data and any comments that may be applicable.

Report Checking

• Once your report is written, make liberal use of slice and dice to check your work.• Check each variable to ensure that the

content is what you expect.• Check counts of data for reasonability.• Look at monthly data to be sure you

understand trends and such.• Double check calculations by doing crosstabs

and by doing hand calculations where appropriate.


Report Checking

• If you are running the same report regularly, keep track of how the data changes over time

• Use that knowledge to know what to expect.• Example of same report being run once per

month, with counts by month. Keeping adding data w/ each new run.


Month 11/10/2011 12/10/2011 1/10/2012

Oct 32 154 156

Nov 24 145

Dec 14

Report Checking

Be careful of row limits!


• Check to make sure you don’t get “Partial Results”

• Your data will be wrong if you use them!

• Tiny little error message at the bottom of the screen!

Report Checking

Row limits can cause analytic problems:


• If you need data that exceeds the row limit: If you have both restricted and unrestricted access, see if

your query can be written if the unrestricted universe. This will give you 490K extra rows to work with.

If you still need more than your row limit allows there are not many good choices.

Some users run their queries in pieces (i.e. month at a time) and then piece results back together.

If you do this, be sure you are authorized. (i.e. both universes are PHI, be careful what you do!).

Also, do very carefully, because you can introduce more human error this way.

Make a checklist of all of the pieces you need, check off that you got them, and run summary queries to confirm you put things together properly.

Data Protection


Row Limits

• M2 obviously contains protected health information (PHI) and personally identifiable information (PII)• PII: Generally defined by the Privacy Act• PHI: Defined by HIPAA.

• Currently both the M2 and M2 Restricted Universes contain PII and PHI

• There are many rules and regulations regarding the use of PII and PHI


Data Protection

When using M2 data, what do you need to know?

• Which data fields are considered PII or PHI?

• Which data fields are sensitive?

• When you use these types of data, how do you protect yourself and the beneficiaries in the data?


PHI or PII?

• PII• Defined by the Privacy Act• Such as names, Social Security Numbers, etc..• ‘Small cell’ (<=30) data fields (i.e. age 90+, or Alaskan Indian

Race).

• DEERS Person ID (EDIPN):• Has recently been classified as PII by the TMA Privacy Office.• DEERS Person ID is available in both the M2 and M2 Restricted

Universes.• Must be handled as you would a social security number.• DEERS Person ID will be encrypted in regular M2 in a future

update.• The DEERS Person ID of the user is visible on the screen in M2.• Take particular caution if you are a beneficiary and an M2 User!


Sensitive Data

• Some data are not covered under laws but are deemed sensitive

• Some examples include:• Procurement sensitive data: Such as Ingredient

cost and dispensing fee in purchased care pharmacy data

• Proprietary Data: Such as CPT Codes and Descriptions.

• Restricted use: Such as cause of death


PII or PHI?

• PHI• Defined by HIPAA.• HIPAA is much stricter than the Privacy Act.• More than just PII. HIPAA seeks to further restrict

deduction of identities, in addition to just restricting access to direct identifiers.

• Stiffer fines for breaches (25K per HIPAA, 1K per Privacy Act).

• HIPAA Limited Data Sets are a type of PHI but less strict than “Full PHI”.

• Most research is done with HIPAA Limited Data Sets (LDS).• Lower hurdle to gain access (Privacy Office).• Still subject to all the rules and regulations of PHI, though.


Specifically Covered Elements

• Names• Any geographic designation lower than state level

(zip codes, catchment area, PRISM area, etc.). Any geography < 20K people.

• Any date related to a person below the level of year (FM, Service Date, Date of Birth, Age (if >89)

• Telephone Numbers, Fax Numbers, Email Addresses• Social Security Numbers • Medical Record Number (FMP/SSN)• Health Plan Beneficiary Number (EDIPN, DEERS

Beneficiary ID)• Any other unique identifying number• (Other HIPAA elements are not generally available in

MHS systems)The PHI bar is set very high by HIPAA


HIPAA

• HIPAA Limited Data Sets (LDS) are PHI, but dates (except DOB) and geographic designations are allowed.

• Absolutely essential elements in health research and analytics

• Especially dates, where the temporal relationship between events must be understood.

• M2 is a PHI system. Even the non-restricted version of M2 does not qualify as a limited data set.


HIPAA

• Under HIPAA, a citizen has the right to information about who has accessed their PHI.

• Implemented as the “Chain of Custody” requirement.• Must document use/release of PHI. Done with a

HIPAA Log.• After writing a report, also consider whether the

data use needs to be tracked.• Must have data use agreements in place to release

external to the organization.• HIPAA breaches are very serious matters. Reporting of

breaches are expected immediately.• The MHS has the biggest HIPAA breach in history, with

CHCS data in San Antonio. Has resulted in a billion dollar lawsuit.


OUCH!


Sample HIPAA Log

Date of FileDisposition DateFed Ex NumberDestruction Witnessed ByDelivery Mechanism

• Date Data Retrieved• User• Company/Organization • Task Order • Project Name • External Client/POC • Received From External Source File? • Sent To External Source?• Date Delivered • External Agency DUA #• File Names and Locations• Protected Info

• Tracking is required by HIPAA for all PHI handling, not just M2

• Well-suited for a spreadsheet


Best Practices

• Have all users in an organization use the same template for a HIPAA log.

• Assign someone to be the HIPAA Log Coordinator at your location.

• Have all users submit the log to the Coordinator on a routine basis for consolidation.

• Review with DUA custodian or supervisor routinely to ensure data are being properly used and documented.


44

• Physically secure your data

• Locked cabinets, locked doors, locked file cabinets

• Implement technological safeguards

• Encryption

• Mark copies FOUO, etc

• Protect passwords/log-on information

• Protect terminal while logged in (lock your workstation)

• Do not leave CAC card unattended

Best Practices


45

• Don’t release (or re-release) PHI data w/o authorization

• Check DUA if a contractor or consider need to know if internal government.

• Don’t publish PHI or PII.

• Limit reports to retrieve the ‘minimum necessary’ information.

• Destroy data when you are done with it (cross-cut shred, overwrite and delete, or deGauss)

• Deliver data securely

• Secure FTP, Encrypted Drives

Best Practices


M2 Deliverable Products


M2 Deliverable Products

• Since M2 allows ad-hoc querying, there is always a risk that reports will not yield the results you want!

• Ad-hoc can be your best friend or your worst enemy!

• A good M2 deliverable product will contain detailed documentation so that recipients of data understand what they got.

• Most users document in spreadsheets, with multiple tabs if needed.

• Screen shot of query panel(s).• Screen shots of additional math/manipulation or language

to describe it.• List of filters used with an indication of why.• Snapshot of the data status table and other metadata.


Screen ShotData Elements Retrieved

Filters Applied

Using the DMIS ID Table


Screen Shot of Additional Calculations

Created a variable called “% Diff”


• Picture of a slice and dice panel.

• Similar to Excel Pivot Tables• Enables users to

reshape/filter data within the M2 application.


Documenting Filters

• Filters or conditions can be applied in many places in M2.

• It’s good practice to document what filters were applied and most importantly, what the filters were intended to do.

• Example “Filter Grid”• Enables the recipient to understand what was

pulled and why, and to check for mistakes. Filters Why ACV Group = 'Prime' Only Prime Enrollees MEPRS1 Code = 'B' Only Ambulatory Care Provider Specialty not between 910 – 999 No unlisted specialties Beneficiary Category in list 'ACT', 'GRD' Only Active Duty and Active Guard/Reserve


Metadata

• Metadata: Data about the data!• Data Status table is a good example.

• Metadata tab would include:• Picture of data status table• Query name and location• When the query was run• How long it took• Etc..


Example Deliverable

• What are the top MS-DRGs in Navy MTFs in FY 2011?

• How many dispositions and bed days?• What was the average length of stay for

each of these MS-DRGs?• What was the case mix?• Let’s review an example deliverable for

this question.


“Final” Tab


“Query Panel” Tab


“Variables” Tab


“Metadata” Tab


“Metadata” Tab


Available Date

FY Updated Object Class

Reported As Of Date

Reported Date Range

1/23/2012FY

2011Direct Care

Inpatient 1/10/2012Oct 04 - Jan

12

Consistency and Repeatability in Analytics (Using M2) wfunk@kennellinc

Documents

Transcript of Consistency and Repeatability in Analytics (Using M2) wfunk@kennellinc