Managing for SuccessData as a Tool in Performance

Evaluation

Sheila Fesko, Ph.D.Susan Foley, Ph.DJean Winsor, Ph.D

Institute for Community Inclusion

Agenda

• Purpose and use of data• Defining complex concepts for data collection• Collecting data on disability status• Data collection options• Developing a plan to collect data

Role of data

• Reporting purposes• Planning purposes—Determining where to

devote resources • Evaluation purposes—have you meet your

goal• Telling your Story—promoting your program

and recruiting new members

What is “data-driven” management

Integration of data into your management practices

• Setting measurable goals & measure progress towards them

• Setting benchmarks or standards

• Using data in presentations to staff, funders

Why is it useful?

• Make decision based on evidence not instinct, assumptions, or perceptions

• Have more information to use for analyzing issues / developing solutions

• Helps managers and funders see big picture, accountability for outcomes

• Helps identify trends over time

• Provides benchmarks for staff

Telling Your Story

• Who are your audiences?• What do you want your message to be?• What information will engage them?

Goal 2 Strategic Plan • Strengthen national service so that

participants engaged in CNCS-supported programs consistently find satisfaction, meaning and opportunity– Experiences that offer unique combination of

professional, educational and life benefits to service participants

– Recruits a diversity of Americans especially those from under-represented populations

Goal 2 Priority Measure

• Percent of service participants engaged in CNCS-supported programs who report having an experience that expands educational, employment or civic opportunities

Types of data

• Demographics– Number of

members/volunteers– Age– Racial Identification – Level of education – Disability

• Program activities– Number of children

tutored– Number of at-risk acres

improved – Number of individuals

who received disaster response and recovery services

Clarifying the concept

• Process of specifying what we mean when we use particular terms.

– Example: “Quality of experience” among members / volunteers. • What does that “mean” to those interested in measuring it?• What does that look like? What are examples of it?

• Produces an agreed upon meaning for a concept for the purposes of research.

. Operationalize your definition

• Things like “satisfaction with life” or “fear of crime” are hard to measure directly, so we have to make inferences.

• Process of defining specific ways to infer the what we are trying to measure.

• Indicators are observations we think reflect the presence or absence of the phenomena to which the concept refers.– How do we “know it when we see it” or “when someone

experiences it”

A Case Example:A Case Example:What if you want to ….What if you want to ….

improve the member experience improve the member experience for people with disabilities?for people with disabilities?

Where to start?

• Answer the Who and What questions first.– Who are they and what are their characteristics? – What do we know about their experience?– What are we doing well?– What are we not doing quite as well?

How many people with disabilities are currently members?

• A big question to answer… how many people are members? – Prevalence (current “cases”) vs incidence (new “cases”).

• Has anyone surveyed their members about disability?

• What was your experience?

First Step:

Understand the concept you are trying to measure

Measuring Identity is complex• What is your age? • What is your sex?• What is your sexual orientation?• What is your race?• What is your ethnicity?• What is your religion?• What is your marital status?

But self-identification alone is not sufficient for disability

prevalence

WHY???

Disability, Identity, and Politics

• Collecting prevalence data on characteristics of people is political.

• Disability is a loaded term and has a very particular history.

• Disability is very culturally nuanced. • Our understanding of what is a disability has

changed over time.

Universal versus Minority Model

• Disability is a universal experience and not just a list of certain impairments– Everyone will have a health issue in their life and very

likely to have a disability at some point

• Disability is on a continuum rather than categorical– Example: vision loss

• Multi-dimensional concept rather than uni-dimensional.

WHO, ICF Model, 2002

Case Examples

• Macular Degeneration• Deafness at Birth• Asthma• Schizophrenia• Head Injury• Attention Deficit Disorder• Multiple Sclerosis

Evolution of the definition

Medical Model

• Physical, mental, or psychological condition that limits a person’s activities.

• Linked to conditions.

• Problem residing in the individual.

Social Model

• Arises from interaction of a person’s functional status with the physical, cultural, and policy environments.

• Neither person nor environment specific.

• Environment should be designed to accommodate all.

Nagi’s theory on disability• Sociologist / epidemiologist - 1st to recognize environment

plays a role in definition of disability.• Disability = a relational concept which goes beyond individual

limitations & attributes & includes actual physical & social environments as well as the reaction of others.

• Possible to have a disability construct used:– Across the US?– Internationally / cross-nationally?– What are the complexities involved?

Factors…. WHO, 2002

Disability is an interplay of biology, social roles, and

environmental conditions

Lots of factors influence whether someone is perceived as a

person with a disability and…

Whether or not they perceive themselves as having a disability.

So… what does this mean?

Measuring

• Using self-identification measures only (Do you have a disability) may leave out very important groups:– People who have the least power– People who equate disability with poor health or with

weakness– People who have conditions that are episodic– People who have conditions associated with stigma– People who disagree that they have a health condition – People who function quite well because of technology and

environmental adaptations.

Conceptualizing Construct of Disability

• Complex task! Implications of the construct - status, identity.

• Context of the definition: social, political, economic, ethical.

• No single definition of disability among large national surveys and polls (different purposes / audiences).

Disability Defined • Americans with Disabilities Act (ADA) of 1990 defines disability as:

– physical or mental impairment that substantially limits one or more life activities,

– a history or record of such an impairment, or – Regarded by others as having such an impairment.

• Under Federal Social Security Disability Act, "disability" means the "inability to engage in any substantial gainful activity by reason of any medically determinable physical or mental impairment which can be expected to last for a continuous period of not less than 12 months or result in death."

• World Health Organization (WHO) created an International Classification of Functioning and Disability (ICF). Breaks disability construct into three dimensions:– 1) Medical or health condition; 2) Functional limitation and participation

restrictions; 3) Environmental restrictions

Once you have the concept… now you need to figure out how

to“operationalize”…write the questions.

Operationalizing construct of Disability• How will you identify disability, based on your

construct / definition?

• How will you measure the construct?

• How many items will you need to do so?

– If unlimited amount …– If only 1 item … or 3, 6, 10 items … – As a stand-alone instrument or subsection of whole

Overview: 3 Types of Disability Definitions

• Functional /Activity / Work Limitation: – Because of a health condition/disability, the person experiences limits in

the ability to walk, talk, think, see, hear, remember, etc. The condition lasts at least six months.

• Specific condition/disease: – The specific disorder (Down Syndrome, Multiple Sclerosis, Macular

Degeneration), it is often paired with a severity and onset measure.

• Self-report: – Do you have a disability?

Functional/Activity Limitation• Easier to measure functional limitation. Many scales exist

(e.g. activities of daily living scales).

• Need to combine this with severity measures.

• Use in self-report surveys has disadvantages because two people with the same functional limitation may report it differently.

– Anchor problem: Who is the person comparing themselves to in assessing their functional impairment.

– Someone who cannot do something may report no difficulty because he / she sees it as not relevant / no difficulty in something not done at all.

Work Limitation Questions• A subset of functional or activity limitation is work limitation.

• (Do you/Does anyone in this household) have a health problem or disability which prevents (you/them) from working or which limits the kind or amount of work (you/they) can do? (Y/N)

• Problems: – It is over-inclusive. People with broken legs, pregnant, short term injuries may

say yes. – It is under-inclusive because people with disabilities who work might say no.

People who have visual impairments, deafness, or learning disabilities often report a condition but say that they have no activity limitation.

– Any others?

Disability as measured by Health Condition only

• Example: Do you have one of the following conditions: – Asthma– Gastrointestinal disorder– Depression– Visual impairment

• Survey measurement issues:– Number of conditions is too large – Ability to determine severity per condition important as not everyone

with a condition has a disability.

Self-report measure• Do you have a disability? AND / OR Do you have a disability or health condition

that causes a functional limitation lasting more than 6 months?• Problems: • Specific groups are less likely to say yes

– Older people– People with cognitive disabilities– People who are deaf– People with a stigmatizing condition

• Some people with disabilities may not fit the definition above.– People with a history of a disability or who may qualify as having a disability because

they are perceived to have a disability. (e.g. people with facial scars or deformity).

• Some people may say yes to it and not have a disability– Particular disorders can not be counted as disability (actively SA, pyromania, “deviance

clause”)– Some claim to be disabled with no confirming diagnosis (LD, ADHD)

What measures are being used?

• In an effort not to reinvent the wheel, let’s look at what is currently being used:

– By who– How (who do they include / exclude)– For what purpose– With what results (for prevalence).

Examples of Federal & International Measures• Behavioral Risk Surveillance Survey (BRFSS)

• Survey of Income and Program Participation (SIPP)

• Current Population Survey (CPS)

• National Health Interview Survey (NHIS)

• Census 2000 Long form & American Community Survey (ACS)

• CDC Healthy People 2010 measures

• UN Measures (with NCHS staff from US)

Federal Measures: BRFSS• RDD telephone survey of non-institutional pop >18 yrs

• 1998 module on disability from 36,842 respondents in 11 states & DC. RR of 59.2%

• Prevalence rates range by state – 21.8% in AL to 17.1% SC

• Used 2 Qs: – Are you limited in any way in any activities because of an impairment

or health problem? (y/n)– If you use any equipment or help from others to get around, what do

you use? (wheelchair, walker, cane, another person)

Federal Measures: SIPP• Not designed as survey of health or disability - but these concepts so closely

connect with program participation it cannot meet goals without including these measures

• Panel survey with topical modules; monthly admin, sample sizes from 14,000 to 36,700 interviewed households. Widely cited prevalence rate of 20.6% based on 1994 data

• Non-institutionalized civilians only – uses a “kitchen sink” definition of disability– Itemized specific functional activities, ADLs, and instrumental activities of daily

living. – Examples:

• Seeing ordinary newspaper print • Hearing normal conversation• Having speech understood

Federal Measures: CPS (1981-2004)• Survey of 50,000 HH only (NONE in institutions, schools,

hospitals) in 754 sample areas.

• Conducted monthly by Census for BLS.

• Purpose - provide primary indicator for labor force / income stats. Q on disability = work limitation:– Does anyone in this HH have a health problem or disability which

prevents them from working or limits the kind of work they can do?– Strengths / weaknesses of this measure?

• In 2002, data showed 22M (10%) ages 16-64 reported work disability

Federal Measures: NHIS• Disability defined as being limited in activity caused

by chronic health condition. Focuses on activity limitation in major activity of life, defined by age:

• <5 years - play• 5-17 yrs - attending school• 18-69 working or keeping home• >70 yrs - ability to care for self / home without assistance from

another person

• Data as of 2002 reported 34M (12%) of non-institutionalized civilians limited in usual activities

Federal Measures: Census 2000• 281.4 M people in 115.9M housing units

• Long form (1/6 HH) had disability Q

• Broken into 6 categories:– Sensory– Physical– Mental– Self-care– Going outside the home– Employment Disability

Federal Measures: ACS

• Designed to replace Census long form.

• Conducted by Census to 3M HHs in 2005. Starting in 2006 – included the 2.5% of US population living in “group quarters.”

• As of 2002, data shows 41M or 13% of non-institutionalized population with disability.

Census 2000 / ACS Measures cont.

• Does this person have any of the following long-lasting conditions:– Blindness, deafness, or a severe vision or hearing impairment?– A condition that substantially limits one or more basic physical

activities such as walking, climbing stairs, reaching, lifting, or carrying?

• IF 15 yrs or older: Because of a physical, mental, or emotional condition lasting 6 months or more, does this person have any difficulty in doing any of the following activities?– Going outside the home alone to shop or visit a doctor’s office?– Working at a job or business?

Federal Measure: CDC Healthy People 2010

Developed standardized items to be used in all Healthy People 2010 surveys.

1. Are you limited in any way in any activities because of a physical, mental, or emotional problems (y/n)

2. Do you now have any health problems that require you to use special equipment such as a cane, wheelchair, a special bed, or special telephone? (y/n)

UN Measures• UN commissioned a team of experts to create a cross-national

disability measure

• Broad design - able to compare across cultures. Draft Qs include (scale for response cats)

– Do you have difficulty seeing, even if wearing glasses?– Do you have any difficulty hearing, even if using a hearing aid? – Do you have any difficulty walking or climbing steps?– Do you have difficulty remembering or concentrating?– Do you have difficulty with self-care, such as washing or dressing?– Because of a physical, mental, or emotional health condition, do you

have difficulty communicating (ex. understanding others or others understanding you).

Challenges posed for data collection

• Measures– With limited space - challenge to construct Qs straightforward / easy

to answer– R fear of potential stigma associated with reporting– Others?

• Administration:– Not all modes are accessible to all pops– Non-response at unit and item level – Others?

Summary: What to Do?• Disability measures are not uniform and there is no consensus about what

questions to use. Pattern of questions may help you get some indication of disability prevalence. Use multiple types of questions within one survey.

• Limit the use of the word “disability” when possible.

• Consider including questions that ask about disability-specific service use in which eligibility is required.– Ever use VR, SPED, DMH, DMR– Ever received SSI, SSDI

• Consider including questions that ask about equipment, devices, and AT.– Wheelchair, scooter, leg braces– Guide dog, service dog, white cane, JAWS, Braille– Hearing aides, CART, ASL interpreters

Data Collection Options

Two kinds of data: Qualitative and Quantitative

• Data come in many shapes, sizes, and formats. Distinction is between numerical (quantitative) and non-numerical (qualitative) data.

• Different data will be needed to:– Answer different types of questions– Measure different kinds of outcomes

• These two types of data– Require different data collection and analysis techniques. – Both are useful and valuable tools. – Each have advantages and disadvantages.

Quantitative Data• In numerical format naturally

– monetary values, counts, dates

• Coded / assigned numerical values for analysis – placement= 1, non-placement=2– coding into categories with numerical representations (e.g. 1 = yes, 2 = no).

• Used for: – Describing information in aggregate, identifying trends overtime, quantifying outcomes,

conducting statistical analysis

• Able to be analyzed using statistics

• Examples include:– N clients served at agency, N staff, % of staff with BA degrees, % of clients placed in jobs,

agency cost per client served

Advantages & disadvantages of using quantitative data

Advantages• Aggregate large volume of data

• Numbers can be “persuasive”

• Track trends over time

• Measure relationship between different variables

Disadvantages• Training / expertise required to

collect, enter, and analyze these data

• May not shed light on the “whole story” or the “why” of a situation

• Participants may feel limited to preset response categories

Administrative Data or Case Record Abstractions

• Collection of existing data from case records or files:– “Abstracting” key variables from the data for reporting /

analytical purposes– Because the data are not collected for research purposes –

files may have missing data or vary in how items were recorded

• Unobtrusive research– study social behavior without affecting it

Value of Record Abstractions / Administrative Data

• Minimal costs, effort to gather these data – no burden on participants

• Provides a snapshot of key indicators for your state or agency

• Gives you ability to aggregate these data for snapshot on progress towards goals / outcomes:

– Recruitment: N or types of agencies tapped for recruitment partnerships, N or type of sites where applications were distributed

– WEBBERS on members / volunteers: gender, age, race, education

– Agencies: N and type of service opportunities members / volunteers placed in

What is a survey?• Way to collect standardized information from large group of

individuals.

• Collection of data from a scientifically selected group of people. Results can be representative of a larger population.

• Data collected are used to address specific issues.

• A standard set of procedures are followed.

Advantages and Disadvantages of Surveys

Advantages• Collecting original data on

population too large to observe directly

• Results can be generalizable to whole population (when using scientific sampling methods)

• Paper / web give Rs the flexibility to return data at their convenience

Disadvantages• Can be extremely costly to

conduct

• Item non-response and unit non-response

• Accounting for sampling bias (based on mode), can leave out some members of the population (reading level, non telephone household, non-English speakers, persons with disability)

4 Modes of Survey Administration

1. Self-administered: Mail

2. Self-administered: Web

3. Interviewer Administered: Telephone

4. Interviewer Administered: In Person

1. Mail Surveys• Most common mode of survey data collection

• Low in cost

• Response Rates generally low – need multiple waves of follow-up

• Used a great deal in business surveys when directed at specific groups (such as members of professional organizations)

• Who does it exclude?

2. Web Surveys• New technology, seen most prevalently in convenience samples

• High costs associated with programming, yet once programmed:

– data available immediately– structure quex. In such a way to eliminate item non-response (benefits /

drawbacks of doing this)– Rs can answer at any time, like paper instrument– Response options can be personalized based on previous responses

• Currently still LARGE bias in general pop using web mode alone, therefore not recc (alone) for general pop study

3. Telephone Surveys• Most large-scale surveys in the US are conducted

by telephone using CATI - improves the quality of the data collection– Must be tested for correct routing / branches of Qs– Avoids an important error - omissions!

• Can increase cooperation rates.

• Faster, less expensive than in-person interviewing.

• Who doesn’t it reach? – When might this be a problem?

4. In-Person Surveys(Interviewer Administered)

• Presence of interviewer may have effects. – Increase in cooperation– Possible to get immediate clarification on issues in the instrument– Possible for bias because of interviewer presence

• High quality of data - training the interviewers in a classroom like environment– Good interviewing techniques stressed

• Professionalism• Avoiding bias

• Why might this mode not get used as often?

Regardless of mode: Use Advance / Cover letters

• Key Components:– Explain the purpose of the survey

– Organization sponsoring the survey & any relevant endorsements or supporters

– Lets person know you will be contacting them (or they may contact you) with any relevant details about items needed for survey

– Provide details on deadlines or submission requests

• Write from reader’s perspective: “Why should I participate?

Schedule of survey data collection

• Set up your calendar with mail dates

• Identify total field period (start to finish)– Allow sufficient time to – Prep mailings– Recruit, hire, train interviewers– Design / test web survey– Process survey returns / enter data

• Goal: work backwards from end goal or deadline

Preparing to field your survey• If interviewer-administered (on phone or in person) you must hire

interviewers and supervisors.

– Train them on your survey– Ensure they have basic interviewer training– Specify DC schedule, QC rates, production rates, and response rate

expectations.

• For mail surveys, training staff on schedule, receipt and follow-up procedures

• For all methods: developing QC process check on completion of work, including collecting and editing documents.

Once the quantitative data are collected …

• Once the data have been: quality checked, edited, and entered you can begin your analysis! Your analysis should focus on answering the questions you posed when designing your data collection forms (abstractions or surveys).

• Spreadsheets may be useful to you for– Simple entry procedures– Few case records– Ease of reporting– Disadvantages of spreadsheets?

• Databases may be useful to you for– Simple entry, once entry form is designed, Minimizing entry error– Ability to: link several datasets, program reports into database / create on-line for field

access– Disadvantages of databases?

Qualitative data• Qualitative data are non-numerical

• Used for: – Examining social world through stories, images, and experience– Probing more deeply into constructs, examining the “how” or “why”

types of questions

• Examples include:

– Transcripts from 1:1 or group interviews– Observations made in the field– Pictures, texts

Why aren’t qualitative data used more?

• Capturing and analyzing qualitative data sets has been a tough business.

• Extremely costly process, quite time consuming, often necessitating small sample sizes.

• “Numbers” can be perceived as more persuasive.

Qualitative interviews• Interaction between participant and interviewer

where interviewer has “general plan” of inquiry – but not set questions

• No specific order of questions

• Interviewer must be knowledgeable in subject matter so they can ask clarifying or probing questions

• Essentially a “conversation” but participant does 95% of the talking

Advantages and disadvantages of 1:1 qualitative interviewing

Advantages

• Participants share info in 1:1 format may not share in group

• Allow participant to explore concepts more freely / fully

• Researchers not limited to script or preset response categories

• Great for exploratory work where you may have limited info on topic

• Focus on verbal and non-verbal cues.

Disadvantages

• Relies heavily on skill and knowledge of interviewer

• Costly to implement – per interview costs may limit sample size

• Due to small N, limits to generalizability

• Large volume of data to transcribe / analyze

Focus groups

• Purpose is to explore rather than describe or explain in a definitive sense– Group of 7-12 people too atypical to generalize to whole population

• Group dynamic may bring up topics or issues that do not occur in an individual interview

• Examples: – member / volunteer service experience: successes, challenges– application experience: how heard of opportunity, why applied– retention issues: why left service, what can be changed– agency partnerships: quality of service provided by members / volunteers

• Group interviews - they are like in-depth interviews – Guided discussion on topics of interest

Advantages & Disadvantages of Focus Groups

Advantages

• Socially oriented research method

• Flexible – group may raise topics researcher didn’t foresee or anticipate

• Speedy results

• Low in cost

Disadvantages

• Less control than individual interviews. Tendency to produce “group think” where people may not readily express ideas that deviate from group’s.

• Data can be difficult to analyze.– Difference between groups

can be troublesome.

• Moderators must be skilled and discussion must be conducted in a conducive environment.

• Groups are difficult to assemble.

Field Observation• Methods of collecting data on people, likely in their natural

settings.

• People from somewhere going somewhere else & sharing what they find

• “Informant” who gives you your data (like a narrator)

– Participant observation: performed by those who take part in the activities they observe. Gain “verstehen” by immersing themselves in the daily lives of those they study.

– Non-participant observation: made by an observer who remains as aloof as possible from those being observed.

Using Field Observation• Decide on a topic where field observations are appropriate

• Identify your research questions and constructs to measure

• Can include narrative and quantitative measures

• Examples can include:– observation of worksite or agency – observation of member / volunteers with those they serve– Attending recruiting events – observing candidates and recruiters

Advantages and disadvantages of field observation

Advantages• Direct observation, rather

than descriptions or interpretations (via interviews) from participants’ bias / perspectives

• Data can richly supplement other sources of info

Disadvantages• Disruption of natural

setting• Time consuming / labor

intensive• Relies heavily on skills of

observer• Can rely on honesty – level

of disclosure of informants• Can have ethical dilemmas

(participant vs. non-participant)

• Act of study can change behavior of those observed

Developing a Data Collection Plan• What do you want to know?• Who will you be collecting information from?• What tool is going to be effective in answering

that question?• What limitations must you consider in

implementation?• What instrument will you use?• How will you analyze the data?