Getting the Most out of Data Collection and Analysis: Part ...€¦ · PART 1 RECAP. www ......

>www.dpw.state.pa.us >www.dhs.pa.gov

Getting the Most out of Data Collection and Analysis:

Part 3- Analyzing Graphed Data to Inform Next Steps

Amy Alford, M.Ed., BCBAClinical ConsultantDecember 14, 2017


• Brief recap from parts 1 & 2 of this series• The 5W’s of data analysis• Visual instead of statistical analysis• Using the data during team meetings• Using the data to make informed, instructional decisions

Agenda


PART 1 RECAP


• The goal for all data collection is to capture quality evidence that allows analysis to lead to the formulation of convincing and credible answers to the questions that have been posed.

• Data collection is more than just a frequency count• Needs > Goals/Objectives > Data Collection Tool• Refine and individualize your data sheet• Training others to correctly use the data sheet will yield

better data• As a professional in this field, you have a responsibility to

collect data- No more excuses!

Recap


PART 2 RECAP


• There are 2 parts to graphing in Excel- 1) setting up the spreadsheet and 2) Data Entry

• Let the data sheet drive Excel formatting (and vice versa)• Choosing to NOT graph and analyze data is NOT acceptable• Use what you know, learn what you don’t Stay relevant and current with changing technologies

• Excel is only ONE program out there- don’t limit yourself if you have access to other programs

• Data collected (and analyzed) should ALWAYS ask more questions

Recap


THE 5 W’S OF DATA ANALYSIS


• Who (should do the analysis)

The person who designed the data collection process (may or may not be those who actually collected the data) OR someone who has a skill set to analyze (may depend on intent)

• What (does data analysis mean)

Count, Mean/Median/Mode, Slope/Level/Variability, rate, identify patterns over time, comparing results to previously determined benchmarks, correlations, statistical significance

• Where (does data analysis take place)

Excel (most common) but many other tools to utilize exist (statistical software, apps on phone/iPad/computer, paper/pen), during team meetings, written in reports

The 5 W’s of Data Analysis


• When/Why (do we analyze data)

Ideally, analysis should begin early and continue often (begin with the end in mind, frame questions to answer as you go along)

Prior to implementing a new program (e.g. to determine baseline, develop hypotheses)

During implementation (to make tweaks to instruction as you go along)

After implementation (to ensure what you did continues to work as intended)

• How (do we analyze data)

Visually Statistically

The 5 W’s of Data Analysis


VISUAL INSTEAD OF STATISTICAL ANALYSIS


• It is relatively easy to learn• Most common program (Excel) is readily available • Easy to disseminate final products • Helps show the patterns without needing to read a novel• Makes data manipulation more efficient to get to the

intended goal• Provides a clear communication tool (for other staff,

participants, families)

Why do we use visual analysis?

11


• Technically, yes…BUT• Beyond formal research studies or analyzing “big data” it is

not something necessary to make informed decisions and plan for next steps

• Stats involves an advanced skill set (and many sleepless nights) and are tricky to interpret

• Advanced computer software often necessary• In our daily work, statistical analysis is not necessary and not

a good use of our time on a daily basis.

Wouldn’t statistical analysis make for a stronger argument?

Correlation does not equal causation!


• Three characteristics of graphed data used to analyze and summarize information:

a) Level

b) Slope

c) Variability

Guidelines For Visual Analysis

14


• Refers to the average level of performance across a period of time.

• Often use the mean or arithmetic average to reflect this value, but we can also use median performance.

– Median is especially important when there is an unusual outlier in your data (for example, an individual has a significant life event that greatly impacts their performance on a single day).

• Important to review any changes that occur immediately after a program or strategy is implemented or modified

– Does it produce an immediate step (up or down) in behavior or skill?

Level

15

>www.dpw.state.pa.us >www.dhs.pa.gov 160 2 4 6 8 12 14 16 181020

30

40

50

60

70

80

90

100

Successive Days

Example: Level

The difference betweenthese two lines is thechange in level.

16


• Refers to trend over time (rate of change)• Many ways to estimate and draw trend from a data set.

– it takes at least 3 and usually 6 data points to establish a trend that is meaningful.

– Not necessary to get too concerned about single days of performance that are unusual from other data points.

– In Excel it’s as simple and clicking a button to “add trend line”

• How fast is performance changing over time?• Look at the graph and judge the general direction in which a

participant’s performance is: – Increasing (accelerating), Decreasing (decelerating) or Staying the same

Slope

17

>www.dpw.state.pa.us >www.dhs.pa.gov 18

Example: Slope/Trend

0

2

4

6

8

10

12

14

16

18

1 2 3 4 5 6

Sessions

# of

exc

essi

ve v

erba

lizat

ions

18


• This refers to the “bounce” (up and down movement) in the data to reveal stability or consistency). Too much bounce may be a concern

• Causes of bounce may include – personal characteristics of the individual (i.e., the “bad day”), changes in difficulty in the

activities, changes in how the data were collected (I.e., yesterday the staff collected data during the small group activity, today individually), and a host of other unknown factors.

– We expect a reasonable amount of “bounce” in the data but if the variability becomes too great, we need to try to understand WHY.

– We might need to examine environmental factors, and other variables as to what is causing the problem. Regardless, once we determine there is too much bounce in the data we need to put things in place that can control the bounce.

• Scan the data and make judgment about the degree of variability, ranging from high to low.

Variability

20

21

0 2 4 6 8 12 14 1610

10

20

3040

50

Successive Days

Occ

urre

nce

Example: Variability

60

70

80

90


Beyond Level, Slope, and Variability

• Average (across time• Count (whole number or percentage)• Themes, patterns across qualitative data• Compare against previous graphs on the same variables• Drill down the data further; Ask others for insight • Geographical Setting/Location patterns


EXAMPLES

0

2

4

6

8

10

12

14

16

18

20

22

6-May 8-May 10-May 13-May 15-May 3-Jun 5-Jun 7-Jun 12-Jun 14-Jun 21-Jun 26-Jun 28-Jun

# ti

mes

eac

h be

havi

or o

ccur

red

Dates Behaviors Occurred at Work

Jon- Hitting and Yelling Behaviors at Work- June, 2013Hitting Yelling

Baseline BSP Implementation-

0

10

20

30

40

50

6-May 8-May 10-May 13-May 15-May 3-Jun 5-Jun 7-Jun 12-Jun 14-Jun 21-Jun 26-Jun 28-Jun

Dur

atio

n of

beh

avio

ral o

ccur

renc

es (m

inut

es)

Dates Behaviors Occurred

Jon- Duration of Behavioral Occurrences at Work- June, 2013

Baseline BSP Implementation

0

2

4

6

8

10

12

14

16

18

# ti

mes

eac

h be

havi

or o

ccur

red

Dates Behaviors Occurred at Work

Jon- Hitting and Yelling Behaviors at Work- July and August, 2013Hitting Yelling

June July August

>www.dpw.state.pa.us >www.dhs.pa.gov 28

0

5

10

15

20

25

30

35

Antecedents Behaviors Consequences

Freq

uenc

y of

Occ

urre

nce

George- Observation Summary of Data Collection at Day Program


USING DATA DURING TEAM MEETINGS


• Take time to learn baseline: You can’t decide where to go next if you don’t know

where you first came from. It may not always be clear how someone is doing

right now until the data is presented Behaviors occur “all the time” should be a red flag to

dig deep

• Share results (or partial results) as soon as possible• Ask open-ended questions• Don’t forget about the indirect data!

Sharing the Results

31


• Revisit the intent/purpose- Why are you here?• What is important to the behavior?• Are you capturing setting events or other factors that play a

role in that person’s day (e.g. sleep, medication refusals, eating habits, daily schedule, other people)

• Can simple coding be done to eliminate writing and not take up as much space

• What are others already collecting (e.g. sleep patterns, incidents, call-logs with family, medications)

• What do you think happened here? Why? Is that typical?

Questions to Engage the Team

32


• Share small pieces of data to evoke conversation amongst team members, without YOU telling the whole story

• Plant the seed, then take notes of the dialogue• Engage team ongoing, not at the end• “Slice the data” in real time (advanced skill- can be done

using pivot tables)• Link areas of need to goals (current or new)

Additional Strategies to Encourage Team Collaboration


EXAMPLES

What questions would you ask?

35

What about now?

36

What if this was the first graph shared with the team?

37

Would this be helpful?

38

0

2

4

6

8

10

12

14

16

18

20

Person 1 Person 2 Person 3 Person 4 Person 5 Average

MAS

Fre

quen

cy S

core

sGeorge- Negative Verbalization Behavior at Day Program-

Motivational Assessment Summary data across staff

Sensory Escape Attention Tangible


USING DATA TO MAKE INFORMED, INSTRUCTIONAL DECISIONS

Data Analysis Decision

Progress-Trend is going up

Make no changes. Continue instruction.If trend is very steep then consider making instruction more difficult to facilitate greater learning.

Mastery-Met criterion

Develop a new plan to maintain and extend performance (e.g., fluency, generalization)

Minimal Progress-Shallow trend-Error patterns

Change Instruction-THINK!Change prompting strategy-Address motivation-Simplify or modify instruction for problem steps/skills-Teach a more simple version of the skill

No Progress-No improvement since baseline-No independent correct responses (may vary depending upon prompting strategy)-No improvement since the last data decision

Change Instruction-THINK!-Simplify skill-Change prompting strategyContinue Instruction and Observe-When progress in other aspects of instruction is noted (e.g., progress within the prompting system).

Regression-Trend is going down

Change Instruction-THINK!What could be the problem?-motivation (e.g., hate you, hate skill, skill not functional or meaningful )

Progress

Progress is being made as the time engaged in activity is going up.

Instructional decision: continue instruction.

Mastery Fluency – speed or ease with which skill is performedSome strategies for promoting fluency:- manipulate reinforcement (e.g., reinforce when dishwasher is loaded versus reinforcing each correct step of the task analysis)-load dishwasher after every meal instead of just after dinner

Generalization- being able to perform the skill in situations other than the “training” situationSome examples:-load dishwasher at volunteer site (senior center)-load dishwasher on Sundays after having lunch with grandparents

Minimal Progress Instructional Decisions:

• Change prompting strategy to include a direct verbal prompt since progress is not being noted using an indirect verbal prompt.

• Errors consistently occurring on steps 6, 7, and 9

• Consider if motivation is a factor and where it can be increased

No ProgressInstructional Decisions:

• Select different activities if lack of interest/motivation is problematic

• Chose a simpler activity if it is too difficult

• Change person who does the activity with participant

RegressionInstructional Decisions:

• Revise the task analysis to simplify, currently too many steps. Decreased steps from 15 to 10.

• Participant expressed dislike of meals currently making, increase motivation by allowing participant to choose what meals to make


• Where should supports be targeted?• When is the best time to teach the skill? And, under what

conditions?• Does there need to be a staffing change?• How can we arrange the environment based upon what is

working (or not) right now• Are there medical underpinnings that first need to be

addressed?• If unclear, is there a need for more data? Are we missing the

perspective of a person who may have better insight?

Additional Informed Decisions To Consider


FINAL THOUGHTS


• Collecting data is good, but doing something with it is necessary

• You don’t need to be a statistician to analyze data (but you should have a few tricks up your sleeve to dig deep into the data)

• Don’t get discouraged by inconclusive results- go back and collect more until the picture is clear

• Sharing analyzed data with teams and opening up the discussion will yield powerful insight you may have otherwise missed

• Thinking about the analysis from the beginning is key

Final Thoughts

We’ve only just begun…


Thank you for joining the Data Series!

Getting the Most out of Data Collection and Analysis: Part ...€¦ · PART 1 RECAP. www ......

Documents

Transcript of Getting the Most out of Data Collection and Analysis: Part ...€¦ · PART 1 RECAP. www ......