Basic Data Collection and Analysis

TAU Breakfast Session 14 April 2011

Fia van Rensburg

1

Purpose • To highlight aspects with regard to monitoring, data

collection and data analysis that are relevant to TAU on an operational level

• To build capacity of support staff within TAU to perform basic monitoring, data collection and data analysis functions

• To encourage TAU support staff to request assistance form the KM unit to advise on monitoring, data collection and data analysis

2

What this presentation is not about

• This presentation does not give detailed information on RBM

• It is not a technical lecture on research methodology and statistics

3

What is happening here?

4

What is happening in the picture?

• We can see different things in this picture: how many children; how many girls and boys; colours of their clothes; colours of the sacks; one fell; how many have long hair, short hair; they are indoors etc.

• This means that in any situation there is a lot of potential data that can be collected, but we are not going to collect all – we need to focus on what it is that we need, what is important to monitor. We need to decide up front what it is that we need to know in order to measure performance and inform decisions.

• The caption of the picture was: “5 out of 6 children like sack-racing”• This statement can be questioned, because it is based on an assumption that the child that

fell does not like sack racing. This may not be true. Therefore, we need to think very carefully of our interpretation of the data.

LEARNING POINT: Data collection can be simple, but be clear on what is data and what is interpretation.

5

Why do we collect data? • To get feedback on how we are doing in relation to the results

we want to achieve (e.g. performance monitoring) • To know where we can improve our services (e.g. customer

satisfaction) • To find out how and where we can work more cost-effectively

(e.g. cost-benefit analysis) • To learn from our experiences and to inform our future planning

(e.g. programme evaluation) • What is it that we don’t we know, that if we know it, will make a

difference if we know it?

6

TAU Context • Building capacity of our clients to deliver services to citizens• How is capacity defined? (individual, organisational,

environmental)• Focus on capacity in: OD, PPM, Strategic Planning, SCM and

Finance Management etc. • Capacity is built through process consulting and sometimes

workshops or capacity building sessions • What is our theory of change? (strategy, and logic model linking

inputs>outputs>immediate, intermediate, ultimate outcomes – Results Based Management)

7

Let’s pause for a moment

8

http://www.helentoons.com/2009/06/where-is-pause-button.html

Background on Results-Based Management

• Before we look at data management on capacity building, we need to take note of how Results Based Management (RBM) works.

• The next four slides (with recognition to Peter Brook – TAU) gives an overview of: – basic terminology; – the results chain; – levels of measurement on the logic model;– indicator definition; and – baselines, targets and measures

9

10

Activities, Outputs and Outcomes are the building blocks of Results Based Management

Activitieswhat we do

Describes a collection of functions (actions, jobs, tasks) that consume inputs/resources required to produce outputs

Outputswhat we produce

The direct products and services generated through processes or activities.

Outcomeswhat we wish to achieve

The effects, benefits or consequences that occur (either in the short, intermediate, or long-term) due to the produced outputs

Every programme undertakes activities that produce outputs that contribute to the achievement of outcomes.

11

How do activities, outputs and outcomes link?

Single Results Chain

Results Hierarchy

Outcome

Output

Activity

Input

12

FINAL OUTCOME

INTERMEDIATEOUTCOMES

OUTPUTS

ACTIVITIES

All levels of the Logic Model should be measured

INPUTS

IMMEDIATE OUTCOMES

Indicators

Indicators

Indicators

Indicators

Indicators

Indicators

Indicator measurement has four different elements

Indicator definition

Baselines

Targets

Measures

1

2

3

4

• The first measurement of an indicator• For comparing later measurements with• For setting targets against

• The score we would like to achieve• Must be time-bound, challenging and achievable

• Actual score for each measurement• Must be measured regularly, so that trends can be established

• Identifies what should be measured, over what time period, and in what units, by who, how frequently, but does not contain a score or a target

Baselines, Targets and Measure are all expressed in the same units

INDICATOR Baseline(date)

2009 Target

2009 Measure

Number of subsidies paid for solar water heaters per annum

5640 (in 2008)

12 000 12 345

Mortality rate of children under 5 years of age

565 per 10 000 births (in 2005)

400 per 10 000 births

385 per 10 000 births

Percentage of learners passing mathematics in Matric

43% (in 2002)

72% 65%

So…now that we have refreshed our memories on RBM…what is the

Programme logic/Theory of Change related to Capacity Development in

TAU?

15

http://www.iconspedia.com/icon/button-refresh-3374.html

Chain of events/levels of evidence Programme chain of events (Theory of Action)

Matching levels of evidence

End Results (Outcomes)

7 Measures of impact on overall problem, ultimate goals, side effects, social and economic consequences

Practice and behaviour change

6 Measures of adoption of new practices and behaviour over time

Knowledge, skill and attitude changes

5 Measures of individual and group changes in knowledge, skills, attitudes and learning

Reactions 4 What participants and clients say about the programme, satisfaction, interest, strengths, weaknesses

Participation 3 Characteristics of Programme participants and clients, numbers, nature of involvement, background

Activities 2 Implementation data on what the programme actually offers or does

Inputs 1 Resources expended, number and types of staff involved, time extended

(Adapted from Patton, 1997: p 235)

16

How do we know we are successful with capacity building?

17

•Ultimate: Behaviour change – actual application and change in systems, practice – CSS, MTR, Evaluation

•Intermediate: Knowledge skills and attitude•Immediate: Reaction to presentation content,

delivery and format Outcome Indicators will

relate toREACTION;

KNOWLEDGE, SKILLS AND

ATTITUDE; and BEHAVIOUR CHANGE

•Number of workshops/capacity building sessions held

•Number of participants attending•Sphere(s) of government covered •Number of departments provinces attending•Level of participants

Output Indicators will relate to

PARTICIPATION

How do we measure the indicators?

Indicator Data Data source

Number of workshops held

Simple count, dates, topics and themes covered

Attendance registers and feedback forms

Number of participants attending

Simple count, disaggregated i.t.o. sex, sphere of government, department/province, level/ designation of participants

Attendance registers and feedback forms

Reaction to presentation

Participants’ rating of content, relevance delivery and format , overall satisfaction

Feedback forms

Extent to which knowledge skills and attitudes have improved

Participants’ rating of improvement in knowledge, skills and personal effectivenessComparative post-intervention assessment against baseline/ stated project outcomes

Feedback forms - also Project Closure Reports, CSS, MTR

Behaviour change over time

Self-reported or others’ reports/observation providing evidence of change in relevant behaviour (comparison pre-and post), level of implementation of proposals, systems, practice in relation to stated objectives

Feedback forms (related to a series of capacity building initiatives implemented over time), SS, MTR, Evaluation

18

How do we collect the data? • Hard copy/electronic questionnaire• Telephonic, face-to-face or focus group interviews • When and how to administer questionnaires/use

interview guides • Questionnaires:

– How many questions? (Not too many, but cover the indicators)

– Open-ended or fixed response categories– Options for fixed responses - even or uneven number,

ranking scales, naming– The rhythm of the questionnaire

19

How do we capture and analyse and present the data?

Quantitative data

• Use MS Excel • Draw up a framework and

decide on codification (remember o response, not applicable)

• Descriptive statistics• Graphs

Qualitative data

• Capture in exact wording • Thematic analysis• Summary

20

…and we need to pause again…to consider a few technicalities…

21

http://www.pvtmurphy.com/Prints/Pause%20Button.htm

What level of measurement/types of data are we working with?

• Nominal: Lowest, least precise level of measurement. Different types of categories of a variable (e.g. colour: red or blue or yellow/sex: male or female/religion: protestant/catholic). Categories are mutually exclusive, not ordered.

• Ordinal: Identifies difference among categories available, and allows categories to be rank ordered (e.g.: level of education: primary, secondary, tertiary)

• Interval: Categories are ranked, and distance between categories is measured. No true zero (e.g. IQ)

• Ratio: Most precise level, rank ordered, distance is precisely measured, and there is an absolute zero (e.g. age).

22

Please note - the level of measurement determines what calculations we can

do with the data…

• The mode is the most frequent score. Can be use for all levels of measurement. (Most

• The median is the middle point – half of cases are above this point and the other half below. Can be used for ordinal, interval and ratio data.

• The mean is the average. Can be used only when we have interval and ratio data.

23

What you can and cannot do…

24

OK TO COMPUTE....

NOMINAL ORDINAL INTERVAL RATIO

frequency distribution Yes Yes Yes Yesmedian and percentiles No Yes Yes Yesadd or subtract No No Yes Yesmean, standard deviation, standard error of the mean

No No Yes Yesratio, or coefficient of variation

No No No Yes

 

How can we present the data analysis?

The most common ways of presenting analysed data are: – Pie charts– Bar graphs – Stacked bar graphs– Line charts

25

Pie Charts• Circles divided into a

number of slices, each slice representing the relative proportion data points falling into a given category.

• Suitable for graphing nominal data and percentages.

26

http://www.mathleague.com/help/data/data.htm

Pie Charts Example: Attendance of TAU Breakfast Session 9

September 2010

7%

71%

14% 7%

Attendance per Unit in TAU

BATASKMOCD

27

Male 38%

Fe-male62%

Gender Breakdown

Bar Graphs • Used to show umber of proportion of

nominal or ordinal data according to a specific attribute (nominal data).

• Often shows # or % of observations per data points

• Switching rows and columns gives different perspectives (see next slide) –it is important to decide if you want to focus on the project phase or what the picture is per portfolio (see example in next slide)

28

http://www.webdonuts.com/2009/08/the-presentation

Bar Graphs Projects per Project Phase

(fictional data)

29

ED &IR IRD G&A JCPS SS0

5

10

15

20

25

30

35

40

45

PipelineApprovedCompletedWithdrawnCurrent

Pipeline Approved CompletedWithdrawn Current 0

5

10

15

20

25

30

35

40

45

ED &IRIRDG&AJCPSSS

Project phase per Portfolio(fictional data)

Stacked Bar Graphs • Can be used for representing multiple categories of

data in a single bar, and each bar is made up of sections that represents the proportion of data falling into a category.

• Can be used to show comparative contributions of different groups or ratings to a total.

• Use the switch rows and columns function to see which display makes sense – in the example in the next slide Participant Feedback (B) may be better to use.

30

Stacked Bar Graphs Participant

Feedback (A)

Signific

ant

Some e

xtent

Limite

d exte

nt

Not at

all

No res

pons

e0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Facilication skills Expectations met Presenter Response Usability of Knowledge Enhanced Un-derstanding

31

Enhan

ced U

nders

tandin

g

Usabil

ity of

Kno

wledge

Presen

ter R

espo

nse

Expec

tation

s met

Facilic

ation

skills

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

No responseNot at allLimited extentSome extentSignificant

Participant Feedback (B)

Line Graphs• A single line is used to connect data points (interval

and nominal data) • Used to present progress over time and trends• More than one variable can be presented on a single

chart – but check that it works. See examples on next slide: The presentation of # of website visits, together with page views per visit and # of seconds per page view does not work. The presentation of Male/Female breakdown over time works well.

• 32

Line Graphs• A single line is used to

connect data points (interval and nominal data)

• Used to present progress over time and trends

http://www.netpaths.net/blog/

33

Line Chart

Website report

Jan

March

May

July

Sept

Nov0

100

200

300

400

500

600

700

800

Website visitsPage views per visit Seconds per page

Staffing Table - Gender Breakdown per Quarter

34

Q1 Q2 Q3 Q40

5

10

15

20

25

30

35

40

45

50

MaleFemale

Qualitative Data • Qualitative data in questionnaires is mostly gathered

through open-ended questions • In the next slide a basic example is given of how

qualitative data can be analysed: (1) Capture the text; (2) Group related themes/comments together; (3) count; (4) analyse; and (5) present the analysis in an appropriate format.

• NB Qualitative date provides more in-depth information, but if applicable, it can be counted, as in the example on the next slide

35

Qualitative data

Future PMIG sessions should focus on….Verbatim • The project charter • How to set up a project office • Stakeholder management • Communication in projects• How to manage a Project

office • Project Office

AnalysisThree priorities were identified for

future PMIG sessions: • Project office set-up and

management (50%)• Communication and

stakeholder management (33.3%)

• Project Charter (16.7%)

36

You can do it• It is not as difficult as we sometimes think to

analyse data, and to represent it in graphs• There are, however a few points of caution

and if you are not sure about something, rather consult a colleague that has a deeper understanding of data analysis.

37

It’s easy?

38

Really?• Our potential President of the Math Club

does not really qualify for this position – he has seriously under-estimated his support.

• Why? …see the next slide

39

Are you sure?   Total Support Percentage

Boys 100 21 21%

Girls 100 30 30%

Total 200 51 51/200= 26%

Boys 300 63 21%

Girls 100 30 30%

Total 400 93 93/400=23%

Boys 100 21 21%

Girls 200 60 30%

Total 300 81 81/300=27%

40

What happened here?• This is a very common mistake• Percentages cannot be added up as percentages• A percentage represents a proportion of a universe, and the

size of the universe needs to be taken into account. • To calculate his support base correctly, our clever friend had

calculate the total number of people that supported him in relation to the total number of boys an girls.

LEARNING POINT: We need to know what calculations we can do with certain types of data. Always double-check. (see notes on types of data)

41

Thank You References: http://graphpad.com.faq.cfm?faq=1089http://www.blueprintusability.com/topics/articlelevels.htmlhttp://highered.mcgraw-hill.com/sites/0073521426/student_view0/ebook16/chapter1/chbody1/levels_of_measurement.htmlhttp://www.socialresearchmethods.net/kb/measlevl.phphttp://courses.csusm.edu/soc201kb/levelofmeasurementrefresher.htmhttp://cnx.org/content/m10809/latest/http://www.usablestats.com/lessons/noirhttp://www.thinkoutsidetheslide.com/articles/using_graphs_and_tables.htmhttp://www.mathleague.com/help/data/data.htmhttp://www.42explore.com/graphs.htmhttp://changingminds.org/explanations/research/measurement/types_data.htm

42