Ten D by Night conference Turin,

19 February 2010

THE RESULTS:

QUALITATIVE EVALUATION

Anita Villerusa, Associate Professor

Dean, Faculty of Public health

Riga Stradins University

Ten D by Night conference Turin,

19 February 2010

Ten D by Night conference Turin,

19 February 2010

What is evaluation?

• Evaluation is collecting data about an intervention and its effects, in order to judge the value of the intervention, through making a comparison, so as to make more informed decisions.

Øvretveit, 1998.

• Program evaluations aim to provide convincing evidence that a program is effective. The standards are specific criteria by which effectiveness is measured.

Fink, 1993.

Ten D by Night conference Turin,

19 February 2010

Essentials of evaluation

Evaluation is : to collect data,

to judge the value,

to make a comparison,

to make more informed decisions.

Audit evaluation – comparison what was intended (objectives) with what happened

Ten D by Night conference Turin,

19 February 2010

Project evaluation framework

Project start

Project processes

Project end

Goals/aims/objectives

Planning

Communication

Management

Training

Piloting

Field work

Outputs

Outcomes:

-New preventive approach

-Changes in alcohol/drug related accident rate

Ten D by Night conference Turin,

19 February 2010

The criteria for evaluating a health

programmeCorrespondence - if the action corresponded to objectives and

procedures?

Outcomes (Short term and long term)

Objective measures

Subjective (retrospective perception)

Efficiency - resources consumed and outputs produced

Acceptability

to assess target population and providers willingness to use and

apply the action

Sustainability

the continuation of the programme, if finance continues, or if

withdrawn

Ten D by Night conference Turin,

19 February 2010

Six standard criteria for programme

evaluations

1. What was done? (description)

2. Did all activities follow the plan and achieve the

objectives? (correspondence to plan and objectives)

3. What were the outcomes? (Did they meet needs of

target group?)

4. Was there any waste of resources, or could have used

resources more efficiently? (efficiency)

5. Did people participate in the project? (acceptability)

6. What threatens the continuation – with and without

continued funding? (sustainability)

Ten D by Night conference Turin,

19 February 2010

Methods for qualitative evaluation

• focus group discussion

• In- depth interviews

• questionnaire also with open- ended

questions for managers and for volunteers

• analyses of different documents

(Analyses of written minutes of meetings;

E- mail Correspondence; reporting tables;

project time table and financial plan, press

releseas, publications etc.

Ten D by Night conference Turin,

19 February 2010

Participants in the evaluation

Participants in the focus group discussion- 10 respondents of the

Ten D by Night Project participants from Italy took part in the focus

group; 3 were project managers and 7 were volunteers

In- depth interviews - Project technical coordinators- S&T Italy

Questionnaires – one questionnaire for managers and one for

volunteers (7 project managers and 22 volunteers completed the

questionnaires from al countries except Spain)

Ten D by Night conference Turin,

19 February 2010

Qualitative evaluation - Results

• Preparation for the work

Training of project workers/volunteers.

The equipment - alcohol tests, drug tests, console for measurement of the reaction time.

Project tools - questionnaires, leaflets, information/consent forms, posters advertising the project and action in particular places and times.

• Sampling recreation places (Selection of night clubs, their representativeness and how they correspond to the target audience)

• Field work

• Advertising the project and rising awareness among the local population.

Ten D by Night conference Turin,

19 February 2010

Involvement of volunteers in the

project

• Poland, Bulgaria, the Netherlands and

Belgium where the project partners were

NGOs they used people who they had

collaborated with in the previously

executed projects as volunteers

• Students

• the number of field workers

Ten D by Night conference Turin,

19 February 2010

Information about the project, training

and preparation for the field work

Project managers

• ~ one hour training;

• Some theoretical part

of training;

• Training on the

practical

implementation of the

tests

Volunteers

• A majority assessed

the level of

information as

sufficient

Ten D by Night conference Turin,

19 February 2010

Evaluation of information volunteers

received about the project, its mission,

objectives and tasks before field work

9

8

8

4

6

5

4

2

2

5

6

7

0% 20% 40% 60% 80% 100%

amount

type

quality

sufficient suff>insuff insuff>suff insufficient

Ten D by Night conference Turin,

19 February 2010

The level of skills regarding the

practicalities of the field work

16

8

10

11

11

1

4

3

5

6

6

2

2

2

5

4

3

4

3

0% 20% 40% 60% 80% 100%

alcohol t.

drug t.

console

communic.

questionn.

sufficient suff>insuff insuff>suff insufficient

Ten D by Night conference Turin,

19 February 2010

Selection of night clubs

• Different opinions

Some difficulties

Great number of rejections

in Italy

Less problems

Poland, Bulgaria and Belgium,

The Netherlands

Ten D by Night conference Turin,

19 February 2010

• The main problem was to choose the clubs . When they were chosen, there was no problem in gaining admission. In other words, we faced no problems”. (Poland, Bulgaria and Belgium )

• “Making appointments was difficult/the clubs and discos were poorly accessible/ it was sometimes difficult to start a discussion about drugs”

Ten D by Night conference Turin,

19 February 2010

Reasons for refusal

• luck of interest;

• unwillingness to collaborate;

• fear about control checks of people;

• possible negative effect on the number of

visitors.

Ten D by Night conference Turin,

19 February 2010

Representativeness and the

target audience

• Difficulties to reach representativeness

• the project succeeded in reaching the maximum number of participants who correspond to the target audience- young people aged from 18 to 24;

• attitudes of the customers during the field work - “Overall good even if some of them found that it took too much time to answer the questionnaire and to do the tests. Most of them liked it and found it interesting to do the tests; they cooperated well”.

Ten D by Night conference Turin,

19 February 2010

Field work

Ten D by Night conference Turin,

19 February 2010

The design and methodology of

the project

• The experts

acknowledged that

the design of the

project was

appropriate and

adequate.

• Volunteers-

“How well did you

understand your role

in the field work – the

tasks and duties to

undertake”. 73% said

very well and 27%

quite well.

Ten D by Night conference Turin,

19 February 2010

Evaluation of the suitability of

the tools

5

16

4

8

7

4

6

6

5

5

1

2

5

1

2

1

1

1

0% 20% 40% 60% 80% 100%

questionn.

alcohol t.

drug t.

console

very good good average bad very bad

Ten D by Night conference Turin,

19 February 2010

Field work

Positives• both the managers and volunteers were satisfied with

the process of the field work

• Good collaboration was established with night clubs

• the level of awareness of the necessity of preventive activities increased within the night clubs

• The informative materials provided proved to be useful and small incentives/presents helped to attract participants

• New and expensive equipment was bought

• It was possible to implement the project in several countries at the same time and compare the results

Ten D by Night conference Turin,

19 February 2010

Field work

Negatives

• In some clubs there was no space where to place questionnaires or it was not convenient to complete them

• There were queues to pass tests and some customers left because they did not want to queue.

• The drug tests took too much time

• the questionnaire was too long and some questions were misleading;

• In some cases there was not enough experience among the field workers how to deal with drunken people

Ten D by Night conference Turin,

19 February 2010

Ethical aspects

The anonymity

The system of anonymity worked very well and was

adequate for the needs of project.

no threats to volunteers

Ten D by Night conference Turin,

19 February 2010

Cultural and religious obstacles

and barriers

• No barriers were noticed.

• men showed a higher level of interest

regarding the intervention.

• there were many cases when women

refused to take part in intervention.

• although there were no restrictions on

gender, gender inequality was observed in

some countries.

Ten D by Night conference Turin,

19 February 2010

Publicity of the project

• radio interviews;

• publications in newspapers;

• the information in web-sitewww.tendbynight.com

• posters and brochures

• The experts interviewed gave 3,7 marks out of 5 for the publicity at national level and 3,5 marks at the international level.

Ten D by Night conference Turin,

19 February 2010

Evaluation of the suitability of the

project materials for field work

1

4

1

4

0

6

8

8

9

8

7

6

8

6

8

6

11

5

3

2

1

2

2

3

1

0% 20% 40% 60% 80% 100%

brochures

logo

posters

webpage

leaflets

gadgets

very good good average bad very bad

Ten D by Night conference Turin,

19 February 2010

Use of results

• for practical purposes

• for scientific and research purposes

• for constructing educational programmes

• The further development of the project

idea

Ten D by Night conference Turin,

19 February 2010

Meetings and collaboration

among project countries

• the meetings were useful and the amount

of them as optimal for the project

• Good collaboration

• Good communication via e-mail

Ten D by Night conference Turin,

19 February 2010

“This was a very good project. It

gives understanding to young people

regarding those things which are

important to their health and safety.

The project gave us the opportunity

to talk about these things … and also

opportunity to talk to each other.”

Ten D by Night conference Turin,

19 February 2010

Some final points

• Some changes in the staff of the project. .

• The decrease of the initial budget by 50

%...

• Some corrections in time table...

All the activities were implemented on time.

Ten D by Night conference Turin,

19 February 2010

Goodbye Turin!