Pilot Project Increasing the Use of Large Scale Surveys in Undergraduate Dissertations in the Social...

Pilot ProjectPilot ProjectIncreasing the Use of Large Scale Increasing the Use of Large Scale

Surveys in Undergraduate Surveys in Undergraduate Dissertations in the Social Dissertations in the Social

SciencesSciences

Mark Brown and Jo WathanMark Brown and Jo WathanSchool of Social Science School of Social Science University of ManchesterUniversity of Manchester

[email protected]@[email protected]@manchester.ac.uk

What we’re presenting today:What we’re presenting today:

Motivation and rationale for the projectMotivation and rationale for the project

About the pilot About the pilot

Taking it forward: from pilot to credit-Taking it forward: from pilot to credit-bearing course bearing course

ContextContext

‘‘British students are usually encouraged to collect their British students are usually encouraged to collect their own data for final year undergraduate projects…This own data for final year undergraduate projects…This contrasts with the USA where most sociology students contrasts with the USA where most sociology students conduct secondary analysis of large-scale survey data… conduct secondary analysis of large-scale survey data… The lack of use of secondary analysis in Britain is The lack of use of secondary analysis in Britain is surprising given that such a large number of high quality surprising given that such a large number of high quality national surveys are readily available and that expertise national surveys are readily available and that expertise in the analysis and data management of large surveys is in the analysis and data management of large surveys is in great demand by employers.’in great demand by employers.’

Sara Arber in Nigel Gilbert (Sara Arber in Nigel Gilbert (2001)Researching Social Life2001)Researching Social Life

What are the constraints?What are the constraints?

Lack of confidence in quantitative skillsLack of confidence in quantitative skills

Perceived lack of relevance/interest of Perceived lack of relevance/interest of quantitative data/analysis quantitative data/analysis

Lack of knowledge about the resources Lack of knowledge about the resources available available

Reason for OptimismReason for Optimism

There is a vast amount of relevant social data on topics There is a vast amount of relevant social data on topics sociology students are interested in.sociology students are interested in.

It’s become much more accessible and user friendly – It’s become much more accessible and user friendly – teaching datasets, user-guides, workbooksteaching datasets, user-guides, workbooks

Students are increasingly aware of the value of attaining Students are increasingly aware of the value of attaining quant skills for their CVs quant skills for their CVs

The numerical skills required are below those needed for The numerical skills required are below those needed for GCSE maths GCSE maths

Let the resources speak for Let the resources speak for themselvesthemselves

Our conviction …Our conviction …

If… students knew what was available (through ESDS), and could engage If… students knew what was available (through ESDS), and could engage hands-on with the materialshands-on with the materials

then… resources would sell themselves. Students would enjoy the learning then… resources would sell themselves. Students would enjoy the learning process, and want to do more. process, and want to do more.

That learning and teaching data analysis (SPSS training) is enhanced if That learning and teaching data analysis (SPSS training) is enhanced if students see it as part of the research process that starts with a research students see it as part of the research process that starts with a research questionquestion

ESDS (and related resources) serve learning and teaching in all parts of that ESDS (and related resources) serve learning and teaching in all parts of that process: process:

Searching for datasetsSearching for datasets Data evaluation: is it fit for purpose? Data evaluation: is it fit for purpose? How to analyse data.How to analyse data.

The pilot projectThe pilot project

Jo ….Jo ….

The pilotThe pilotTo increase use of secondary analysis in To increase use of secondary analysis in dissertationsdissertations

Take on <20 sociology 2Take on <20 sociology 2ndnd years in yr 2 years in yr 2 Provide basic training in group settingProvide basic training in group setting For those continuing to a dissertation one to For those continuing to a dissertation one to

one supportone support

Production of 3 topic workbooks for Production of 3 topic workbooks for dissemination through ESDS Governmentdissemination through ESDS Government

External evaluationExternal evaluation

Pathways through the projectPathways through the project

Yr 2

Sum

mer vac

Xm

as

W1 W2 W3 W4 W5

WORKSHOPS

AllDissertation based on secondary analysis

£100

Yr 3: Sem 1 Yr 3: Sem 2

Research training module

Dissertation project

proposal

DROP IN SUPPORT

+£100 when submitted

(+ enter for £500 prize)

Dissertation deadline

Intake Intake What was the highest qualification you gained in math’s

at school?

GCSE (or equivalent) 13

Grade A3

Grade B8

Grade C1

Grade D1

A level (or equivalent 1

Grade A0

Grade B 0

Grade C1

Grade D0

Agree

Disagree

Not sure

I lack the knowledge about the datasets available to me 8 3 3

I lack the SPSS skills required 6 3 5

I lack the statistical skills and understanding required 7 2 5

I would struggle with the interpretation of the output 6 4 4

I do not find survey analysis interesting 2 5 7

Survey analysis will not answer my questions of interest 2 4 8

Topics and data for dissertation Topics and data for dissertation proposalsproposals

Class homogamyClass homogamy SARsSARs

Attitudes to crimeAttitudes to crime BCSBCS

Background to understanding gender Background to understanding gender differences in marketsdifferences in markets

LFSLFS

Sexual health (esp. young)Sexual health (esp. young) Sexual attitudes Sexual attitudes and behaviourand behaviour

Intergenerational differences in attitudesIntergenerational differences in attitudes BSAS /YPSASBSAS /YPSAS

Relationship between poverty and crimeRelationship between poverty and crime BCSBCS

Background to study into the girl guide Background to study into the girl guide movement movement

Young people Young people social attitudessocial attitudes

WorkshopsWorkshops

Year 2: Research benefit of secondary analysis Year 2: Research benefit of secondary analysis & using online data – one off& using online data – one off

Then in year 3, fortnightly 3 hour sessions from Then in year 3, fortnightly 3 hour sessions from 28/9. Mixture of presentation, group discussion 28/9. Mixture of presentation, group discussion and hands-on. Based on BCS exemplar and hands-on. Based on BCS exemplar workbooks. workbooks. Year 3/1:Thinking about your research question Year 3/1:Thinking about your research question & locating data in ESDS& locating data in ESDSYear 3/2:frequencies and bar charts, weightingYear 3/2:frequencies and bar charts, weightingYear 3/3:recodes and select if Year 3/3:recodes and select if Year 3/4: crosstabs, continuous data and Year 3/4: crosstabs, continuous data and reportingreporting

WorkbooksWorkbooksAvailable at Available at

http://course-data.ccsr.ac.uk/sasshttp://course-data.ccsr.ac.uk/sass

Based on British Crime Based on British Crime Survey teaching dataset Survey teaching dataset 20022002

Chosen after consulting Chosen after consulting on topics of intereston topics of interest

AttendanceAttendance•14 original participants

•Lowest participation in workshop 11 (plus apology)

•9 attended all 5 workshops

•Biggest drop at deadline for submitting dissertation proposal

Did they the workshops useful?Did they the workshops useful?How useful did you find the workshops?

0

1

2

3

4

5

6

7

8

very useful of some use not useful did not answer

co

un

t

Will they be using secondary analysis Will they be using secondary analysis in their dissertations?in their dissertations?

Using in your dissertation?

0

1

2

3

4

5

6

7

Yes:mainsource

Yes:mixedmethods

Yes:smallcomponent

No No answer

co

un

t

Before and after affectsBefore and after affects

Subjective changeSubjective change

Conducted Conducted anonymously anonymously More honest?More honest? Not possible to look at Not possible to look at

individual changeindividual change

Confidence measured Confidence measured on scale out of 5:on scale out of 5:

5 =very able to5 =very able to

44

33

22

1 = unable to1 = unable to

Locating dataLocating data

Confidence locating data

0

2

4

6

8

10

12

14

16

locate data before locate data after

co

un

t

5

4

3

2

1

Evaluating the suitability of a survey datasetEvaluating the suitability of a survey datasetConfidence: evaluating a dataset

0

2

4

6

8

10

12

14

16

evaluate data before evaluate data after

co

un

t

5

4

3

2

1

Open SPSS and run some frequency Open SPSS and run some frequency tables and graphs for single variablestables and graphs for single variables

Freq/graphs in SPSS

0

2

4

6

8

10

12

14

16

spss freq/grph before spss freq/grph after

co

un

t

5

4

3

2

1

Summary statistics (mean, median etc...)Summary statistics (mean, median etc...)

Summary statistics

0

2

4

6

8

10

12

14

16

sum stats before sum stats after

co

un

t

5

4

3

2

1

SubsettingSubsettingSubsetting

0

2

4

6

8

10

12

14

16

select before select after

co

un

t

5

4

3

2

1

Carrying out a simple recodeCarrying out a simple recodeRecoding

0

2

4

6

8

10

12

14

16

recode before recode after

co

un

t

5

4

3

2

1

Interpret a crosstabInterpret a crosstabInterpret a crosstab

0

2

4

6

8

10

12

14

16

before after

co

un

t

5

4

3

2

1

Did it work?Did it work?

High level of enthusiasm amongst studentsHigh level of enthusiasm amongst studentsStudents focused on potential for their Students focused on potential for their dissertationsdissertationsMonetary reward became secondary for most Monetary reward became secondary for most but not allbut not allStudents found it helpful and improved their Students found it helpful and improved their skillsskillsRequired one-to-one supportRequired one-to-one supportFormat not idealFormat not idealFive dissertations submitted with a secondary Five dissertations submitted with a secondary analysis componentanalysis component

Back to MarkBack to Mark

Building from the pilotBuilding from the pilot

Recognition that pilot model (with financial Recognition that pilot model (with financial incentive) is unsustainableincentive) is unsustainable

Major objective was to bring training within Major objective was to bring training within the formal curriculum – develop a credit-the formal curriculum – develop a credit-bearing coursebearing course

Incorporating within the curriculumIncorporating within the curriculumManchester Sociology (BSocSc)Manchester Sociology (BSocSc)

Research DesignResearch Design

Secondary Analysis of Secondary Analysis of Social SurveysSocial Surveys

Qualitative MethodsQualitative Methods Quantitative MethodsQuantitative Methods

Life Stories and Life Stories and Biographical researchBiographical research

or

compulsory

Year 2 Teaching in Research Methods

Course content: Course content: From research question to data analysis. From research question to data analysis.

Built around the resources of ESDSBuilt around the resources of ESDSTOPICTOPIC PRACTICALPRACTICAL RESOURCE FOR RESOURCE FOR

PRACTICALPRACTICAL

11 Survey data: a Survey data: a goldmine for Social goldmine for Social Scientists Scientists

What data is out there What data is out there on my topic? on my topic?

ESDS Data SearchESDS Data Search

Data ArchiveData Archive

Question BankQuestion Bank

22 The right question the The right question the right dataright data

Is it fit for purpose? Is it fit for purpose? evaluating a survey evaluating a survey datasetdataset

ESDS data pagesESDS data pages

Survey home pagesSurvey home pages

NESTARNESTAR

33 Analysing data 1Analysing data 1 First steps in data First steps in data analysisanalysis

ESDS Teaching Dataset ESDS Teaching Dataset (BCS 2002) in SPSS(BCS 2002) in SPSS

Exercises available in Exercises available in themed workbooksthemed workbooks

44 Analysing data 2Analysing data 2 Filtering and recodingFiltering and recoding

55 Analysing data 3Analysing data 3 Controls and statistical Controls and statistical significance significance

AssessmentAssessment Part 1: Finding and evaluating data for a research Part 1: Finding and evaluating data for a research

question of choicequestion of choiceTOPICTOPIC PRACTICALPRACTICAL RESOURCE FOR RESOURCE FOR

PRACTICALPRACTICAL










NESTARNESTAR






AssessmentAssessment Part 2: Data AnalysisPart 2: Data Analysis

TOPICTOPIC PRACTICALPRACTICAL RESOURCE FOR RESOURCE FOR PRACTICALPRACTICAL










NESTARNESTAR






Encouraging statisticsEncouraging statistics

First year: 6 studentsFirst year: 6 students

This year: 21 students (third of compulsory This year: 21 students (third of compulsory class) class)

Positive student feedback – enjoyment as well Positive student feedback – enjoyment as well as attainmentas attainment

Will monitor to see if year 2 interest converts to a Will monitor to see if year 2 interest converts to a quantitative dissertation in year 3.quantitative dissertation in year 3.

Student feedbackStudent feedback

AgreeAgree Mostly Mostly AgreeAgree

Neither Neither Agree or Agree or DisagreeDisagree

Mostly Mostly disagreedisagree

DisagreeDisagree

The material was The material was intellectually stimulatingintellectually stimulating

1010 77 -- -- --

On-line material On-line material significantly enhanced significantly enhanced my learning my learning

1515 22 -- -- --

The study materials were The study materials were helpfulhelpful

1414 22 11 -- --

In conclusionIn conclusionExcellent resources now exist to support methods teachers who Excellent resources now exist to support methods teachers who want to introduce more hands-on learning with real data (for want to introduce more hands-on learning with real data (for both classroom and self-study). both classroom and self-study).

Materials easily customised to meet needs of different classes Materials easily customised to meet needs of different classes (by level and topic) (by level and topic)

Our experience suggests we should be less defensive about Our experience suggests we should be less defensive about selling quantitative methods to students: selling quantitative methods to students: Challenge the stereotype that sociology students must find it Challenge the stereotype that sociology students must find it

boring, irrelevant, difficult boring, irrelevant, difficult When learning is enriched by engagement with ESDS When learning is enriched by engagement with ESDS

resources and real survey data, sociology students show resources and real survey data, sociology students show interest and confidence in considering a dissertation based interest and confidence in considering a dissertation based on the secondary analysis of survey data.on the secondary analysis of survey data.

Pilot Project Increasing the Use of Large Scale Surveys in Undergraduate Dissertations in the Social...

Documents

Transcript of Pilot Project Increasing the Use of Large Scale Surveys in Undergraduate Dissertations in the Social...