Australasian journal of Economics Education 13, Number... · 1. Report research on the teaching of...

Australasian Journal of Economics Education

Volume 13 Number 1 March 2016

ARTICLES

A Report on the Development of Ross Guest

Learning Standards for Economics Allan P Layton

in Australia

The Impact of Lecture Capture on Chris Jones

Student Performance Matthew Olczak

Selecting Sessional Tutors: A New Carl W. Sherwood

and Effective Process Bruce Littleboy

BOOK REVIEW

Teaching Innovations in Economics Peter Docherty

M. K. Salemi & W. B. Walstad

ISSN 1448-448 X

Editorial Executive Co-Editor Professor Rod O’Donnell Telephone: (+61 2) 9514 7738 Email: [email protected] Co-Editor Dr Peter Docherty Telephone: (+61 2) 9514 7780 Email: [email protected] Co-Editor Mr Joseph Macri Telephone: (+61 2) 9850 6069 Email: [email protected]

Editorial Board Professor William J. Baumol, New York University, USA. Professor Harry Bloch, Curtin University of Technology, Australia. Professor Bruce Chapman, Australian National University. Professor Kenneth Clements, University of Western Australia. Professor David Colander, Middlebury College, Vermont, USA. Professor John Foster, University of Queensland, Australia. Professor Andrew Hannan, University of Plymouth, UK. Professor Yujiro Hayami, Foundation for Advanced Studies in

International Development, Japan. Professor Tim Hazledine, University of Auckland, New Zealand. Professor K.L. Krishna, Delhi School of Economics, India. Professor Alan Luke, Nanyang Technological University, Singapore. Professor Rod O’Donnell, Macquarie University, Australia. Professor David Round, University of South Australia. Professor Daniel Rubinfeld, University of California, Berkeley, USA. Professor Warren Samuels, Michigan State University, USA. Professor Amartya Sen, Harvard University, USA. Professor John Siegfried, Vanderbilt University, USA. Professor Jim Taylor, University of Lancaster, UK.

Secondary School Teaching Representatives Mr Doug Cave, Queensland Economics Teachers Association. Mr Ian Searle, Brisbane Boys Grammar.

AUSTRALASIAN JOURNAL OF ECONOMICS EDUCATION

MISSION STATEMENT The Australasian Journal of Economics Education is a peer-reviewed journal that publishes papers on all aspects of economics education. With a view to fostering scholarship in the teaching and learning of economics, it provides a forum for publishing high quality papers and seeks to bring the results to a widening audience. Given both the increasing diversity of the student clientele, and increasing calls for greater attention to the quality of tertiary teaching, this Journal seeks to foster debate on such issues as teaching techniques, innovations in the teaching of economics, student responses to such teaching, and the incentive systems which influence the academic teaching environment. The AJEE is interested in research involving both quantitative and qualitative analyses and also in interpretative analyses based on case studies. While the Journal is Australasian-focussed, it encourages contributions from other countries in order to promote an international perspective on the issues that confront the economics discipline. AJEE aspires to: 1. Report research on the teaching of economics, and cultivate heightened interest in the teaching of economics and the scholarship of teaching. Pedagogical issues will be a central feature, and will encompass work on the teaching of economics in diverse contexts, including large and small classes, undergraduate and postgraduate classes, distance learning, issues confronting foreign students on-shore and off-shore, and issues related to the teaching of fee-paying MBA and other post-graduate groups from diverse disciplinary backgrounds. Though economics is the prime focus, consideration will also be given to work on other subjects that have a demonstrated relevance for the teaching of economics. Such issues will also involve evolutionary issues in the teaching of economics, in terms both of effective ways to teach evolving theory and of evolving technology with which to teach that theory (including on-line teaching). Recognition will be given to the fact that economics as a discipline has not fared well in CEQ results (course experience questionnaire

results) since the reporting of those results began in Australia. Nor has economics teaching typically been well received in the USA or UK, according to survey evidence. In that context the relevance to teaching of changing administrative arrangements in universities will also be highlighted (eg in terms of contemporary quality assurance procedures and other government policy changes in Australia and New Zealand). 2. Report research on the nexus between teaching and research (including research on the diverse, changing and potentially conflicting incentives within the academic industry). Papers exploring the extent to which research and teaching activities are complementary or competitive will be welcomed. 3. Recognise the relevance of some more deep-seated implicit assumptions and issues of economic philosophy embedded in what is commonly taught, (as in Sen’s work on economics and ethics, for example). Inter alia, the question arises as to the way in which students respond to economics taught as a path to scientific certainty, as against economics taught as reflecting unsettled debate and vigorous controversy. 4. Recognise the place of history in the teaching of economics. Both HET and economic history tend to play a diminishing role in professional economics training, as emphasis on technique dominates. This a-historical approach to the teaching of economics has been criticised by many influential economists (including Joan Robinson, Leontief, Myrdal, Colander, and Robert Clower in his acerbic remarks about the value of much that is published in such prestigious journals as the AER). This line of criticism has been continued in the recent growth of heterodox economics associations in a number of countries (including one for Australia and New Zealand) and on the web through the Post Autistic Economics (PAE) newsletter. Historical and institutional factors will thus provide one focal interest. 5. Recognise interdisciplinary issues important to the presentation of economics in various contexts. On the one hand, economics students are not systematically exposed to the insights of other social sciences and the conformity or otherwise of their conclusions with those of economics. On the other hand, other disciplines within the social sciences and humanities (e.g. the Social Work profession) do not always include even an introduction to economics for their students, notwithstanding that economic issues are often very important

determinants of the environment within which they operate. More fundamentally, questions arise as to whether social science is more than the sum of its respective parts, and as to whether the roots of economics can be fully understood in isolation from the history not only of economics but also of politics and philosophy. 6. Establish a link to the teaching of economics in the secondary schools, given that tertiary enrolments in economics reflect fluctuating enrolments in economics in the secondary schools. 7. Encourage on-going surveys of student response to the teaching of economics across Australasian (and other) institutions, including response to experimental teaching and to differences between institutional approaches. (c.f. Colander and Klamer’s 1988 survey of economics students at USA ivy league institutions.) 8. Monitor trends in the teaching of economics both globally and in the Australian and New Zealand university systems (such as enrolments, staff-student ratios, international-domestic student ratios, offshore offerings etc), and the implications of those trends for various funding arrangements. 9. Promote a series of papers on specialised themes within the overall province of the teaching of economics e.g. on the teaching of Principles courses, the teaching of History of Economic Thought, the teaching of intermediate microeconomics and macroeconomics, the teaching of development economics, and likewise regarding teaching in such streams as Quantitative Methods, large first year classes, non-English speaking background students, the teaching of economics to non-economists, product differentiation in teaching economics, and professional education in economics in executive education programs outside conventional university contexts. 10. Monitor the measuring and rewarding of quality (economics) teaching within Australasian universities.

AUSTRALASIAN JOURNAL OF

ECONOMICS EDUCATION

Volume 13, Number 1

March, 2016

CONTENTS

ARTICLES

A Report on the Development of Learning

Standards for Economics in Australia

Ross Guest

Allan P Layton

1

The Impact of Lecture Capture on Student

Performance

Chris Jones

Matthew Olczak

13

Selecting Sessional Tutors: A New and

Effective Process

Carl W. Sherwood

Bruce Littleboy

30

BOOK REVIEW

Teaching Innovations in Economics

M. K. Salemi & W. B. Walstad

Peter Docherty

49


Volume 13, Number 1, 2016, pp.1-12

A REPORT ON THE DEVELOPMENT OF

LEARNING STANDARDS FOR ECONOMICS IN

AUSTRALIA*

Ross Guest

Griffith University

Allan P Layton

University of Southern Queensland

ABSTRACT

Guest (2013) motivated the development of learning standards for Economics in

Australia. Since then, under the auspices of an Australian Office for Learning and

Teaching (OLT) National Senior Teaching Fellowship, with support from the

Australian Business Deans Council (ABDC) and endorsement by the Economics

Society of Australia, a set of Learning Standards for Economics in Australia has

been developed through a collaborative and highly consultative process. In

addition, a set of exemplar assessment items has also been developed which align

with, and link to, the developed Learning Standards. This paper provides an update

on this development including: details of the process by which the Learning

Standards were developed, a brief discussion of the Learning Standards themselves

and of the accompanying assessment items, and, finally, a discussion of the

intended impact of the whole process on the teaching and learning of Economics

at Australian universities.

Keywords: learning standards, learning outcomes, student assessment.

JEL classifications: A22.

* Correspondence: Ross Guest, Griffith Business School, Griffith University, Business 3

(G06), 2.12, Gold Coast Campus, Parklands Drive, Southport, QLD, 4222, Australia.

Phone: (07) 555 28783; E-mail: [email protected]. This paper draws on material

from Guest (2013, 2014) and the Economics Learning Standards website

http://www.Economicslearningstandards.com. The authors acknowledge contributions

from members of the Economics Learning Standards Working Party and Expert Advisory

Group – listed on the website - who all contributed to the development of the Economics

Learning Standards.

ISSN 1448-4498 © 2016 Australasian Journal of Economics Education

http://www.economicslearningstandards.com/

2 R. Guest & A.P. Layton

1. INTRODUCTION

As discussed in Guest (2013, 2014), under the rubric of the Tertiary

Education Quality Standards Agency (TEQSA) Act 2011, Learning

Standards are “threshold standards”. This approach requires, inter alia,

that “the academic standards intended to be achieved by students and

the standards actually achieved by students in the course of study be

benchmarked against similar accredited courses of study offered by

other higher education providers” (Australian Government 2011, p. 17).

Furthermore, whilst the articulation of general teaching and learning

standards falls under the ambit of the Higher Education Standards Panel

(HESP), it is generally acknowledged that the TEQSA Act will

necessitate each discipline developing its own Learning Standards

consonant with the HESP general standards.

As has been acknowledged elsewhere, it is very important to note

that, whilst the Act uses the term “learning standards” it is not intended

that the ultimate outcome be standardization across Australia’s tertiary

institutions at either the general level of Learning Standards, nor at the

discipline level. On the contrary, learning standards describe what is

essential for graduates to have attained at graduation and this leaves

considerable room for diversity across institutions. Indeed, as will be

seen later, the Economics Learning Standards have been developed in

such a manner as to allow for great diversity in curricula across the

sector, whilst nonetheless allowing for the sort of benchmarking as

envisaged by the Act. Furthermore, as will be evident later, the term

“standards” may even be misleading and a more appropriate descriptor,

as least in the case of the Economics Learning Standards described

below, might be “learning outcomes”.

In the sections below, we will, in brief, traverse again some of the

motivations for developing Learning Standards in Economics (Section

2), provide a brief summary of the process used to develop the

Economics Learning Standards, the Learning Standards themselves, an

overview and flavour of the assessment exemplars (Section 3), the

intended impact of the Learning Standards on teaching and learning

(Section 4), with brief concluding remarks following in Section 5.

2. THE NEED FOR AUSTRALIAN LEARNING STANDARDS

The motivation for Learning Standards in Economics is threefold:

legislative compliance, educational improvement through curriculum

renewal, and earlier precedent developments in cognate disciplines. A

Learning Standards for Economics 3

widespread movement towards discipline-specific learning standards in

higher education has occurred both nationally and internationally over

the past decade (Guest 2013, 2014).

At the international level, these include: the OECD’s Assessment of

Higher Education Learning Outcomes (AHELO) Feasibility Study,

(which included an Economics strand); the Tuning Project which arose

from the European Bologna Process, and which interestingly overtly

avoids the term “standards” but defines “intended, expected or desired

learning outcomes” in Economics. The UK’s Subject Benchmark

Statements - including Economics - have been in place since 2000, and

the Economics Statement was revised in 2007.

At the national level, the Federally-funded Learning and Teaching

Academic Standards (LTAS) project developed a set of threshold

learning outcomes for a number of disciplines, with Accounting being

chosen as the first discipline from Business and Management to

produce a set of LTAS, and which were published in December 2010

(Hancock & Freeman 2010) with support from the Australian Business

Deans Council (ABDC). Since then, learning standards have been

developed, also under the auspices of the ABDC, for the business

disciplines of Marketing, Tourism Hospitality and Events Management

and Finance.

As previously mentioned, a key legislative catalyst for the

development of Economics Learning Standards was the TEQSA

legislation. Under this legislation all higher education institutions are

required to be able to demonstrate: (i) that their internal processes for

the design and approval of each degree take account of external

standards, and (ii) that the outcomes achieved by their students are

benchmarked against external standards. Learning Standards, such as

those provided in this document, developed through extensive

consultation with the discipline community, provide one reference

point for benchmarking. Another driver for these Learning Standards is

the Australian Qualifications Framework (AQF), which specifies

descriptors of learning outcomes in terms of knowledge, skills and

application abilities for graduates. These are generic descriptors, first

introduced in 1995, but revised in 2011. The AQF was the starting point

for the LTAS discipline-specific statements of threshold learning

outcomes, where the term “threshold” meant minimum.

Learning Standards provide guidance to a range of national and

international stakeholders, including academics designing new


programs, academics wanting to benchmark their existing programs,

employers who want assurance about what Economics graduates know

and can do, prospective students and secondary school course advisors

who want to know what Economics at tertiary level is about, and the

wider discipline community who wish to have assurance about the

relevance and value of Economics learning outcomes. Also, learning

standards provide the opportunity for widespread curriculum renewal,

with emphasis on ensuring that graduates of Economics have attained

the core skills and knowledge that are needed in order to effectively

practice the discipline of Economics.

Evidence from employers and elsewhere, both nationally and

internationally, indicates a need for such a renewed focus (Guest &

Duhs 2002; O’Doherty et al. 2007; Walstad & Rebeck 2002). In

particular this evidence suggests that, according to national and

international evidence about the lack of ability of Economics graduates

to address real world contexts, a refocussing on deeper understanding

and application of core knowledge and skills is needed. A UK study

(O’Doherty et al. 2007) found that “Economics graduates are not

particularly good at applying knowledge” to any “real world” situation.

Similarly US Economics graduates were found to have a weak

understanding of fundamental economic concepts and only a “slightly

better” ability to apply economic principles to solve economic problems

(Walstad & Rebeck 2002). In the Australian context, former Australian

Productivity Commission Chairman Gary Banks has argued that:

. . . in many cases the sort of Economics needed to inform policy

decisions is not very complicated or sophisticated. Much public policy

could go a long way with a few basic principles or precepts . . . . . But

we see policy proposals and decisions that violate those principles

almost on a daily basis.

(Banks 2011, p. 2)

3. THE DEVELOPMENT PROCESS AND OUTCOMES

The following four principles were considered fundamental to guiding

the development of the Learning Standards:

1. Learning Standards should reflect the minimum learning

outcomes that all graduates are expected to have attained.

Additional learning outcomes and those of an aspirational nature

are outside the scope of the Learning Standards;

2. Learning Standards should recognise both diversity among

Economics programs as well as the need to enable learning


outcomes to be compared across institutions and student cohorts.

However, the standards will not prescribe a set of topics, learning

activities or assessment items;

3. Learning Standards should be consistent with AQF standards and

should be informed by international standards;

4. The process should be collaborative, evidence-based, transparent

and iterative, incorporating feedback from the discipline

community including academics and other economists from the

private and public sectors.

Guest (2014) describes the process in detail which can be summarised

here as follows. The approach was based on extensive consultation and

engagement with the Economics discipline community, both nationally

and internationally, followed by extensive dissemination activities.

A working party of academic economists - chaired by co-author, Ross

Guest - was appointed in late 2012 (by expressions of interest to

Australian academic departments followed by appointment by a

selection panel) to develop the learning standards for Australian

Economics degrees at Bachelor and Coursework Masters level. At the

same time an Expert Advisory Group (EAG) was established which was

chaired by co-author, Allan Layton.

See Guest (2014) for a complete list of members of both the working

party and the EAG. The Working Party held nine meetings from late

2012 through 2013. In April 2013 the first draft went to the EAG for

consideration, and in May 2013, a series of in-person workshops were

held in university departments covering all States including the ACT.

Each workshop was an open forum where the aim, process and

outcomes to that point were explained, followed by a discussion. The

aim of the workshop was to capture feedback to be shared with the

Working Party in developing the next draft. There were 19 such

workshops held over a six month period. A Symposium entitled

Learning Standards in Economics for Australia: A Draft for Debate and

Discussion was then conducted at the Australian Conference of

Economists (ACE 2013) in July, 2013. The symposium panel consisted

of members from the Working Party and the co-authors. All interested

stakeholders from the academic, private and public sectors attending the

conference were invited to participate in the ongoing discussion.

In September 2013 a second draft was distributed through a survey

of Australian economists (mainly academics). Over 800 respondents

were targeted and 137 responses were received. Consultation was also


held with the external evaluator of the National Fellowship, Professor

John Sloman, and other UK Economists at the Developments in

Economics Education conference at the University of Exeter. In late

2013 further revisions were made and in November 2013 the final

version – being considered final by both the Working Party and the

EAG - was disseminated to primary stakeholders and posted on the

website (http://www.Economicslearningstandards.com/). At that point

the Learning Standards were also presented at meetings of the ESA by

Ross Guest and of the ABDC by the ABDC Teaching Fellow, Mark

Freeman, who was also on the Working Party. Both the ABDC and the

ESA endorsed the Learning Standards at their respective meetings.

(a) The Outcomes: Economics Learning Standards for Australia

The Learning Standards apply to a program of study badged as an

Economics program. This would typically be the case for a Bachelor

(or Master) of Economics, or Bachelor (or Master) of

Commerce/Business degree with a Major in Economics. These

programs may carry a reference to Economics in the degree

nomenclature, such as Bachelor of Commerce (Economics). The

Learning Standards have been developed at both bachelor level (AQF

Level 7) and Masters (coursework) level (AQF Level 9). ‘Masters level’

applies to both entry-level Masters degrees and advanced Masters

degrees. The Learning Standards presented here do not apply to

research Masters or Honours degree programs in Economics.

The question then is: what constitutes the study of Economics? The

Learning Standards define Economics as “the study of the factors that

influence income, wealth and well-being. Its aim is to analyse and

understand the allocation, distribution and utilisation of resources and

their consequences for economic and social well-being”. The Learning

Standards provide a further contextual statement as follows. Economics

provides analytical methods to address problems and issues in society.

Analytical methods refer to theoretical models and empirical tools for

explaining and predicting both microeconomic and macroeconomic

behaviour. Microeconomics is concerned with the behaviour of

individual consumers, workers, firms, markets and industries, and the

way they interact. It is also concerned with the role of government

regulation in moderating behaviour and interaction among these

groups. Macroeconomics refers to the analysis of the behaviour of

economy-wide phenomena such as unemployment, inflation, economic

growth, the distribution of income and wealth, financial markets,


exchange rates, international trade in goods, services and capital, and

the role of government policies, including fiscal and monetary policies,

in influencing these phenomena.

(b) The Learning Standards

The Economics Learning Standards are defined in terms of a set of

minimum learning outcomes rather than aspirational outcomes.

Bachelor and Masters learning standards are distinguished in terms of

knowledge and skills and their application. Compared with Bachelor

graduates, Masters graduates are expected to have attained knowledge

that is more complex, more integrated and more inclusive of recent

developments in the discipline. They should also be able to analyse

more critically and reflectively, communicate to wider audiences, and

be able to plan and execute a research-based project or piece of

scholarship.

The five learning domains and associated learning outcomes for

Bachelor and Masters graduates are summarised in Table 1. Whilst the

learning outcome domains are listed separately, this is simply for

taxonomic purposes. The work of Economics graduates often draws on

several of the learning domains simultaneously. Furthermore,

associated with the Learning Standards described in Table 1, there is

also a set of Economics concepts which are outlined as pertaining to the

Knowledge domain. These are included in Appendix A of the Learning

Standards document located on the website.

In addition, the full standards come with a set of notional micro and

macroeconomic examples of what could/should be expected of

graduates, one for each learning domain. For example, in the

Application domain (refer to Table 1), a macroeconomic example

provided in the Learning Standards is:

Bachelor graduates will be able to identify and evaluate possible causes

of a change in the rate of unemployment, including changes to labour

force participation, and broader economic influences such as

government policy and macroeconomic aggregates.

Masters graduates will have a more advanced understanding of

unemployment as a labour market outcome, and will be able to discuss

debates over the causes of the different types of unemployment.

Finally, the full set of Learning Standards also provide a set of real

life exemplar tasks – provided by members of the EAG – in which

graduate economists can expect to find themselves engaged, and as


Table 1: The Economics Learning Standards for Higher Education

Learning

Domain

Learning Outcomes

Bachelor Degree Masters Degree

Knowledge Bachelor graduates will be able to:

identify, coherently explain and

synthesise core economic concepts.

Masters graduates will be able to:

identify, coherently explain and

synthesise core and advanced

economic concepts, including

recent developments in the

discipline.

Application Bachelor graduates will be able to:

frame problems in terms of core

economic concepts and principles;

apply economic reasoning and

analytical skills, in order to make

informed judgments and decisions.


frame and critically analyse

problems in terms of core and

advanced economic concepts and

principles;

apply advanced economic

reasoning and analytical skills,

including quantitative techniques

where appropriate, in order to

make informed judgments and

decisions;

plan and execute a research-

based project.

Data Analysis

Bachelor graduates will be:

able to use economic data to address

typical problems faced by

economists;

aware of, and able to implement,

basic empirical techniques and

interpret the results.


select and apply an appropriate

empirical method to address

typical problems faced by

economists;

critically evaluate the results.

Communication Bachelor graduates will be able to: present a clear and coherent

exposition of economic knowledge,

ideas and empirical evidence both

orally and in writing, individually or

in collaborative contexts.

Masters graduates will be able to: communicate complex ideas

clearly and coherently, in written

form and interactive oral form to

expert and non-expert audiences,

individually or in collaborative

contexts.

Reflection

Bachelor graduates will be able to

reflect on: the nature and implications of

assumptions and value judgments in

economic analysis and policy;

interactions between economic

thinking and economic events, both

historical and contemporary;

the responsibilities of economists

and their role in society.

Masters graduates will be able to

reflect on and evaluate: the nature and implications of

assumptions and value

judgments in economic analysis

and policy;

interactions between economic

thinking and economic events,

both historical and

contemporary;

the responsibilities of economists

and their role in society.


noted in the Learning Standards, such tasks will often draw on more

than one and often all of the five Learning Standards domains.

(c) Assessment of Learning Outcomes

The OLT subsequently funded a follow up project related to

assessment, which was co-led by the two co-authors. The Project

produced a set of exemplar assessment items which overtly aligned with

the newly developed Economics Learning Standards.

The purpose of the developed repository of assessment exemplars

was to illustrate how to design Economics assessment tasks which

explicitly and transparently map to the Learning Standards discussed

above. Full details are provided on the website. The assessment

repository was developed in a collaborative way by a core assessment

project team of academic economists drawn from the initial Economics

Learning Standards Working Party and the EAG as well as other

advisors on assessment design and business education.

For the Learning Standards to be fit for purpose, the assessment tasks

in each unit of study must be explicitly and transparently designed to

address particular Learning Standards, such that achievement of all of

these Learning Standards can be demonstrated by the end of the degree

program. This was the primary motivation for developing the set of

assessment task exemplars located on the Learning Standards website –

that is, to provide guidance to those academic economists who want to

ensure that their assessment is explicitly and transparently linked to the

Learning Standards.

Importantly, the exemplars seek to be authentic in that the tasks are

set in the context of real world issues/situations, and focus on the

application of knowledge and skills. The exemplars also illustrate

innovative types of tasks, including approaches for assessing

communication skills, both written and oral. The tasks typically ask

students to appropriately frame the problem or issue in terms of

economic principles, apply those principles and communicate the

outcomes to appropriate audiences. The tasks may also require an

ability to reflect on changes in economic thinking over time, and on the

interplay between economic events and development of theory and

policy.

The exemplars cover the broad areas of macroeconomics,

microeconomics, data analysis/econometrics, international economics

and economic thought. In each exemplar, an overview of the item is

provided first which outlines what the assessment task seeks to achieve


and how this is to be accomplished in terms of the use of the assessment

instrument. Details of the task are then provided assuming a Bachelor’s

assessment level with each part of the assessment item referencing

particular Learning Standards it is addressing. It is then demonstrated

how the item can be adapted to cater for the Master’s level (with similar

reference to the Learning Standards), followed by suggestions about

how the item might be used – perhaps in some adapted form - for

different purposes, and/or how it might be adapted to assess different

Learning Standards at different levels.

In total there are 18 assessment exemplars provided on the website

across the areas listed earlier. Readers are invited to visit the website1

to interrogate any exemplars potentially of interest to them to get a

sense of the resource. It is hoped that academic economists will find

this assessment repository a useful resource in guiding development of

their own assessment tasks that are explicitly and transparently linked

with the Economics Learning Standards.

4. INTENDED IMPACT OF LEARNING STANDARDS

The whole point of learning standards is, ultimately, to improve the

learning – and the teaching - of Economics, and, in particular, to

improve the ability of Economics graduates to practice the discipline of

Economics in a range of professional contexts and as citizens. Guest

(2013) sets out the case for this, citing national and international

evidence that Economics graduates fall short of expectations of

employers, academics and indeed themselves in their ability to apply

economic principles to real world problems outside of the classroom.

This evidence is based on surveys of Economics graduates and

employers (cited in detail in Guest 2013). It is also worth noting that

student enrolments in Economics programs have struggled for several

decades in the face of competition from other business disciplines

(again cited in Guest 2013).

All of this evidence points to a need for curriculum renewal. However

as we discovered in the 12 month process of producing the Australian

Economics Learning Standards, curriculum debates are fraught.

Considerable evidence suggests that ‘less is more’ in relation to content

coverage (Guest 2013). That is, learning is deeper, more lasting and

more useful when students are given more time on task, and more

1 http://www.Economicslearningstandards.com/search-for-an-assessment-exemplar -via-

field-of-Economics.html.

http://www.economicslearningstandards.com/search-for-an-assessment-exemplar


opportunities to apply principles in a range of new contexts. This

necessarily however comes at the cost of breadth of topic coverage.

The problem then becomes that it is very hard to get agreement about

what content should be cut, and what core skills should be emphasised.

The pluralist or heterodox school argues for a broad range of economic

thinking perspectives to potentially be brought to bear on an economic

problem or issue (Guest 2013). This tends to imply a broadening rather

than narrowing of content, taking in schools of thought including

Austrian, institutionalist, evolutionary, behavioural, post Keynesian,

feminist and Marxist approaches, as well as perspectives from

alternative social science disciplines. It is fair to say that the set of

developed Economics Learning Standards tend toward the

traditional/orthodox perspectives on Economics, but not entirely – the

learning domain, “reflection”, recognises the contestability of

assumptions and value judgements, as well as the evolution of

economic thinking.

The Australian Economics Learning Standards focus on application

of core knowledge and skills. It is hoped that the examples in the

Learning Standards of the way in which each of the learning domains

could be operationalised will focus teachers and students on real world

applications. Similarly, the exemplars of assessment items, produced

subsequently and posted on the website, are explicitly focussed on

applications using real data/contexts. Finally, the Learning Standards

also include the important domain of “communication”, reflecting

feedback from employers that graduates – and not just of Economics

programs - fall short of expectations in their ability to present ideas and

analytical results effectively to a range of audiences.

5. CONCLUDING REMARKS

The Economics Learning Standards for higher education in Australia

are certainly not meant to be the last word on minimum learning

outcomes in Economics in this country. They will evolve as the

discipline itself evolves and as the student body evolves in terms of

prior learning and preparation for tertiary education. The Economics

Society of Australia and the Australian Business Deans Council, both

of which supported the development of the Learning Standards, have

an important role to play in leadership of the evolution of learning

standards in Economics. Indeed the success of the Learning Standards

will depend on the commitment of the whole of the academic

Economics community. It is important that the approach to Learning


Standards is not one of simple compliance, but rather continues as a

dynamic driver of curriculum renewal along the lines suggested above.

REFERENCES

Australian Government (2011) Higher Education Standards Framework

(Threshold Standards) – F2012L00003, available at www.comlaw.

gov.au/details/F2012L00003/Html/Text#_Toc311791711.

Banks, G. (2011) ‘Does Australian Public Policy Get the Economics It

Deserves?’, Agenda, 18 (3), pp.21-29.

Guest, R. (2013) “Towards Learning Standards in Economics in Australia”,

Economic Papers, 32 (1), pp.51–66.

Guest, R. (2014) Economics Learning Standards for Australian Higher

Education, Australian Government Office for Learning and Teaching,

Final Report, available at http://www.olt.gov.au/resource-economics-

learning-standards-Australian-higher-education.

Guest, R. and Duhs, A. (2002) “Economics Teaching in Australian

Universities: Rewards and Outcomes”, Economic Record, 78 (241),

pp.147–60.

Hancock, P. and Freeman, M. (2010) Learning and Teaching Academic

Standards Project Business, Management and Economics. Learning and

Teaching Academic Standards Statement for Accounting, Strawberry

Hills, Australia: Australian Learning and Teaching Council, available at:

http://www.olt.gov.au/resource-accounting-ltas-statement-altc-2010.

O’Doherty, R., Street, D. and Webber, C. (2007) The Skills and Knowledge

of the Graduate Economist: Findings of a Survey Conducted on Behalf of

the Royal Economic Society and the Economics Network, Bristol, UK.

Walstad, W.B. and Rebeck, K. (2002) “Assessing the Economic Knowledge

and Economic Opinions of Adults”, Quarterly Review of Economics and

Finance, 42, pp.921–35.

http://www.comlaw/



THE IMPACT OF LECTURE CAPTURE ON

STUDENT PERFORMANCE*

Chris Jones and Matthew Olczak

Aston Business School,

Aston University, Birmingham, UK

ABSTRACT

This paper investigates whether watching online recordings of live lecturers

improves student performance on a large first year introductory economics module

taught in a Business School. Our results show that performance on other modules

and previous experience of economics are the key determinants of performance on

this module. However, we also show that watching the lecture recordings may go

some way to counteracting a student’s lack of previous experience in the subject.

Keywords: technology, lecture recording, lecture capture, and student performance.


1. INTRODUCTION

Technology that enables live lectures to be recorded and subsequently

made available for students to watch online has become widely used in

university teaching.1 Certainly from our experience at Aston

University, lecture capture has developed from initial experimentation

to virtually all lectures over a period of less than five years.

Furthermore, a broad range of evidence discussed below suggests that

this innovation is extremely popular with students. However, the

perceived benefits also appear to differ significantly across the diverse

* Correspondence: Matthew Olczak, Aston Business School, Aston University,

Birmingham, B4 7ET, UK, email: [email protected] Tel: +44(0)121 204 3107. Chris

Jones may be contacted at Aston Business School, Aston University, Birmingham, B4

7ET, UK, email: [email protected] Tel: +44(0)121 204 3036. Thanks to two

anonymous referees for comments and suggestions.

1 See Stephenson & Cortinhas (2013) for a detailed overview of how this technology

works.


14 C. Jones & M. Olczak

student population. Furthermore, the views of instructors are less

positive.

Given these mixed views, there is clearly the need for direct evidence

that investigates the effect of watching lecture recordings on student

performance. Therefore, the aim of this paper is to investigate this in a

large first year introductory economics module taught in a Business

School. We estimate the average effect watching the recordings has on

student performance in this module and, in addition, consider whether

this effect differs depending upon the characteristics of the student.

Our paper relates to a growing body of literature that tries to assess

the impact of various technologies on student learning of business and

economics. The approach typically taken has been to compare a control

group who have face-to-face lectures with a treatment group who only

receive online lectures. In contrast, we are interested in examining the

effect of watching lecture recordings on student performance when this

technology is made available to all students taking the module.

Therefore, more closely related to our methodology are a number of

papers, described in detail below, that examine student choice over

whether or not to make use of the lecture recordings provided.

The remainder of the paper proceeds as follows. In section 2 we

summarise the alternative views in the literature of lecture capture and

outline the existing literature that assesses the impact of technology on

student learning. In section 3 we provide an overview of the module

studied and outline our empirical methodology. In section 4 we first

show that the majority of the students on the module made at least some

use of the lecture recordings; some for a considerable amount of time.

Interestingly, these students were more likely to have no previous

experience of economics. We then report our baseline econometric

results. These show that performance on other modules and previous

experience of economics are the key determinants of performance on

this module. However, we also find that watching the lecture recordings

has a small positive effect on performance and that this may go some

way to counteracting a student’s lack of previous experience in the

subject. Nevertheless, this finding should be interpreted with some

caution since there is an obvious concern that it may be affected by self-

selection bias. This is because the students that watched the recordings

may have been more able students who would have performed better in

the module regardless of whether or not they watched the recordings.2

2 See Greene (2003, p.788) for a formal demonstration that this results in an upward bias.

Lecture Capture & Student Performance 15

Below we outline how we attempt to control for this potential bias.

However, despite this, our finding of the impact of watching lecture

recordings is best interpreted as an upper bound and therefore any

significant effect can be interpreted as being small. At the end of section

4 we discuss various robustness checks on our baseline results. Finally,

in section 5 we make some concluding remarks.

2. RELATED LITERATURE

Our treatment of the literature is divided into that which considers

alternative views of lecture capture and that which assesses the impact

of lecture capture on student learning.

2.1 Alternative Views of Lecture Capture

The literature which addressed the issue of how lecture capture is

perceived may also be divided into that which reflects the views of

students, the heterogeneity of the benefits that students perceive from

lecture capture, and that which reflects the views of instructors. Each of

these issues is addressed separately.

2.1.1 Popularity of lecture capture with students

A reasonably large body of survey evidence from a range of disciplines

shows how popular lecture capture is with students. For example,

Gosper et al. (2008) reports the results of a large survey conducted

across a number of Australian universities and a wide range of

disciplines showing that the majority of students perceived their

experience of lecture capture to have been positive and claimed that it

helped them to achieve better results.3 Furthermore, in our module

student feedback the overall satisfaction score was high and many

comments commended the provision of lecture recordings.

However, evidence on how much students actually value and make

use of recorded lectures is mixed. Taplin et al. (2011) try to more

accurately assess student valuation of lecture recordings by measuring

their willingness to pay for them. They found that many accountancy

students had a low valuation both in terms of willingness to pay and the

fact that they would only on average be prepared to substitute 2 of 11

tutorials for the provision of recorded lectures (Taplin et al. 2011,

p.183).4 Survey evidence in Wong (2013) also suggests that students

found other online resources, such as solutions to exercises, lecture

3 See also other consistent evidence in for example Taplin et al. (2011) and Elliott & Neal

(2016). 4 This evidence is also confirmed for a wider range of disciplines in Taplin et al. (2014).


notes and other information more useful than recordings. In contrast, in

Elliott & Neal (2016) most students made at least some use of the

recordings and typically viewed a reasonably large proportion of the

lectures, whereas usage was much lower in Larkin (2010) and Taplin et

al. (2011). It is also clear that students value face-to-face class time and

don’t regard recordings as a substitute (Wong 2013; Gosper et al. 2008;

Larkin 2010; and Taplin et al. 2011).

2.1.2 Heterogeneity of perceived benefits from lecture capture

Not only is the evidence on valuation and usage of lecture recordings

mixed, it also appears that the benefits can differ substantially across

students. For example, Wong (2013) highlights the fact that

technological innovations can be especially useful as support for a

diverse group of students, in particular those who are in employment as

well as studying. The survey results in Flores & Savage (2007) show

that students who substitute online recordings for attending the live

lecture valued the availability of recorded lectures highly. Furthermore,

watching the lectures improved performance. However, they also find

that a nontrivial subset of students do not use the technology in this way

and therefore don’t value its provision. Likewise, despite the Taplin et

al. (2011, 2014) estimates of student valuation of lecture recordings

being low, their results suggest that the overall benefits of providing the

recordings exceed the costs. This is because a few students value their

provision very highly. As they go on to discuss, this raises equity and

ethical issues.

2.1.3 Instructor views on lecture capture and its impact

Instructor views on lecture capture also appear to be less positive than

those of the students. In Gosper et al. (2008, pp.22-23) only 29% of the

staff surveyed stated that their overall experience was almost always

positive.5 Furthermore, whilst 49% of staff said the technology made it

easier for students to study, only 30% agreed that it helped the students

to achieve better results. This compares to 67% of students who claimed

that it had helped them to achieve better results. Staff concerns included

a negative effect on student attendance, reduced communication with

5 Interestingly, good experiences were positively correlated with staff having chosen to

introduce the technology rather than being required to do so. They also find that about

one third of the lecturers using recording technology had made no substantial change to

their teaching approach as a result. In contrast, they argue that such technological

innovations should ideally impact on the whole course design.


students and deterioration in their ability to motivate and inspire

students.

In terms of a negative effect on attendance, the actual evidence is

mixed (see, for example, Taplin et al. 2011).6 This may be because, as

Larkin (2010) points out it is conceivable that watching recordings

enables students to see what they are missing and gain confidence in

the subject and that this will actually encourage them to subsequently

attend lectures.

Like Gosper et al. (2008), a number of other papers have also raised

similar concerns about the impact of lecture recordings on staff-student

interaction, and questioned the effect on student engagement (see, for

example, Flores & Savage 2007; Taplin et al. 2014 and Bennett &

Maniar 2007). On the other hand, Larkin (2010) highlights the fact that

recordings provide an opportunity for the lecturer to listen back and

reflect on their teaching. In addition, the lecturer can potentially

respond to student requests for additional material by providing

supplementary online recordings. Therefore, in this way interaction

between students and lecturers can actually be enhanced.

Overall, given these mixed views on the benefits of watching lecture

recordings and the likelihood that the benefits that do arise may differ

across students, there is clearly the need for direct evidence on the effect

on student performance.

2.2 Assessing the Impact of Technology on Student Learning

There is now a growing literature that tries to assess the impact of

various technologies on student learning of business and economics

(see Agarwal & Day 1998, as an example of one of the first papers).

Specifically on lecture recordings, Savage (2009) did not find evidence

that students for whom recordings were made available performed

significantly better than a control group in which no recordings were

available. In addition, student attendance did not differ between the two

groups. In contrast, Wong (2013) finds that the introduction of a range

of technologies coincided with an increase in module pass rates and that

the best students used the online resources much more than students that

failed the module. A number of other papers haven taken a similar

approach by comparing a control group who have face-to-face lectures

with another group who only get online lectures. For example, Brown

6 In addition, the actual evidence in Gosper et al. (2008) was also mixed and in Larkin

(2010) and Stephenson & Cortinhas, (2013) the survey evidence attendance suggests it

was not affected.


& Liedholm (2002) find that student performance in the latter group

was inferior, especially on questions that aimed to assess a deeper level

of understanding. In addition, there was also some indication that the

students taught online worked less hard on the course. More generally,

the results on the impact of teaching online on performance are mixed

(see Williams et al. 2012 for a summary). This suggests that, whilst the

availability of recorded lectures may be a useful complement to live

lectures, caution should be exercised in any moves towards replacing

these with recorded material.

More closely related to our methodology are papers that examine

student choice over whether or not to make use of lecture recordings

when they are provided to complement live lectures. Chen & Lin (2012)

examine evidence from an intermediate microeconomics course in

Taiwan. They find that most lecture views occurred during the revision

period; and on average, watching the relevant recording just prior to the

exam improved performance by around 5%. Whereas, viewings

immediately after the lecture did not have a positive impact on student

performance. Their data also shows that it was the students with poorer

attendance records that made the most use of lecture recordings. Crucial

to their approach is the need to isolate the impact of a particular lecture

for understanding a given topic.7 However, if key threshold concepts

are introduced which, once understood, facilitate the understanding of

subsequent material, then this approach becomes less valid. Evidence

suggests this may well be the case in first year introductory economics

modules (see for example Shanahan et al. 2006). Therefore, we instead

focus on the impact recorded lecture viewings have on overall module

performance.

Williams et al. (2012) use data on self-reported lecture attendance to

examine whether viewing lecture recordings are a substitute or

complement to lecture attendance and to test the impact this choice has

on student performance in their first year undergraduate economics

class. They identify two groups of students. Firstly, those that attended

very few lectures, some because of employment commitments, and

report that they use the recordings to catch-up. Secondly, those that

attended the most lectures and used the recordings to revisit material.

Whilst the first group of students were the ones that made most use of

7 This is because they attempt to control for the possible self-selection bias discussed

earlier by looking at the variation in a given student’s lecture viewings and matching this

with their performance on the relevant exam questions.


the recordings, they show that it was the latter group of students that

derived the most benefit. Recordings were useful as a substitute, but not

enough to eradicate the negative effect that non-attendance has on

performance. In our concluding section we compare our results with

those of Chen & Lin (2012) and Williams et al. (2012).

3. DATA AND METHODOLOGY

Economic Environment of Business is a first year Business School

module that provides an introduction to economics for students that will

typically not go on to major in economics. The teaching of the module

comprises 11 lectures, one of which is a revision session delivered at

the end of term, and 5 one hour tutorials per student taken fortnightly.

Each lecture lasted between 1 and 2 hours and every lecture was

recorded live using the Panopto software that the university has

adopted. The recordings were then made available for the students to

watch (but not download) from the university’s Virtual Learning

Environment (Blackboard) shortly after the lecture.8 This meant that

we could readily obtain detailed data on which students watched the

lecture recordings and for how long. The module had one piece of

assessment at the end of term - a 2 hour examination. This comprised a

40 question multiple-choice test (accounting for 40% of the total marks)

to assess breadth of knowledge and then two essays (a choice of one

from two on micro and one from two on macroeconomics) to assess

depth of knowledge. Our sample comprises the 380 students that sat this

examination for the first time (i.e. not for reassessment) in the Spring

term of 2012.

We are interested in explaining a student’s module performance (yi),

which will be measured by their overall examination mark. This will be

modelled as:

𝑦𝑖 = 𝛼𝑂𝑀𝑂𝐷𝑖 + 𝑥𝑖′𝛽 + 𝛿𝑊𝐴𝑇𝐶𝐻𝑖 + 휀𝑖 (1)

Most importantly given our focus, WATCHi is a binary variable

capturing whether or not student i decided to watch the online lecture

recordings. The variable OMODi is student i’s average year mark across

all the other modules they study in their first year (excluding this

module). This is an important control variable as it will pick up a 8 The recording software captures both the lecture and the PowerPoint slides. Unless

students request that the recording is paused, student interaction with the lecturer could

be captured. However, the microphone may not pick up some audience participation.


student’s overall ability and level of engagement. Furthermore, it is

important to note that lecture recordings were not provided in any of

these other modules. This means that including this variable helps to

alleviate concerns that our results will be affected by the self-selection

bias discussed above. We return to discuss the possibility of self-

selection bias at the end of the paper. In addition, xi is a vector of other

characteristics of student i (summarised below). We also include fixed

effects to control for the student’s degree programme, parental

occupational class, school and ethnicity. Finally, εi is an error term.

Table 1: Descriptive Statistics for the Module Intake

Variable Obs Mean Std. Dev. Min Max

EXAM MARK (%)

380

58.35

13.77

9

95

GENDER (1 if male) 380 0.47 0.50 0 1

OVERSEAS (1 if overseas

student) 380 0.15 0.36 0 1

AGE (on entry in years) 380 19.1 2.37 17 42

OMOD (Average mark (%)

for all other first year

modules taken) 380 57.53 11.49 12.29 81.47

ECON (1 if prior study of

economics) 380 0.15 0.36 0 1

Table 1 summarises the key characteristics obtained from the

university records of the 380 students in our sample and their overall

exam performance. As Table 1 shows, the average overall mark for the

module was 58% and this was the same as the overall average mark

across all other modules.9 Most of the students had come to university

soon after completing their schooling and the majority had not studied

economics prior to joining the university.10 Finally, overseas students

(defined as those from outside the European Union) represented a

relatively small, but non-trivial proportion of the students.

9 This explains why the data is from Spring 2012. After this date lecture capture use

became more widespread and was then made compulsory. 10 For those that had studied Economics this was typically via an AS- or full A-level or

as part of the International Baccalaureate.


Table 2: Descriptive Statistics for Students Who Watched the Recordings

Variable Observations Mean

Std.

Dev. Min Max

MINS (total mins watched) 235 452.19 498.10 0.52 2668.27

NUM LECS (no. lectures

watched) 235 5.28 3.45 1 11

4. RESULTS

In total 235 (59%) of students watched part of at least one lecture

recording. Table 2 provides further information on exactly how much

these students watched the recordings. Out of the 235 students who

watched the lecture recordings, on average students watched about 5

of the lectures and for a total of around 7.5 hours. Figure 1 plots total

viewing activity across the term up to the examination date. It is clear

that the viewings peak substantially leading up to the exam. This is

entirely consistent with other evidence in the literature (for example

Elliott & Neal 2016 and Williams et al. 2012). Furthermore, consistent

with this previous evidence, there is also some indication that students

were being selective and only watching specific parts of the lectures.

Figure 1: Number of Minutes Watched Across the Term

Table 2 also suggests that a few students watched for a very short

period of time. This was perhaps to check that the recording had

worked, with the intention to go back and watch at a later date.

However, it was clearly not for sufficient time to benefit from using the

0

2000

4000

6000

8000

10000

12000

26/01/2012

05/02/2012

15/02/2012

25/02/2012

06/03/2012

16/03/2012

26/03/2012

05/04/2012

15/04/2012

25/04/2012

05/05/2012

Min

ute

s


resource. In total, there were 18 students who viewed some of the

recordings, but for a total of less than 15 minutes. Therefore, in our

baseline specification we define having watched lecture capture as

having watched more than 15 minutes in total.11 Table 3 compares the

key characteristics of the students between those that did and did not

watch the lecture recordings for more than 15 minutes.

This shows that the students that made use of the recordings did better

across all the other first year modules. Therefore, this provides

additional evidence of the importance of including this control variable

in our subsequent econometric analysis. Interestingly, the students that

made use of lecture capture were less likely to have previously studied

economics and more likely to be home students.

We will discuss these findings further below. It is also worth noting

that in contrast, the 18 students who watched the recordings for less

than 15 minutes were more likely to have studied economics previously

than those that didn’t watch at all.12 Therefore, this provides some

additional justification for treating this small group of students

differently.

The first results column of Table 4 reports Ordinary Least Squares

(OLS)13 estimation results for our baseline specification where, as

outlined above, the WATCH variable captures the students who watched

a total of more than 15 minutes. These students represent 55% of our

sample and had a mean viewing time of just over 8 hours. It is clear

from these results that previous experience of economics and

performing well across all the other first year modules are key

determinants of student performance on the module. In contrast, a

student’s age, gender or whether or not they are an overseas student

appear to have no significant effect on their performance.

Most importantly, the key coefficient of interest on the variable

WATCH is also positive and significant and at the 10% level (p =

11 Since only 2 students watched for between 15 and 30 minutes, our results do not differ

substantially if we instead made the cut off at 30 minutes. 12 One possible explanation is that these students were keen at the start of the module, but

become disengaged early on in the term. Some evidence to support this is provided by the

fact that their brief viewings of the recordings were typically early on in the term. The

possible disengagement of this subset of students clearly merits further investigation. 13 Our dependent variable is bounded between 0 and 100. However, using OLS does not

appear to be problematic since the predicted values from our baseline specification all lie

within this interval.


Table 3: Conditional Mean Values of Key Student Characteristics

Students that watched

< 15 mins Recordings

Students that watched

> 15 mins Recordings

OMOD ** 56.073 58.725

ECON ** 0.192 0.111

GENDER 0.494 0.447

OVERSEAS ** 0.198 0.111

AGE 19.169 19.043

Two-sample t-test (allowing for unequal variance) testing whether means are

significantly different: *** p < 0.01; ** p < 0.05; and * p < 0.1.

0.053).14 However, the size of this effect is relatively small (95%

confidence interval of -0.031 to 4.146). Since most of the students that

watched for more than 15 minutes had not previously studied

economics,15 this result suggests that watching the lecture recordings

may partially offset their lack of previous experience. More

specifically, ceteris paribus, not having previously studied economics

is estimated to reduce a student’s expected examination performance by

6 marks, whereas watching the recordings for more than 15 minutes

increases this by 2 marks.

Therefore, the results from our baseline specification suggest that

watching the lecture recordings may have a small positive effect on

student performance.16 In order to test more precisely whether it

matters how much the students watched the lecture recordings, in

columns 2 and 3 of Table 4, we replace the WATCH variable with the

total number of minutes and lectures viewed as reported in Table 2. The

14 If we instead exclude the 18 students that watched the recordings but for less than 15

minutes then the size of the estimated effect from watching the recordings increases and

is slightly more significant (p = 0.045). 15 If we don’t control for previous experience of economics, the WATCH coefficient is

insignificant. This suggests that previous experience of the subject is the key determinant

of performance. In other unreported results we have also tried interacting the WATCH

variable with ECON. However, given that very few students with previous experience of

economics made use of the lecture recordings, it is not possible to disentangle the two

effects. 16 We have also allowed the effect of watching lecture capture to vary according to the

student’s performance on other modules, whether they had previously studied economics,

whether they were an overseas student and by gender. However, there was no evidence

to suggest that the effect of watching the recordings depends upon these characteristics.


Table 4: OLS Regressions of Overall Examination Performance

Regression (1) Regression (2) Regression (3)

OMOD 0.776*** 0.788*** 0.782***

(0.0539) (0.0541) (0.0545)

WATCH 2.058*

(1.062)

MINS 0.00108

(0.00113)

NUM LECS 0.202

(0.132)

ECON 6.349*** 6.048*** 6.242***

(1.408) (1.408) (1.419)

GENDER 0.463 0.421 0.422

(1.011) (1.017) (1.015)

OVERSEAS -2.296 -2.188 -2.231

(1.725) (1.724) (1.721)

AGE -0.158 -0.164 -0.161

(0.298) (0.285) (0.290)

Programme Yes Yes Yes

Occ. Class Yes Yes Yes

School Yes Yes Yes

Ethnicity Yes Yes Yes

CONSTANT 8.429 7.218 8.351

(13.554) (13.392) (13.425)

Observations 380 380 380

Adj. R-squared 0.571 0.567 0.567

Robust standard errors in parentheses. These are calculated using the standard

Huber-White procedure to correct for possible heteroskedasticity. Symbol ***

indicates p < 0.01; ** p < 0.05; and * p < 0.1.

results show that neither of these variables has a significant impact on

performance. One interpretation of this is that it supports the evidence

that students are selective in the material that they watch. It may be

that additional information on how the students use the recordings, for

example on the extent to which students focus their viewings on the

parts of the lectures covering key threshold concepts and specific advice

on assessment, would provide further insights. As outlined in section 3,

we also include fixed effects to control for the student’s degree

programme, parental occupational class, school and ethnicity. To ease

the burden on the reader, we don’t report estimates for these here.

However, the full results for our baseline specification are reported in

the Appendix.


6. CONCLUSION

Consistent with Chen & Lin (2012) and Williams et al. (2012) we find

that watching lecture recordings may have a positive effect on student

performance. Also, in line with Chen and Lin, our results suggest that

watching the recordings during the revision period accounts for much

of this. However, our estimated effect on student performance is smaller

than in these previous studies. In Williams et al. (2012), despite using

the recordings less, it is the students that attended most lectures and then

use the recordings to revisit material that derived the most benefit from

watching the recordings. Recordings were useful as a substitute, but not

enough to eradicate the negative effect non-attendance has on

performance. Since we do not have data on attendance the smaller effect

that we estimate may partly be attributable to the effect of non-

attendance.

Our results also highlight another heterogeneous impact from

providing lecture recordings. In our sample, students with no previous

experience of economics were more likely to watch the lecture

recordings and our results indicate that this may have gone some way

to counteracting this. Our analysis of which students made use of the

recordings also shows that these were less likely to be overseas

students. This suggests that there may be scope for investigating how

to encourage this subset of students to make more use of the technology.

Overall, especially given the possibility that self-selection bias is still

present, our results suggest that whilst there may be some small benefits

for student performance from providing lecture recordings, this

technology is unlikely to result in substantial improvements. However,

additional evidence would be beneficial. Whilst perhaps difficult to

implement due to ethical issues, comparisons between a treatment and

a control group who (unlike in earlier research) receive the same live

lectures, but differ in terms of whether or not they are provided with

lecture recordings, would be one useful way to further evaluate the

benefits of this technology. Such an approach has been used to examine

other teaching innovations (see, for example, McMahon 2011).

APPENDIX

This appendix provides details of regressions that included fixed effects to

control for the students’ degree programme, parental occupational

class, school and ethnicity that were not reported in the main text.


Table 5: OLS Regression of Overall Exam Performance with

Fixed Effects for Control Variables

Explanatory Variable Coefficient Estimate (s.e.)

OMOD 0.776 (0.0539)***

WATCH 2.058 (1.062)*

ECON 6.349 (1.408)***

GENDER 0.463 (1.011)

OVERSEAS -2.296 (1.725)

AGE -0.158 (0.298)

Programme

BSc Business Computing & IT -5.689 (4.633)

BSc Business & Computer Science -2.919 (3.503)

BSc Business & French -7.515 (3.254)**

BSc Business & Int. relations -2.749 (3.187)

BSc Business & Mathematics 6.363 (3.110)**

BSc Business & Politics -0.615 (2.682)

BSc Business & Psychology 4.881 (2.674)*

BSc Business and Public Policy Management 4.125 (2.469)*

BSc Business & Sociology 1.019 (7.314)

BSc Business & Spanish 3.794 (3.576)

BSc Computing for Business 0.586 (2.604)

BSc Human Resource Management -0.620 (2.465)

BSc International Business and Modern

Languages 3.859 (2.176)*

BSc International Business & Management 2.972 (1.882)

BSc Marketing 2.967 (2.034)

LL.B. Law with Management -1.900 (2.448)

Occ. Class

Higher Managerial 8.480 (6.003)

Intermediate Occupations 7.329 (5.934)

Lower Managerial Occupations 7.948 (5.929)

Lower Supervisory and Technical Occupations 5.857 (6.643)

Semi-routine Occupations 7.112 (5.995)

Small Employers and Own Account Workers 8.361 (5.906)


School Coefficient Estimate (s.e.)

Comprehensive School -3.973 (5.318)

Grammar School -9.473 (5.950)

Independent School -9.272 (5.290)*

Language School -10.333 (5.596)*

Sixth Form College -5.466 (4.795)

Special School 9.719 (5.757)*

Tertiary School -8.422 (5.331)

Unknown -4.879 (4.754)

Ethnicity

Asian Other 0.971 (4.721)

Bangladeshi 0.963 (6.253)

Black Other 8.780 (18.179)

Black or Black British - African 2.527 (4.196)

Black or Black British - Caribbean -0.338 (4.013)

Chinese 7.334* (4.177)

Indian 3.282 (3.535)

Information Refused 4.586 (4.364)

Other -2.457 (5.671)

Other Mixed Background 3.665 (5.012)

Pakistani 3.691 (4.241)

White 2.503 (4.450)

White & Black African 1.067 (5.248)

White & Black Caribbean -9.117 (6.270)

White British 2.551 (3.552)

White Other 3.073 (4.106)

CONSTANT 8.429 (13.554)

Observations 380

Adj. R-squared 0.571

Ethnicity classifications are those used by Aston University for reporting to the UK

Higher Education Statistics Agency (HESA). See http://www.aston.ac.uk/registry/for-

students/facts-figures/ as well as https://www.hesa.ac.uk/data-and-analysis/students.

Reference category for Programme is ABS Undergraduate Exchange, Occ. Class is

Unknown Category, School is Art Design and Performing Arts and Ethnicity is White

Irish. Robust standard errors in parentheses. Symbol *** indicates p < 0.01;

** p < 0.05; * p < 0.1.

http://www.aston.ac.uk/registry/for-%20students/facts-figures/

http://www.aston.ac.uk/registry/for-%20students/facts-figures/


REFERENCES

Agarwal, R. and Day, A. E. (1998) “The Impact of the Internet on Economic

Education”, Journal of Economic Education, 29 (2), pp.99-110.

Bennett, E. and Maniar, N. (2007) Are Videoed Lectures an Effective

Teaching Tool?, Mimeo, http://podcastingforpp.pbworks.com/f/Bennett

%20plymouth.pdf.

Brown, W.B. and Liedholm, C.E. (2002) “Can Web Courses Replace the

Classroom in Principles of Microeconomics?”, American Economic

Review, 92 (2), pp.444-448.

Chen, J. and Lin, T-F. (2012) “Do Supplementary Online Recorded Lectures

Help Students Learn Microeconomics?”, International Review of

Economics Education, 11 (1), pp.6-15.

Elliott, C. and Neal, D. (2016) “Evaluating the Use of Lecture Capture Using

a Revealed Preference Approach”, Active Learning in Higher Education,

17 (2), pp.153-167.

Flores, N. and Savage, S.J. (2007) “Student Demand for Streaming Lecture

Video: Empirical Evidence from Undergraduate Economics Classes”,

International Review of Economics Education, 6 (2), pp.57-78.

Gosper, M., Green, D., McNeill, M., Phillips, R., Preston, G. and Woo, K.

(2008) The Impact of Web-based Lecture Technologies on Current and

Future Practices in Learning and Teaching: Report for Australian

Learning and Teaching Council, available at https://www.mq.edu.au/

ltc/altc/wblt/research/report.html.

Greene, W. H. (2003) Econometric Analysis, 5th edtion, Upper Saddle River,

N.J.: Prentice Hall.

Larkin, H.E. (2010) “ ‘But They Won’t Come to Lectures …’ - The Impact

of Audio Recorded Lectures on Student Experience and Attendance”,

Australasian Journal of Educational Technology, 26 (2), pp.238-49.

McMahon, M.F. (2011) “Classroom Games in Economics: A Quantitative

Assessment of the ‘Beer Game’ ”, Warwick Economics Research Paper,

No.964, Warwick University, UK.

Savage, S.J. (2009) “The Effect of Information Technology on Economic

Education”, Journal of Economic Education, 40 (4), pp.337-53.

Shanahan, M. P., Foster, G. and Meyer, J. H. F. (2006) “Operationalising a

Threshold Concept in Economics: A Pilot Study using Multiple Choice

Questions on Opportunity Costs”, International Review of Economics

Education, 5 (2), pp.29–57.

Stephenson, J. and Cortinhas, C. (2013) “Creative Uses of In-class

Technology”, The Handbook for Economics Lecturers, Economics

Network, available at https://www.economicsnetwork.ac.uk/handbook/

technology.

http://podcastingforpp.pbworks.com/f/Bennett

https://www.mq/

https://www.economicsnetwork.ac.uk/


Taplin, R.H, Kerr, R. and Brown, A.M. (2014) “Opportunity Costs

Associated with the Provision of Student Services: A Case Study of Web-

based Learning Technology”, Higher Education, 68, pp.15-28.

Taplin, R.H, Low, L.H. and Brown, A.M. (2011) “Students’ Satisfaction and

Valuation of Web-based Lecture Recording Technologies”, Australasian

Journal of Educational Technology, 27 (2), pp.175-91.

Williams, A., Birch, E. and Hancock, P. (2012) “The Impact of Online

Lecture Recordings on Student Performance”, Australasian Journal of

Educational Technology, 28 (2), pp.199-213.

Wong, L. (2013) “Student Engagement with Online Resources and its Impact

on Learning Outcomes”, Journal of Information Technology Education:

Innovations in Practice, 12, pp.129-46.



SELECTING SESSIONAL TUTORS:

A NEW AND EFFECTIVE PROCESS*

Carl W. Sherwood & Bruce Littleboy

School of Economics,

University of Queensland

ABSTRACT

Devising an effective way to select high-performing sessional (casual) tutors from

a pool of applicants is relatively unexplored. Standard methods are to rank them

by their grade point average or to interview them one at a time. We report on a

new selection process based on observing a small group activity where applicants

discuss an unseen newspaper article and devise a tutorial question based on it. By

selecting tutors who demonstrate good knowledge, communication and inter-

personal skills, student evaluations of tutors have increased. Costs and benefits of

switching to this new process are identified and discussed.

Keywords: tutorials, small group teaching, sessional teaching, tutor selection.


1. INTRODUCTION

Teachers and administrators aspire to improve the quality of the entire

learning process. We focus here on tutorials, small classes conducted

to support lectures. Tutor training is widely recognised as important,

but the quality and effectiveness of tutor training may vary. Training

and support is also costly. Better and more cost-effective tutors are

more likely to emerge if the applicants with greater potential are

initially selected. We argue that ‘better’ tutors are those who are

* Correspondence: Carl W. Sherwood, School of Economics, University of Queensland,

St Lucia, QLD 4072, Australia; Email: [email protected]; tel: +61-7-3365-6563;

fax: +61-7-3365-7299; Bruce Littleboy may be contacted at School of Economics,

University of Queensland, St Lucia, QLD 4072, Australia; Email: [email protected].

We are most grateful for the referee’s helpful comments and suggestions.


mailto:[email protected]

Selecting Sessional Tutors 31

already well organised, articulate and have strong interpersonal skills.

The task is then to identify such applicants before hiring them.

Applicants for senior teaching or research positions typically deliver a

staff seminar, but tutors are not similarly screened prior to appointment.

Remarkably little attention has been devoted to selecting tutors from

a pool of applicants. In the corporate world, the filtering of prospective

employees can be elaborate. However, in universities, tutors are

regarded as at the bottom of the academic hierarchy. Yet tutorials are

often students’ most direct source of engagement with a course,

especially when enrolments are very large. If tutorials matter, so does

the selection of tutors. Whatever ‘good teaching’ is, obtaining high tutor

evaluations nowadays from students is a managerial imperative. This

paper shows how the median overall evaluation scores of tutors can be

raised by changing the selection process.

We have located no literature that directly relates to any tutor

selection process and consequently, no analysis of subsequent tutor

performance. In particular, there is no analysis relating to tutor

communication and interpersonal skills during interview and to

subsequent tutor evaluation by students. There is a broader literature

on training and certification to achieve competence; for example,

Angrist & Guryan (2004), Becker (1997, p. 1351). Bosshardt & Watts

(2001) note that while students value speaking-skills and enthusiasm,

they value preparation and organisation more. Ragan & Walia (2010)

likewise point to the value of preparation. Bianchini et al. (2013) show

that instructor characteristics such as age and seniority, and perhaps

gender, can affect student evaluations. Korur & Eryilmaz (2012) report

on the ethnic and gender preferences of students for teachers in Turkey.

Some of these criteria are clearly inappropriate for appointing tutors.

This paper discusses a pilot selection process used during 2012 and

2013. Rising student tutor evaluations of tutors has encouraged its

ongoing use. Additional data have been collected since 2013 with the

intention to perform a more detailed statistical analysis of the selection

process and tutor evaluations. Here we simply provide an analysis

using descriptive statistics after the first year of implementation.

Processes for selecting tutors often falls into one of two conventional

kinds. One is to rank applicants using administrative criteria such as

Grade Point Average (GPA). The other is to judge an applicant’s

performance in front of an interview panel. We argue for a third

method, a small-group interview that allows better identification of

32 C.W. Sherwood & B. Littleboy

those likely to receive high student evaluation scores. We hold that

tutor skills revolving around communication, organisational, and

interpersonal skills matter in a tutor far more than grade point average

(GPA) or performance in a standard interview. Good teachers have an

ability to communicate well, and this attribute is looked for in

applicants. Successful applicants later only receive one day of training

before taking their first tutorial, so choosing well is important.

We aim to find tutors with an aptitude for communicating and an

ability to facilitate work in smaller groups. Tutorials typically have up

to 25 students. We assigned a group of three applicants a task that

relates to a graduate attribute outcome, namely an ability to apply

economic concepts in practice. We assigned an unseen newspaper

article story to the group and invited them to design a suitable tutorial

question based on the economic issues that the story related to. A panel

observed how well they responded.

First, we aim to identify those tutors likely to be high performers and

to reduce the number performing below a specified benchmark.

Success is measured by increases in the median overall tutor

effectiveness using Tutor Evaluation survey data. Second, we

investigate which of the criteria may help account for any improvement

in tutor evaluations. The criteria used to evaluate applicants in

interviews deliberately match criteria students use in official surveys to

evaluate tutor effectiveness. Third, we investigate the link between an

applicant’s GPA and their perceived effectiveness as a tutor. There are

three hypotheses:

Hypothesis 1: The new process increases the median

evaluation score.

Hypothesis 2: Good communication skills are essential for

being a good tutor.

Hypothesis 3: An applicant’s GPA is a necessary but not

sufficient condition for being an outstanding

tutor.

The following section describes the old selection process and its

shortcomings. The next section explains how the new interview process

works. A statistical analysis section then evaluates the results of the

new process. We then consider the wider costs and benefits of making

the change to the new interview process. Conclusions and reflections

follow.


2. THE OLD SELECTION PROCESS

Since at least 2001, the School has advertised in October for tutors for

the following year. There are more high-quality applications each year

than available tutoring positions. Applicants were filtered according to

their overall course work GPA. Applicants with a GPA of 6 or more

were typically shortlisted and interviewed. Some categories of

applicants were given priority for an interview such as the fourth-year

Honours and PhD students. A tutorial position for these students would

provide income during their studies while also giving them teaching

experience useful for possible future academic careers.

After applicants were shortlisted, a panel of two academic staff and

at least one administrative staff-member interviewed each applicant

individually. This would take place in November and lasted for about

15 minutes. The interview used standard questions such as: What do

you think makes a good tutor? Why do you want to be a tutor? How

would you encourage students to participate in a tutorial? But clichéd

questions invite clichéd responses, and those already interviewed were

clearly informing those yet to be interviewed of the questions being

asked. Applicants had limited opportunities to demonstrate their ability

to perform in a tutorial setting. The process was also time-consuming

and regarded as a chore. The same people were reused as interviewers

as it was difficult to encourage other staff members to participate.

Typically, those involved were lecturers from very large undergraduate

courses that required many tutors, a senior School administrator, and

the administrative staff who interacted regularly with the tutors. While

promoting consistency within and between years, the selection process

was not transparent or even known to exist by most staff in the School.

At the end of the interviews in November, the great majority of

applicants interviewed were offered positions. In essence, the real

selection process was occurring pre-interview by administrative staff

and was driven by applicants’ GPAs. The absolute minimum number

of suitable applicants were shortlisted for an interview, and the process

had simply became a ritual, a procedure used to confirm each

applicant’s suitability. By 2011, up to 60 applicants were required to

be interviewed to fill an estimated 50 to 60 new tutor positions in 2012

(with around 30 existing tutors continuing). This ritual had also now

become very time consuming. With only 3 applicants interviewed per

hour, the process was requiring up to 20 hours to complete. The process

was clearly in need of change.


3. THE NEW SELECTION PROCESS

To widen the pool of applicants able to be interviewed, the overall GPA

threshold required was lowered from 6 to 5.5. This meant that almost

100 applicants were interviewed at the end of 2012. This was easily the

most applicants the School ever interviewed (about double the number

from 2011). To do so, the interview process needed to be completely

redesigned.

The new group-based interview process was designed so each

interview session was strictly limited to 20 minutes, followed by 10

minutes for staff to discuss the applicants. Each group interviewed

consisted of 3 applicants, and so 6 applicants could be processed in an

hour. Another room was used to conduct a concurrent interview,

thereby ensuring 12 applicants per hour could be interviewed. On any

one day, interviews were conducted for 1.5 hours, resulting in a total of

18 applicants being process per day. Therefore, by conducting

interviews on five different days across a two week period, up to 90

applicants were able to be interviewed.

The group-based interview involved organizing three randomly

assigned applicants to arrive at the interview and sit around a table. A

minimum of two applicants in a group was permitted if an applicant

failed to arrive, but a group of four was not allowed. Scribble paper and

pens were provided, and a whiteboard was also available for use by the

applicants. A chief interviewer welcomed the applicants and read out

brief instructions on what was required. The group’s task was to

collaboratively create a tutorial question from a randomly assigned

newspaper article presented to them within ten minutes.

We selected nine different newspaper articles prior to the interviews,

with each article of similar length and complexity. Some of the titles

were “Nowhere to run if you're on a menu”,1 “Hospital parking fees

enough to make you sick”,2 and “Easter holidays to deliver fuel price

hikes for motorists”.3 Each interview group was assigned an article so

as to reduce the chance that later groups received the same article.

When two different groups were being observed, in two different

rooms, both rooms used the same article.

Applicants were not forewarned of the newspaper article they would

be given or who would be in their group. They were advised of the

1 N. Mirosch, Courier Mail, 23 March 2011. 2 P. Mickelburough, Herald Sun, 17 March 2012. 3 A. Kelly, The Sunday Mail (Qld), 1 April, 2012.


interview process and requirements, but they were prevented from

preparing or sharing model answers. The three applicants were in a

sense directly competing in the interview for limited positions.

However, those who dominated or tried too hard to impress were likely

to be assessed lower. Tutors need to work well with students, other

tutors, lecturers and course administrators. The process thus aims to

identify natural collaborators and facilitators. The selection approach is

based on the belief that good teaching requires both an individual ability

and teamwork.

During each interview, there would be the chief interviewer present

in all interviews, while other observers would assist the chief

interviewer. These observers typically consisted of another two

academics and one administrative staff, giving a total of four staff in

one interview. At the end of the collaborative 10 minutes working time,

each applicant was then given the opportunity to individually answer a

question asked by the chief interviewer. Each applicant had to answer

the same question. At least three questions were asked, with a different

applicant given the opportunity to answer each question first. The

questions involved each applicant reflecting on the interview process,

what they had learned, and how their experiences in the interview could

help them as a tutor. This question time was limited to 10 minutes. At

the end of these questions, the 20 minute interview was closed by the

chief interviewer, the applicants thanked for their contributions and

interest in the position, and the applicants left the room. The staff

members in the room then recorded a score for each applicant using a

scoring sheet with certain criteria.

There were five criteria used to score and rank the applicants,

namely:

i. appropriateness of their questions and answers;

ii. communication skills;

iii. interpersonal skills;

iv. provides evidence of encouraging student participation;

v. potential for being a tutor.

Each applicant was scored out of five on each criterion by each staff

member, giving a maximum possible score of 25. Both the observers

and chief interviewer provisionally scored each applicant

independently. Then the observers and chief interviewer discussed all

three applicant’s performance for no longer than five to six minutes.


The chief interviewer then determined and recorded a score for each

applicant taking into account the observers’ scores and comments.

Brief qualitative comments were also recorded on all marking sheets

for each applicant. In practice, it was found a score of 21 would see an

interviewee offered a position. This new tutor selection process was

aimed at eliminating applicants assessed as likely to perform below the

School’s high level of expectation. Many of the rejected applicants

interviewed would likely be good tutors, but competition is always

strong.

4. DATA COLLECTION

At the University of Queensland, the Teaching & Educational

Development Institute (TEDI) prepares and processes a paper-based

student evaluation survey on each tutor’s performance. This is done for

each tutorial at the end of every semester. The results are then made

available electronically to the School and a copy provided to individual

tutors several weeks later. These surveys are the key source of data

used. We collected data for each semester from semester 1, 2011 to

semester 2, 2013. TEDI’s Tutor Evaluation survey uses a 5 point Likert

Scale (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 =

strongly agree) to generate various TEVAL (Teaching Evaluation)

scores. These scores are an average of all student responses to survey

questions in a particular tutorial group. The survey asks students to rate

the extent to which:

Q1. The tutor was well prepared;

Q2. The tutor communicated clearly;

Q3. The tutor was approachable;

Q4. The tutor inspired me to learn;

Q5. The tutor encouraged student input;

Q6. The tutor treated students with respect;

Q7. The tutor gave helpful advice; and

Q8. Overall, how would you rate this tutor?

The vast majority of tutors typically take a minimum of two tutorial

groups each semester, with about 10 to 15 students per tutorial

completing each tutor evaluation at the end of the semester. Therefore,

a tutor employed for two semesters typically receives about 50

responses during a year to each survey question. It is Q8 score on the

TEVAL that typically attracts the most attention from administrative


and teaching staff in assessing a tutor’s overall effectiveness and

performance. A Q8 TEVAL score of 4 is regarded as satisfactory and

deserving automatic re-appointment, subject to enrolment numbers, in

the following semester. Tutors with a score at or below approximately

3.8 each semester are categorised as underperforming. Prior to 2013,

tutors scoring below 4 were normally offered support or mentored to

help them to improve. However, if no improvement occurred, it seemed

these underperforming tutors remained in the system.

5. DATA ANALYSIS

We analyse the data in relation to each of the hypotheses outlined in the

Introduction.

Hypothesis 1: The new process increases the median

evaluation score.

In 2011, 407 tutorials were evaluated involving 96 different tutors. The

median number of responses per class was 12.5. In 2012, 367 tutorials

were evaluated with 98 different tutors. All tutors employed during

2011 and 2012 were interviewed using the old selection process. In

2013, 427 tutorials were evaluated with 96 different tutors consisting of

45 interviewed using the new process and 51 continuing tutors

appointed using old selection process. The median number of responses

per tutorial group in 2013 was 10.8.

Figure 1: Relative Frequencies of Q8 TEVAL Scores in

Various Cohorts of Tutors.

0%

5%

10%

15%

20%

25%

30%

35%

3.2 3.4 3.6 3.8 4 4.2 4.4 4.6 4.8 5

Rel

ati

ve

Fre

qu

ency

Q8 TEVAL Score

2011 old process 2012 old process

2013 new process 2013 old process


Using 2011 and 2012 as benchmark control years, and 2013 as the

intervention year, the distribution of Q8 TEVAL average scores for all

tutors, a measure of their overall effectiveness, is presented in Figure 1.

In reading Figure 1, consider the four bars shown where the Q8 TEVAL

score is 5. The four relative frequency values, for each year, have been

calculated using all tutors’ Q8 TEVAL average scores that were more

than 4.80 and less than or equal to 5.

Figure 1 shows the proportion of tutors and their overall effectiveness

score in a particular year. The proportions of tutors recruited using the

new interview process in 2013 (the solid black bar), who score at 4.4,

4.6, and 4.8, are clearly higher than the proportions where tutors were

recruited under the old interview process used in 2011 and 2012. In

addition, for these same scores of 4.4, 4.6 and 4.8, the proportions of

tutors recruited under the new process in 2013 are also clearly higher

than the proportion of tutors recruited under the old process and tutoring

in 2013. The distribution has shifted toward the right reflecting an

increase in the median evaluation of the median score of the tutors

recruited under the new process compared to the old process. This is

confirmed by the data presented in Table 1 using individual student

evaluation data, with the median Q8 score being 4.57 in 2013 (new

process) compared to 4.42 and 4.41 in 2011 and 2012 respectively using

the old process.

Table 1: Summary Statistics for Q8 TEVAL Scores and Different

Tutor Cohorts.

Cohort

Year

Interview

System Used

Number of

Tutors

Median

TEVAL

Score

Mean

TEVAL

Score

Standard

Deviation

2011 Old 96 4.42 4.36 0.32

2012 Old 98 4.41 4.36 0.39

2013 New 45 4.57 4.49 0.27

2013 Old 51 4.48 4.41 0.40

It is noted that our tutors under the old system had a reputation for

being high quality and performing well. Indeed, a median score of 4.4

for almost 100 tutors would generally be considered impressive. It was

anticipated that it could be difficult to observe any clear improvement.


Figure 2: Cumulative Frequency Distribution (% Cohort with

Q8 TEVAL Score at a Particular Value or Better).

Indeed, there are about 8% more tutors scoring at 4.8, recruited under

the new process in 2013, compared to the next best cohort of old-system

tutors (also 2013). At scores of 4.4 and 4.6 this margin is just less than

5% and 3% respectively. Similarly, at the highest scores of 4.8 and 5.0,

the new system cohort tutors are not appreciably superior. Indeed, these

old process tutors outperform the new process tutors in 2013. This

likely reflects the fact that better performing tutors are more likely to be

retained over time.

An alternative approach to evaluate the impact of the intervention is

presented in Figure 2. This allows for comparison of the cumulative

frequencies of Q8 TEVAL scores of tutors received from students. In

reading Figure 2, the cumulative frequencies for each year represent the

proportion of tutors scoring at a particular score or better. For example,

in 2013 using the new process, for a Q8 TEVAL score of 4.4, 91% of

tutors scored this value or more.

The scores for tutors recruited using the new process in 2013 are

shown, along with the scores for tutors from the old system employed

in 2013, 2012, and 2011 to benchmark against. What this reveals is that

91% of scores using the new interview process in 2013 were 4.4 or

better. This compares with only 79% of scores in 2013 from the old

process, and 71% and 69% of scores using the old process in 2012 and

2011 respectively. A cohort of tutors with 91% scoring at a TEVAL

score of 4.4 or better is considered to be excellent. It is rare to encounter

0%

20%

40%

60%

80%

100%

120%

3.2 3.4 3.6 3.8 4 4.2 4.4 4.6 4.8 5

Cu

mu

lati

ve

per

cen

tag

e o

f co

ho

rt

Q8 TEVAL Score

2011 old process 2012 old process

2013 new process 2013 old process


individual tutors able to achieve scores of over 4.8, let alone 5.

Moreover, when appointing around 100 tutors per year, it would be

extremely difficult to devise any selection process able to identify

sufficient tutors to reach such high TEVAL scores. However, the

incremental improvements that have been observed, where the

proportion of tutors scoring 4.4 has increased from 70% to over 90%,

has effectively shifted the quality of the School’s tutors to a higher

level. The median TEVAL increased from around 4.4 to almost 4.6,

which is a change from very good to outstanding. This improvement is

also reflected in the lower standard deviation value reported in

Table 1 for the new process compared to any value obtained under the

old process.

Hypothesis 2: Good communication skills are essential in being a

good tutor.

Over many years, student written and verbal feedback on tutor

performances have indicated students decided not to attend tutorials

because they simply could not understand what their tutor was trying to

say. This could result from not being able to clearly communicate

technical content in meaningful ways as well as not speaking clearly.

To evaluate the link between communication skills and being a good

tutor, TEVAL data relating to Q2 and Q8 were analysed. It revealed

that the correlation coefficient between communication (Q2 TEVAL)

and overall effectiveness as a tutor (Q8 TEVAL) was 0.91 (n= 96), 0.91

(n=98) and 0.94 (n=96) for tutors appointed in 2011, 2012, and 2013

respectively. There is clearly a strong positive correlation between a

tutor’s ability to communicate and their overall effectiveness score.

The new interview process pays particular attention to an applicants’

communication skills. Indeed, the five criteria used in the new selection

process in essence revolve around communication. Figure 3 shows the

relative frequency distribution of Q2 TEVAL scores (communication

ability) for the benchmark year of 2011, using the old process, and 2013

using the new process. This clearly shows that the proportion of tutors

having a Q2 TEVAL score from 4.2 to 5 is higher using the new process

compared to the old process. Figure 4 presents an alternative to

appreciate the impact on the new process on tutors’ communication

skills using a cumulative distribution. This reveals that more than 80%

of tutors have a Q2 TEVAL score of 4.4 or higher using the new process

compared to 65% using the old process. This again reflects an increase

in the median Q2 TEVAL scores, and is captured by the descriptive


Figure 3: Relative Frequencies of Q2 TEVAL Scores for

Tutors in 2011 and 2013.

Figure 4: Cumulative Frequency Distribution (% Cohort

with Q2 TEVAL Score at a Particular Value or

Better).

statistics presented in Table 2. Presumably, it would be desirable, other

things equal, to have tutors with higher communication skills, even if

student evaluations of tutor overall effectiveness were unaffected.

Hypothesis 3: An applicant’s GPA is a necessary but not sufficient

condition for being an outstanding tutor.

0%

5%

10%

15%

20%

25%

30%

35%

3.4 3.6 3.8 4 4.2 4.4 4.6 4.8 5

Rel

ati

ve

Fre

qu

ency

Q2 TEVAL Score

2011 old process 2013 new process

0%

20%

40%

60%

80%

100%

120%

3.4 3.6 3.8 4 4.2 4.4 4.6 4.8 5Cu

mu

lati

ve

per

cen

tag

e o

f co

ho

rt

Q2 TEVAL Score

2011 old process 2013 new process


If GPA could be found to be strongly and positively correlated with

tutor performance, then the interview process arguably could be

discarded. GPA data on tutors recruited for semester 1, 2013 for both

their overall program to the end of 2012, and their GPA on only the

economics courses they had completed were compiled. These were

compared with their semester 1 Q8 TEVAL and Q2 TEVAL scores.

The details of the correlation analysis is shown in Table 3. This reveals

that the tutor’s overall GPA is a weak predictor of a tutor’s overall

effectiveness (Q8) and communication skills (Q2). Further, the tutor’s

GPA for economics-coded courses is even a less reliable predictor.

Table 2: Summary Statistics for Q2 TEVAL Scores and Different

Tutor Cohorts.

Cohort

Year

Interview

System

Used

Number of

Tutors

Median

TEVAL

Score

Mean

TEVAL

Score

Standard

Deviation

2011 Old 96 4.37 4.30 0.38

2013 New 45 4.46 4.42 0.36

Table 3: Correlation Analysis for New Tutors who Started in 2013 (n=40).

Program GPA ECON GPA Q8 TEVAL

Average

Q2 TEVAL

Average

Program GPA 1

ECON GPA 0.74 1

Q8 TEVAL

average

0.22 0.07 1

Q2 TEVAL

average

0.25 0.11 0.93 1

Table 3 reveals there is only a weak positive correlation between both

program GPA and Economic (ECON) GPA scores with a tutor’s Q8

TEVAL score (0.22 and 0.07) and Q2 TEVAL score (0.25 and 0.11).

For tutors newly employed during 2013, their GPA for the economic

courses completed was also obtained, and it revealed that GPAs for

economics courses are even less important than the overall GPA. (There


were 5 tutors appointed in 2013 whose evaluations were not included.

Either they were not currently economics students or their program

GPAs were not available).

This analysis suggests there is no strong evidence for selecting tutors

based on GPA alone. Instead, the much stronger correlation between

the Q8 TEVAL (effective tutors) and Q2 TEVAL (communication)

scores of 0.93 suggests applicants’ communication abilities are a key

element needed as a second filter of applicants. A first filter, using a

GPA threshold as an indicator, still needs to be applied to at least ensure

the applicant knows the course material.

These findings overall are quite stark, but it is improper to move from

finding correlations to attributing causality. Here, however, there is a

simple and intuitive connection between cause and effect. Good

communication is not a controversial attribute of a good tutor.

Furthermore, there is quite a deal of overlap between the interview’s

selection criteria and the criteria on the evaluation forms, which is not

accidental. However, to claim that causality flows from the new

selection process to an improvement in tutor evaluations is warranted

only if other factors may reasonably be regarded as constant. (Clearly,

reverse causality is not an issue here as the prior decision consciously

to change the process was made independently). Other factors appear

to be substantially the same. There are no grounds to suppose that there

has been a sudden and inexplicable improvement in the quality of the

applicant pool. (An administrative officer confirms that the overall

profile of the applicants appears comparable over the last few years).

Neither is there anecdotal evidence of a sudden and mysterious rise in

the students’ appreciation of tutors of the kind selected and trained by

the School. We do not currently have access to data about survey

evaluations of tutors in other Schools, so we cannot gauge relative

performance or test whether numerical evaluations are somehow

framed by experience elsewhere.

Neither has the composition of the student intake suddenly changed.

Although tutorial attendance has declined a little, it is implausible that

the drop-outs had a particular distaste towards tutors of the kind

selected. (Our results should be viewed against the background of

declining average class-attendance rates across the campus, perhaps

reflecting improved internet access or the need to earn income). Tutor

training has also evolved over the period, but it has not been

transformed. The results are so clear and immediate that it is reasonable


to attribute the improvement in survey outcomes to the new tutor-

selection process. It is difficult to improve on something that is already

operating very well, but overall improvement has occurred. It may be

possible to move our tutors to an even higher level.

6. WEIGHING THE COSTS AND BENEFITS

Change requires that the extra benefits are expected to exceed the extra

costs. The new process appears no more costly in terms of material and

administrative resources. On the cost side, there are small additional

financial costs (such as photocopying of newspaper articles), and there

are extra time costs in preparing the interview activities, collating the

paperwork and in recruiting (voluntary) academic observers.

Comparing the new and old interview systems, there is some extra

administrative load, but the total commitment in time, once the

procedures are settled, is about the same and possibly less. There is

more paper per applicant to collate and store, but there is also a better

trail of documentation that justifies the decisions reached. The greater

rate of throughput means that junior administrative staff can better

manage the flows of people who attend the sessions. A senior

administrative officer no longer attends the interviews, a considerable

advantage. Instead of interviews taking at least 15 minutes per

applicant as under the old system, they take 10 minutes per applicant

under the new. Quicker processing of applicants has permitted

interviewing more diverse applicants. Several students with a high

GPA, who would likely have done well under the old process, did

poorly under the new process.

There has been an increase in the time devoted by academics, but this

involves not only a time cost but also diverse benefits to the volunteer

that were not anticipated. Academic staff had many conversations at

the end of each interview session about the strengths and weaknesses

of the applicants. In particular, in a School that currently has many new

academic appointments, this was an opportunity for real collegiality.

Academics, beyond those teaching large first year courses, saw how

well or poorly their star students in later year courses performed. This

prompted reflection on how they designed, prepared and delivered their

own courses.

The sessions are far more unpredictable and engaging for the

academic staff than the former, more bureaucratic, method. Applicants

soon realised that questions from others required them to respond with

intelligent answers framed creatively in the specific context. The


experience for the academic staff is unlike being dragooned to form part

of an old-style interview panel. During November 2013, 22 academic

staff (3 professors, 6 associate professors, 3 senior lecturers, and 10

lecturers) and 3 administrative staff participated. This is more than

eight times the number of academics involved in the old process.

Previously, it was also unlikely for any professors or associate

professors to have been involved, especially if they were not lecturing

in the large courses requiring tutors. Instead, coordinators in the first-

year courses ranked the applicants. This was often regarded as one of

their administrative duties as well as being in their interests to monitor

the quality of the intake. Having academic staff acting as observers

during interviews has clearly encouraged increased staff involvement.

We hope that positive experiences by academic staff may also result in

wider participation during our tutor training (and induction) day, and

perhaps to some revision of their own teaching aspirations and

practices.

Observers quickly gain experience as to how to evaluate the

applicants. In turn, this allows them to be promoted to chief

interviewer, which means the system tends naturally to spread the

workload over time. Promotion of observers to chief interviewer

occurred for interviews conducted at the end of 2012 and 2013.

Previous observers volunteered for the role of chief interviewer, simply

for the experience, on three separate occasions. The transition was

easily implemented and the observer was keen to participate in the role

again.

Importantly, the new tutor selection process also gives a head start to

subsequent tutor training. Applicants are given practical experience in

group work during the interview process. We confirmed what we

expected: as students, most applicants had not undertaken anything like

the interview activity during their classes. They realised how valuable

it can be as a learning experience (or had the good sense to say so). In

addition, academic staff observing the process also realised the

potential of using such activities in their classes and tutorials. If similar

group exercises were embedded in tutorials, then key graduate

attributes, notably an ability to apply theory to given problems, would

be demonstrably embedded in the program. One applicant observed

astutely that the interview set-up reversed the more standard teaching

process of using an example to fit a given concept.


The sessions are open to wider scrutiny and are observably arm’s-

length. In former practice, interview performance has not been the

exclusive criterion for tutor selection. Applicants who were Honours

or PhD students received automatic preference. (Honours students are

required to add an extra year to their three-year undergraduate

program). Positions are no longer offered on the basis of student

category. Those interviewees that ‘fail’ (score below about 20.5) are

not offered a tutoring position in the large first-year courses.

Supervisors of Honours, Masters or PhD students have no scope to

lobby or alter the rankings if their protégé does not ‘pass’ the interview.

They may, however, hire them for their own courses and accept

responsibility for their performance.

Some may believe it is more reliable having applicants assessed by

the same interview panel and under conditions as close as possible to

identical. However, any advantages offered by uniformity in this

selection process may be overstated. Uniformity can lead to a

predictable selection system meaning applicants can more readily

simulate the attributes desired. All sessions in the new selection process

are conducted according to the same protocols. However, what varies

are the newspaper articles, questions and knowledge applicants must

draw upon, and the combination of observers in each session. The

observers provide moderation. Almost all interviews are currently

conducted by only two chief interviewers, so basic consistency is

readily established.

Applicants with poor language skills or shy dispositions tend to fall

in the rankings. Selecting for desired attributes (interpersonal,

organisational and communication skills) still results in a gender-

balanced and ethnically diverse range of tutors. Note that a more

diverse range of staff involved in a more open selection process is also

likely to guard against interviewer bias.

7. REFLECTIONS AND CONCLUSIONS

The new process is quicker, more engaging for a wider range of

participants, and it delivers better results. Although tutor performance

is multi-factorial, which can make reliable statistical associations

difficult to identify, the net benefits of switching to the new process of

selection seem clear. When resources for training are scarce, it makes

sense to select those who are more likely to emerge as good tutors.

Since a tutor position is essentially the bottom rung of the academic

ladder, and appointments are increasingly on a casual basis of payment


per hour, university administrators and academic staff perhaps pay

rather less attention to ranking applicants to perform tasks widely

regarded as being less skilled and of low financial value. Risks of

deterioration in the quality of the tutorial experience need actively to be

countered by devoting greater care to tutor selection.

Those entrusted to decide appointments should use criteria that

withstand inspection. Selection criteria need to align with goals. If the

goal of providing tutorials is remedial and intended for students in

difficulty, then patience and clarity in a tutor are at a premium.

Guidance during laboratory or practical work calls on different skills

and conducting fieldwork still others. There appears to have been little

published reflection, and still less empirical work, done on framing

selection practices to suit whatever learning or motivational goals are

assigned to tutorials.

A potential objection to a more targeted selection process is that we

should be more neutral about traits and should not select on too narrow

a range of preconceived criteria. A portfolio of different types of tutors

may be appropriate, at least in high-enrolment courses in which means

exist for students to migrate towards tutors with qualities they find

congenial. So far as we know, however, Australian universities do not

yet provide quiet tutors for shy students or run designated bilingual

classes specifically for international students, but this day may come.

Change requires that the extra benefits are expected to exceed the extra

costs. The new process appears no more costly in terms of material and

administrative resources. On the cost side, there are small additional

financial costs (such as photocopying of newspaper articles), and there

are extra time costs in preparing the interview activities, collating the

paperwork and in recruiting (voluntary) academic observers.

REFERENCES

Angrist, J.D. and Guryan, J. (2004) “Teacher Testing, Teacher Education

and Teacher Characteristics”, American Economic Review, 94 (2), pp.241-

246.

Becker, W.E. (1997) “Teaching Economics to Undergraduates”, Journal of

Economic Literature, 35 (3), pp.1347-1373.

Bianchini, S., Lissoni, F., and Pezzoni, M. (2013) “Instructor Characteristics

and Students’ Evaluation of Teaching Effectiveness: Evidence from an

Italian Engineering School”, European Journal of Engineering

Education, 38 (1), pp.38-57.


Bosshardt W. and Watts, M. (2001) “Comparing Student and Instructor

Evaluations of Teaching”, Journal of Economic Education, 32 (1), pp.3-

17.

Korur, F. and Eryilmaz, A. (2012) “Teachers’ and Students’ Perceptions of

Effective Physics Teacher Characteristics”, Eurasian Journal of

Educational Research, 46, pp.101-120.

Ragan, J.F. and Walia, B. (2010) “Differences in Student Evaluations of

Principles and Other Economics Courses and the Allocation of Faculty

across Courses”, Journal of Economic Education, 41 (4), pp.335-352.



BOOK REVIEW

Review of Salemi M.K. & Walstad W.B. (eds.) (2010), Teaching

Innovations in Economics: Strategies and Applications for

Interactive Instruction, Cheltenham, UK & Northampton MA,

USA: Edward Elgar, 274 pp + xiii.

Peter Docherty

Business School Economics Group,

University of Technology, Sydney

Teaching Innovations in Economics is an extremely useful book that

every thoughtful economics instructor ought to read. It is a set of

collected essays produced from the Teaching Innovations Program

(TIP) initiative of the American Economic Association’s Committee on

Economic Education between 2004 and 2010. As explained in Michael

Salemi’s opening chapter, this program was taught by a number of

leading economic educators in the United States, and was aimed at

enhancing the teaching skills of university and college instructors. It

thus represented a type of professional development program for these

instructors. Salemi describes its three part structure as made up: firstly,

of a set of ten face to face workshops that explored a range of non-

traditional pedagogical strategies; secondly, of a set of follow-up, on-

line modules completed when participants returned to their respective

universities; and thirdly, of projects that applied one or more of the

pedagogical strategies considered in the face to face and online sessions

to participants’ own classrooms.

The first three chapters of the book explore the TIP initiative in

further detail. Salemi’s opening chapter provides some background to

the initiative, describes its overall objectives, and outlines the structure

of the workshop phase. Mark Maier and Tisha Emerson’s second

chapter examines the on-line phase of the program and describes how

it built upon the workshops that participants had attended in phase one.

KimMarie McGoldrick’s chapter then discusses the program’s project

phase but she does this within a broader discussion of the scholarship

50 P. Docherty

of teaching and learning in economics. The last chapter of the volume

by William Walstad reflects back on the program at its conclusion and

carefully considers participant feedback. The intervening seven

chapters, all authored by program participants, report on a variety of the

projects conducted during stage three of the program. They include such

initiatives as cooperative learning, use of experiments in the classroom,

discussion techniques, formative assessment strategies, context rich

problems and case studies, and techniques designed for large classes.

One of the most valuable features of the book is its detailed description

and evaluation of these techniques. Each of these chapters provides a

rationale for the initiative described, a detailed description of the

initiative itself, and some kind of evaluation of or reflection on the

effectiveness of the strategy.

A number of the chapters on specific pedagogical strategies are

particularly interesting. Two addressed the issue of involving students

in class discussion, one of the most obvious strategies for increasing

student engagement. These were Chapter 6 entitled Classroom

Discussion by Michael Salemi, Kirsten Madden, Roisin O’Sullivan and

Prathibha Joshi, and Chapter 9 entitled Case Use in Economics

Instruction by Patrick Conway, Derek Stimel, Ann E. Davis and

Monica Hartmann.

In the first of these, Salemi et al. make a distinction between

structured and unstructured discussion. Structured discussion requires

students to read an article or book chapter prior to the class and answer

a series of questions which focus on either some factual dimension of

the reading or a key concept developed in the reading. Students are then

asked to contribute to answering these questions in class and the

instructor withholds any evaluative comment at least until towards the

end of the class to see how students react to one another’s answers.

Alternatively students may be required to interpret the article and

explain their interpretation. Unstructured discussion on the other hand,

also requires students to read an article in advance of the class but poses

a smaller number of general questions, and requires the instructor to

provide more context and guidance as the discussion progresses.

Three case studies that employed various discussion techniques and

collected feedback data from students are provided by the authors of

this chapter, reflecting their experiences in History of Economic

Thought, Principles of Macroeconomics and Intermediate

Macroeconomics classes. They conclude that discussion techniques are

Book Review - Teaching Innovations in Economics 51

less preferred by more junior students, especially those in first year, but

that exam performance is enhanced compared to classes where lecturing

is the only pedagogical technique employed. They also find that student

feedback improves as discussion is used more frequently and students

learn how to engage with it more effectively.

This chapter is useful because it formalises one of the most basic but

productive pedagogical strategies for moving teachers from simply

lecturing, towards a mode of delivery that more actively involves

students in classroom learning. William Becker has consistently argued

that such a movement needs to be one of the highest priorities for

enhancing university economics instruction, both because active

engagement of students is more educationally effective than simply

lecturing, and because economists under-utilise active techniques in

favour of lecturing (see, for example, Becker 1997, pp.1354-55; and

2000, p.113). Classroom discussion is a technique that some of us use

somewhat intuitively; in fact the description of structured discussion

provided by Salemi et al. sounds remarkably like the tutorials of

between 10 and 15 students I taught in first year economics at the

University of Sydney at the beginning of my career. To some extent one

may wonder whether this approach is really that innovative. But by

formalising the technique, Salemi et al. also extend it, because

formalisation allows us to analyse how the method works and thus to

consider possibilities for variation and adaptation that have the potential

for even more interesting classroom experiences.

The chapter on Case Use in Economics, extends the range of

possibilities for classroom discussion even further, but also relates these

possibilities to the additional idea that what one discusses is important.

Conway et al. argue that discussing real world phenomena provides

additional motivation for students who seem to care about the

applicability of the ideas they encounter in university courses. The case

method, used so widely in management education, enables the

instructor, they suggest, to bring together active student involvement in

class discussion, a consideration of real economic phenomena, and

greater use of formative assessment structures that provide on-going

feedback to students as the case unfolds.

I have personally been convinced of the value of the case method in

economics for some time, partly because of the number of MBA

students taught by my department and partly because of the career-

focused undergraduates that my university attracts. The value of

52 P. Docherty

looking at real cases is also supported philosophically since the

objective of economics is to explain actual economic phenomena. Even

high theory, for which I had a passion as a graduate student, is

ultimately of value only if it provides insight into the nature of

economic reality. Teaching students good theory and using that theory

to examine actual economic problems, whether of a business or a policy

nature, does not seem to me a difficult proposition to support, whoever

are the students under consideration. The Conway chapter is thus one

of the most useful chapters in the book from this perspective. If I have

any criticism, it is that the particular cases chosen by the authors do not

go far enough in terms of realism. The case reported to have been

employed in a macroeconomics subject of whether hypothetical George

should accept baseball cards as payment for washing a car, while

perhaps highlighting the essential features of money, is nonetheless a

relatively trivial and overly individualised case for macroeconomics, in

my view. Something like an examination of whether the FOMC should

purchase more securities in the face of continuing weakness in the U.S.

economy after the Global Financial Crisis (see Whiting 2006 for a case

along these lines) would be much more realistic.

The chapter by Mark Maier, Joann Bangs, Niels-Hugo Blunch and

Brian Peterson on Context-rich Problems to some degree unpacks the

value that active student engagement and real world problems have for

learning. This chapter argues that providing students with a well-

developed context that requires them to sift what is important and what

isn’t from the information at their disposal, and to use economic

principles in this identification process, helps them to understand these

principles more thoroughly and also develops application skills they

will use after graduation. The chapter’s emphasis, however, is mainly

on hypothetical scenarios designed to highlight the particular concepts

under consideration. My feeling, once again, is that while such

techniques may be more interesting and productive than simply

lecturing the textbook model, looking at real problems has an

authenticity and integrity that is hard to beat.

The chapters discussed so far tend to focus on the primary means

used to have students confront economic ideas that are new to them.

The chapter by William Walstad, Michael Curme, Katherine Silz

Carson and Indradeep Ghosh examines the question of how assessment

structures can be used formatively as part of the learning process rather

than simply summatively to evaluate post-facto what students have


learned (Ramsden 1992, p.184). A range of methods and course

contexts is explored in this chapter for using formative assessment

methods from in-class quizzes in microeconomic principles to short

assignments in econometrics, to multiple writing assignments in

monetary economics.

But a curious and disappointing shortcoming characterises this

chapter. A fundamental dimension of the distinction between formative

and summative assessment is that formative assessment provides

students with feedback that they may use to modify their understanding

of the material. A well-structured formative assessment regime also

gives students the opportunity to use this modified understanding in

further assessment tasks (Sadler 1998). The nature and quality of

feedback is thus central to the effectiveness of formative assessment as

a teaching/learning device. But this point is almost completely absent

from the chapter. In the opening section that lays the conceptual

foundations for the chapter, the definition of summative assessment

offered focuses only on its ex-post dimension and not on the more

central dimension of taking stock of student knowledge or skill

development. In describing formative assessment, the concept of

feedback is raised, but it is principally feedback received by the

instructor about the student (an essentially summative idea) that the

instructor may then use to modify his or her approach later in the course.

While this is certainly a useful aspect of formative assessment

structures, it is not the kind of feedback that lies at their heart, and

careful consideration of feedback to students on their learning is

essentially absent from the opening section.

To a large degree the same neglect characterises each of the

assessment initiatives described in the rest of the chapter. The test-retest

strategy of the microeconomic principles course involved a test on day

one before anything in the course had been learned at all, and a retest

on the final day. This seemed like a more elaborate summative

procedure that tested the increment to student knowledge developed

across the course rather than a formative strategy, although the group

project of looking at one of the questions in the pre-test did move in the

right formative direction. Little information about the nature of

feedback from the pre-test was provided in the discussion of the

initiative. The regular in-class testing in the econometrics initiative may

well have provided effective formative feedback if the final assessment

was properly aligned with these earlier assessments and feedback drew

54 P. Docherty

student attention to the knowledge or skills that were the focus of this

alignment, but insufficient information was provided to determine

whether this was the case. The most interesting assessment design was

the monetary economics writing assignment where students had three

opportunities to write across two assignments and a seemingly aligned

final exam. Here again, the nature of feedback to students on the earlier

writing opportunities and the ability of students to feed learning forward

into the next assessment task did not appear to be an important focus of

the authors’ discussion of this initiative. In the end, it was difficult not

to be disappointed with this chapter.

By contrast, it was good to see a chapter on interactive approaches in

large class teaching and the various suggestions offered by Gail Hoyt

and her co-authors provide a useful guide to making this challenging

teaching assignment more productive for the student and more

enjoyable for the instructor. I was also convinced by Denise Hazlett and

her co-authors that class experiments can be extremely effective for

teaching microeconomics in particular, because the class becomes a

micro-market and the behaviour under investigation is more directly

observable.

A set of questions, however, ought to be posed and carefully

considered in every course where class experiments are deployed.

These questions involve: the degree to which the Hawthorn effect

operates in such circumstances and whether the contrived nature of the

experimental micro-market potentially alters behaviour; a discussion of

what it actually is that is being directly observed, and what assumptions

are being explicitly made about such observations; and a consideration

of the differences that real economic institutions and contexts might

make to the behaviour in question.

I also found KimMarie McGoldrick and her co-authors’ chapter on

Co-operative Learning useful, especially the section outlining the key

elements that should characterise an effective design of this strategy.

The ability to work co-operatively is clearly a skill that most business

students will need after graduation, but designing effective group work

is a bane for many business school instructors. The set up cost for such

approaches to teaching is unfortunately high but the potential for much

more interesting student learning experiences is also significant, and

having some direction for shaping these approaches that avoids wasting

time on less effective strategies is an extremely useful contribution of

the McGoldrick et al. chapter.


Overall, Salemi and Walstad’s book is a useful contribution to

thinking about economics education. The range of pedagogical

techniques considered, the rationales outlined, and the evidence

collected from student feedback and performance on many of the

initiatives trialled, provide useful stimuli for university teachers of

economics to reflect upon and to use in designing their own approaches

to teaching.

REFERENCES

Becker W.E. (1997), “Teaching Economics to Undergraduates”, Journal of

Economic Literature, 35, September, pp.1347-73.

Becker W.E. (2000), “Teaching Economics in the 21st Century”, Journal of

Economic Perspectives, 14 (1), pp.109-119.

Ramsden P. (1992) Learning to Teach in Higher Education, London:

Routledge.

Sadler D.R. (1998), “Formative Assessment: Revisiting the Territory”,

Assessment in Education: Principles, Policy & Practice, 5 (1), pp.77-84.

Whiting C. (2006), “Data-based Active Learning in the Principles of

Macroeconomics Course: A Mock FOMC Meeting”, Journal of Economic

Education, 37 (2), pp.171-177.

Australasian journal of Economics Education 13, Number... · 1. Report research on the teaching of...

Documents

Transcript of Australasian journal of Economics Education 13, Number... · 1. Report research on the teaching of...