Bolouri qualitative method
Transcript of Bolouri qualitative method
Maryam Bolouri
Mixed method Research
Toward New Designs in
Applied LinguisticsHashemi &Babaii
MLJ 97,4, 2013
Movement in inquiry in AL
• Emergence of Mixed method• Driving forces behind this shift:1.Advances in natural science2.Development and challenges of social and
educational research3.Criticism against the dichotomy of
qualitative and quantitive research methods
Statement of problem:
• How these 2 methods can be integrated usefully in a way to enhance the research tools?
• How these 2 methods can be integrated usefully in a way to enhance the quality of obtained findings?
Purpose of the study:
• This study attempted to be a first move to study
1.The current status of mixed methods research in AL
2.Explore the nature of mixed research designs, sampling designs and the quality of inferences that can be obtained
Review of literature• Mixed method definition:
1.Simple definition by Bergman (2008): the combination of at least one qualitative and at least one quantitive component in a single research project or program
2.Technical definition by Hanson et. al (2005): the collection, analysis and integration of quantitive and qualitative data in a single or multiphase study
3.Extended definition by Tashakkori & Creswell (2007a): research in which the investigator collects, and analyzes data integrate the findings and draws inferences using both qualitative and quantitive approaches in a single study or program of inquiry
Mixed method origins• Mixed method research has evolved from mono method
research.• The first instances of mixing designs can be traced back to
1800s.• It has been influenced by the emergence of the
“Triangulation”• Navigation: taking bearings on 2 landmarks in order to
locate one’s position• Land surveying: determine the position of point C if we
know the positions of points A and B• Original definition: the use of multiple forms of qualitative
research methods in favor of richly developed multidimensional descriptions of the world
Theoretical issues in mixed method research
• Incompatibility thesis • commensurability validity• Pragmatist position (dialectical, balanced,
transformative or critical interpretive view):• From practice oriented perspective. It is not a major
hindrance, the value is evident and they can coexist in a dialectical relationship.
• The findings can enhance the understanding.• The absent one would remain partially answered or
wholly unanswered questions.
Methodological issues:in mixed method research
• Bazeley (2004): corruption of methods“It more that using a variety of methods….it is an attempt
to serve the dual purposes of generalization and in-depth understanding…”
• High quality mixed method research:Integrations happens to RQ, units of analysis, sample,
instrumentation, data collection methods and analytic strategies
Mixed method validity issues: (Onwuegbuzie &Johnson, 2006): legitimation
• Complementary strength and non-overlapping weakness of both are combined, so the validity
assessment is important and complex.• (Tashakkori & Teddlie, 2008): inference quality• It gathers info that result in meta inferences that
neither of method can do alone. They are an overall conclusion through integration.
Methodological issues:in mixed method research
Methodological issues:in mixed method research
• (Tashakkori & Teddlie, 2003): Inference• A researcher’s construction of the
relationship among people, events, and variables as well as the his/her
construction of respondent’s perceptions, behaviors, and feelings, and how there
relate to each other in a coherent and systematic manner
Research questions:
1. What kinds of research designs are used when method mixing is utilized in AL research?
2. What kinds of sampling designs are used in mixed methods research in the field of AL?
3. How are general conclusions or meta inferences developed in this type of research?
Methodology
1. Data gathering• How to select a representative sample
of all journals?2. Data analysis• What frameworks and typologies
inform data analysis?
Data gathering• 7 comprehensive international peer reviewed AL
journals• Over a period of 14 years 1995---2008 • Journal selection: based on 5 sources:1.The list of professional journals published by MLJ- Weber &
Campbell, 20042.Evaluation of AL journals -Egbert (2007)3.Examination of the frequency of the appearance of ELT
journals selected for presentation in lg teaching btw 1996-2002-Jung, (2004)
4.A study on the use of qualitative and quantitative methods in 332 articles over a 7 year period- Lazarton (2000)
5.Expert judgment (2 PhD holders with research experience of more than 10 years and currently active in the field of AL)
Data gathering (cont.)• The first four sources yielded a comprehensive list of
64 journals• Expert judgment• This exclusion was based on the following five
reasons:1.An indirect relationship to the field of AL such as:
computers and compositions, British journal of educational psychology.
2.Publishing in lg other than English3.Insufficient info in the methodology section of
articles4.Lesser prominence on the international scene5.Strongly delimited research focus such as:
assessing writing, lg and cognitive process
The final list of Journals1. Applied linguistics-AL2. English for specific purpose-ESP3. Language learning-LL4. Language teaching research-LTR5. Language testing-LT6. The modern language journal-MLJ7. TESOL quarterly-TQ
The process of gathering the data sources
• The electronic versions• Screening phase (abstracts and
methodology)• Search function of adobe reader
A total of 273 articles were identified
Review of Data gathering
Data Analysis1. Rigorous qualitative content analysis(rich description of mixing in the process of data
collection and data analysis)Content analysis was based on both:• Manifest content: the concrete terms used
for coding• Latent content: the coder’s interpretation of
the content2. Make use of code-sheets and respective
categories based on three FWs
Creswell et al. (2008) analysis of research designs
• Concurrent designs and Sequential designs• Concurrent triangulation design (parallel)• Concurrent embedded design( btw pre and
post tests)
• Sequential explanatory design : quan->qual
• Sequential exploratory design: qual->quan
• Sequential embedded design: before and after the intervention--qual
Collins et al. (2007) 2 dimensional model of mixed methods
sampling• Time orientation of components (concurrent or sequential)• Relationship of the qualitative and quantitative
samples 1.Identical 2.Multilevel3.Nested4.Parallel
Data Analysis (cont.)• Content analysis based on FWs• Consider the arguments of validity and
inference quality
Tashakkori and Teddlie’s (2008) integrative model of inference quality
in mixed method • It consists of 2 broad categories of 1.Design quality2.Integrative rigor
Data Analysis (cont.)Design quality
1. Design suitability2. Design adequacy
3. Within design consistency
4. Analytic adequacy
Integrative rigor
1. Interpretive consistency
2. Theoretical consistency
3. Interpretive agreement 4. Interpretive
distinctiveness5. Integrative efficacy
What is Integrative efficacy?
• The degree to which inferences made in each strand of mixed methods study are
effectively integrated into a theoretically consistent meta-inference.
• Simply speaking:
effective integration of both qual and quan findings
extended definition of Tashkkori & Creswell (2007a)
1. Mixing should happen at different stages of a particular study
2. Meta inference must be developed based on data integration both qual and quan
It led to the exclusion of 68 articles and the final sample included 205 articles
The reliability of the coding • 1) An independent coder with qualitative
background coded a sample of articles ( 10% of the sample 20 articles)
• 2) Cohen’s Kappa was calculated . It was • 0.82 for coding of research designs • 0.71 for coding of sampling designs.
• 3) These indices were checked against Altman (1991) criteria and proved to be acceptable.
Review of Data Analysis
Research question1: What kinds of research designs are used when method mixing is
utilized in AL research?
Research question1: What kinds of research designs are used when method mixing is
utilized in AL research?• The direct use of mixed designs was directly
acknowledged in only a few studies• . Triangulation at the level of analysis is
referred to as triangulation by analyses(68 articles out of 136 articles that used triangulation)• The term triangulation was used With different wording
such as: Multiple data source, multiple measures, qualitative and quantitive data
• the emergence of mixed method designs that were not in Creswell et al. typology (7 articles) next table
Research question2: What kinds of sampling designs are used in mixed methods research in
the field of AL?
Research question3: How are general conclusions or meta-inferences developed in
this type of research?
• The analyses of discussion and conclusion sections revealed that data integration and consolidation were not presented
• These meta-inferences were offered by researchers in order to:
1.enrich their discussions and conclusions
2.present informative summaries of the findings
Discussions: 1- Use of mixed research design
Concurrent triangulation design (66%)1. Providing the supplementary data in multilevel D.A2. Needs analysis3. Program evaluation (collect data from various sources or by
multiple methods)Lg Testing and ESP journals (Need assessment /program
evaluation)Adopting a particular design could be a practical
considerationsConcurrent embedded designs(MLJ and lg learning)Sequential explanatory design
Discussions: 1- Use of mixed research design
• Sequential exploratory design:• development of questionnaire tests inventories • validating the measurement instruments• Sequential embedded designs • examine cross sections of the test development
and validation process.• ethnographical studies in order to develop
patterns in a more rigorous way.
Discussions 2-Use of mixed sampling design
This study (2013)• Concurrent Identical designs• multilevel concurrent>>>
Concurrent nested• Why: • different samples of articles in
terms of disciplines, size, purpose, rationale for mixing
• Why concurrent identical:• Practicality, feasibility, time
span
Collins et al (2007)• Concurrent Identical
designs• Concurrent
nested>>>multilevel concurrent
Discussions :3-Inference quality
• Why there was scant attention to the concept of integrative efficacy among the articles?
• The concept hasn’t been introduced appropriately to AL research
• It hasn’t been widely used in AL research• It can be explained by Tashakkori and Creswell
distinction btw • Methodology: integration of the 2 approaches
throughout the study• Method: collecting and analyzing different types of
qual and quan data
Implications of the study
• 1) To explore complicated process and outcomes• 2) Concurrent triangulation designs (DA, GA, NA,
…)• 3) Concurrent embedded design (Lg development
or a process within an experiment)• 4) Sequential explanatory designs:• To explore a subsample from large samples used in qual
phase• Analysis of outliers-extreme cases-inconsistent
components or incompatible subsystems• Further analysis of performance on quan measurement
instruments and evaluate DIF items after statistical analyses
Implications of the study(Cont.)
• Sequential exploratory designs: validating evaluation criteria such as tests, surveys, questionnaire, scales or band descriptors
• Sequential embedded design: longitudinal studies of development process in SLA analyze cross sections of test development and
validation processto investigate social aspects of lg learning such as T ID,
L ID, T D, L D, interaction patterns,…
Summary• Innovative mixed method can be used in
design and sampling in AL based on • Research purpose• Requirements of the research contextHigh quality integration in mixed method
depends crucially on design quality and quality of meta-inferences
• This can happen as result of integration of qual and quan components in all of the stages of the study
Penny for thoughts
• What are the evaluative criteria for investigating the integration quality in AL research?
• What is the tendency of some journals in use of particular designs?
(As it was not the main focus of this study, more accurate accounts of it would require further research.)
Thank You for your
patience