Quantitative Research: Surveys and Experiments
-
Upload
martin-kretzer -
Category
Education
-
view
304 -
download
1
Transcript of Quantitative Research: Surveys and Experiments
IS541, Lecture 6a: “Quantitative Research: Surveys and Experiments”
Master Management, Master Business Informatics; March 4th, 2015
Martin KretzerChair of Information Systems IV, Business School and
Institute for Enterprise Systems (InES), University of Mannheim
Overall Course Structure
#9 Final
Assignm
ent#5a Literature Review Intro
#5b Literature Review RBD
2ManTIS FSS 2015 - Quantitative
Research
#7a Design Science Intro
#7b Design Science RBD
#8a Qualitative Research Intro
#8b Qualitative Research RBD
#6a Quantitative Research Intro
#6b Quantitative Research RBD
#1 Introduction
#2 Theories
#3 Methods
#4 Scientific Writing
and Publishing
Goals of this Lecture
A
Understand the basics of survey-
based research and experiments
Know the research process of
surveys and experiments
Data gathering and analysis
Be aware of important quality
criteria for quantitative research
Learn best practices
Get to know popular software tools
for analyzing quantitative data3ManTIS FSS 2015 - Quantitative
Research
Agenda
Agenda
1 Basics of Quantitative Research
2 Surveys
3 Experiments
4 Software Demo and Hands-On Exercise (SmartPLS, R Studio)
5 Summary
6 Supplementary Material on Quality Criteria
7 References
4ManTIS FSS 2015 - Quantitative Research
The Two Main Paradigms of Empirical Work
Quantitative Qualitative
Mostly deductive (theory first) Mostly deductive (observation first)
Statistical generalizability Analytical generalizability
Linear, pre-planned research design Evolving, iterative research design
High number of observations Focused number of observations
Statistical analyses Conceptual analyses
Independent of context Context-dependent
Reliability is key Authenticity is key
Source: Denzin and Lincoln (2011), Neumann (2000)
After spring breakToday
5ManTIS FSS 2015 - Quantitative
Research
Characteristics of Quantitative Research
Strongly connected to a positivist epistemological stance ( lecture 2)
– Objective reality can be captured and translated into testable hypotheses
– Researcher can capture empirical data that allows them to make inferences about that reality
Generally emphasizes high “n”
– Numbers represent values and levels of theoretical constructs and concepts
– Interpretation of the numbers is viewed as strong scientific evidence of how a phenomenon works
– Aims for statistical generalizability to make predictions among unobserved members of the
population
Strongly relies on statistical tools as an essential element in the researcher's toolkit
6
Source: Straub et al. (2005)
ManTIS FSS 2015 - Quantitative
Research
Quantitative Research Methods (revisited)
Simulation Field Experiment / Quasi Experiment
State a hypothesis
Imitate some real process or action to prove your hypothesis
Suitable for observing correlation between variables
Strengths: allows estimation and prediction
Conducted in field settings, e.g. real organization
Rare, because of the difficulties associated with manipulating
treatments and controlling for extraneous effects in a field
setting
Strengths: high internal and external validity
Survey Laboratory Experiment
Collect self reported data of people
Standardized questionnaire or interview
Suited to study preferences, thoughts, and behavior of
people
Suited for descriptive, exploratory or explanatory research
Strengths: collect unobservable data (thoughts); remote
collection; convenient for respondents; can detect small
effects
Independent variables are manipulated by the researcher (as
treatments)
Subjects are randomly assigned to treatment
Results of the treatments are observed
Suited for explanatory research to examine individual cause-
effect relationships in detail
Strengths: influence of individual factors can be well
explained; very high internal validity (causality)
Source: Bhattacherjee (2012), Myers (2009)
7ManTIS FSS 2015 - Quantitative
Research
Examples for Quantitative Studies
8
Survey
– Goal: Identify drivers for user satisfaction
– Build a questionnaire for (1) measuring satisfaction with a new IS and (2) measuring the constructs that you expect could influence satisfaction.
– Distribute the questionnaire to your sample (e.g. people in your organization)
– Analyze answered questionnaires by analyzing which drivers can be associated with satisfaction (e.g., through computing correlation)
Experiment
– Goal: Examine the effects of your new online shop
– Build your treatment groups, e.g.:
• Online shop that recommends items
• Online shop that does not recommend items
– Randomly assign individuals to a treatment group
– After 4 weeks check whether your recommender increased sales
Simulation
– Goal: Examine the effect of a new algorithm
– Build your treatment groups (e.g., old algorithm, new algorithm) and check for differences
– Can be referred to as „computational experiment“
ManTIS FSS 2015 - Quantitative
Research
conceptual
observable
unobservable
Theory’s Basic Constituents and Mechanisms
Source: Bhattacherjee (2012, p. 39)
Empirical
Plane
Theoretical
Plane Construct A Construct BProposition
Independent
Variable
Dependent
VariableHypothesis
External validity
Internal validity
9ManTIS FSS 2015 - Quantitative
Research
Classes of Variables
10
IntelligenceAcademic
Achievement
Earning
Potential
Effort
Independent
Variable
Moderating
Variable
Mediating
Variable
Dependent
Variable
Source: Bhattacherjee (2012, p. 12)
ManTIS FSS 2015 - Quantitative
Research
Internal Validity and External Validity
Internal validity External validity
= Causality
Does a change in X really cause a change in Y?
Three conditions
1. Covariation of cause and effect
2. Temporal precedence
3. No plausible alternative explanation
Note: causality <> correlation!
= Generalizability
Can the observed association be generalized from
sample to the population or further contexts?
Source: Bhattacherjee (2012), Myers (2009)
11ManTIS FSS 2015 - Quantitative
Research
Internal Validity and External Validity
12Source: Bhattacherjee (2012, p. 38)
•There is no single „best“ research method!•You need to know the strengths and limitations of your research method!•Combinations might make sense (also mixing quantitative and qualitative research methods; see Venkatesh et al. 2013 for further information)
ManTIS FSS 2015 - Quantitative
Research
Internal Validity (Causality)
Why is internal validity of cross-sectional field surveys typically limited?
– Independent variable Y cannot be manipulated
– Cause and effect are measured at the same time you do not know whether X
causes Y or Y causes X
Why is internal validity of laboratory experiments typically high?
– Independent variable Y can be manipulated via a treatment
– Effect can be observed after a certain point in time
– External factors can be controlled
13ManTIS FSS 2015 - Quantitative
Research
External Validity (Generalizability)
Why is external validity of laboratory experiments typically limited?– Artifical treatments
– External factors are controlled But: in real settings external factors cannot be controlled!
Why is external validity of cross-sectional surveys typically high?– Data from a wide variety of individuals or firms is collected
Qualitative research: Why may single case studies have higher generalizability than multiple case studies?– In qualitative research studies, you have to clearly describe the context of your
study
– The more detailed you can describe the context, the better you can explain to which further cases your results can be generalized!
14ManTIS FSS 2015 - Quantitative
Research
conceptual
observable
unobservable
Theory’s Basic Constituents and Mechanisms (cont’d)
Source: Bhattacherjee (2012, p. 39)
Empirical
Plane
Theoretical
Plane Construct A Construct BProposition
Independent
Variable
Dependent
VariableHypothesis
External validity
• Internal validity
• Statistical conclusion validity
Construct
validity
Construct
validity
15ManTIS FSS 2015 - Quantitative
Research
Construct validity and statistical conclusion validity
Construct validity Statistical conclusion validity
How well is our measurement scale measuring the
independent variable and the theoretical construct
that it is expected to measure?
Are the statistical conclusions really valid?
Did we select the right statistical method for testing
the hypothesis?
Does the sample meet the requirements?
(only relevant for quantitative research)
Source: Bhattacherjee (2012), Myers (2009)
16ManTIS FSS 2015 - Quantitative
Research
Agenda
Agenda
1 Basics of Quantitative Research
2 Surveys
3 Experiments
4 Software Demo and Hands-On Exercise (SmartPLS, R Studio)
5 Summary
6 Supplementary Material on Quality Criteria
7 References
17ManTIS FSS 2015 - Quantitative Research
Using surveys: advantages and disadvantages
Biases
Non-response bias
Common method bias
Sampling bias
Social desirability bias
Flexible and Efficient High Volume Data Analysis
Application across all research
phases
Measure unobservable data
(Preferences, Attitudes, Traits)
Economical in terms of
researcher time, effort and cost
Can be administered to a
high number of subjects
Remote data collection
Comparability through
standardization
Well established quality
criteria
Statistical tests
Detect small effects with
large samples
Richness of Data
Only answers to standardized questions
Interpretation and context of respondent
missing
Source: Bhattacherjee (2012)
18ManTIS FSS 2015 - Quantitative
Research
Structural and measurement model
19
Structural model defines abstract relationship between constructs
Measurement model contains empirically observed variables
Legend:
latent exogenous variable
latent endogenous variable
x measurable exogenous indicator
y measurable endogenous indicator
path coefficient between latent exogenous and
endogenous variables
path coefficient between latent variables and
measurable indicators
measurement error
y2y1
3
Measurement Model (Outer Model)
Structural Model (Inner Model)
3 4
4
x2x1
1
1 2
2
Source: Williams et al. (2009)
ManTIS FSS 2015 - Quantitative
Research
Attention: List of terms that are frequently used
interchangeably in positivistic reseach!
Measurable indicator (e.g., x1, x2, x3, y1, y2, y3)
– „Question“
– Indicator
– Item
– Measure
– Measurable variable you should not use this term!
Independent variable (IV) (e.g., x)
– (Latent) Exogenous variable
– Factor (typically refers to an IV that uses a nominal scale in experiments)
– Antecedent
– Cause
Dependent variable (DV) (e.g., y)
– (Latent) Endogenous variable
– Outcome
– Effect
Moderation effect
– Interaction effect
20ManTIS FSS 2015 - Quantitative
Research
Scale development for measurement indicators
21
Definition Scaling Process: Process of developing scales to measure indicators (items)
Rating scales are attached to items (unidimensional or multidimensional scaling)
Scales define what the respondent can choose to answer
Depending on the chosen scale certain statistic analysis are possible
Description Example
Equ
al A
ppea
ring
Participants rate items with
“agree” or “disagree”
Items only “appear” equal; in fact they represent
different values for measuring a certain concept
Thurstone
agree disagree
I like doing sports. X
I like swimming. X
Sum
mat
ive
/ Cum
ulat
ive
Likert
Participants rate items on a 5-point or
7-point scale
Scale ranges from “strongly agree” to “strongly
disagree”
strongly strongly
agree … neutral … disagree
I like …. X
I like…. X
Guttman
Goal: Cumulative scale
Participants rate items with “yes”/“no”
Creates a sorted matrix or table (see example)
yes no
Do you mind immigrants in your city? X
Would you live next to an immigrant? X
Would you marry an immigrant? X
Source: Bhattacherjee (2012, pp. 48-50)
ManTIS FSS 2015 - Quantitative Research
Number of Items per Construct
Why is the number of items for measuring a variable so important?
– Problem with only one item: risk of random error may be high
– Problem with too many items: people will stop answering your survey
How many questions ( items) should I ask?
– Almost no risk for random erorr: 1 item
– Stable constructs: 3-6 items
– Rather new constructs: at least 8 items
Examples:
– Age 1 item
– Perceived ease of use (Davis 1989):
• First pretest: 16 items
• Final items: 6 items
– Perceived ease of use (today): usually 3 items
22ManTIS FSS 2015 - Quantitative
Research
Quality Criteria: Measurement Model Validity
Summary of quality criteria for the measurement model
23ManTIS FSS 2015 - Quantitative
Research
Quality criterion Recommendation
Convergent validity of a
single indicator
High average factor loadings: λ > 0.7
Narrow range of factor loadings: λmax - λmin < 0.2
Convergent validity of a
single construct
High average variance extracted: AVE > 0.5
High composite reliability: ρc > 0.8
High communality index of the construct
Convergent validity of the
measurement modelHigh average communality index
Discriminant validity of a
single indicatorEach item loading is greater than all cross-loadings
Discriminant validity of a
single construct
A construct’s AVE is greater than the squared construct’s
correlation with any other construct
No common method bias
Substantive factor loadings are greater than method factor
loadings
Method factor loadings are not significant while
substantive factor loadings are significant
(Details in section 6 “Supplementary Material on Quality Criteria”)
Quality Criteria: Structural Model Validity
24ManTIS FSS 2015 - Quantitative
Research
Summary of quality criteria for the structural model (PLS regression analysis)
Quality criterion Recommendation
Direct effectHigh standardized path estimates
Bootstrap algorithm and t-test
Moderating effect ANOVA and F-test or product-indicator approach in conjunction with PLS
Predicting power
High variance explained (R² > 0.2)
High effect size 𝑓²: 0.02 small effect; 0.15 medium effect; 0.35 large effect
High redundancy
Global quality of structural
modelHigh goodness of fit
No multicollinearity
No perfect correlation between independent variables: Standardized path
estimates < 0.8
High tolerance of independent variables: tolerance > 0.1
Small variance inflation factor of independent variables: VIF < 10
(Details in section 6 “Supplementary Material on Quality Criteria”)
Golden Circle Analysis – Survey Example
Golden Circle Analysis (GCA)
“Privacy Concerns and Privacy-Protective
Behavior in Synchronous Online Social
Interactions”
– Authors: Z.J. Jiang, C.S. Heng, B.C.F. Choi
– Year: 2013
– Outlet: Information Systems Research (ISR)
• Vol. 24, No. 3, pp. 579-595
25ManTIS FSS 2015 - Quantitative
Research
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
GCA Survey: Brief Summary
Individuals‘ behavior is at times inconsistent with their privacy
concerns (e.g., they disclose private information in synchronous
online social interaction although they know the risks)
Focus: privacy concerns versus social rewards
Students conduct 3 chatroom sessions afterwards a survey is
administered
Findings:
– Individuals use self-disclosure and misrepresentation to protect their privacy
– Social rewards explain deviations
26
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Survey: Motivation
Privacy trade-off in the context of online social interactions
– Online social interactions may generatie multiple benefits:
synchronous exchange of information, sharing of cultural artifacts,
self-presentation, feedback from peers, socio-emotional support,
a borderless „space“
– However, 33% of internet users are concerned about their privacy
in online social interactions ( the paper lists numerous threats)
– Ironically, many users are still likely to disclose private information
even if they become aware of the risks
27
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Survey: Research Question
28
The paper clearly states a research question:
– Why is users‘ privacy behavior at times inconsistent with their
privacy concerns?
– P. 580, end of second paragraph
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Survey: Research Objective
29
The paper provides a clear definition of its research
objectives
– P. 580, last paragraph
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Survey: Theory Base
Paper is not based on a single, specific theory
However, the paper integrates two research streams to build a research model:
– Hyperpersonal framework• Approach for understanding how users of mediated communications experience relational
intimacy
– Privacy calculus • Relational privacy trade-off: privacy concerns versus rewards from disclosing private
information
Research Model:
30
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Survey: Research Design – Data Collection
Pilot: 3 rounds of preliminary tests to compare and evaluate different
methods of data collection (Appendix A)
Research Design
– Sample: 251 students in Singapore
– Three online chat sessions, each lasting 1 hour
Survey
– Survey after the end of the third chat session
– All items measured on a 7-point Likert scale ranging from 1 „strongly
disagree“ to 7 „strongly agree“. E.g., privacy concerns awareness:
31
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Survey: Research Design – Data Analysis
Data Analysis:
– Partial least squares (PLS) regression
– Measurement model assessment:
• Individual item reliability
• Internal consistency
• Discriminant validity
• Item loadings, cross-loadings, composite reliability, average variance
extracted (AVE)
– Structural model assessment
• Correlations
• Path coefficients and hypotheses testing all hypotheses were confirmed
• R²
• Sobel tests to examine whether privacy concerns and social rewards fully
mediate the effects
• Confirmatory factor analysis (CFA) for two models in order to test for
common method bias
32
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Survey: Research Design – Data Analysis
Measurement model assessment:
33
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Survey: Research Design – Data Analysis
Measurement model and structural model assessment:
34
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Survey: Research Design – Data Analysis
Structural model assessment:
35
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Survey: Contribution
Extension of the privacy calculus perspective to the context of
synchronous online social interactions
– This contribution is valuable, because past research „has
predominantly applied the privacy calculus to commercial contexts“ (p.
590)
– „In the absence of monetary or tangible rewards, social rewards are
just as attractive in balancing privacy concerns and governing
individuals‘ behavior.“ (p. 590)
Identification of four antecedents (hyperpersonal framework) of
privacy concerns and social rewards
Disclosure and nondisclosre are not the only two possible actions
stemming from privacy protection misrepresentation is a third
action (and independent from the established two actions)
Explanation of the different roles „anonymity of self“ and
„anonymity of others“ in online social interactions
36
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Survey: Analysis of Links
Practical motivation results in the research question (RQ)
Theoretical motivation builds on RQ and results in three
clearly stated research objectives
To address the research objectives, two research streams
are reviewed and integrated into one model
A survey for confirming the model seems to be an excellent
choice
The contribution section (section 6.2) summarizes the survey
results and explains how they extend existing literature.
Thereby, it directly addresses the RQ and the motivation of
the paper ( “golden circle”)
37
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Survey: Summary of Analysis
All elements of the GCA are addressed in the paper
The reader can easily follow the central theme („Roter
Faden“)
Contribution seems valuable
10 Hypotheses are confirmed and typical quality criteria for
surveys is met
Limitations and future research directions are also oulined
38
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
Agenda
Agenda
1 Basics of Quantitative Research
2 Surveys
3 Experiments
4 Software Demo and Hands-On Exercise (SmartPLS, R Studio)
5 Summary
6 Supplementary Material on Quality Criteria
7 References
39ManTIS FSS 2015 - Quantitative Research
Experiment Designs
Between-subjects design
– = Participants can be part of only one treatment group (and are then compared to the „control
group“)
– Advantage: no carryover effects
Within-subjects design
– = Every single participant is subject to each treatment (incl. The control)
– Advantage: statistical significance
Mixed design
– E.g., between-subjects design for independent variable A and within-subjects design for
independent variable B
– Example: Mixed experiment design of Master thesis on the next few slides
40ManTIS FSS 2015 - Quantitative
Research
Mixed Design Experiment Example: Model
41
Perceived ease of use
Expertise
Factor 1
Business Intelligence
Client
Factor 2
Report Recommendation
Factor 3
Potential interaction effects:
• Factor 1 * Factor 2
• Factor 1 * Factor 3
• Factor 2 * Factor 3
• Factor 1 * Factor 2 * Factor 3
• 7 effects that should be tested!
Design:
• Factor 1 (Expertise): within-subjects variable
• Factor 2 (BI Client): within-subjects variable
• Factor 3 (Rep. Rec.): between subjects variable
ManTIS FSS 2015 - Quantitative
Research
Mixed Design Experiment Example: Design Introduction
User Expertise Ease of Use of
SAP
BusinessObjects
Ease of Use of
Microsoft Excel
Ease of Use of
Tableau Desktop
Perceived ease of use
of recommendation
Counterbalanced
Counterbalanced
Factor 2: Business Intelligence Client
Factor 3: Recommender
Factor 1:
User Expertise
42ManTIS FSS 2015 - Quantitative
Research
Mixed Design Experiment Example: ANOVA Results
Statistical analysis of multiple factors (i.e., nominally scaled independent
variables) on one independent variable mostly Analysis of Variance
(ANOVA )
If you have multiple independent variables Multiple ANOVA
(MANOVA)
43
Df Sum Sq Mean Sq F value P(>F)
Between-subjects:
EXP 4 0.42 0.106 0.048 0.995
CLIENT 2 6.93 3.464 1.587 0.227
EXP*CLIENT 8 24.23 3.029 1.387 0.256
Residuals 22 48.03 2.183
Within-subjects:
REC 1 4.879 4.879 4.010 0.058+
REC*EXP 4 14.629 3.657 3.006 0.040*
REC*CLIENT 2 0.496 0.248 0.204 0.817
REC*EXP*CLIENT 8 13.952 1.744 1.433 0.238
Residuals 22 26.769 1.217
Dependent variable: perceived ease of use; n=37. Significance: *p<0.05; +p<0.10
ManTIS FSS 2015 - Quantitative
Research
Mixed Design Experiment Example: Graph. Results
44
(M)ANOVA only indicates effects but no directions of the effect!
Thus, you need to draw the effect and interpret the figure!
Note: If the lines in your graphic are parallel, then there is no interaction effect at all!
ManTIS FSS 2015 - Quantitative
Research
Common ways for increasing internal validity of experiments
• Manipulate independent variables in one or more levels (treatment)
• Compare the effects of the treatments against a control group
• In experimental designs subjects must recognize different treatmentsTreatment
• Eliminate extraneous variables by holding them constant
• For example restricting a study to a single genderElimination
• Consider additional extraneous variables
• Separately estimate their effects on the dependent variable (e.g., via factorial designs where one factor is gender)
Inclusion
• Measure extraneous variables
• Use them as covariates during the statistical testing processStatistical control
• Cancel out effects of extraneous variables through a process of random sampling (if random nature is proven)
• Two types: random selection, random assignmentRandomization
• Randomize the order of experimental treatments
• Reduces error due to carryover effectsCounterbalance
45
Source: Bhattacherjee (2012, pp. 39-40)
ManTIS FSS 2015 - Quantitative
Research
Golden Circle Analysis – Experiment Example
Golden Circle Analysis (GCA)
“The Nature and Consequences of Trade-Off
Transparency in the Context of
Recommendation Agents”
– Authors: J.D. Xu, I. Benbasat, R.T. Confetelli
– Year: 2014
– Outlet: Management Information Systems Quarterly
(MISQ)
• Vol. 38, No. 2, pp. 379-406
46ManTIS FSS 2015 - Quantitative
Research
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
GCA Experiment: Brief Summary
The authors investigate the impact of a novel design
feature for a recommendation agent (RA): trade-off
transparency (TOT)
The TOT design feature directly influences „consumer‘s perceived
product diagnosticity“ and „perceived enjoyment“
The authors find that there exists an optimal maximum in TOT
Furthermore, the authors identify diagnosticity and enjoyment as two
antecedents for „perceived decision quality“ and „perceived decision
effort“
47
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Experiment: Motivation
The authors reference four sources which indicate the
economic necessity of RAs for online shops. But: poorly
designed RAs have negative effects!
Influence of specific design attributes of RAs on decision
making and other outcomes is still not well understood
Overall many sources that indicate benefit and importance
of RAs
RAs ability to capture consumer‘s product attribute
preferences is identified as a „central function of RAs“
Explanation of potential benefits that might arise if users
have better knowledge about pros and cons of different
laptops when browsing an online shop
48
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Experiment: Motivation
Explicit identification of a research gap that needs to be
filled:
– P. 380, last sentence of third paragraph
49
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Experiment: Research Question
50
No specific research question statedBrief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Experiment: Research Objective
51
Three research objectives are stated specifically
1. Examine the impact of a trade-off transparent RA on perceived
enjoyment and perceived product diagnosticity. The context are
laptops in an online shop. Example trade-off: price vs. hard-drive
capacity; weight vs. screen size
2. Examine whether there is an optimal maximum of TOT
3. Extend and challenge the effort-accuracy framework because the RA
enables more accurate decisions to be made without simultaneously
increasing efforts
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Experiment: Theory Base
Stimulus-Organism-Response (S-O-R) model
– Adopted from marketing and psychology research
– External cues (e.g., design features of online shops) influence a consumer‘s affective and/or
cognitive processes; which in turn determine the consumer‘s behavioral and/or internal
response
– Overarching framework for the authors‘ own theoretical model Operationalization:
• Stimulus: trade-off transparency feature of an online RA
• Organism: the user‘s enjoyment (affective system), perceived product diagnosticity
(cognitive system)
• Response: the user‘s perceived decision quality and perceived decision effort
Cognitive load theory
– TOT improves decisions up to a certian point. After that point, TOT overburdens users‘
cognitive limitations and is counterproductive
Effort-Accuracy framework
– An increase in decision accuracy is accompanied by an increasing in the decision makers
efforts (the „longstanding effort-accuracy conflict“)
52
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Experiment: Theory Base
Proposed Theoretical Model– Enjoyment an affective reaction (i.e., an emotional response when interacting with a stimulus)
– Product diagnosticity = extent to which a consumer believes that a system is helpful for fully evaluating a product a cognitive reaction (i.e., a user‘s mental process when interacting with the stimulus)
Based on their proposed theoretical model, the authors develop 10 hypotheses– Note: The authors hypothesize inverted U-shaped curves as the level of trade-off increases on
perceived enjoyment (H3) and perceived product diagnosticity (H4) --> assumption: an optimal maximum exists!
– Research model on next few slides
53
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Experiment: Research Design – Data Collection
Implementation of design feature
– Horizontal scales with „slider“ represent the value of each product attribute
– If the user moves one slider, other sliders will automatically be moved, too the
user can directly observe the trade-off dependencies between several attributes
– The greater the TOT, the more sliders are moved automatically
– In total, the online shop offers 50 laptops
54
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Experiment: Research Design – Data Collection
160 participants (of which 131 are undergraduate students)
Treatment groups:
– „Self-Shoppers“ versus „Friend Shoppers“• 50% of participants are asked to shop for themselves: if participants are
shopping for themselves, their initial product preference can be compared with their final attribute preferences
• 50% of participants are asked to shop for a fictitious fried: prior research indicates that shopping for friends helps minimize the effects of negative emotions when making attribute trade-offs
– Trade-off transparency• Low (25% of participants)
• Medium (25% of participants)
• High (25% of participants)
• Control no specific trade-off transparency (25% of participants)
Between-subjects design – Each participant is assigned to one of the 2*4 = 8 treatment groups
– 160 / 8 = 20 participants per group a power analysis test indicates sufficient statistical power (0.80) to detect a medium effect size (f=0.25)
55
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Experiment: Research Design – Data Collection
Instructions for participants
Experimental procedure
– Questionnaire related to demographic and control variables website training
random assignment to a group Questionnaire related to DVs
56
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
Instruction for “Self-Shoppers” Instruction for “Friend Shoppers”
After the instruction, the user selects a
value range (in USD) for eight laptop
attributes
GCA Experiment: Research Design – Data Analysis
Statistical analysis
– MANOVA for testing the effects of trade-off transparency
– PLS regression for testing
• All items from previous literature (p. 391, tbl. 4)
57
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Experiment: Research Design – Data Analysis
Manipulation check
– Numbers of shown trade-offs was measured
– Users‘ awareness of trade-offs was measured
Effect of Trade-Off Transparency levels
– MANOVA (incl. Pillari‘s trace, Wilk‘s lambda, Hotelling‘s trace, Roy‘s largest
root) results are significant further ANOVAs on the two DV‘s separately
– Product diagnosticity• s
58
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Experiment: Research Design – Data Analysis
PLS results
– Measurement model assessment: loadings and cross loadings
– Structural model assessment: composite reliability, Cronbach‘s alpha, AVE,
path coefficients, R²
– Since this GCA focuses on the paper as an exemplary experiment paper, the
PLS results of the questionnaire are not presented in detail for PLS
analysis, please have a look at the survey GCA
59
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Experiment: Contribution
Results– Trade-off transparent RA improves perceived enjoyment and perceived product
diagnosticity
– Medium level of TOT has the best effect
– Besides H8, all hypotheses are confirmed
TOT feature helps to identify how users‘ attribute choices are related to, and are constrained by, one another
Prior research proposed that trade-off awareness creates unfavorable feelings (p. 400; Luce et al., 1999) – But: the authors show that the TOT feature creates positive emotions!
– Reasons: additional content is conveyed (i.e., relationship among product attribute values) and the interactive presentation
Contribution to task complexity literature (in particular coordinative complexity as one dimension of task complexity) by analyzing how the different number of revealed trade-off relationships influencesn users‘ evaluations
Both enjoyment and product diagnosticity improve perceived decision quality without increasing perceived decision effort
In addition, some practical contributions are derived
60
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Experiment: Analysis of Links
The motivation identifies a research gap which is directly addressed by the
research objectives
Three theory bases are referenced and integrated in order to develop and
propose a model that adresses the three research objectives
To investigate the proposed model, the authors select a confirmatory research
design. In particular, they select a combination of an experiment and a survey.
The impact of the TOT feature is tested using a between-subjects experiment
design. Further effects are tested using a survey design.
The contributions section directly builds on the analysis of the experiment (and
the survey)
Furthermore, the contributions link back to the motivation by answering the three
identified research objectives
61
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
GCA Experiment: Summary of Analysis
Although a research question is not explicitly stated, the
reader can easily follow the authors‘ work
GCA elements are addressed and links between them are
straightforward
Quality criteria for experiments (e.g., manipulation check)
and surveys (e.g., item loadings) are reported
9 of 10 hypotheses are confirmed and the rejected
hypotheses intuitevely seems to be true in a real setting, too
Limitations are outlined (e.g., students as subjects, little or
no experience with the RA, laptops is a very customizable
product)
Overall, the inferences drawn appear to be valid
62
Brief summary
of article
Motivation
Research Question
Research Objective
Theory Base
Research Design
Contribution
Analysis of Links
Summary of
Analysis
ManTIS FSS 2015 - Quantitative
Research
Agenda
Agenda
1 Basics of Quantitative Research
2 Surveys
3 Experiments
4 Software Demo and Hands-On Exercise (SmartPLS, R Studio)
5 Summary
6 Supplementary Material on Quality Criteria
7 References
63ManTIS FSS 2015 - Quantitative Research
Software Tool Presentation
Live Demo
– Survey PLS regression in software „SmartPLS Version 2“
– Experiment MANOVA in software „R Studio“
64ManTIS FSS 2015 - Quantitative
Research
Recommended Materials
PLS algorithm and SmartPLS software:
– Hair, J.F., Hult, T.M., Ringle, C.M., Sarstedt, M. 2013. A Primer on Partial Least
Squares Structural Equation Modeling (PLS-SEM), Sage Publications.
– https://www.youtube.com/user/Gaskination/playlists
Statistical programming language R:
– Just search for it using Google and/or YouTube
65ManTIS FSS 2015 - Quantitative
Research
Agenda
Agenda
1 Basics of Quantitative Research
2 Surveys
3 Experiments
4 Software Demo and Hands-On Exercise (SmartPLS, R Studio)
5 Summary
6 Supplementary Material on Quality Criteria
7 References
66ManTIS FSS 2015 - Quantitative Research
Today‘s Lecture in Review
67
You learned about the foundations of quantitative, survey-based research
You have some advice on study design
You know about the fundamental process of study design and execution
You are familiar with the most important steps for validating your study
You know the basic quality criteria and strategies to ensure them ( more details in supplementary slides!)
You have the basic tools for the discussion of the survey-based papers
You have seen popular software tools for conducting quantitative research
ManTIS FSS 2015 - Quantitative
Research
What did we exclude?
Sampling– In IS important: Who are you talking to? Users? Developers? What is your subject‘s expertise? How often
are they using the IS of interest?...
– E.g. surveying SAP employees about ERP software would probably cause a huge error
Scale development– In case you need to examine new items in your thesis, your supervisor will explain you how to do this
we assumed all questions can be taken from previous literature
– Formative versus reflective measures as long as you can take your questions from preivous literature,
this should not bother you too much
Statistical analyses: Regression, PLS/CB-SEM, (M)AN(C)OVA…– There are multiple specialization courses offered at the University that you can take for this
– Your thesis supervisor can help you in choosing an appropriate statistical method
68ManTIS FSS 2015 - Quantitative
Research
Questions, Comments, Observations69
ManTIS FSS 2015 - Quantitative
Research
Homework until Next Week
Write a Golden Circle Analysis (3 pages text) for one of the following papers
and be able to present and discuss your analysis:
–Survey:
• Li, X., Po-An Hsieh, J. J., and Rai, A. 2013. „Motivational Differences Across Post-
Acceptance Information System Usage Behaviors: An Investigation in the Business
Intelligence Systems Context,“ Information Systems Research (24:3), pp. 659-682
• Note: supplementary material in additional pdf-file!
–Field experiment:
• Martin, S. L., Liao, H., and Campbell, E. M. 2013. „Directive versus Empowering
Leadership: A Field Experiment Comparing Impacts on Task Proficiency and Proactivity,“
Academy of Management Journal (56:5), pp1372-1395
–Experiment + Survey:
• Sun, H. 2013. „A Longitudinal Study of Herd Behavior in the Adoption and Continued Use
of Technology,“ MIS Quarterly (37:4), pp. 1013-1041
70ManTIS FSS 2015 - Quantitative
Research
Agenda
Agenda
1 Basics of Quantitative Research
2 Surveys
3 Experiments
4 Software Demo and Hands-On Exercise (SmartPLS, R Studio)
5 Summary
6 Supplementary Material on Quality Criteria
7 References
71ManTIS FSS 2015 - Quantitative Research
Quality Criteria: Intro to Quality Criteria
Constructs (theoretical level)
– Constructs are imaginary creations in
our minds
– Definitions or constructs are not
objective, but shared (“inter-
subjective”) agreements
Two forms of constructs
– Unidimensional constructs have a
single underlying dimension
– Multidimensional constructs consist of
two or more underlying dimensions
Unobservable theoretical constructs
are translated into indicators
Indicators are questions that can be
empirically observed and measured
Example: socioeconomic status is
measured by asking for
– Family income
– Education
– Occupation
Can be measured multidimensional
or unidimensional
Definition Conceptualization Mental process translating imprecise
concepts into precise definitions
Understand and define what is included
and excluded in a concept
Definition Operationalization Process of developing indicators to
measure abstract constructs
Is based on conceptualization
72
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity
Quality Criteria: Intro to Quality Criteria
Three major types of validity in quantitative research (Cook and Campbell 1979; Shadish et al. 2002):
– Design validity
– Measurement validity
– Inferential validity
Design validity refers to internal validity (causality and control for alternative explanations) and external validity (generalizability)– See previous slides
– E.g., students as participants valid design? potential issues are usually mentioned in the limitation section
Measurement validity estimates how well items measure what they are purported to measure according to their definitions
Inferential validity, also called statistical conclusion validity, refers to the correct application of statistical procedures to find relationships.
Note: This summary of quality criteria focuses on PLS-SEM.– PLS is a prediction-oriented variance-based approach that focuses on endogenous target constructs in the
model and aims at maximizing their explained variance, i.e., their R² value (Hair et al. 2012a).
– PLS has become a quasi-standard (e.g., Bagozzi and Yi 2012; Hair et al. 2012b; Ringle et al. 2012; Shook et al. 2004; Steenkamp and Baumgartner 2000)
– I do not provide a detailed comparison of the two approaches, because it would go beyond the scope of this presentation (e.g., Chin and Newsted 1999; Chin et al. 2003; Marcoulides et al. 2009; Qureshi and Compeau2009) and there is still an ongoing debate about strengths and weaknesses of the two approaches (Goodhue et al. 2012a, 2012b; Marcoulides et al. 2012)
73ManTIS FSS 2015 - Quantitative
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity
Quality Criteria: Measurement Model Validity
Two quality goals (Bagozzi and Yi 2012):
– Discriminant validity
– Convergent validity
Discriminant validity refers to the degree to which a construct is more strongly related
to its own indicators than with any other construct (Chin 2010)
– Discriminant validity assures that indicators are assigned to the correct construct
and multiple constructs do not overlap in their definitions
Convergent validity refers to the degree to which a block of items – usually all
indicators of a specific construct – agree (i.e., converge) in their representation of the
construct they are supposed to measure (Chin 2010)
– Convergent validity assures that a set of indicators measures the same construct
74ManTIS FSS 2015 - Quantitative
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity
Quality Criteria: Measurement Model Validity
Convergent validity (1/4)
A quasi-standard is reporting factor loadings to show that various indicators are
measuring the same construct.
If loadings would be mixed and have a wide range (e.g., varying from 0.5 to 0.9), this
would raise concern about whether the indicators are a homogenous set that primarily
captures the phenomenon of interest (Chin 2010).
Literature argues that all indicators should be significant, exceed 0.7, and the
difference between indicators measuring the same construct should not exceed 0.2
(Bagozzi and Yi 2012; Chin 2010; Fornell and Larcker 1981).
Similarly, on the construct level, the shared variance of a set of indicators in relation to
their shared variance plus measurement errors, attempts to measure the amount of
variance that a construct extracts from its indicators, so-called average variance
extracted (AVE). It is computed as
where 𝜆𝑖 is the factor loading connecting an indicator to its hypothesized factor and
𝜃𝑖𝑖 is the variance of the error term corresponding to the indicator (Bagozzi and Yi
2012; Fornell and Larcker 1981).75ManTIS FSS 2015 - Quantitative
Research
iii
i
iindicatorfactor
factorAVE
var
var2
2
_
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity
Quality Criteria: Measurement Model Validity
Convergent validity (2/4)
At least 50% variance of items should be accounted for, leading to a minimum AVE of
0.5 (Bagozzi and Yi 2012; Chin 2010; Hair et al. 2014).
Although many researchers have compared the square root of AVE to construct
correlations, you can equivalently compare the AVE to squared correlations among
constructs as this has two advantages (Chin 2010):
– The shared variance is represented in terms of percentage overlap
– Differences are easier to distinguish
76ManTIS FSS 2015 - Quantitative
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity
Quality Criteria: Measurement Model Validity
Convergent validity (3/4)
Another index measuring a construct’s reliability based on convergent validity of its
indicators, is the composite reliability index 𝜌𝐶 , given by
where 𝜆𝑖𝑗 refers to factor loading i on factor j and 𝜃𝑖𝑖 is the variance of the error term
corresponding to the indicator (Bagozzi and Yi 2012; Werts et al. 1974)
In contrast to Cronbach’s alpha, which is a minimum estimate of reliability, composite
reliability does not assume that all items are equally weighted and thus can be a better
estimate of reliability. Like AVE, 𝜌𝐶 is only applicable for constructs measured with
reflective indicators, too. According to recommendations in literature, 𝜌𝐶 should be at
least greater than 0.6 for new constructs (Hair et al. 2014) and greater than 0.8 for
stable constructs (Fornell and Larcker 1981).
77ManTIS FSS 2015 - Quantitative
Research
iiij
ij
compositeC
factor
factor
var
var2
2
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity
Quality Criteria: Measurement Model Validity
Convergent validity (4/4)
Furthermore, the average variance an indicator explains is measured through the
communality index and is computed as
where pq is the amount of indicators of the latent variable q and is the
squared factor loading of one indicator of q (Vinzi et al. 2010).
Regarding the entire measurement model, the average communality index is defined
as
where p is the total number of indicators in the model, J is the amount of latent
variables, and pj is the amount of indicators of J (Vinzi et al. 2010).
Note: Although the communality index indicates quality of constructs and the average communality index indicates
the quality of the overall measurement model, communality scores are frequently reported together with quality
criteria of the structural model. The reason for this is, that, based on communality, further indices indicating quality of
the structural model can be calculated and, for the reader, data analysis is be easier to understand if communality
and indices based on communality are reported together.
78ManTIS FSS 2015 - Quantitative
Research
qp
p
qpq
q
q xcorp
com1
2 ,1
qpqxcor ,2
J
j
jj compp
com1
*1
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity
Quality Criteria: Measurement Model Validity
Discriminant validity
Recent research recommends to prove that a construct is more correlated with its own
indicators than with the indicators of another construct (Chin 2010). Otherwise there
would be the option that multiple constructs share the same types of indicators and
thus would not be conceptually distinct.
Prior research recommends comparing all constructs’ AVE indices with their respective
squared correlations to other constructs – or equivalently all constructs’ square root of
the AVE indices with their respective correlations to other constructs (Chin 2010;
Fornell and Larcker 1981). If a construct’s AVE score is higher than all squared
correlations to other constructs, the construct is more strongly related to its own
indicators than to the indicators of another construct.
Furthermore, literature recommends comparison of correlations between indicators
and constructs in order to argue for discriminant validity (Chin 2010). That is, loadings
of an indicator to the construct it is supposed to measure (i.e., factor loadings) should
be greater than all loadings of the same indicator to other constructs (i.e., cross
loadings).
79ManTIS FSS 2015 - Quantitative
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity
Quality Criteria: Measurement Model Validity
Common method bias (1/2)
Besides convergent and discriminant validity, bias caused by common method
variance (CMV) can be a potential threat, because the exogenous and endogenous
variables are not obtained from different sources (Podsakoff et al. 2003).
The effects of an unmeasured latent methods factor are controlled by including a
common method factor in the PLS model whose indicators included all the principal
constructs’ indicators and should not be significant (Liang et al. 2007; Podsakoff et al.
2003; Richardson et al. 2009; Williams et al. 2003).
80ManTIS FSS 2015 - Quantitative
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity
Quality Criteria: Measurement Model Validity
Common method bias (2/2)
Podsakoff et al. (2003) suggest adding a latent method factor to the structural model
which is measured using all indicators. The figure below (adopted from Liang et al.
2007) shows an example with the exogenous variable A and the endogenous variable
B, indicators a1, a2, b1, and b2, measurement errors 𝑒1𝑎, 𝑒2
𝑎, 𝑒1𝑏 and 𝑒2
𝑏, and factor
loadings 𝜆1𝑎, 𝜆2
𝑎, 𝜆1𝑏 and 𝜆2
𝑏.
81ManTIS FSS 2015 - Quantitative
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity
Quality Criteria: Measurement Model Validity
Summary of quality criteria for the measurement model
82ManTIS FSS 2015 - Quantitative
Research
Quality criterion Recommendation
Convergent validity of a
single indicator
High average factor loadings: λ > 0.7
Narrow range of factor loadings: λmax - λmin < 0.2
Convergent validity of a
single construct
High average variance extracted: AVE > 0.5
High composite reliability: ρc > 0.8
High communality index of the construct
Convergent validity of the
measurement modelHigh average communality index
Discriminant validity of a
single indicatorEach item loading is greater than all cross-loadings
Discriminant validity of a
single construct
A construct’s AVE is greater than the squared construct’s
correlation with any other construct
No common method bias
Substantive factor loadings are greater than method factor
loadings
Method factor loadings are not significant while
substantive factor loadings are significant
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity
Quality Criteria: Structural Model Validity
Estimation of standardized path estimates (PLS)
The distribution-free PLS approach estimates standardized path estimates based on
shared variances of the associated constructs
The significance of these estimates are typically assessed using a nonparametric
bootstrapping algorithm (Chin 1998; Chin 2010)
This algorithm is based on n samples with m cases each (Efron and Tibshirani 1993).
– First, for each case all indicators are replaced with a value from their confidence
intervals
– Then, based on m values per indicator, a value for the sample is computed
– This procedure continues until n samples are calculated
The accuracy of the bootstrapping algorithm increases with the amount of cases and
samples on which it is based. Literature recommends using default software properties
for the amount of samples and cases when performing bootstrapping analyzes,
because then research results would be comparable (Temme et al. 2010).
For instance, the bootstrapping algorithm implemented in the software tool
“SmartPLS” estimates (per default) the significance of standardized paths based on
200 samples with 100 cases each
83ManTIS FSS 2015 - Quantitative
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity
Quality Criteria: Structural Model Validity
Interaction effects
Note: Broad IS research has predominantly employed multiple regression- and
ANOVA-based analytic techniques to investigate interaction terms
– You do not have to use PLS for testing interaction effects!
Recent literature suggests to use the product-indicator approach in conjunction with
PLS as described by Chin et al. (2003)
– Advantages:
• This approach requires fewer indicators per construct and a smaller sample
size to find true interaction scores
• Furthermore, it is able to handle measurement error, produce consistent
results, and has a smaller tendency to underestimate paths coefficients
– Disadvantage:
• However, it has a slight tendency to overestimate factor loadings
84ManTIS FSS 2015 - Quantitative
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity
Quality Criteria: Structural Model Validity
Predictive Power (1/2)
Besides strengths of associations between various constructs, predictive power of the
structural model needs to be quantified.
Commonly predictive power is reported through R² values of the endogenous
constructs. Falk and Miller (1992) recommend that R² values should be greater than
0.1. Hair et al. (2014) recommend that R² values should be greater than 0.2.
A change in the R² values can further be explored to see whether a particular variable
has a significant effect on another particular variable (Chin 2010). Specifically, the
effect size 𝑓2 should be calculated:
where 𝑅𝑖𝑛𝑐𝑙𝑢𝑑𝑒𝑑2 and 𝑅𝑒𝑥𝑐𝑙𝑢𝑑𝑒𝑑
2 are the R² values provided on the dependent latent
variable when the predicting latent variable is used or omitted in the structural equation
respectively.
According to Cohen (1988), an effect size 𝑓2 of 0.02, 0.15, and 0.35 can be
interpreted as a small, medium, or large impact.
85ManTIS FSS 2015 - Quantitative
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
2
222
1 included
excludedincluded
R
RRf
Design validity
Quality Criteria: Structural Model Validity
Predictive Power (2/2)
Besides the R² scores, the redundancy index attempts to measure the quality of the
structural model for an endogenous construct, too (Tenenhaus et al. 2005).
While the R² scores only consider relationships predicting one endogenous construct,
the redundancy index regards the entire structural model (Vinzi et al. 2010).
Furthermore, the redundancy index combines (a part of) the quality of the
measurement model (i.e., communality index) with (a part of) the quality of the
structural model (i.e., R² values):
Likewise, the quality of the overall structural model is expressed by the average
redundancy (Vinzi et al. 2010) computed as
where J is the total number of endogenous latent constructs in the model (Vinzi et al.
2010).
86ManTIS FSS 2015 - Quantitative
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
2
jjj Rcomred
J
j
jredJ
red1
1
Design validity
Quality Criteria: Structural Model Validity
Overall (i.e., measurement model + structural model) fit criterion
Furthermore, a global criterion of goodness of fit (GoF) which takes into account the
model performance in both the measurement and the structural model can be
computed
GoF provides a single measure for the overall prediction performance of the model
(Tenenhaus et al. 2005; Vinzi et al. 2010)
GoF is computed as the geometric mean of the average communality and the average
R² value:
Since PLS does not optimize any global function, there is no index that can provide
the user with a global validation of the model (as it is instead the case with 𝜒² [Chi
Square] and related measures; Tenenhaus et al. 2005)
However, the GoF index represents an operational solution to this problem as it may
be meant as an index for validating the PLS model globally (Duarte and Raposo 2010)
87ManTIS FSS 2015 - Quantitative
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
2RcomGoF Design validity
Quality Criteria: Structural Model Validity
Multicollinearity (1/3)
While the indices introduced on the previous slides favor strong correlations between
independent and dependent variables, very strong correlations between several
independent variables (commonly known as multicollinearity) are undesired, because
for each regression coefficient there would be an infinite number of combinations of
coefficients that would work equally well thus making it impossible to obtain unique
estimates of the regression coefficients (Field et al. 2012)
– In other words, if there are two predictors that are perfectly correlated, then the
regression coefficients for each variable would be interchangeable
– This could lead to reduced statistical power, untrustworthy regression coefficients,
high sensitivity to small changes in the data, and difficulties to assess the individual
importance of independent variables (Field et al. 2012)
Consequence: correlations between independent variables should be smaller than 0.8
(Field et al. 2012).
88ManTIS FSS 2015 - Quantitative
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity
Quality Criteria: Structural Model Validity
Multicollinearity (2/3)
Multicollinearity can be detected through the tolerance and variance inflation factor
indices.
Tolerance is the proportion of variance in an independent variable which is not
predicted by the other independent variables (Clark-Carter 2010).
In order to calculate a certain independent variable’s tolerance, that variable is treated
as dependent variable with all other independent variables as predictors. The obtained
R² is then used to determine the variable’s tolerance index:
Similarly, the variance inflation factor (VIF) is computed as
89ManTIS FSS 2015 - Quantitative
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
21 Rtolerance
21
11
RtoleranceVIF
Design validity
Quality Criteria: Structural Model Validity
Multicollinearity (3/3)
Recent literature argues that tolerance should be greater than 0.1, meaning that at
least 10% of an independent variable’s variance should not be explained by other
independent variables yet (Clark-Carter 2010; Meyers et al. 2006)
Equivalently, VIF should be smaller than 10 (Stevens 2002).
However, O’Brien (2007) argues that the stability of estimated coefficients can be
influenced by other factors. Hence, the variance of the regression coefficients would
be reduced and VIF values of 40 or more could still be acceptable.
90ManTIS FSS 2015 - Quantitative
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity
Quality Criteria: Structural Model Validity
91ManTIS FSS 2015 - Quantitative
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Summary of quality criteria for the structural model (PLS regression analysis)
Quality criterion Recommendation
Direct effectHigh standardized path estimates
Bootstrap algorithm and t-test
Moderating effect ANOVA and F-test or product-indicator approach in conjunction with PLS
Predicting power
High variance explained (R² > 0.2)
High effect size 𝑓²: 0.02 small effect; 0.15 medium effect; 0.35 large effect
High redundancy
Global quality of structural
modelHigh goodness of fit
No multicollinearity
No perfect correlation between independent variables: Standardized path
estimates < 0.8
High tolerance of independent variables: tolerance > 0.1
Small variance inflation factor of independent variables: VIF < 10
Design validity
Quality Criteria: Design Validity
92
• No responses due to a systematic reason
• E.g., dissatisfied customers tend to be more vocalNon-response bias
• Parts of the population are excluded
• E.g., online surveys exclude people without webSampling bias
• Tendency to portray oneself socially desirable
• E.g., “Have you ever downloaded illegal music?”Social desirability bias
• Participants might not remember certain events
• E.g., “For which tasks have you used your personal computer ten years ago?”
Recall bias
• Variables measured with an identical method, and
• Variables measured at the same timeCommon method bias
Source: Bhattacherjee (2012)
ManTIS FSS 2015 - Quantitative
Research
Intro to
Quality Criteria
Measurement
model validity
Structural
model validity
Design validity
Agenda
Agenda
1 Basics of Quantitative Research
2 Surveys
3 Experiments
4 Software Demo and Hands-On Exercise (SmartPLS, R Studio)
5 Summary
6 Supplementary Material on Quality Criteria
7 References
93ManTIS FSS 2015 - Quantitative Research
References (1/4)
94
Bagozzi, R. P., and Yi, Y. 2012. “Specification, Evaluation, and Interpretation of Structural Equation Models,” Journal of the Academy of Marketing Science 40(8), pp. 8-34.
Bhattacherjee, A. 2012. Social Science Research: Principles, Methods, and Practices, (2. ed.). Tampa, FL, USA: Open Access Textbooks.
Chin, W. W. 1998. “The Partial Least Squares Approach to Structural Equation Modeling,” in Modern Methods for Business Research, G. A. Marcoulides (ed.), Mahwah, NJ:
pp. 295-336.
Chin, W. W., and Newsted, P. R. 1999. “Structural Equation Modeling Analysis with Small Samples Using Partial Least Squares,” in Statistical Strategies for Small Sample
Research, R. Hoyle (ed.), Thousand Oaks, CA: Sage Publications, pp. 307-341.
Chin, W. W., Marcolin, B. L., and Newsted, P. R. 2003. “A Partial Least Squares Latent Variable Modeling Approach for Measuring Interaction Effects: Results from a Monte
Carlo Simulation Study and an Electronic-Mail Emotion / Adoption Study,” Information Systems Research (14:2), pp. 189-217.
Chin, W. W. 2010. “How to Write Up and Report PLS Analyses,” in Handbook of Partial Least Squares. Concepts, Methods and Applications, V. E. Vinzi, W. W. Chin, J.
Henseler and H. Wang (eds.), Heidelberg, Germany: Springer, pp. 655-690.
Clark-Carter, D. 2010. Quantitative Psychological Research. The Complete Student’s Companion (3rd ed.), Hove, UK: Psychology Press.
Cohen, J. 1988. Statistical Power Analysis for the Behavioral Sciences (2nd ed.), Hillsdale, NJ: Erlbaum.
Cook, T. D., and Campbell, D. T. 1979. Quasi-Experimentation: Design and Analysis Issues for Field Settings, Boston, MA: Houghton Mifflin Company.
Cronbach, L.J., 1951. Coefficient alpha and the internal structure of tests. Psychometrika 16 (3), 297–334.
Denzin, N. K., and Lincoln, Y. S. 2011. The SAGE Handbook of Qualitative Research, (4. ed.), Thousand Oaks, CA, USA: Sage Publications.
Duarte, P. A. O. and Raposo, M. L. B. 2010. "A PLS Model to Study Brand Preference: An Application to the Mobile Phone Market," in Handbook of Partial Least Squares.
Concepts, Methods and Applications, V. E. Vinzi, W. W. Chin, J. Henseler and H. Wang (eds.), Heidelberg, Germany: Springer, pp. 449-485.
Efron, B., and Tibshirani, R. J. 1993. An Introduction to the Bootstrap. Monographs on Statistics and Applied Probability, New York, NY: Chapman & Hall.
Falk, R. F. and Miller, N. B. 1992: A Primer for Soft Modeling, Ohio: The University of Akron Press.
Field, A., Miles, J., and Field, Z. 2012. Discovering Statistics Using R, London, UK: Sage Publications.
Fornell, C., and Larcker D. F. 1981. “Evaluating Structural Equation Models with Unobserved Variables and Measurement Error,” Journal of Marketing Research (18), pp. 39-
50.
Goodhue, D. L., Lewis, W., and Thompson, R. 2012a. “Comparing PLS to Regression and LISREL: A Response to Marcoulides, Chin, and Saunders,” MIS Quarterly (35:3),
pp. 703-716.
Goodhue, D. L., Lewis, W., and Thompson, R. 2012b. “Does PLS have Advantages for Small Sample Size or Non-Normal Data?” MIS Quarterly (36:3), pp. 981-1001.
ManTIS FSS 2015 - Quantitative
Research
References (2/4)
95
Goodhue, D., Lewis, W., and Thompson, R. 2006. "PLS, Small Sample Size, and Statistical Power in MIS Research," 39. Annual Hawaii International Conference on System
Sciences (HICSS 2006), Kauai, HI, USA, pp. 202b-202b.
Guba, E.G., and Lincoln, Y.S. 1994. "Competing Paradigms in Qualitative Research," in: Handbook of Qualitative Research, N.K. Denzin and Y.S. Lincoln (eds.). Thousand
Oaks, CA, USA: Sage, pp. 105-117.
Hair, J. F., Ringle, C. M., and Sarstedt, M. 2012a. “Partial Least Squares: The Better Approach to Structural Equation Modeling?” Long Range Planning (45), pp. 312-319.
Hair, J. F., Sarstedt, M., Ringle, C. M., and Mena, J. A. 2012b. “An Assessment of the Use of Partial Least Squares Structural Equation Modeling in Marketing Research,”
Journal of the Academy of Marketing Science (40:3), pp. 414-433.
Hair, J. F., Hult, T. M., Ringle, C. M., Sarstedt, M. 2013. A Primer on Partial Least Squares Structural Equation Modeling (PLS-SEM): Sage Publications.
Jiang, Z. J., Heng, C. S., and Choi, B. C. F. 2013. „Privacy Concerns and Privacy-Protective Behavior in Synchronous Online Social Interactions,“ Information Systems
Research (24:3), pp. 579-595.
Lewis, B. R., Templeton, G. F., and Byrd, T. A. 2005. “A Methodology for Construct Development in MIS Research,” European Journal of Information Systems (14:4), pp. 388-
400.
Li, X., Po-An Hsieh, J. J., and Rai, A. 2013. „Motivational Differences Across Post-Acceptance Information System Usage Behaviors: An Investigation in the Business
Intelligence Systems Context,“ Information Systems Research (24:3), pp. 659-682.
Liang, H., Saraf, N., Hu, Q., and Xue, Y. 2007. “Assimilation of Enterprise Systems: The Effect of Institutional Pressures and the Mediating Role of Top Management,” MIS
Quarterly (31:1), pp. 59-87.
MacKenzie, S.B., Podsakoff, P.M., and Podsakoff, N.P. 2011. "Construct Measurement and Validation Procedures in MIS and Behavioral Research: Integrating New and
Existing Techniques," MIS Quarterly (35:2), pp. 293-A295.
Marcoulides, G. A., Chin, W. W., and Saunders, C. 2012. “When Imprecise Statistical Statements become Problematic: A Response to Goodhue, Lewis, and Thompson,” MIS
Quarterly (36:3), pp. 717-728.
Marcoulides, G. A., Chin, W. W., and Saunders, C. 2009. “A Critical Look at Partial Least Squares Modeling,” MIS Quarterly (33:1), pp. 171-175.
Marcoulides, G.A., and Saunders, C. 2006. "PLS: A Silver Bullet?," MIS Quarterly (30:2), pp. iii-ix.
Martin, S. L., Liao, H., and Campbell, E. M. 2013. „Directive versus Empowering Leadership: A Field Experiment Comparing Impacts on Task Proficiency and Proactivity,“
Academy of Management Journal (56:5), pp1372-1395.
Meyers, L. S., Gamst, G., and Guarino, A. J. 2006. Applied Multivariate Research. Design and Interpretation, Thousand Oaks, CA: Sage Publications.
Monette, D. R., Sullivan, T. J., and DeJong, C. R 2010. Applied Social Research. A Tool for the Human Services, (8. ed.), Belmont, CA, USA: Cengage Learning.
Myers, M. D. 2009. Qualitative Research in Business & Management, Illustrated edition: Sage Publications.
ManTIS FSS 2015 - Quantitative
Research
References (3/4)
96
Neuman, W.L. 2000. Social Research Methods: Quantitative and Qualitative Approaches, (4. ed.). Boston, MA, USA: Allyn and Bacon.
O’Brien, R. M. 2007. “A Caution Regarding Rules of Thumb for Variance Inflation Factors,” Quality and Quantity (41), pp. 673-690.
Pagano, R. R. 2010. Understanding Statistics in the Behavioral Sciences, (10. ed.), Belmont, CA, USA: Cengage Learning.
Petter, S., Straub, D. W., and Rai, A. 2007. “Specifying formative Constructs in Information Systems Research,” MIS Quarterly (31:4), pp. 623-656.
Podsakoff, P. M., MacKenzie, S. B., Lee, Y.-J., and Podsakoff, N. P. 2003. “Common Method Biases in Behavioral Research: A Critical Review of the Literature and
Recommended Remedies,” Journal of Applied Psychology (88:5), pp. 879-903.
Qureshi, I., and Compeau, D. 2009. “Assessing Between-Group Differences in Information Systems Research: A Comparison of Covariance- and Component-based SEM,”
MIS Quarterly (33:1), pp. 197-214.
Richardson, H. A., Simmering, M. J., and Sturman, M. C. 2009. “A Tale of Three Perspectives: Examining Post Hoc Statistical Techniques for Detection and Correction of
Common Method Variance,” Organizational Research Methods (12:4), pp. 762-800.
Ringle, C. M., Sarstedt, M, and Straub, D. W. 2012. “A Critical Look at the Use of PLS-SEM in MIS Quarterly,” MIS Quarterly (36:1), iii-xiv.
Shadish, W. R., Cook, T. D., and Campbell, D. T. 2002. Experimental and Quasi-experimental Designs for Generalized Causal Inference, Boston, MA: Houghton-Mifflin.
Shook, C.L., Ketchen, D.J., Hult, T., and Kacmar, K.M. 2004. “An assessment of the use of structural equation modeling in strategic management research,” Strategic
Management Journal, 25(4), pp. 397-404.
Sun, H. 2013. „A Longitudinal Study of Herd Behavior in the Adoption and Continued Use of Technology,“ MIS Quarterly (37:4), pp. 1013-1041.
Steenkamp, J.-B., and Baumgartner, H. 2000. “On the Use of Structural Equation Models for Marketing and Modeling,” International Journal of Research in Marketing (17:2/3),
pp. 195-202.
Stevens, J. P. 2002. Applied Multivariate Statistics for the Social Sciences (4th ed.), Mahwah, NJ: Earlbaum.
Straub, D.W., Gefen, D., and Boudreau, M.-C. 2005. "Quantitative Research," in: Research in Information Systems: A Handbook for Research Supervisors and Their Students,
D. Avison and J. Pries-Heje (eds.). Amsterdam, The Netherlands: Elsevier, pp. 221-238.
Temme, D., Kreis, H., and Hildebrandt, L. 2010. “A Comparison of Current PLS Path Modeling Software: Features, Ease-of-Use, and Performance,” in Handbook of Partial
Least Squares. Concepts, Methods and Applications, V. E. Vinzi, W. W. Chin, J. Henseler and H. Wang (eds.), Heidelberg, Germany: Springer, pp. 737-756.
Tenenhaus, M., Vinzi, V. E., Chatelin, Y.-M., and Lauro, C. 2005. “PLS Path Modeling,” Computational Statistics & Data Analysis (48), pp. 159-205.
Venkatesh, V., Brown, S. A., Bala, H. 2013. Bridging the Qualitative-Quantitative Divide: Guidelines for Conducting Mixed Methods Research in Information Systems,“ MIS
Quarterly (37:1), pp. 21-54.
ManTIS FSS 2015 - Quantitative
Research
References (4/4)
97
Vinzi, V. E., Trinchera, L., and Amato, S. 2010. “PLS Path Modeling: From Foundations to Recent Developments and Open Issues for Model Assessment and Improvement,” in
Handbook of Partial Least Squares. Concepts, Methods and Applications, V. E. Vinzi, W. W. Chin, J. Henseler and H. Wang (eds.), Heidelberg, Germany: Springer, pp. 47-82.
Werts, C. E., Linn, R. L., and Joereskog, K. G. 1974. “Intraclass Reliability Estimates: Testing Structural Assumptions,” Educational and Psychological Measurement (34), pp.
25-33.
Williams, L. J., Edwards, J. R., and Vandenberg, R. J. 2003. “Recent Advances in Causal Modeling Methods in Organizational and Management Research,” Journal of
Management (29:6), pp. 903-936.
Williams, L.J., Vandenberg, R.J., and Edwards, J.R. 2009. "Structural Equation Modeling in Management Research: A Guide for Improved Analysis," The Academy of
Management Annals (3:1), January, pp 543-604.
Xu, J. D., Benbasat, I., and Cenfetelli, R. T. 2014. „The Nature and Consequences of Trade-Off Transparency in the Context of Recommendation Agents,“ MIS Quarterly
(38:2), pp. 379-406.
ManTIS FSS 2015 - Quantitative
Research
Contact
Martin KretzerResearch Assistant
Consultation hour: per request
E-Mail: [email protected]
ManTIS 2015 - Overview and
Registration98