Post on 15-Dec-2015
Two Specific Questions
• How can MIS be identified within academia?• What differentiates high and low quality MIS
research?
Method
• Determine fields related to MIS (Katerattankul, Han, & Rea, 2006)
• Gather article attributes from the Top 6-9 journals in each of these related fields and MIS
DisciplinesMIS
EducationAccounting
Computer ScienceEconomicsSociology
PsychologyLibrary Science
HealthcareCommunication
ManagementMarketing
Electrical Engineering
Data
• Scrape ISIknowledge.com• 102,388 articles• Attributes analyzed included
– Title– Publication– Abstract– Keywords– Citations per Year– References to other articles– Many more
Coded Articles
• 50 citation classics were randomly chosen from the MIS articles
• Matched with 50 non-citation classics on journal and publication year
• Coded each of these 100 articles in groups of 3 after a training session and 2 trials
• Attributes coded– Theoretical contribution– Type of article (Empirical, Theoretical, Review,
Methodological)– Type of study
9
Analysis of Research Paper Abstracts
• Determine disciplines similar to MIS– Comparative definition of MIS discipline
• 13 Disciplines– MIS, Accounting, Communication, …
• Variables– 3 Numeric variables
• No. of authors• No. of pages (end page – start page = no. of pages)• No. of total citations (received to date)
– 817 Text variables - nouns and noun phrases• Extracted from abstracts
10
Descriptive Statistics
• 13 Disciplines; 38,642 Papers
DisciplineNo of
Papers1 Accounting 8152 Communication 6693 Computer Science 1,2254 Economics 5,2865 Education 2,2646 Electrical Engineering 4,2467 Healthcare 1,8768 Library Science 3,6969 Management 3,510
10 Marketing 3,79211 MIS 5,78812 Psychology 3,28213 Sociology 2,193
Total 38,642
11
Framework for Analysis
MIS Mgmt Psychology Computer Science…
Global Vocabulary (817 distinct terms)
Bag-of-Words for Each Paper
Cluster Analysis
Extract nouns and noun phrases by term frequency (TF) for each discipline
Extract most frequent 150 terms from each disciplineResult: 817 distinct terms
Build a bag-of-words model for each paper
Apply cluster analysis to bag-of-words from papers
12
5 Naturally Formed Clusters
Total # of papers: 38,642
• No. of papers / cluster
IS for Deci-sion
Support; 5524
Or-ga-nizati
onal
Be-havior, 6196Ele
c Eng & Heal-thcare,9270
Econ & Acct; 9555
What MISis NOT,8097
13
1 Info Systems for Decision Support
10 Top Keywords from Abstracts
Decision Support System (DSS)
Information
System
Software
Organization
Database
Web
Collaboration
Knowledge
Information retrieval
● Core: Library Science● Communication-based● Not psychology
14
2 Organizational Behavior
10 Top Keywords from Abstracts
Transformational leadership
Leader-member exchange (LMX)
Relational uncertainty
Organizational citizenship behavior
(OCB)
Organizational commitment
Leadership
Satisfaction
Culture
Meta-analysis
Social movement
● Human side● Sociology in business school● Collaborative
15
3 Electrical Engineering & Healthcare
10 Top Keywords from Abstracts
Inverter
Induction motor
Sensor
Topology
Mobile robot
Neural network
Architecture
System
Support vector machine (SVM)
Genetic algorithm (GA)
● Technical side● Data-driven● Not human
16
4 Economics & Accounting
10 Top Keywords from Abstracts
Earnings announcement
Financial Accounting Standard Board (FASB)
Sarbanes-Oxley Act (SOX)
Audit fee
Equilibrium
Valuation
Private information
Bidder
Earnings forecast
Incentive
● Econ & Acct very similar● No psychology● Numbers-based
17
5 What MIS is NOT
10 Top Keywords from Abstracts
Somatic symptom
Body mass index
Bipolar disorder
Anxiety disorder
(Major) depression
(Psychiatric, Mental) disorder
Physical activity
Medication
Blood pressure
Competitive intelligence (CI)
● Outside business school● Stress related● MIS does not research
Keyword Analysis in a Nutshell
• Questions to be asked and addressed:– How to represent a discipline?
• Vector Space Model
– Based on the representation, how to compare the relations/similarities among different disciplines?
• Cosine Similarity
– How’s the relations/similarities between MIS and the other disciplines evolve over time?
Vector Space Model
= <w11, w12, … , w1x>
= < w21, w22, … , w2x >
MISComputer Science
MIS
Computer Science
K1 KxK2
w11 w12 w1x
w21 w22 w2x
Dn
D1D1D2
D1
K1, K2, …, Km
Dn
D1D1D2
D1
K1, K2, …, Km
…
…
…
Similarity of MIS with the other Areas(measurement unit: each year)
1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 20100
0.1
0.2
0.3
0.4
0.5
accounting communication computer science economics educationelectrical engineering medical informatics management marketing psychologysociology
Sim
ilarit
y
Similarity of MIS with the other Areas(measurement unit: every two years)
1991-1992
1993-1994
1995-1996
1997-1998
1999-2000
2001-2002
2003-2004
2005-2006
2007-2008
2009-2010
0.0
0.1
0.2
0.3
0.4
0.5
accounting communication computer science economics educationelectrical engineering healthcare management marketing psychologysociology
computer science
marketing
management
healthcare
sociologyeducationpsychology
electronical engineeringaccounting
economics
Sim
ilarit
y
Interaction of MIS vs others
• Indicators:– MIS Contribution (CMIS)
– MIS Consumption (MISC)
Contribution to MIS
Who are buying ideas?
CMIS
MIS Contribution
1971~1975 1976~1980 1981~1985 1986~1990 1991~1995 1996~2000 2001~2005 2006~20100
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Accounting
Communication
Computer Science
Economics
Education
Electrical Engineering
Healthcare
Library Science
Management
Marketing
Psychology
Sociology
Management
Marketing
MIS
Con
trib
ution
MIS Consumption
1971~1975 1976~1980 1981~1985 1986~1990 1991~1995 1996~2000 2001~2005 2006~20100
0.2
0.4
0.6
0.8
1
1.2
Accounting
Communication
Computer Science
Economics
Education
Electrical Engineering
Healthcare
Library Science
Management
Marketing
Psychology
Sociology
Healthcare
Marketing
Library sci-ence
Electrical Engineering
Education
MIS
Con
sum
ption
MIS Consumption
1986~1990 1991~1995 1996~2000 2001~2005 2006~20100
0.1
0.2
0.3
0.4
0.5
0.6
Accounting
Communication
Computer Science
Economics
Education
Electrical Engineering
Healthcare
Library Science
Management
Marketing
Psychology
Sociology
Marketing
MIS
Con
sum
ption
Healthcare
Education
Library science
Overview
• Identify factors that determine high quality MIS articles
• “High quality” = 100 or more citations• Logistic regression models• Dependent variable is binary variable called “quality”
– 1 = high quality– 0 = not high quality
Analysis
• Analysis used 6 models– 2 “standard” models
• 5 or 6 explicit variables from ISI data set
– 4 “conceptual phrase” models• Numerous phrases derived from article title, author
keywords and ISI keywords generated by text mining
Two “Standard” Models
“Standard” model• Years since publication• Number of references• Number of authors• Number of pages• Type of document
“Standard” + name model• Years since publication• Number of references• Number of authors• Number of pages• Type of document• Name of journal*
* Name of journal suspected of dominating “standard” model
Four “Conceptual Phrase ” Models
Steps to find new possible “conceptual phrase” variables 1. Text-mine fields for most frequently used terms in
– Article titles– Author keywords– ISI keywords
2. Group terms into conceptual phrases3. Add conceptual phrases to “standard” models
– “standard” + title– “standard” + author keywords– “standard” + ISI keywords– “standard” + title + author keywords + ISI keywords
Compare Model Performance
Type Model Performance (R2)
Standard Standard 0.135
Standard Standard + Journal Name 0.304
Conceptual Phrase Standard + Article Title 0.302
Conceptual Phrase Standard + Author Keyword 0.232
Conceptual Phrase Standard + ISI Keyword 0.247
Conceptual Phrase Standard + Title + Author + ISI 0.479
Key Factors in “High Quality”
Factors Evidence
Number of pages Coefficient = 0.053
Number of references Coefficient = 0.023
Age of paper Coefficient = 0.052
Keywords (see table)
Name of journal (see table)
Factor: KeywordsResearch Contributions• study• Investigation• research
Theoretical Background • theory • perspective • building
Research Domains• computing• commerce
Title Author Keyword ISI Keywordcomputing cost economics
building online researchinfluence computing userssuccess quality computer
commerce user theoryresearch research strategytheory technology information
technology mediainvestigation perspective
study
Title Author Keyword ISI Keywordsupport decision design
information systems impact
Positive Factors
Negative Factors
Factor: Name of JournalJournal Citations
MeanCitationsMedian
Citations Mode
MIS Quarterly 60.31 31 0
Information Systems Research 49.11 30 12
Journal of Management Information Systems 24.48 13 0, 13
Information & Management 14.31 8 0
Journal of Strategic Information Systems 12.86 7 0
European Journal of Information Systems 12.55 8 0
Decision Support Systems 12.17 7 2
Information Systems 8.68 3 0
*626 articles published in MISQ received a total of 37,754 citations. The top 10 most cited MISQ articles received more than 20% of the total citations of MISQ articles.
10 Most Cited Articles – MISQ
Article Name Citations Received
Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology 2298
User acceptance of information technology: Toward a unified view 897
Review: Knowledge management and knowledge management systems: Conceptual foundations and research 693
Computer Self-Efficacy – Development of a Measure and Initial Test 499Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology – A Replication 454
Trust and TAM in online shopping: An integrated model 448
The Measurement of End-User Computing Satisfaction 413A set of principles for conducting and evaluating interpretive field studies in information systems 404
Task-Technology Fit and Individual-Performance 398
The Case Research Strategy in Studies of Information Systems 395
Article Name Journal Citations Received
Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology MISQ 2298User acceptance of information technology: Toward a unified view MISQ 897Understanding Information Technology Usage - A Test of Competing Models ISR 760Review: Knowledge management and knowledge management systems: Conceptual foundations and research MISQ 693Computer Self-Efficacy – Development of a Measure and Initial Test MISQ 499The DeLone and McLean model of Information Systems Success: a Ten-Year Update JMIS 463Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology – A Replication MISQ 454
Trust and TAM in online shopping: An Integrated Model MISQ 448
The Measurement of End-User Computing Satisfaction MISQ 413
A set of principles for conducting and evaluating interpretive field studies in information systems MISQ 404
10 Most Cited Articles – All Journals
Matched Pair Logistic Regression
• Articles paired on year and journal• One highly cited (>=100 citations)• One non-highly cited (<100 citations)• Analyze variables to determine what is
significant in predicting the highly cited paper
Theory Testing > Theory Building?
• Technology evolves, so a lot of work is done in applying old theories to new technological paradigms
• Many theories used in IS are borrowed from other fields, so building is not as prevalent as testing it in the IS domain
Conditional Logistic Regression
coef exp(coef) se(coef) z pTheory Testing -1.924 0.1460 1.26 -1.53 0.130Theory Building -2.380 0.0926 1.33 -1.79 0.074Testing*Building 0.944 2.5706 0.47 2.01 0.044
Possible Reasons
• Neither Theory Testing nor Theory Building is enough to constitute a valuable paper alone.
• A strong combination of Testing and Building produce the most valuable works
Matched Pairs Logistic Regression Management Empirical Articles
coef exp(coef) se(coef) zBuild New Theory*** 1.153236 3.168429 0.107841 10.694Test Existing Theory*** 0.660693 1.936134 0.104642 6.314
Refs*** 0.016899 1.017043 0.004754 3.555
Reading Comp* 0.197758 1.218667 0.077736 2.544
Validity*** 1.133152 3.105429 0.165915 6.83
Rsquare= 0.277 (max possible= 0.454 )Likelihood ratio test= 492.1 on 6 df, p=0Wald test = 173.1 on 6 df, p=0Score (logrank) test = 349.6 on 6 df, p=0
Non-Empirical Articlescoef exp(coef) se(coef) z Pr(>|z|)
Theoretical Contribution*** 0.289556 1.335835 0.05806 4.987 6.13E-07
Refs** 0.010592 1.010648 0.003714 2.852 0.00435
Reading Comp* 0.243718 1.275984 0.110311 2.209 0.02715
Rsquare= 0.128 (max possible= 0.39 )Likelihood ratio test= 63.33 on 4 df, p=5.783e-13Wald test = 36.87 on 4 df, p=1.920e-07Score (logrank) test = 51.62 on 4 df, p=1.656e-10
Conclusions
• MIS seems to be a multidisciplinary and maturing field
• There seem to be at least two identifiable areas within MIS – Behavioral and Technical
• Over time MIS seems to be becoming more behavioral and less technical
• Although theory testing has been more important than theory building in the past, as the discipline matures it is likely that theory building will emerge as the dominant paradigm for research in MIS