Introduction to Design Science methodologyroelw/Introduction... · Research problemsin design...
Transcript of Introduction to Design Science methodologyroelw/Introduction... · Research problemsin design...
Introduction to Design ScienceMethodology
Prof. Dr. Roel WieringaUniversity of TwenteThe Netherlands
26‐11‐2014 Hanze Hogeschool 1
• R.J. Wieringa. Design Science Methodology forInformation Systems and Software Engineering. Springer, 2014. http://link.springer.com/book/10.1007/978‐3‐662‐43839‐8
• Slides and other material at http://wwwhome.ewi.utwente.nl/~roelw/
26‐11‐2014 Hanze Hogeschool 2
Outline
• Design science– Design cycle– Empirical cycle
• Theories– Structure– Function
26‐11‐2014 Hanze Hogeschool 3
Outline
• Design science– Design cycle– Empirical cycle
• Theories– Structure– Function
26‐11‐2014 Hanze Hogeschool 4
• Design science is the design and investigation of artifacts in context
526‐11‐2014 Hanze Hogeschool
• Design science is the design and investigation of artifacts in context
626‐11‐2014 Hanze Hogeschool
7
Subjects of design science
Artifact:
SW component/system,HW component/system,Organization,Business process,Service,Method, Technique,Conceptual structure, ...
Problem context:
SW components & systems, HW components & systems,Organizations,Business processes, Services,Methods, Techniques,Conceptual structures, People,Values, Desires, Fears, Goals, Norms, Budgets,...
Interaction
Something to be designedSomething to be influenced26‐11‐2014 Hanze Hogeschool
• Design science is the design and investigation of artifacts in context
826‐11‐2014 Hanze Hogeschool
Research problems in design science
• “Design a DoA estimation system for satellite TV reception in a car.”
• “Design a multi‐agent aircraft taxi‐route planning system for use on airports”
• “Design an assurance method for data location compliance for CSPs”
• “Is the DoA estimation accurate enough?”
• “Is this agent routing algorithm deadlock‐free?”
• “Is the method usable and useful for cloud service providers?
9
To design an artifactto improve a
problem context
To answer knowledgequestions about the artifact in
context
Problems & Artifactsto investigate
Knowledge,Design problems
The design researcher iterates over these two activities
26‐11‐2014 Hanze Hogeschool
Design science
10
The context of design research
Improvement design Answering knowledge questions
Social context:Location of stakeholders
Knowledge context:Mathematics, social science, natural science, design science, design specifications, useful facts, practical knowledge, common sense, other beliefs
Existing problem‐solving knowledge, Old designs
Existing answers to knowledge questions
Goals, budgets Designs
New problem‐solving knowledge, New designs
New answers to knowledge questions
26‐11‐2014 Hanze Hogeschool
Roel1
Slide 10
Roel1 kniowledge base --- knowledge context
also:make a diagram of interaction context, social context, knowledge contextRoel Wieringa; 23-11-2012
Template for design problems
• Improve <problem context> • by <treating it with a (re)designed artifact> • such that <artifact requirements>• in order to <stakeholder goals>
• Improve my health • by taking a medicine • such that my headache disappears • in order for me to get back to work
• Improve the course admin• By integrating with student admin• Such that less errors• In order to reduce workload
26‐11‐2014 Hanze Hogeschool 11
Knowledge questions• Descriptive questions:
– What happened?– When?– Where?– What components were involved?– Who was involved?– etc.
• Explanatory questions: – Why? – What has caused the phenomena? – Which mechanisms produced the phenomena? – For what reasons did people do this?
26‐11‐2014 Hanze Hogeschool 12
Example knowledge questions• Effect:
– What is the execution time of the DoA algorithm?
– What is its accuracy?
• Trade‐off:– Comparison between algorithms
on these two variables– Comparison between versions of
one algorithm
• Sensitivity:– Assumptions about car speed?– Assumptions about processor?
• Stakeholders:― Who are affected by the DoA
algorithm?
• Goals:– What are their goals?
• Contribution evaluation(after DOA algorithm is in use)– How well does the DoA
algorithm contribute to these goals?
26‐11‐2014 Hanze Hogeschool 13
Effect questions
– Central effect question• Effect question: Context X Artifact→ Effects?
– Generalizations• Trade‐off question: Context X Alternative artifact → Effects?• Sensitivity question: Other context X artifact → Effects?
– Trade‐off and sensitivity analysis • Generalize to a larger class of artifacts and contexts• Compare different artefacts• Uncover context assumptions
26‐11‐2014 Hanze Hogeschool 14
Contribution questions– Preliminary questions:
• Stakeholder question: Who are the stakeholders?• Goal question: What are their goals?
– Central contribution question:• Contribution question: Do Effects contribute to Stakeholder
goals?
26‐11‐2014 Hanze Hogeschool 15
Outline
• Design science– Design cycle– Empirical cycle
• Theories– Structure– Function
26‐11‐2014 Hanze Hogeschool 16
26‐11‐2014 Hanze Hogeschool 17
Implementation evaluation starts a new cycle
Implementation evaluation = Problem investigation
Treatment designDesign validation
Designimplementation
•Stakeholders? Goals? •Conceptual problem framework?•Phenomena? Causes? Effects?•Effects contribute to Goals?
•Specify requirements!•Requirements contribute to goals?•Available treatments?•Design new ones!
•Context & Artifact → Effects?•Effects satisfy Requirements?•Trade‐offs for different artifacts?•Sensitivity for different Contexts?
Legend: Knowledge questions? Actions!
26‐11‐2014 Hanze Hogeschool 18
Design research projects
• Design research projects iterate one or more times throughthe design cycle.
Implementation evaluation = Problem investigation
Treatment designDesign validation
Designimplementation
Design cycleskips introduction in original problemcontext
Concurrent engineering• Development may be organized concurrently with successive
versions of the artifact
26‐11‐2014 Hanze Hogeschool 19
Time
Problem investigationTreatment designDesign validationImplementation
Evaluation
Tasks
Systems engineering
• Cycles of systems engineering– High level goals, high level requirements– Iterative refinement until– Low‐level approved interfaces, low‐level implementedspecs.
• Shown on next slide
26‐11‐2014 Hanze Hogeschool 20
26‐11‐2014 Hanze Hogeschool 21
• Iteratively reduce uncertainty about the problem• Once the goals are clear enough, reduce risk of choosing the wrong treatment
Ill‐understood problem
Better understood problem
Treatment idea
Validation
Even betterunderstood problem
Treatment specification
Validation
Still betterunderstood problem
OperationalTreatmentspecification
Validation
Goals and requirementsOperational concept
Feasibility
Prototype
Time
Clear problem, clear goals Solution1 spec Validation Implementation1 Eval
Clear goals, risky treatment Solution2 spec Validation Implementation2 Eval
Clear goals, acceptable risk Solution3 spec Validation Implementation3 Eval
Early requirements Validation
Process versus logic• Process is the performance of tasks• Logic is the structure of valid reasoning• Engineering cycle is structure of valid reasoning about an
artifact– Rational decision making– Justification of a design
• Engineering process is the performance of engineering tasks– Management of finite resources and risk over time.– Finite resources (time, money, people, knowledge, capital, …)– Risk (undesirable events may happen)– Risk appetite (willingness of decision maker to run risks)
26‐11‐2014 Hanze Hogeschool 22
Outline
• Design science– Design cycle– Empirical cycle
• Theories– Structure– Function
26‐11‐2014 Hanze Hogeschool 23
24
To design an artifactto improve a
problem context
To answer knowledgequestions about the artifact in
context
Problems & Artifactsto investigate
Knowledge,Design problems
26‐11‐2014 Hanze Hogeschool
Checklist questions about research context
26‐11‐2014 Hanze Hogeschool 25
1. Improvement goal?2. Knowledge goal?3. Current knowledge?
Engineering cycle Empirical cycle
17. Contribution to knowledge goal?18. Contribution to improvement goal?
4. … ….16. …
26‐11‐2014 Hanze Hogeschool 26
Research problem analysis4. Conceptual framework?5. Research questions?6. Population?
Research execution11. What happened?
Research & inference design7. Object of study?8. Treatment specification?9. Measurement specification?10. Inference?
Data analysis12. Data?13. Observations?14. Explanations?15. Generalizations?16. Answers?
Empirical cycle
New research problem
Design justification7. Object of study justification?8. Treatment specification justification?9. Measurement specification justification?10. Inference justification?
The research setup
• Case‐based research:– Objects of study are investigated one by one, in sequence– E.g. observational case studies, simulation, action research
• Sample‐based research:– Sample of objects of study are investigated as a whole– E.g. statistical surveys, randomized controlled experiments
26‐11‐2014 Hanze Hogeschool 27
Outline
• Design science– Design cycle– Empirical cycle
• Theories– Structure– Function
26‐11‐2014 Hanze Hogeschool 28
• A theory is a belief that there is a pattern in phenomena
26‐11‐2014 Hanze Hogeschool 29
Theories in popular discourse
• In popular discourse:– A theory is an unvalidated speculation:
• “There is a secret government department that monitors all our email”
– An idealization not applicable to the real world:• “Merging two faculties reduces cost in theory, not in practice.” • “Traffic rules are fine in theory, but not on the street”.
– An opinion (value judgment) that may be denied by otheropinions (value judgments).
• “The Dutch lost the game because the players earn too much.”• “You should buy a Mac, then you will not lose your files anymore”
26‐11‐2014 Hanze Hogeschool 30
• A scientific theory is a theory that– Has survived tests against experience– Has survived criticism by critical peers
26‐11‐2014 Hanze Hogeschool 31
Sciences of the middle range
27 November 2014 © R.J. Wieringa 32
Generalization
Realism
Universalgeneralization
Existentialgeneralization
Casedescription
Idealized conditions Realistic conditions Conditions of practice
Basic sciencesPhysics, Chemistry, parts of Biology
Special sciences (about the earth):Biology, Psychology, Sociology, …Applied sciences:Astronomy, Geology, Meteorology, Political sciences, Management science, …Design sciences:Software engineering, Information systems, Computer sciences, Electrical engineering, Mechanical enghineering, ...
Case research:Engineering, Consultancy, Psychotherapy, Health care, Management, Politics, ...
Outline
• Design science– Design cycle– Empirical cycle
• Theories– Structure– Function
26‐11‐2014 Hanze Hogeschool 33
The structure of scientific theories
1. Conceptual framework– E.g. The concepts of beamforming, of multi‐agent planning, of
location compliance
2. Generalizations stated in terms of these concepts, thatexpress beliefs that go beyond experienced phenomena.
3. Each generalization has a scope. The scope of a theory is the union of the scope of all of its generalizations.
– The generalizations of a theory usually all have the same scope.
26‐11‐2014 Hanze Hogeschool 34
Variable‐based theory• Conceptual framework defines constructs and variables.• Variables have data types and scales• Generalizations stated in terms of variables
– Case based (about class of similar cases)– Sample‐based (statistical generalization)
• Examples:– DOA descriptive theory: e.g. performance graphs– DOA causal theory: e.g. change in angle of incidence causes change in
phase difference– Introduction of agile development causes customer satisfaction to
increase– Programmer productivity correlates well with conscientiousness
26‐11‐2014 Hanze Hogeschool 35
Architectural theory
• Conceptual framework defines an architecture in terms of components and relationships
• Components have capabilities• Generalizations stated in terms of capabilities and interactions
of components• Examples:
– DOA mechanistic theory: e.g. input‐output relation is explained bystructure of the algorithm
– In agile development for SME, the SME does not put customer on‐site. SME resources are limited and focus is on business.
– Introduction of change control board reduces requirements creep.
26‐11‐2014 Hanze Hogeschool 36
The structure of design theories
1. Conceptual artifact framework– Usually a conceptual problem framework is assumed
2. Generalizations of the form Artifact X Context → Effects3. The scope of the theory is usually expressed in terms of
constraints on artifact design and assumptions about the context
26‐11‐2014 Hanze Hogeschool 37
Outline
• Design science– Design cycle– Empirical cycle
• Theories– Structure– Function
26‐11‐2014 Hanze Hogeschool 38
The functions of scientific theories
• To analyze a conceptual structure• To describe phenomena (descriptive statistics, interpretation)• To explain phenomena• To predict phenomena (analytical or statistical generalizations
of a theory, or use models)• To design an artifact by which to treat a problem
26‐11‐2014 Hanze Hogeschool 39
The functions of scientific theories
• To analyze a conceptual structure• To describe phenomena (descriptive statistics, interpretation)• To explain phenomena• To predict phenomena (analytical or statistical generalizations
of a theory, or use models)• To design an artifact by which to treat a problem
26‐11‐2014 Hanze Hogeschool 40
Causal explanations (cause‐effect relation between variables)
• If Y has been caused by X, then Y changed because X changedearlier in a particular way
• Examples– Light is on because switch was turned– Cost increased because the organization had to perform additional
tasks• Causation may be nondeterministic
– Forward nondeterminism: X sometimes causes Y– Backward: Y is sometimes caused by X
• In the field, the causal influence of X on Y may be swamped bymany other causal influences. – Lab research versus field research
26‐11‐2014 Hanze Hogeschool 41
Mechanistic explanations (interactionsbetween components)
• If system phenomenon E was produced by the interaction of system components C1, …, Cn, then C1, …, Cn is called a mechanistic explanation of E.
• Examples– Light is on because it is connected by to electricity supply when switch
was turned on– Cost increased because new people had to be hired to perform
additional tasks
• May be nondeterministic• May be interfered with by other mechanisms in the field
26‐11‐2014 Hanze Hogeschool 42
• Single‐case mechanism experiments: Investigate underlyingmechanisms (interaction between components) in single cases– Test a single instance of the artifact in the lab/field– Technical action research: Use the artifact to solve real‐world problem
• Statistical difference‐making experiments: Investigate averageeffects (average difference between treating and not treating) in large samples
26‐11‐2014 Hanze Hogeschool 43
Small samples
Large samples
Population
More realistic conditions of practice
Larger generalizations
Idealized Practical
Statistical difference‐making experiments
Single‐case mechanismexperiments
Technical action research
Case‐based inference Sample‐based inferenceNo control overobject of study(observational study)
Observational case study Survey
Control over object of study (experimentalstudy)
Single‐case mechanismexperiment,Technical action research
Statistical difference‐making experiment
26‐11‐2014 Hanze Hogeschool 44
Summary of research designs and research goals
26‐11‐2014 Hanze Hogeschool 45
Research goalsResearch designs
Problem research / Evaluation research
Treatment survey Validation research
Survey To survey problemowners / implementations
To survey possibletreatments
To survey expert opinion
Observationalcase study
To study a problem/ Implementation
Single‐casemechanismexperiment
To diagnose a problem / Test animplementation in context
To test an artifactwithout context
To validate an artifactin context
Technical Action Research (TAR)
To use and validatean artifact in practice
Statistical difference‐making experiment
To comparetreatments on random samples
To comparetreatments on random samples
Take‐home
• Design problems vs knowledge questions• Middle‐range theories• Case‐based research and sample‐basedresearch
• Scaling up
26‐11‐2014 Hanze Hogeschool 46
Extra material
• Scientific inference
26‐11‐2014 Hanze Hogeschool 47
Four kinds of inference
• All inferences are fallible. • Validity is degree of support for the inference. Cannot be
total.
27 November 2014 © R.J. Wieringa 48
Case‐based inference
27 November 2014 © R.J. Wieringa 49
Sample‐based inference
27 November 2014 © R.J. Wieringa 50