Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis
-
Upload
upton-skinner -
Category
Documents
-
view
21 -
download
0
description
Transcript of Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis
![Page 1: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/1.jpg)
Interactive Goal Model Analysis Applied - Systematic Procedures
versus Ad hoc Analysis
Jennifer Horkoff1
Eric Yu2
Arup Ghose1
Department of Computer Science1
Faculty of Information2
[email protected] [email protected] [email protected] of Toronto
November 10, 2010PoEM’10
![Page 2: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/2.jpg)
Goal Modeling Used as a tool for system analysis and design in an
enterprise Captures social-driven goals which motivate design
or redesign First sub-model of Enterprise Knowledge
Development (EKD) method Used in several Requirements Engineering
frameworks i* (Yu, 97) Tropos (Bresciani et al., 94) GBRAM (Antón et al., 98) KAOS (Dardenne & van Lamsweerde, 93) GRL (Liu & Yu, 03) Etc.
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 2
![Page 3: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/3.jpg)
Goal Model Analysis Work has argued that more utility can be gained from
goal models by applying systematic analysis Many different types of analysis procedures have been
introduced (metrics, model checking, simulation, planning, satisfaction propagation)
Most of the work in goal model analysis focuses on the analytical power and mechanisms of the procedures
What are the benefits of goal model analysis? Do these benefits apply only to a systematic procedure?
Or also to ad-hoc (no systematic procedure) analysis?
Focus: interactive satisfaction propagation
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 3
![Page 4: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/4.jpg)
Hypotheses: Benefits of Systematic, Interactive Goal Model Analysis Previous work by the authors has introduced interactive,
qualitative goal model analysis aimed for early enterprise analysis (CAiSE’09 Forum, PoEM’09, IJISMD)
Hypotheses concerning benefits of interactive analysis developed through application of several case studies (PoEM’09, PST’06, REFSQ’08, HICSS’07, RE’05) Analysis: aids in finding non-obvious answers to domain
analysis questions Model Iteration: prompts improvements in the model Elicitation: leads to further elicitation of information in the
domain Domain Knowledge: leads to a better understanding of the
domain
In this work we design and administer studies to test these hypotheses
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 4
![Page 5: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/5.jpg)
Background: i* Models We use i* as an example goal modeling framework
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 5
![Page 6: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/6.jpg)
“Real” Example: inflo Case Study
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 6
![Page 7: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/7.jpg)
Background: Interactive Satisfaction Analysis Forward: A question/
scenario/ alternative is placed on the model and its affects are propagated “forward” through model links
Interactive: user input (human judgment) is used to decide on partial or conflicting evidence “What is the resulting value?”
Publications: CAiSE’09 Forum, PoEM’09, IJISMD
Additional procedure for “backward” analysis, allows “is this possible?” questions
Publications: istar’08, ER’10
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 7
Human Judgment
Human Judgment
What if…?
![Page 8: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/8.jpg)
Case Study Design One group study involving “inflo” “back-of-the-envelope”
calculation and modeling tool (case = group) Four grad students, 1 professor, and 1 facilitator Three two hour modeling sessions + one hour analysis
session Most of each session devoted to developing the model &
discussion with analysis at the end of each session Ten two-hour sessions with an individual and a facilitator
(case = individual) Five used systematic forward and backward analysis
implemented in OpenOME Five were allowed to analyze the models as they liked
Individual study design was modified midway through Divided into Round 1 and Round 2
Studies were both exploratory and confirmatoryInteractive Goal Model Analysis Applied - Horkoff, Yu, Ghose 8
![Page 9: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/9.jpg)
Individual Studies (Round 1) Participants: students who had i* experience in
system analysis courses or through i*-related projects Purposive selection: wanted subjects with some i*
knowledge but not much analysis experience Training:
Participants given 10 minutes of i* training (including analysis labels)
Systematic participants given 15 minutes of analysis training using the tool
Model Domain: ICSE Greening models, large to medium models created by others
Analysis Questions: 12 questions provided 2 for each analysis direction (forward, backward) per model
* 3 models
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 9
![Page 10: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/10.jpg)
ICSE Greening Example: Conference Experience Chair
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 10
![Page 11: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/11.jpg)
Individual Studies Intermediate (Round 1) results:
Models were too complicated Too many analysis questions Participants unfamiliar with domain Didn’t “care” about judgment decisions Made very few changes to models (too afraid to change
other’s work? too intimidated to change complex models?)
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 11
![Page 12: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/12.jpg)
Individual Studies (Round 2) Round 2 Changes (last 4/10 participants)
Model Domain: Asked participants to create their own models describing student life
Group case study showed that participants had trouble finding analysis questions over their own model
Created Analysis Methodology to help guide the analysis
Extreme test conditions (all alternatives/targets satisfied/denied)
Analyze likely alternatives/targets Analyze domain-driven questions
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 12
![Page 13: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/13.jpg)
Data Capture Analysis: captured answers to analysis questions Model Iteration: quantitative counts of model changes
for each stage in the studies Elicitation: captured lists of questions asked about the
domain in each stage Domain Knowledge: follow-up questions about
experience
Recorded and analyzed other interesting qualitative findings
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 13
![Page 14: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/14.jpg)
Results
![Page 15: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/15.jpg)
Analysis Analysis: aids in finding non-obvious answers to
domain analysis questions Some participants gave explicit answers, others had
difficultly producing answers Some referred to analysis labels in the model as answers
to the question Only some participants were able to interpret analysis
results in the context of the domain Generally, difficulty in mapping the model to the domain Conclusion: knowledge of i* and the domain may
have a significant effect on the ability to apply and interpret analysis
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 15
![Page 16: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/16.jpg)
Model Iteration & Elicitation Model Iteration: prompts improvements in the model Elicitation: leads to further elicitation of information in the domain
Few changes, few differences between ad hoc & systematic, familiar and unfamiliar domain, forward backward
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 16
# Model Changes # Questions Asked
Treatment Partic.Forward Questions
Backward Questions
Forward Questions
Backward Questions Round
Ad-hoc
P1 59 10 10 1
1P4 0 0 1 0P5 5 13 6 6P7 2 5 0 0
2P9 0 5 0 0
Systematic
P2 0 0 2 3
1P3 0 0 2 0P6 0 3 5 1P8 0 0 2 2
2P10 0 0 0 1
![Page 17: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/17.jpg)
Model Iteration & Elicitation Conflicts with previous results (PoEM’09, PST’06, etc.),
Why? Underlying theory: interactive analysis prompts users to
notice differences between mental domain model and physical model Evaluation did not reveal differences between the mental and
physical model, or these differences existed, but were not used to modify the model
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 17
![Page 18: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/18.jpg)
Model Iteration & Elicitation Previous studies were conducted by i*/modeling “experts”
who had commitment to the project Conclusion: Model iteration and elicitation relies on:
More extensive knowledge of syntax and analysis procedures
More extensive knowledge of the domain “buy-in”/caring about a real problem
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 18
![Page 19: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/19.jpg)
Domain Knowledge Domain Knowledge: leads to a better
understanding of the domain Follow-up question: “do you feel that you have a better
understanding of the model and the domain after this exercise?”
7/10 participants said yes (mix of ad-hoc and systematic participants)
Conclusion: both ad-hoc and systematic knowledge can help improve domain knowledge
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 19
![Page 20: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/20.jpg)
Additional Findings Promoted Discussion in Group Setting: human judgment
caused discussion among participants Example: “what is meant by Flexibility?”
Model Interpretation Consistency i* syntax leaves room for interpretation Results shows a variety of interpretations when propagating
analysis labels with ad-hoc analysis Conclusion: systematic analysis provokes a more consistent
interpretation of the model Coverage of Model Analysis
Results show significant differences in the coverage of analysis across the model with systematic vs. ad-hoc analysis
Model Completeness and Analysis Analysis may not be useful until the model is sufficiently complete Some participants noticed incompleteness in the model(s) after
applying analysis
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 20
![Page 21: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/21.jpg)
Conclusions and Future Work Designed and administered studies to test perceived benefits
of interactive goal model analysis Initial Hypotheses: Analysis, Model Iteration, Elicitation,
Domain Knowledge Benefits dependent on:
Knowledge of i* and i* evaluation Presence of an experienced facilitator Domain expertise/buy-in The presence of a real motivating problem
Discovered benefits: Interpretation Consistency, Coverage of Model Analysis, Model Completeness
Several threats to validity (construct, internal, external, reliability) described in the paper
Future Work More realistic action-research type studies Better tool support – make the tool the expert?
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 21
![Page 22: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/22.jpg)
Thank youQuestions?
[email protected] www.cs.utoronto.ca/~jenhork
[email protected] www.cs.utoronto.ca/~eric
OpenOME: https://se.cs.toronto.edu/trac/ome
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 22
![Page 23: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/23.jpg)
Outline Goal Modeling Goal Model Analysis Hypotheses: Benefits of Systematic, Interactive
Goal Model Analysis Background: i* Syntax Background: Interactive Goal Model Analysis Case Study Design
Group study Individual Studies
Results Threats to Validity Conclusions and Future Work
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 23
![Page 24: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/24.jpg)
Goal Model Analysis Work has argued that more utility can be gained from
goal models by applying systematic analysis Many different types of analysis procedures have been
introduced Metrics (Franch, 06) (Kaiya, 02) Model checking (Fuxman et al., 03) (Giorgini et al., 04) Simulation (Gans et al., 03) (Wang & Lesperance, 01) Planning (Bryl et al., 06) (Asnar et al., 07) Satisfaction Propagation (Chung et al., 00) (Giorgini et al.,
05) Most of this work focuses on the analytical power and
mechanisms of the procedures What are the benefits of goal model analysis? Do these benefits apply only to a systematic procedure?
Or also to ad-hoc (no systematic procedure) analysis?Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 24
![Page 25: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/25.jpg)
inflo (Group) Case Study inflo: “back-of-the-envelope” calculation and
modeling tool Support informed debate over issues like carbon
footprint calculations Four grad students, 1 professor, and 1 facilitator Three two hour modeling sessions + one hour
analysis session Most of each session devoted to developing the
model & discussion Used systematic model analysis at the end of
each session
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 25
![Page 26: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/26.jpg)
Individual Studies (Round 1) Analysis Questions: 12 questions provided
4 per model (3 models) 2 for each analysis direction (forward, backward) per
model Example (forward):
“If every task of the Sustainability Chair and Local Chair is performed, will goals related to sustainability be sufficiently satisfied?”
Example (backward): “What must be done in order to Encourage informal
and spontaneous introductions and Make conference participation fun?”
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 26
![Page 27: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/27.jpg)
Analysis Methodology 1. Alternative Effects (Forward Analysis)
a) Implement as much as possible: all leaves are satisfied b) Implement as little as possible: all leaves are denied c) Reasonable Implementation Alternatives: Evaluate likely
alternatives 2. Achievement Possibilities (Backward Analysis)
a) Maximum targets: all roots must be fully satisfied. Is this possible? How?
b) Minimum targets: lowest permissible values for the roots. Is this possible? How?
c) Iteration over minimum targets: try gradually increasing the targets in order to find maximum targets which still allow a solution.
3. Domain-Driven Analysis (Mixed) a) Use the model to answer interesting domain-driven
questionsInteractive Goal Model Analysis Applied - Horkoff, Yu, Ghose 27
![Page 28: Interactive Goal Model Analysis Applied - Systematic Procedures versus Ad hoc Analysis](https://reader036.fdocuments.us/reader036/viewer/2022081520/56812ebc550346895d945e74/html5/thumbnails/28.jpg)
Threats to Validity Construct Validity
Model changes may not be beneficial Internal Validity
Presence of facilitator Think-aloud protocol Choice of model domain
External Validity Used students Used i* - generalize to other goal model frameworks?
Reliability Facilitator was i* & evaluation expert
Interactive Goal Model Analysis Applied - Horkoff, Yu, Ghose 28