Design, Implementation and Evaluation of a Collaborative ...€¦ · Software Engineering for...

Software Engineering for Business Information Systems (sebis) Department of InformaticsTechnische Universität München, Germany

wwwmatthes.in.tum.de

Design, Implementation and Evaluation of a Collaborative Synthesis Stage in the Context of Computer-Supported Morphological AnalysisBastian Rodehüser | Final Presentation Master Thesis | 05.12.2016

1. Introduction § What is Morphological Analysis?§ What Support Systems for Morphological Analysis do exist?

2. Collaborative Morphological Analysis§ Group Decisions§ Real-Time-Delphi Technique§ Computer-Supported Collaborative Cross-Consistency Assessment§ Computer-Supported Collaborative Constraint Management

3. Evaluation§ Hypotheses§ Solution Space and Actions§ Satisfaction with process, decision and application

4. Conclusion

Overview

© sebis 2Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016

What is Morphological Analysis?

§ Creativity technique for groups§ Iterative process

© sebis 3

The general iterative process model cf. Zec et al. 2015

Synthesis

à Cross-Consistency Assessmentà Constraint Management

Analysis

Define parameters &

values

Exploration

Examine consistent

configurations

Bastian Rodehüser | Final Presentation Master Thesis | 05.12.2016


© sebis 4

Decomposition of system• Identification of parameters• Definition of alternatives/ values for each

parameter

Morphological Box/Field

Motorization Range Operation area Number of Passengers

Control

CombustionEngine

< 100 km Private City Traffic

1 Driver

Electric Motor 100 - 299 km Travelling 2 Driver / Autopilot

Fuel Cell 300 - 599 km Delivery Vehicle 4-5 Only AutopilotHybrid 600 - 999 km > 5

> 1000 km

K is the set of simple configurations.à Cardinality: |K| = 720



© sebis 5

LK

In synthesis phase, all configurations in K are examined if they are consistent.

All consistent configurations of K are elementof the solution space L.à 𝐿 ⊆ K

Elements of L can be ordered.à Rank order




Cross-Consistency Assessment• Configurations with incompatible parameter values

can be excluded from further consideration• Rank order by consistency scores

Cross-Consistency Matrix


© sebis 7

Constraint Management• Positive Constraints:

• All configurations that fit to a positive constraint receive a recommendation point for the rank order

• Negative Constraints: • All configurations that fit to a negative

constraint are excluded from the solution space

Motorization: Electric Motor OR Fuel CellOperation Area: Delivery DeviceNumber of Passengers: 1 OR 2

NOT Range: <100 kmNOT Range: 100-299 km

Example constraint;can be both, positive and negative


Captures 32 of 720 configurations!


© sebis 8

Exploration of the remaining set (L) of consistent configurations


Existing Support Systems

§ Existing Support Systems (e.g. MA/Carma, Parmenides Eidos):

à [Analysis] Help to collect parameters and their corresponding valuesà [Synthesis] Offer tools for Cross-Consistency Assessment

• Calculate the solution space

à [Exploration] Help to explore the solution space of consistent configurations

§ BUT: existing Support Systems are not collaborative!àGroup members have to use these applications at

• the same place and • the same time

àThey have to discuss and to agree on inputs to the applicationàSoftware does not help to enhance social-psychological process gains und

to avoid process losses


Computer-Supported Synthesis in Practice

© sebis 10

• Cross-Consistency Assessment (three approaches)1. For each cell: Discussion in group and building consensus; input by

moderator2. Individual work by participants; discussion of contradictions and building

consensus; input by moderator3. Individual work by participants; use of all created individual solution spaces

in exploration phase

• Weaknesses• complex (example: 143 cells)• time-consuming• labor-intensive

Not suitable for many participants respectively high complex projects


cf. interviews by Rodehüser (2016, appendix master thesis)




4. Conclusion

Overview


§ Advantages1. Combination and integration of:

a) Different knowledgeb) Different ideasc) Different perspectives

2. Identification and integration of individual points of view Ø higher acceptance of decision

§ Quality• Comprehensive exchange of ideas and information is important• Set union of all individual idea / information sets

§ Preferences• Case: similar information set but different individual preferences

Ø Cause: different objectives / different risk attitudesØ Aggregation of individual preferences to group preferenceØ Discussions and use of voting procedures

Group Decisions


Real-time Delphi technique


Goal: Avoid process losses | support process gains | achieve comprehensive information exchange

cf. Gordon and Pease 2006cf. Gordon 2009cf. Gnatzy et al. 2011

Implementation


LIVE DEMO




4. Conclusion

Overview


Hypotheses

The process of Cross-Consistency Assessment is easier to learn than the process of Constraint Management.

Both implemented methods are suitable for solving hidden profiles.

Both implemented methods are suitable for creating a consistent solution space.

Constraint Management can be carried out in a shorter time than Cross-Consistency Assessment.

© sebis 16

1

2

3


4

Test Procedure

§ Morphological Box is given

§ Every participants got individual information• unshared• sharedà Hidden Profiles in distribution of information

§ “Correct” solution space exists

§ 4 group with 4 participants• 2 groups conduct Cross-Consistency Assessment• 2 groups conduct Constraint Management


Solution Space over time


Number of Actions


Solution Space: Time


Left: Last change of solution spaceRight: Last action

→ Does not support hypothesis 4. (Constraint Management can be carried out in a shorter time than Cross-Consistency Assessment.)

→ Future research: time advantage with more complex Morphological Boxes or more experienced teams?

Solution Space: Quality


→ Created solution spaces approximate to the sample solution.

→ Supports hypothesis 3. (Both methods are suitable for creating a consistent solution space.)

Resolution of Hidden Profiles


Information3 Information4 Information5 Information6 Information9

Group Correct SI CAC SI CAC SI CAC SI CAC SI CAC

CCA1 3/5

CCA2 4/5

CM1 4/5

CM1 5/5

SI = Shared Information (transformation of unshared information into shared information)

CAC = Correct Assessment/Constraint(creation of correct assessment/constraint related to information)

→ Supports hypothesis 2. (Both methods are suitable for solving hidden profiles.)→ All unshared information could be transformed into shared information.→ This has led to 16 of 20 correct assessments/constraints.

Satisfaction with process


1,0

1,5

2,0

2,5

3,0

3,5

4,0

4,5

5,0

1efficient5inefficient

1coordinated5uncoordinated

1fair5unfair

1understandable5confusing

1satisfying5dissatisfying

Howwouldyoudescribeyourgroup'sproblemsolvingprocess?

CCA CM CCA1 CCA2 CM1 CM2

→ Can indicate the correctness of hypothesis 4 (Constraint Management can be carried out in a shorter time than Cross-Consistency Assessment), if the problem complex is large.

Questions by Green und Taber (1980, p. 102) and Reinig (2003, p. 82)

Satisfaction with application (1/2)


1,01,52,02,53,03,54,04,55,0

1StronglyDisagree5StronglyAgree





IthinkthatIwouldliketousethis

productfrequently.

Ithoughttheproductwaseasytouse.

Ifoundthevariousfunctionsintheproductwerewell

integrated.

Iimaginethatmostpeoplewouldlearntousethisproductvery

quickly.

Ifeltveryconfidentusingtheproduct.


→ Supports hypothesis 1. (The process of Cross-Consistency Assessment is easier to learn than the process of Constraint Management.)

Questions by Brooke (1996), Bangor et al. (2008), Bangor et al. (2009)

Satisfaction with application (2/2)


1,01,52,02,53,03,54,04,55,0






Ifoundtheproductunnecessarilycomplex.

IthinkthatIwouldneedthesupportofatechnicalpersontobeabletousethis

product.

Ithoughttherewastoomuch

inconsistencyinthisproduct.

Ifoundtheproductveryawkwardtouse.

IneededtolearnalotofthingsbeforeI

couldgetgoingwiththisproduct.


→ Previous knowledge is important.

Questions by Brooke (1996), Bangor et al. (2008), Bangor et al. (2009)

System Usability Scale (SUS)


83,171,6

88,877,5

66,376,9

0,0

10,0

20,0

30,0

40,0

50,0

60,0

70,0

80,0

90,0

100,0


Notacceptable

Marginal

Acceptable

Calculation according to Brooke (1996, p. 194)

1. Introduction § What is Morphological Analysis?§ What Support Systems for Morphological Analysis do exist?§ Social Psychology Considerations

2. Collaborative Morphological Analysis§ Quality of Group Decisions§ Real-Time-Delphi Technique§ Computer-Supported Collaborative Cross-Consistency Assessment§ Computer-Supported Collaborative Constraint Management


4. Conclusion

Overview


Summary & Conclusion (1/2)


§ Methods• Cross-Consistency Assessment

- More structured- Higher certainty that no inconsistent configurations are element of the solution space- Encourages participants to consider all parameters and values

• Constraint Management- More powerful- More effective under certain conditions

§ Both methods can be combined (cf. master thesis)

§ Existing support systems• No support of decision-making, consensus-building and collaborative work• No enhancement of process gains and no avoidance of process losses• Only support of Cross-Consistency Assessment (time-consuming / labor-intensive

à suitable for few participants)

§ Good group decisions require a comprehensive information exchange

Summary & Conclusion (2/2)


§ Web Application• Frameworks: Meteor & React• Implements both methods• Based on Real-Time-Delphi approach à collaborative work is possible

- Enhancement of process gains and avoidance of process losses- Encourages participants to a comprehensive information exchange

§ The results suggest that hypothesis 1 is correct. (The implemented Cross-Consistency Assessment seems to be easier to learn)

§ Both implemented methods are suitable to create consistent solution space (Hypothesis 2)

§ Both implemented methods are suitable to solve hidden profiles (Hypothesis 3)

§ Hypothesis 4 could not be verified. (Constraint Management can be carried out in a shorter time than Cross-Consistency Assessment)

§ Future Research• Varied group size• Varied group experience• Varied group knowledge• Varied problem complexities

Literature§ Gnatzy, Tobias; Warth, Johannes; Gracht, Heiko von der; Darkow, Inga-Lena (2011): Validating an

innovative real-time Delphi approach - A methodological comparison between real-time and conventional Delphi studies. In: Technological Forecasting and Social Change 78 (9), S. 1681–1694.

§ Gordon, Theodore J.; Pease, Adam (2006): RT Delphi. An efficient, “round-less” almost real time Delphi method. In Technological Forecasting and Social Change 73 (4), pp. 321–333.

§ Gordon, Theodore J. (2009): The Real-Time Delphi Method. Available online at http://www.millennium-project.org/millennium/RTD-method.pdf.

§ Green, Stephen G.; Taber, Thomas D. (1980): The effects of three social decision schemes on decision group process. In: Organizational Behavior and Human Performance 25 (1), S. 97–106.

§ Reinig, Bruce A. (2003): Toward an Understanding of Satisfaction with the Process and Outcomes of Teamwork. In: Journal of Management Information Systems 19 (4), S. 65–83.

§ Ritchey, Tom (2006): Problem structuring using computer-aided morphological analysis. Available online at http://www.swemorph.com/pdf/psm-gma.pdf.

§ Ritchey, Tom (2015): Principles of Cross-Consistency Assessment in General Morphological Modelling. In : Acta morphologica generalis. AMG ; on-line journal of the Swedish Morphological Society. Vällingby: Swedish Morphological Society (Vol. 4 No. 2).

§ Stasser, Garold; Stewart, Dennis (1992): Discovery of hidden profiles by decision-making groups. Solving a problem versus making a judgment. In: Journal of Personality and Social Psychology 63 (3), S. 426–434.

§ Zec, Marin; Schneider, Alexander W.; Matthes, Florian (2015): Towards a Process Model for Computer-Supported Collaborative Morphological Analysis.


Technische Universität MünchenDepartment of InformaticsChair of Software Engineering for Business Information Systems

Boltzmannstraße 385748 Garching bei München


Bastian RodehüserB. Sc.

[email protected]

Thank you for your attention! Any questions?

Technische Universität MünchenDepartment of InformaticsChair of Software Engineering for Business Information Systems

Boltzmannstraße 385748 Garching bei München


Bastian RodehüserB. Sc.

[email protected]

BACKUP

• How to avoid process losses with an application?• Coordination Losses• Production Blocking• Social Loafing• Free Riding• Dispensability Effect• Sucker Effect• Cognitive Restriction• Hidden Profiles• Common-Knowledge-Effect• Discussion Distortion• Assessment Distortion• Conformity• Group Polarization

• How to support process gains with an application?• Social Competition• Social Compensation• Köhler Effect• Cognitive Stimulation

Social Psychology Considerations


Computer-Supported Collaborative Morphological Analysis

© sebis 34

Synthesis

- Cross-Consistency Assessment- Constraint Management

Analysis

Define parameters & values

Exploration

Examine consistent

configurations

Process model for CMA derived from Delphi technique cf. Zec et al. 2015

General Delphi Techniqueà Round based

1. Individual judgements and justifications2. Anonymous summary by a facilitator


Real-time Delphi technique§ Technique to conduct group assessments

§ Round-less

§ For each element that shall be assessed there are two phases for each participant:

1. Each participant makes an assessment without the knowledge of other assessments in the group

2. After the first own assessment participants get further information:

§ The average of all of the responses of the group so far§ The number of responses made so far§ Reasons that others have given for their responses

Responses that stay in conflict to the group answer should be marked.

Participants should observe marked responses§ Reassess own response§ Provide a comprehensible justification for the own response

§ Process can be synchronous or asynchronous

§ Participants remain anonymous

© sebis 35

cf. Gordon and Pease 2006cf. Gordon 2009


Constraint Management Process


Hidden Profile

“In a hidden profile, a superior decision alternative exists but its superiority is hidden from individual group members because they each have only a portion of information that supports this superior alternative.”

(Stasser and Steward 1992, p. 426)


Type of Actions


CCA

CM

Satisfaction with decision


1,0

1,5

2,0

2,5

3,0

3,5

4,0

4,5

5,0

1verydissatisfied5verysatisfied

1notatall5toaverygreat

extend


extend


extend


extend

Howsatisfiedordissatisfiedareyouwiththequalityof

yourgroup’ssolution?

Towhatextentdoesthefinalsolution

reflectyourinputs?

Towhatextentdoyoufeelcommittedtothegroupsolution?

Towhatextentareyouconfidentthatthegroupsolutionis

correct?

Towhatextentdoyoufeelpersonallyresponsibleforthecorrectnessofthegroupsolution?


Questions by Green und Taber (1980, p. 102) and Reinig (2003, p. 82)

Design, Implementation and Evaluation of a Collaborative ...€¦ · Software Engineering for...

Documents

Transcript of Design, Implementation and Evaluation of a Collaborative ...€¦ · Software Engineering for...