Cognitive Feedback in GDSS: Improving Control and Convergence · (GDSS), little research has...

Calhoun: The NPS Institutional Archive

Faculty and Researcher Publications Faculty and Researcher Publications Collection

1993-03

Cognitive Feedback in GDSS: Improving Control and Convergence

Sengupta, Kishore

MIS Quarterly

MIS Quarterly/March 1993, p. 87-113

http://hdl.handle.net/10945/48631

Cognitive Feedback in GDSS

Cognitive Feedbackin GDSS: ImprovingControi andConvergence

By: Kishore SenguptaDepartment of Administrative

SciencesNaval Postgraduate SchoolCode AS/SEMonterey, California 93943-5000U.S.A.

Dov Te'eniDepartment of Management

information and Decision SystemsWeatherhead School of

ManagementCase Western Reserve University699 Enterprise HaiiCieveiand, Ohio 44106 U.S.A.

AbstractCognitive feedback in group decision making isinformation that provides decision makers witha better understanding of their own decision processes and that of the other group members. Itappears to be an effective aid in group decisionmaking. Although it has been suggested as a po-tential feature of group decision support systems(GDSS), little research has examined its use andimpact. This article investigates the effect of com-puter generated cognitive feedback in computer-supported group decision processes. It viewsgroup decision making as a combination of in-dividual and collective activity. The article testswhether cognitive feedback can enhance controlover the individual and collective decision mak-ing processes and can facilitate the process ofconvergence among group members. In alaboratory experiment with groups of three deci-sion makers. 15 groups received online cognitivefeedback and 15 groups did not. Users receiv-ing cognitive feedback maintained a higher levelof control over the decision-making process astheir decision strategies converged. This re-

search indicates that (1) developers should in-clude cognitive feedback as an integral part ofthe GDSS at every level, and (2) they shoulddesign the human-computer interaction so thereis an intuitive and effective transition across thecomponents of feedback at all levels. Research-ers should extend the concepts explored here toother models of conflict that deal with ill-structured decisions, as well as study the impactof cognitive feedback over time. Finally, research-ers trying to enhance the capabilities of GDSSshould continue examining how to take advan-tage of the differences between individual,interpersonal, and collective decision making.

Keywords: Group decision making, group deci-sion support systems, group pro-cesses, cognitive feedback

ACM Categories: H.O, H.4

IntroductionFeedback to users about their decision processeshas received relatively little attention in the groupdecision support systems (GDSS) literature. In-terestingly, findings from studies at the level ofindividual support systems (Kleinmuntz, 1985;Te'eni, 1991; 1992) indicate that computerizedsystems offer opportunities for providing feed-back effectively—opportunities that rely on in-stant retrieval and computation that are im-practical under manual conditions. This is impor-tant because feedback in individual decisionmaking has been shown to increase decisionquality and confidence (Hogarth, 1987). DeSanc-tis and Gallupe (1987) have suggested that feed-back about decision processes might also beprovided in GDSS environments, particularly incognitive conflict tasks—tasks that involve theresolution of conflicting viewpoints as opposedto conflicting motives. This study takes some firststeps in conceptualizing and testing the impactof feedback on GDSS.

Studying feedback in a group context, as op-posed to an individual, requires consideration ofadditional levels of analysis and decision-makingactivity. Group decision making incorporates notonly individual activity but also interpersonal andcollective aspects of behavior:

1. At the individual level, group members pro-cess information individually, concentratingonly on their own decision-making processes.

MIS Quarterly/March 1993 87


2. At the interpersonal level, they begin to learnabout the opinions of other members and takethem into account in their own decision-making processes to make an individualdecision.

3. At the collective level, the group exchangesand processes information as a collective ac-tivity in order to make a group decision.

Figure 1 depicts these levels as concentric circlesto denote that the higher levels encapsulate thelower levels of activity.

Nadler (1979) suggests that feedback can be pro-vided regarding all three levels of group decisionmaking. Information technology makes this per-spective particularly plausible. Group memberscan work individually at workstations to analyzeproblems, receive information about the decision-making processes of other group membersthrough a computer network, and then engagein an electronic meeting, using the informationgenerated individually whenever necessary.Some collaborative tools, such as Colab (Stefik,et al., 1987) and Groupware (Ellis, et al,, 1991),are designed to support such a scenario. Thisarticle takes this approach one step further. Bymodeling the decision-making process with quan-titative techniques and using computers to in-stantly generate feedback about the process,decision makers can use the feedback to affectthe outcome of the very same process.

Past research on using computers to providefeedback about decision making has concen-trated on feedback about the outcome of thedecision-making process (Tindale, 1989). In con-trast, this study concentrates on feedback aboutthe decision-making process itself. Feedbackabout the decision-making process is providedinteractively as an integral part of the individualand group processes (loosely, this is what wemean by cognitive feedback).

The objective in this paper is to conceptualizeand empirically test the effects of feedback oncognitive aspects of decision-making behavior.The paper looks at the effects of cognitive feed-back on phenomena that exist at each level ofthe decision making activity: individual, interper-sonal, and collective. The next section developsthe conceptual link between feedback and thethree levels of decision-making activity. Thesubsequent sections describe an experiment andthe experimental results, followed by a discus-sion of the results and their implications for prac-tice and future research.

Cognitive Feedback andCognitive ConflictTwo theoretical frameworks will be employed inanalyzing the effects of cognitive feedback at the

Collective

Interpersonal

Individual

Figure 1. Levels of Anaiysis (n the Context of a GDSS

88 MIS Quarterly/March 1993


individual, interpersonal, and collective levels.Social judgment theory (Hammond, et al., 1980)is suitable for understanding how individuals takeinto account the decisions of others but is not richenough to describe the dynamics of collective de-cision making. In contrast, social decisionschemes theory (Davis, 1973) is concerned withthe interaction between group members thatleads to a collective decision. Both are quan-titative frameworks that can serve as platformsfor developing hypotheses and empirical testing.

In order to clarify the ensuing discussion, con-sider a group-based procedure for personnelscreening. A group of decision makers meets toscreen candidates from a pool of applicants foran entry-level position (such sessions are fre-quently conducted by large corporations prior topersonal interviews). Before the meeting, eachmember rates the applicants on the basis of threecues: work experience, test scores, and educa-tion. Each decision maker works individually ona computer system that presents the three cuesfor each applicant and records the decisionmaker's rating. At the meeting, the group collec-tively reviews the ratings given by each groupmember and agrees on a collective rating foreach applicant.

This discussion assumes that individuals for-mulate a decision strategy before executing it(Payne, 1982). The individual's decision strategydescribes a sequence of subtasks and the nec-essary control over their execution. An exampleof a decision strategy would be to compute aweighted average of the three cues using equalweights, and then take the integer of the resultas the rating. Individual group members mayhave different decision strategies with differentweighting schemes.

We can now define cognitive feedback more con-cisely as information about the decision maker'sdecision strategy and the extent to which thestrategy is applied accurately. In contrast to out-come feedback, which describes the accuracy ofa decision, cognitive feedback provides decisionmakers with insight into their decision processes(Balzer, et al., 1989). Cognitive feedback is ef-fective in enhancing the quality of the decisionprocess by clarifying the decision maker's inten-tions and controlling their implementation (Doher-ty and Balzer, 1988). In the personnel screeningexample, cognitive feedback can take severalforms. Cognitive feedback at the individual level

may be information about relative weights as-signed to cues by a decision maker in rating agroup of applicants, the functional relations be-tween cues and the decision criterion, and theintegration rules for combining information. Cog-nitive feedback at the interpersonal level may bea measure of agreement among group memberson their individual ratings. Cognitive feedback atthe group level may be the consistency withwhich the group has been making collective deci-sions. The next two subsections relate cognitivefeedback, in turn, to the individual and interper-sonal ievels (using social judgment theory) andto the collective level (using social decisionschemes theory).

Cognitive feedback at theindividuai and interpersonal levelsGroup decisions are often characterized bycognitive conflicts, i.e., conflicts among groupmembers that exist despite similar interests. Inother words, the existence of interest differentials(or strategic motives) among players is not a nec-essary condition for conflict to exist. Cognitiveconflicts arise when group members differ in theirunderstanding of the problem (even when theirrespective interests converge) and are due tocognitive limitations (Brehmer, 1976; Hammond,1965). In the personnel screening task, groupmembers may use different strategies in ratingcandidates and, hence, disagree, but they mayfail to realize exactly how their strategies differ.Cognitive conflicts appear regularly in group deci-sion making, constituting a more persistent prob-lem than is generally realized (Baike, et al., 1973;Carroll, et al., 1988).

On the whole, cognitive conflicts should beminimized (Hammond, 1965). A better under-standing of other views is important for individualperformance (Piaget, 1954) as well as group per-formance (Hirokawa and Pace, 1983). An under-standing of other views may reveal shortcomingsin an individual's own strategy or disagreementbetween members over values or assumptions.(This is not to say that conflict in general disruptseffective decision making. Indeed, several tech-niques for injecting conflict into group processeshave been advocated. However, these tech-niques are predicated on the assumption that thegroup members are aware of the conflicting viewsand understand them.) Moreover, a lower level

M/S Quarterly/March 1993 89


of cognitive conflict will usually make it easier toattain consensus when consensus is feasible(Castore and Murnighan, 1978; Svenson, 1989).

According to social judgment theory (Hammond,et al., 1980), two constructs determine the levelof cognitive conflict experienced by a group. Atthe level of individual analysis, cognitive controlis the extent to which the decision maker con-trols the execution of his or her decision strategy.At the level of interpersonal analysis, strategyconvergence is the degree of similarity betweenthe decision strategies of the group members. Itcaptures differences in how each group memberintegrates cues in forming an overall rating, e.g.,one member may weigh work experience moreheavily than education, whereas another membermay do the reverse. Cognitive conflict is the resultof incomplete cognitive control, a lack of strategyconvergence, or both. We will review past workthat links cognitive control and strategy con-vergence to cognitive conflict and then argue thatcognitive feedback enhances cognitive controland strategy convergence and, thereby, reducescognitive conflict. The upper right corner ofFigure 2 shows the relationships among theelements of social judgment theory.

Both cognitive control and strategy convergenceare affected by cognitive limitations. Decisionmakers lack insight into their decision strategies(Hoffman, 1960; Hogarth, 1987) and are inconsis-tent in executing their intended decision strate-gies because of a limited capacity to controlexecution (Bowman, 1963; Dawes and Corrigan,1974). Even more so, individuals lack insight in-to the strategies of others. Thus, two individualsengaged in formulating a joint decision are unlike-ly to know the extent to which their respectivestrategies differ (BaIke, et al., 1973). In theabsence of interest differentials, strategy con-vergence will depend primarily on understand-ing the strategies of other group members.

Research on cognitive conflict has uncovered tworelated phenomena: (1) the level of cognitive con-flict in collective decision making is initially highand gradually declines, and (2) the structure ofthe conflict changes over time. Initially, cognitiveconflict emerges from a disagreement over deci-sion strategy. Over time, the strategy differencestend to decrease, but in the process, decisionmakers tend to lose cognitive control. Brehmer(1972) attributes this behavior to cognitive limita-tions. In order to apply the newly acquired

strategies consistently, decision makers increas-ingly have to use new cues in place of the cuesthey had used initially, but they apparently failto do so in a systematic manner. Instead, theydecrease their dependency on the original cueswithout a corresponding increase in the use ofnew cues. In doing so, decision makers lose theircognitive control and cannot apply their knowl-edge of decision strategies in a consistent man-ner. As a result, the reduction in cognitive conflictdue to converging strategies is counter-affectedby lower cognitive control. These findings havebeen replicated over a variety of task conditions,individual differences, and payoff conditions(Brehmer, 1976).

Regarding the effects of cognitive feedback (seeFigure 2), research on individual decision mak-ing has shown that cognitive feedback canincrease cognitive control (Hammond and Sum-mers, 1972) and facilitate the learning of newdecision strategies (Hogarth, 1987; Jacoby, et al.,1984). We hypothesize that these effects carryover to the group context. In order to attaincognitive control, decision makers need to knowtheir own decision strategy, to know that they arediverging from the strategy, and to know how theyare diverging. Cognitive feedback provides thisinformation and is therefore expected to enhancecognitive control and, moreover, maintain cog-nitive control uniformly over time. That is, in con-trast to an expected decrease in cognitive controlas strategies converge, we hypothesize that deci-sion makers that are provided with cognitive feed-back will maintain control while converging intheir strategies. Recall that in the absence of adecision aid. decision makers lose cognitive con-trol as they execute new strategies but regaincontrol with the repeated use of the samestrategy. These arguments are formulated ashypotheses in the context of an experimentaldesign that has two conditions (with and withoutcognitive feedback), which are measured overtime.

H1a: Cognitive feedback will increasecognitive controi.

H1b: Cognitive feedback wiii result in auniformiy iiigh ievel of cognitive controlover time.

To attain strategy convergence, the decisionmaker needs to know the other members' deci-sion strategies and to be aware of the strategy



HypothesizedRelationships

Social Judgment Theory

Social Decision SchemesTheory

Figure 2. impact of Cognitive Feedback at Different Leveis of Anaiysis

differences among members. Apparently, givenenough time, group members will learn abouteach other's viewpoint even if structured infor-mation is not provided (Brehmer, 1976). However,Jacoby, et al. (1984) stress the need for cognitivefeedback that can explain deviations in decisionsin order tc efficiently learn the appropriate deci-sion strategy for new situations. Brehmer (1979)also suggests that cognitive feedback, such asinformation abcut the relative weights for cuesand the integration rules for combining informa-tion, is typically needed to learn new decisionstrategies. Furthermore, such information isneeded to compare the new decision strategywith other decision strategies.

iH2a: Cognit ive feedback wiii increasestrategy convergence.

H2b: Strategy convergence wiii increase overtime regardiess of whether cognitivefeedback is provided.

Most of the studies mentioned above have beenconducted with a regression formulation of socialjudgment theory (cf. Balzer, et al., 1989; Ham-mond, etal., 1980). This formulation provides in-dices of decision-making activity at the individuallevel (cognitive control) and the interpersonallevel (strategy convergence). It is particularly ef-fective in modeling the information integrationphase of decision making, which is prevalent inthe personnel screening example (Einhorn, et at.,1979; Goldberg, 1970). An extended version ofthis framework is used in this study (seeAppendix).



In summary, it is hypothesized that cognitivefeedback will increase cognitive control andstrategy convergence of decision makers. It isalso hypothesized that cognitive feedback willchange their behavior over time. Furthermore, aslong as strategies do not diverge, we should ex-pect higher cognitive control to be reflected inlower cognitive conflict. The next section iden-tifies an aspect of the collective level of decisionmaking that may also be affected by cognitivefeedback.

Cognitive feedback at thecollective levelRecall the example of personnel screening. Afterthe interpersonal activity, the group meets faceto face to reach a collective rating for each jobapplicant. In contrast to the interpersonal activi-ty, the group is viewed as a level of analysis withits own laws of functioning (Von Cranach, et al.,1986). Although these laws include social andtask-related aspects of the collective activity(De Keyser, 1983), we concentrate only on task-related laws that bear on the cognitive aspectsof group decision making. Unlike the interper-sonal activity, strategy at the collective phasemust include group decision rules for combiningindividual opinions into a collective decision.

Of the several theories that build on the Idea ofgroup decision rules, social decision schemestheory is the most reievant to this discussion (seeStasser, et al., 1989, for a review of the theoryand its recent extensions). This theory assumesthat (1) groups adopt group decision rules to yieldgroup outcomes; (2) the same groups will use dif-ferent rules in different situations; and (3) differentgroups may use different rules in the same situa-tions. For example, for tasks in which memberswill recognize a correct answer when it is pro-posed, a good description is a continuous debateuntil truth "wins" unanimously. For judgmenttasks, such as the personnel screening task inwhich there is no recognizable correct answer,a voting scheme may be more appropriate(Laughlin and Adamapoulos, 1982). Social deci-sion schemes do not necessarily change indi-vidual decision strategies. Indeed, some groupdecision rules can produce group/individual dif-ferences without members changing but merelyacquiescing (Davis, 1973).

The group members' personal consequences ofhaving participated in collective decision mayresult in post-decisional conflict, i.e., a situationin which a person has committed to a decisionthat to him or her seems incorrect (Janis andfvlann, 1977). This happens when there is publicconformity without private acceptance, and it canbe distinguished from the situation of public con-formity with private acceptance by removingthe group influence and eliciting individualbehavior to see whether the compliant behaviordisappears (Festinger, 1953).

A group decision rule specifies how to combinethe members' decision outcomes into a groupoutcome (e.g., an average of the individualratings of applicants) or how to combine mem-bers' decision strategies into a group decisionstrategy that will be applied to the specific cases(e.g., a majority rule on the appropriate weightsto use in evaluating applicants). Like individualdecision-making activity, collective activity alsorequires control to ensure the correct executionof group decision rules (Johnson and Davis,1972;Savoyant, 1984). Indeed, effective groupsusually exercise some form of control (Edingerand Patterson, 1983; Hirokawa and Pace, 1983).Groups, like individuals, suffer from cognitivelimitations in exercising collective control (Chalosand Pickard, 1985; Steiner, 1972). For example,there are cognitive limitations on the task-relatedcommunication between the group members dur-ing the collective activity. In general, groupmembers will be unaware of differences in deci-sion strategies. Janis (1972) and Allison andMessick (1987) suggest that this occurs becausegroups tend to overestimate their unanimity.Stasser, et al. (1989) note that group discussionsare usually dominated by information that isshared and supportive of existing preferences,barring the discussion of unshared informationthat may reveal distinct decision strategies.Cognitive feedback that describes the members'decision strategies and the differences betweenthem should, therefore, reduce such processlosses. The lower right corner of Figure 2 showsthis relationship.

The objective here is to posit the impact ofcognitive feedback on collective control. To doso, two dimensions to characterize collective con-trol are introduced; level and degree. The levelof collective control refers to whether the groupdecision rules determine how to combine in-


Cognitive Feedback iti GDSS

dividuals' decision outcomes (denoted as a lowerlevel) or individuals' decision strategies (denotedas a higher level). The degree of collective con-trol is the extent that a group decision rule ex-ists and is executed. For higher levels ofcollective control, a high degree of collective con-trol includes control over the execution of thecombined decision strategy, which is analogousto cognitive control at the interpersonal level.

Because cognitive feedback includes informationabout decision strategies, we hypothesize thatcognitive feedback will increase the level of col-lective cognitive control by focusing the group'sdecision-making activity on decision strategies asopposed to decision outcomes (Te'eni, 1991;Vallacher and Wegner, 1987). This should resultin the formulation of group decision rules on howto combine decision strategies and how to con-trol the execution of the combined strategy.Following the rationale of Hypothesis la, cogni-tive feedback should also increase the degree ofcollective control.

H3a: Cognitive feedback will increase theievel of collective control.

H3b: Cognitive feedback wiii increase thedegree of coifectlve control.

Social decision schemes theory cannot predictpost-decisional conflict. It, does however suggestthat in situations of cognitive conflict, as opposedto interest differentials, a higher level of mutualunderstanding will lead to personal change ratherthan acquiescence, fn other words, rather thancomply with the collective decision but retain theirpersonal decision strategies, group members willchange their personal decision strategies (Davis,1973). We should therefore expect that a higherlevel of collective controi. if exercised, will resultin a lower levei of post-decisionai conflict, asshown in the iower right corner of Figure 2.

In summary, social decision schemes theory sug-gests a phenomenon (coiiective control) at thecollective level of analysis that may be affectedby cognitive feedback. We introduced two dimen-sions of coiiective controi (levei and degree) andhypothesized that cognitive feedback wouid in-crease control along both dimensions. The nextsection discusses the methodoiogy used to testthe three hypotheses presented above.

MethodThis section describes an experiment based onthe personnel screening task used to illustrateconcepts in eariier sections. Piiot tests with eightgroups (four In each condition) were conductedin order to test the software and ensure that sub-jects could foiiow the instructions. The ex-perimental procedures and software interfaceswere revised as a result of the piiot studies. Thefinai form of the experiment is described below.

Experimental designNinety graduate and undergraduate subjects (66maies and 24 femaies) participated in the experi-ment. The subjects were students majoring in in-formation systems, and they received coursecredit for the experiment. Subject to students'time constraints, they were randomly assignedto 30 groups of three members. The groups wererandomiy divided into two conditions: 15 groupsthat were provided with cognitive feedback and15 groups that were not. Other than the avaiiabiii-ty of feedback data (described iater m detail), thehuman-computer interaction in both conditionswas identical.

The experiment had three stages: training, ex-perimentai tasks, and post-experimentaldebriefing.

The purpose of the training session was to en-sure that each group member used a differentset of weights for aggregating the cues, therebycreating cognitive conflict within groups as astarting condition of the experiment (cf. Brehmer,1980). Subjects were asked to learn, through aseries of examples, the decision strategy foliowedby a hypothetical team of experts. The systemprovided each subject with a series of three-cuedescriptions of appiicants, one at a time, andasked the subject to rate each appiicant. (in ourexample, the three cues were work experience,test scores, and education.) The system then pro-vided the "actuai" rating, i.e., the rating givento that applicant by the hypothetical "commit-tee." This process was repeated for 60 triais.Committee scores given to different members ofa group were made up of different sets ofweights. For exampie, the committee scores forMember 1 of a group used weights of 0.8, 0.1,and 0.1 for the three cues, whereas scores forfvlember 3 used weights of 0.1, 0.1, and 0.8,



respectively. Subjects performed the task in-dividuaiiy, and no time limit was stipulated. Thepredictability of the task was high {H^ = 0.90).The training took between 10 and 20 minutes.

The effectiveness of the training can be evaiuatedin two ways. We can ascertain the match be-tween a subject's decision strategy and that ofthe committee (Brehmer and iHagafors, 1986).Our training produced a high matching value(0.95) and is consistent with past findings on high-predictability tasks (Brehmer and iHagafors, 1986;Hammond and Summers. 1972). The effective-ness of training can also be measured throughthe ievei of cognitive confiict in the group at theend of the training task. Our training produceda high levef of cognitive conflict (0.61), consis-tent with vaiues obtained in past studies (cf.Brehmer. 1976). Before starting on the ex-perimental tasks, subjects in the feedback con-dition were also trained on how to interpret anduse the cognitive feedback.

The experimental tasks inciuded repeated in-stances of the personnel screening task. Thisstage was organized in four blocks, each blockcontaining 10 cases, i.e., 10 applicants to berated. The use of blocks enables an analysis ofbehavior over time. There is no agreement on theoptimai number of blocks and triais within biocks(see Adeiman. 1981; Hoffman, 1960; Schmitt,et ai., 1976). The pilot study used five blocks (of10 triais each), but the sessions proved to be tooiong for the subjects, and the number ot biockswas reduced to four.

At the debriefing stage, subjects were asked toprovide information on decision strategies andthe use of feedback. Subjects described throughself-reports whether they adopted explicit de-cision strategies in the individuai and coiiectivephases of the experiment, and if they did. whatthose strategies were. Cognitive feedback sub-jects aiso provided descriptions of how they usedthe feedback provided to them. Additionaiiy, eachsubject was given 10 cases (i.e., appiicants) andwas asked to rate these cases (these ratings weretaken to ascertain the post-decisional confiict).

The three stages of the experiment followed oneafter the other, in continuous sessions that iastedfrom 50 to 90 minutes in total. The sessions wereconducted in a closed room in the presence ofat ieast one experimenter.

Experimental taskSubjects were asked to screen appiicants whoseprofiles were given as three integer evaiuationsof work experience, test scores, and education.Tasks with a simiiar structure have been used ex-tensively in research on individual and groupdecision making (e.g., Tindale, 1989; Zigurs,et ai., 1988) as weii as in research on personneiseiection (e.g., Dougherty, et ai., 1986; Lane, etai., 1983). Additionaiiy, personnei screening taskswith a simiiar structure are frequentiy used bylarge corporations and are performed in groupsettings.

Subjects worked with personal computers (iBM-compatibie) that were linked through a iocal areanetwork. Three computers were arranged in a20 X 15 room so subjects couid see oniy their ownscreens. Tabie 1 describes the experimentaitasks. At the beginning of each biock. each groupmember worked on a separate machine to rate10 applicants. The system presented each ap-plicant's profiie through three cues (each on a1-9 scale), and the subject keyed in an overaiirating of that applicant (aiso on a 1-9 scaie). Thechoice of the number of cues and the structureof the scaies is in conformance with previousstudies (e.g., Dougherty, et ai,, 1986).

Once a subject had compieted the rating processfor an applicant, the subject was provided accessto information concerning the ratings provided byother group members (and cognitive feedback forsubjects in the cognitive feedback condition).While no face-to-face communication took piaceat this stage, subjects had the opportunity to con-sider new information and adapt their own be-havior by changing ratings. The subjects couldrevise their own decisions as often as they wishedin view of updated information concerning the be-havior of others (and cognitive feedback whereapplicabie).

After aii members compieted their individualratings, group members joined together at asingie machine to formulate a coiiective rating foreach applicant. There were no restrictions oncommunication while subjects arrived at their coi-iective rating. Each group had a scribe who keyedin the ratings once the group as a whoie agreedon a vaiue. The scribe was chosen at random.An analysis of the resuits showed no differencesin performance between the scribe and othermembers.



Tabie 1. Sequence of Events Within Each Blocit of the Experimentai Tasks

At the start of each block, each individuai is given 10 cases. The individuai does thefoiiowing;

Ratingindividuai (1) Each subject rates aii 10 cases.

Subjects receive additionai information concerning the ratings provided by othergroup members (and cognitive feedback if in cognitive feedback condition).Subject revises ratings as desired.Subject is free to iterate between steps (1) and (3) untii he or she indicates thatthe ratings are finai.

(2)

(3)(4)

After aii members of a group have finaiized their ratings, the group works togetherto arrive at a group rating for the same 10 cases.

Coiiective (1) Group makes estimates on ail 10 cases.Rating (2) Groups in cognitive feedback condition receive feedback.

(3) Group revises ratings as desired.(4) Group iterates between steps (1) and (3) untii the group members indicate that

the ratings are finai.

Human-computer interactionFor subjects who did not receive cognitive feed-back, the computer screen used to complete thetask appeared as in Figure 3. While subjectsworked individualiy, the human-computerinteraction was as follows:

Subject S begins the interaction with a set ofonline instructions. After pressing any key, Ssees 10 cases (i.e.. applicants) in the bottom leftwindow of the screen. S formuiates a judgmenton each case, rating each applicant with a valueon a 1-9 scale. S can move up or down usingthe arrow keys and change ratings by typing overprevious vaiues. The bottom row always containsinstructions on what is to be done next. Whensatisfied with the 10 ratings, S presses the ENDkey. The screen is then refreshed with updatedinformation (whatever information in the remain-ing windows of Figure 3 is avaiiable at that time).S then evaiuates the information, and has theopportunity to revise any ratings. This processcan be repeated untii S presses the HOME key.which signifies that S has finaiized his or herdecisions.

in order to minimize iearning probiems. the in-terface for the support systems was the same inboth phases. The foiiowing describes how groupMember 1 performs the task, beginning with theinterpersonal phase. Concentrate on the four win-

dows on the bottom ieft side of Figure 3. in theleft-most window, the numbers 6 ,1 , and 4 on thefirst iine are an appiicant's evaiuations on workexperience, test score, and education. Member1 studies these evaluations and gives this appii-cant an overaii rating of "3 . " which is shown inthe second window. When information is avail-abie from Member 2, it is dispiayed in the thirdwindow at Member 1's request (simiiariy. Mem-ber 3's ratings are dispiayed in the fourth win-dow). Member 1 continues to rate the appiicantsand revise the ratings untii he or she is satisfiedwith the ratings. When aii the group membersfinish their individuai work, they begin the collec-tive rating. At that time, the group observes theinformation in the first four windows and inputsits ratings in the fifth window (entitied "GroupEstimate").

For subjects in the cognitive feedback condition,the screen used to complete the task appearedas in Figure 4. In comparison with Figure 3,severai forms of cognitive feedback have beenadded in the three windows in the upper part ofthe screen. The cognitive feedback, along withthe rest of the screen, was updated upon request.

Cognitive feedback was defined eariier as struc-tured information about the decision strategy andits execution, it consisted of information on deci-



WORK TEST EDUCA-EXP. SCORES TION

6

1

8

6

3

9

1

1

3

2

1

9

2

6

6

3

4

6

2

9

4

1

7

1

1

7

9

5

8

9

MEMBER r SESTIMATE

3

3

4

3

5

4

3

5

3

5

MEMBER 2-SESTIMATE

5

4

5

4

2

4

3

2

6

6

MEMBER 3'SESTIMATE

3

4

4

5

4

5

9

1

8

6

GROUPESTIMATE

3

3

5

5

2

5

3

2

8

6

Enter a number between I and 9, then press END. Press HOME if the estimates are final

Figure 3. Screen for No Feedback Subjects

sion strategies, cognitive controi, and strategyconvergence, which were computed accordingto the formula of the regression formuiationdescribed in the Appendix. To the extent possi-bie, we used graphicai displays that have beenfound to be effective for approximate com-parisons of quantitative information (Brehmer,1984).

The relative weights given to cues are siiown ashorizontai stacked bars (Figure 4, top center win-dow). The system caiculated cue weights asfollows. Beta weights were caicuiated from amuitiple regression of cue vaiues and the sub-ject's estimates. Each weight was then trans-formed to represent percentages of the sum ofsquared weights (cf. Hoffman, et a!., 1981). Forexampie, the first beta vaiue was transformed asfoiiows:

Thus, beta vaiues of 0.4.0.8, and 0.2 were shownas 0.19, 0.76. and 0.05. respectively. Trans-formed weights on the three cues were thendisplayed by the system as a horizontai stackedbar.

Cognitive control is presented through individuaiconsistency scores and an index of cognitive con-troi, i.e., the cognitive controi exercised by the deci-sion maker, The individuai consistency scores arethe ratings that wouid be given by decision makersif they were entireiy consistent in weighting thecues. The scores are dispiayed in the second col-umns of the four windows on the bottom of Figure4. For example. Member 1 rated the first and sec-ond appiicants with "3 , " and the respective con-sistency scores were " 3 " and "6." The horizontaibars, iabeied as consistency in the top rightwindow, dispiay, for each member, the index ofhis/her cognitive controi.'

The verticai bars (Figure 4. top ieft window)depict, for eacii of tiie three possibie pairs, anindex of strategy convergence, iabeied as the

' Computations of the index of cognitive control are shown inthe Appendix. The indices of matching and consistency arecalculated on a conlinuum of 0-1. A score of 0 means com-pletely inconsistent (or no match at all, as the case may bB),and a score of 1 means complete consistency Of matchingSubjects usually start off ai the lower end. The idea of thefeedback is lo move them to as high a level as possible.



Match Between

1 & 2 2 & 3 1 & 3

I1

Weights Given to Cues Consistency

0 1

Work Test Educa-Exp. Scores tion

1

Member l'sEstimate

Member 2'sEstimate

Member 3'sEstimate

GroupEstimate

Enter a number between 1 and 9, then press END. Press HOME if the estimates are final

Note; Weight for wo?-k experience

Weight for educatioti

Weight for test scores

Figure 4. Screen for Cognitive Feedback Subjects

match between two members. The index of strat-egy convergence represents the extent to whichthe decision strategies of two decision makersare similar {the computation of the index ofstrategy convergence is shown in the Appendix).

Dependent measuresTable 2 summarizes the dependent measuresused in the study. In order to test Hypothesis 1.we calculated an index of cognitive control ineach block for each subject. For Hypothesis 2,an index of strategy convergence was computed

for each group in each block. Strategy con-vergence is the average of strategy convergenceindices between the possible pair combinationsof members. Additionally, we computed the in-dex of cognitive conflict to test the social judg-ment theory predictions related to it, as shownin Figure 2. The above measures were derivedfrom estimates made in the final iteration withineach block.

For process and manipulation checks, threevariables were used to capture subjects' decisionbehavior and their use of cognitive feedback: thenumber of iterations made by each individual dur-



ing each block, the average time taken for eachiteration, and attention to feedback. The numberof iterations indicates the extent to which a sub-ject formulated and revised his or her own deci-sion strategy. The average time for an iterationis a surrogate measure for the cognitive effort re-quired in that iteration. Since the interpretationand use of feedback requires cognitive re-sources, we expect cognitive feedback subjectsto iterate more often and spend more time periteration than those not receiving cognitive feed-back. Attention to feedback was assessedthrough a short questionnaire given during thedebriefing stage. The questionnaire showed eachelement of feedback and asked the participantto identify it on a multiple choice format. Theywere then asked if they used the feedback inmaking their decisions, and in what manner.

For Hypothesis 3, the group decision rule wasdetermined from the experimenter's notes andcross-checked with self-reports by group mem-bers. The self-reports on the formulation and useof group decision rules were matched with theexperimenter's notes in the following manner.Subjects' responses were compared with the ex-perimenter's notes and aggregated into an agree-ment matrix. The proportion of agreement andthe Kappa coefficient were then compared. Theraw proportion of agreement was 0.86. The Kap-pa coefficient, which measures the proportion ofagreement obtained after separating the agree-ment attributable to chance {cf. Cohen, 1960) was0.79, with a less than 0.01 probability that the trueKappa coefficient was outside the range 0.765and 0.843. Information on group decision ruleswas separated into (a) whether groups formulated

Table 2. Measures

Test

Hypothesis 1

Hypothesis 2

Process andmanipulation checks(individual andinterpersonal levels)

Hypothesis 3

Process andmanipulation checks(collective level)

(1)

(2)

(3)

(1)

(2)

(1)

(2)

1

Measure

Index of cognitive control

Index of strategy convergence (amonggroup members)

Computer log on number of iterations ineach block and its association withcognitive control

Computer log on average time for eachiteration and its association with cognitivecontrol

Self-report on subjects" attention tocognitive feedback

Group decision rules: experimenter'sevaluation

Group decision rules; members' self-reports

Group's use of cognitive feedback:experimenter's evaluation

Group's use of cognitive feedback:members' self-reports

Activity/Levelof Analysts

Individual

Interpersonal

Individual

Individual

Individual

Collective

Collective

Collective

Collective


Cogrjitive Feedback in GDSS

an explicit decision strategy, and (b) at whichpoint of time (i.e, block) in the experiment theydid so.

The data on group decision rules were then com-bined to create a social decision schemes matrix(Tindaie. 1989). A social decision schemes matrixenables us to analyze processes through whichgroup members combine their individual deci-sions to form a group decision. For ascertaininga group's level of control, we counted the numberof instances where group members decided oncue weights first and ratings later and the numberof times when group members decided on ratingswithout explicitly considering cue weights. Agroup's degree of controi was evaluated by deter-mining when, if at all, a group decision rule wasexplicitly formed and followed. Additionally, post-decisional conflict was measured in each groupby computing the distance (the inverse of themultiple correlation) between the group decisionstrategy and that of each member as derivedfrom the ratings given during the post-experimental debriefing stage.

For the process and manipulation checksassociated with the collective level of analysis,we evaluated the extent to which groups usedcognitive feedback. The group's use of cognitivefeedback was determined from the experi-menter's notes on specific references made bygroup members about feedback, and cross-checked with self-reports of group members. Forexample, if a group member directed the group'sattention to consistency scores, the group wasconsidered to have used feedback in that block.We ascertained whether groups used cognitivefeedback and the blocks in which they used suchfeedback.

Results

Cognitive feedback at theindividual and interpersonal levelsFigure 5 depicts the cognitive control of in-dividuals at the end of training and during the in-terpersonal phase. As Table 3a shows, subjectsin the cognitive feedback condition exhibitedhigher cognitive control than subjects in the nofeedback condition, across all blocks. Table 3bsummarizes the MANOVA, which shows that theoverall cognitive controi of subjects was

significantly higher in the feedback condition(p < 0.001). Hypothesis la is thus supported.Furthermore, the possibility that individualcognitive controi was a function of membershipin a particular group is ruled out.

Figure 5 also indicates that individuals in thecognitive feedback condition reached a high de-gree of cognitive control at the end of the firstblock (mean = 0.92, block 1) and maintained itover the remaining blocks (mean = 0,92, block4). Subjects in the no feedback condition, on theother hand, appeared to have lost cognitive con-trol in the second block and then recovered itgradually. The trend analysis in Tabie 3c indi-cates a quadratic trend. Table 3b shows a signifi-cant btock'feedback interaction (p < 0.01),thereby suggesting that the quadratic trend isprimarily present in the no feedback condition.Thus, the analysis confirms that while cognitivecontrol in the cognitive feedback condition wasuniformly high, control in the no-feedback con-dition decreased initially and then increasedgradually. Hypothesis 1b is thus supported.

Figure 6 and Table 4a, which show the meansof strategy convergence, indicate that cognitivefeedback groups had higher strategy conver-gence than the no feedback groups. However,as the MANOVA in Table 4b indicates, the dif-ference was not significant. Thus, Hypothesis 2ais not supported by the data. Hypothesis 2bpredicts that strategy convergence increasesover time. The MANOVA in Table 4b shows asignificant within-subjects effect (p < 0.001),confirming our prediction.

Figure 2 posits an association between cognitiveconflict and cognitive control, and between cog-nitive conflict and strategy convergence, respec-tively. The analysis shows a high negativecorrelation between cognitive control andcognitive conflict ( -0 .41 , p < 0.001), therebyconfirming that high cognitive control is associ-ated with low cognitive conflict. The correlationbetween strategy convergence and cognitiveconflict is also negative {-0.39, p < 0.01), in-dicating that high strategy convergence isassociated with low cognitive conflict, as shownin Figure 2.

Overall, the process and manipulation checks in-dicate that individuals used the cognitive feed-back. Computer logs show that subjects in thecognitive feedback condition iterated through


Cognitive Feedback it) GDSS

coO

COl

oo

1.00

0.90

0.80

0.70

0.60

Training 2Block

•- Cognitive Feedback+ No Feedback

Figure 5. Cognitive Controi of individuals

their decisions more often than others. Theymade, on the average, 3.5 iterations per blockin their decision processes, in comparison withan average of 1.9 iterations for subjects notreceiving cognitive feedback {F[1,88] = 37.49,p < 0.001). Within the cognitive feedback con-dition, a comparison of the iteration patterns ofthe five best subjects in terms of cognitive con-trol scores with the five worst subjects revealsthat subjects with the best scores iterated, onaverage, significantly more often than those withthe worst scores (p < 0.05). The correspondingdifference in the no feedback condition is notsignificant. Thus, cognitive feedback is associ-ated with more iterations, and more iterations areassociated with higher cognitive control.

The logs also show that subjects receivingcognitive feedback spent more time adaptingtheir decisions, an average of 176 seconds foreach iteration, compared with an average of 136

seconds in the no feedback condition(F[1,88I = 21.19, p < 0.001). A test of com-parison {like the one stated above) shows asignificant difference {p < 0.05) between thebest and worst subjects in the cognitive feedbackcondition, but none in the no feedback condition.Thus, there is a clear association between timeper iteration and cognitive control when cognitivefeedback is provided.

Self-reports indicate that the subjects attendedto and used the feedback. In answering the ques-tionnaire on the use of feedback, most subjects(40 out of 45) indicated that they used the feed-back provided. Their responses indicate thatsuch feedback was used in formulating and revis-ing decision strategies and ensuring cognitivecontrol. In summary, the iteration measures andself-reports indicate that the feedback providedwas used by subjects to enhance their decisionmaking.



Tabie 3. Effect of Cognitive Feedback on Subjects' Cognitive Control

3a. Means (and Standard Deviations)

Cognitive Feedback

No Cognitive Feedback

N

46

45

Biock 1

0.92(0.06)

0.81(0.07)

Biock 2

0.91(0.08)

0.70(0.05)

Biock 3

0.91{0.07)

0.76(0.10)

Biock 4

0,94(0.06)

0.85(0.06)

3b. iMuitivarjate Analysis of Variance forDependent Variabie: Cognitive Controi (n = 90)

Source ofVariation

Between SubjectsCognitive FeedbackCognitive Feedback{Group)Subjects Within Cells

Within SubjectsBlockBlock'Cognltive FeedbackBlock'Cognitive Feedback(Group)

SS

1.8940.7781.356

0.6050.7990.279

Degrees ofFreedom

12860

3, 583, 58

84, 174

F-Vaiue

83.801.23

12.614.841.11

p

0.00010.2485

0.00010.00450.2872

3c. Analysis of Variance for Tests on Trends forDependent Variabie: Cognitive Controi

Trend

Linear

Quadratic

Source ofVariation

MeanCognitive FeedbackCognitive Feedback(Group)Subjects Within Cells

MeanCognitive FeedbackCognitive Feedback(Group)Subjects Within Cells

SS

0.0550.0150.3470.614

0.3490.1580.2930.700

Degrees ofFreedom

11

2860

11

2860

F-Vaiue

5.391.441.21

29.9013.510.90

p

0.02370.23510.2614

0.00010.00050.6153



1.00

0)ocajI—

>

CoO>̂o

• t—'

to

0.90

0.80

0.70

0.60

Training 2

Block

-m- Cognitive Feedback+ No Feedback

Figure 6. Interpersonal Strategy Convergence

Cognitive feedback at thecoiiective ievel

Hypothesis 3 predicts the effect of feedback ongroup decision rules. A log-linear analysis of thedata shows that the proportion of groups that firstconsidered decision strategies (and then derivedratings after agreeing on cue weights) was sig-nificantly higher in the cognitive feedback con-dition (G-Square = 27.31; p < 0.04). In sharpcontrast, the proportion of groups that consideredratings without any explicit formulation of deci-sion strategies was significantly higher in the nofeedback condition (G-Square = 31.27;p < 0.01). Thus, Hypothesis 3a is supported bythe evidence.

Tables 5a and 5b provide information on thedegree of group control. Table 5a shows whethergroups formulated an explicit decision strategyto guide their decisions. Significantly moregroups in the feedback condition formulateda decision strategy (Chi-Square(i) = 3.59,p < 0.059). As Table 5b indicates, feedbackgroups also managed to arrive at a strategy muchearlier in the decision process (Chi-Square(2)= 5.664, p < 0.059). Hypothesis 3b is support-ed. Moreover, as Table 6 indicates, groups in thefeedback condition actually used feedback, con-firming that the experimental manipulation wassuccessful.

Furthermore, there was a strong negative correla-tion between the level of collective control and



Table 4. Effect of Cognitive Feedback on Strategy Convergence

4a. Means (and Standard Deviations)

Cognitive Feedback


N

16

15

Biock 1

0.70(0.04)

0.67(0.05)

Biock 2

0.80(0.05)

0.74(0.04)

Biock 3

0.89(0.05)

0.81(0.06)

Biock 4

0.95(0.04)

0.91(0.07)

4b. iMuitivariate Anaiysis of Variance forDependent Variabie: Strategy Convergence (n = 30)

Source ofVariation SS

Degrees ofFreedom F-Vaiue

Between SubjectsCognitive FeedbackSubjects Within Cells

Within SubjectsBlockBlock'Cognitive Feedback

0.0921.581

0.3880.990

128

3, 263, 26

1.64

13.680.08

0.2113

0.00010.9682

post-decisional conflict ( - 0.45) and between thedegree of collective control and post-decisionalconflict ( - 0.40). These results are consistent withthe relationship predicated in Figure 2 betweencollective control and post-decisional conflict.

Discussion and ConclusionThe objective of this paper was to study the im-pact of computer-based cognitive feedback onthe individual, interpersonal, and collective leveisof decision-making activity. Building on socialjudgment theory (Hammond, et al., 1980), it ex-plains how cognitive feedback enhances an in-dividual's cognitive control and facilitates strategyconvergence between group members. Bothtypes of impact reduce cognitive conflict. Usingthe social decision schemes theory (Davis, 1973),the paper also examines how cognitive feedbackenhances a group's control over group decisionrules. These relationships were then tested in alaboratory experiment.

Summary of findingsGroups of three worked on a personnel screen-ing task. Group members rated the applicants in-dividually and then reached a unified grouprating. The results of the experiment show thatthe impact of cognitive feedback on cognitivecontrol was statistically significant. Furthermore,cognitive feedback was hypothesized to maintaina high level of cognitive control over time, despitethe adoption of new decision strategies in the pro-cess of strategy convergence. This was also con-firmed. The impact of cognitive feedback onstrategy convergence, although positive, was notstatistically significant. The predicted negative as-sociations between cognitive control and cogni-tive conflict and between strategy convergenceand cognitive conflict were supported.

At the collective level, it was hypothesized thatcognitive feedback would increase the degreeand level of collective control. As predicted,groups that received cognitive feedback for-



mulated group decision rules more frequentlythan groups that did not, and this difference wasstatistically significant. Moreover, the decisionrules that cognitive feedback groups formulatedspecified the creation of collective decisionstrategies, as opposed to the combination of deci-sion outcomes (ratings), which was more typicalof the groups that did not receive cognitive feed-back. Furthermore, the predicted negative asso-

ciation between collective control andpost-decisional conflict was supported.

Limitations of the resultsThe generatizability of the results is constrainedprimarily by the nature of the task and subjects.The personnel screening task was relativelystructured; the relevant information was pre-

Tabie 5. Effect of Cognitive Feedback on Coiiective Controi

5a. Groups' Formuiation of Decision Ruie

Cognitive Feedback


Decision

Yes

12

7

Strategy Formulated?No

3

9

Note: This table indicates the number of groups that formulated a decision strategy in the course ofthe experiment. Thus, 12 groups in the cognitive feedback condition formulated a decision strategy.Chi-Square = 3.59; d.f. = 1; p = 0.058.

5b. Time of inttiai Formuiation of Decision Ruie

Biocic 1 Biocii 2 Block 3 Biocit 4

Cognitive Feedback


Note: This table indicates the block in which a group first formulated its decision strategy. Thus, ninegroups in the cognitive feedback condition formulated their initial decision strategy in block 1,Chi-Square = 5.664; d.f. = 2; p = 0.059.

Tabie 6. Use of Cognitive Feedback by Groups

Cognitive

Cognitive

Feedback

Feedback

Used

Not Used

Biock 1

14

1

Block 2

12

3

Biock 3

12

3

Biock 4

13

2

Note: This table indicates the number of groups that used cognitive feedback in a particular block. Thus,14 groups used cognitive feedback during Block 1.



sented in the same format and the same strategycould be applied to new cases. Many such tasksexist and are supported by computerized sys-tems, e.g., student screening, loan evaluation,investment portfolio, and technical ratings of ven-dor proposals for government contracts. Pro-viding timely information on the process ofdecision making shouid be helpful in unstruc-tured tasks as well. This may be difficult to in-vestigate empirically, but on the basis of ourpreliminary findings, it may prove to be a fruitfulchallenge.

The second limitation was the sample population.The subjects were students with no personalstakes in the outcome of their deliberations. Theanalysis of the post-experimental debriefing,together with the experimenters' observations,indicated that the subjects were interested in theexperiment and devoted the time and effort need-ed for serious deliberations. Nevertheless,replication with "real" and experienced stakeholders is clearly needed.

impiications for the designof systemsHow can these findings be translated to practicaluse? Figure 1 depicts three leveis of informationprocessing found in many situations of groupdecision making. We believe that our discussionpoints to two guidelines for the design and im-plementation of GDSS: (1) include cognitive feed-back as an integral part of the GDSS at everylevel; and (2) design the human-computer interac-tion so there is an intuitive and effective transi-tion across the components of feedback at alllevels. To date, there is no formal methodologyfor developing GDSSs, but we believe that theseguidelines should be part of any futuremethodology.

This approach has been used successfully in thedevelopment of SPIDER, a computer-basedsystem for supporting business planning that isdistributed across organizational functions(Boland, et al., forthcoming). At the individuallevel, the user engages in a private, self-refiectiveplanning process that concentrates on ananalysis based on the viewpoint of one particularorganizational function. At the interpersonal level,users engage in dialogue to incorporate view-points of other functions into their own thinkingprocesses. It wouid have been a mistake to

assume that the interpersonal level can be sup-ported by simply adding a mailing facility to theindividual level system. A dedicated task analysisat the interpersonal level suggested a set oftechniques that help assimilate new viewpointsinto the old ones, techniques that had no placeat the individual level. One such technique is acomparison between old and new cognitivemaps. Conversely, typical GDSSs cannot be ex-pected to support work at the individual level bysimply adapting GDSS facilities to individualdecision support.

It is not enough, however, to support each levelof information processing separately by providingappropriate modes of human-computer interac-tion. The different modes must be designed con-currently to facilitate an easy and effectivetransition between interaction modes. One can-not build these modes in isolation and then hopefor a smooth integration. Having satisfied the in-formation requirements for each level, it is impor-tant to strive for consistency across levels withoutsacrificing functionality. Such consistency, bothin terms of the format and the content of infor-mation, facilitates an easier transition betweenlevels. This approach to system design is com-monplace whenever the same user is expectedto work in several interaction modes (Shneider-man, 1992). This study has shown that this ap-proach is especially important in the transitionbetween the individual, interpersonal, and collec-tive levels because of the loss of cognitive controlthat the transition causes.

This study has shown that feedback about thedecision-making process is an effective tech-nique for helping the user regain control duringthe transition between levels. It is expected thatless structured tasks will require even more feed-back to balance the loss of control (Te'eni, 1992).Designers should look for additional techniquesto facilitate an easier transition, such as tech-niques for translating the new information fromother collaborators Into familiar structures. Arelatively easy example is an automatic transla-tion of formats, say, from tables to graphs. Thiswould make it possible to convert the incomingmessages (at the interpersonal level) into thesame format used by the receiver at the individuallevel. More complex translations of the contentof incoming messages that make the messagemore comparable to the receiver's existingknowledge may also be helpful in facilitating an



easier transition. Advances in data modeling aremaking such translations increasingly feasible.

Directions for future researchThis study demonstrates that computer-generated feedback is an effective decision aidfor dealing with cognitive conflict tasks. This con-clusion is in line with DeSanctis and Gallupe's(1987) anaiysis of the possible roles of GDSS ina model of group tasks based on McGrath (1984).The model classifies group tasks according to thegroup's purpose in decision-retated meetings:generating ideas, choosing alternatives, andnegotiating solutions. In relating our study toMcGrath's model, cognitive conflict tasks con-stitute a sub-class of negotiation tasks. It is im-portant to limit our conclusions to the negotiatingaspect of group decision making. Indeed, to theextent that control inhibits creativity, providingcognitive feedback too early in the phase ofgenerating ideas may actually be counter-productive (Hogarth, 1982). Moreover, becausethis study adopted a linear model view of deci-sion making, the conciusions should be limitedto tasks where a similar perspective is ap-propriate. Future research should attempt to ex-tend these concepts to other models of conflictthat deal with ill-structured decisions (Hogarth,1987; Janis and Mann, 1977).

This study also suggests the need to carefully ex-amine the impact of cognitive feedback over time.As Figure 5 indicates, subjects not receivingcognitive feedback gradually recovered theircognitive control over time (albeit to a level lowerthan those that received cognitive feedback).Also, a few groups stopped using cognitive feed-back after the first block (Table 6). This is perhapsindicative of over-confidence, a bias commonlyobserved in situations involving repetitive deci-sions (Hogarth, 1987). Thus, the results highlighta design dilemma of whether to try to correctusers' biases by directing their attention to feed-back over time. Future research should focus onthe effect of time in determining the relativeusefulness of different forms of feedback.

From a methodological perspective, this studydemonstrates the utility of a multi-level frameworkfor studying the use of GDSS (Figure 2). It facili-tates the investigation of behavior at multipleleveis, thereby bringing us closer to a realisticunderstanding of the behavior of GDSS users.

In addition, this study has made an attempt atstudying the process of computer-aided decisionmaking. Future studies may benefit by adoptinga more intensive method of process tracing, suchas think aloud protocols, especially for more com-plex models of decision making where it is dif-ficult to infer decision strategies from outcomes.Protocol analyses may also reveal how and whythe use of feedback changes over time.

More research is also needed to understand howthe technology should be designed so as tocapitalize on the advantages of both interper-sonal decision making and collective decisionmaking (Te'eni, 1992). In particular, futureresearch should examine the unresolved ques-tions of the timing and format of feedback. Im-mediate feedback, while more effective thandelayed feedback, also imposes greater cognitiveload on decision makers (Jacoby, et al.. 1984).Thus, the trade-offs between effort and accuracyentailed in presenting immediate versus delayedfeedback need to be understood. Additionally,because the format of presentation of informa-tion affects decision behavior (Payne, 1982),future research should investigate the differen-tial effectiveness of multiple formats of presen-tation of feedback. Guidelines are also neededon general GDSS design issues, such as the fre-quency of information refreshes, the exchangeof interpersonal information (cf. Ellis, et al., 1991),and the manner in which cognitive feedbackmechanisms should be incorporated within theoveraii architecture of GDSS- We hope thesequestions will trigger future research.

AcknowledgementsBoth authors contributed equally to this article.The authors wish to thank Suzanne Rodriguezand William Haga for their comments on anearlier version of the paper.

References

Adeiman. L. "The Influence of Formal, Substan-tive and Contextual Task Properties on theRelative Effectiveness of Different Forms ofFeedback in Multiple-Cue Probability Learn-ing Tasks.'" Qrganizational Behavior andHuman Performance (27:3). June 1981, pp.423-442.

Allison, S. and Messick, D. "From Individual In-



puts to Group Outputs, and Back Again:Group Processes and Inferences about GroupMembers." in Review of Personality andSocial Psychology, Vol. 8, C. Hendrick (ed.),Sage Publications, Beverly Hills, CA, 1987,pp. 59-62.

BaIke, W., Hammond, K., and Meyer, G. "AnAlternate Approach to Labor-ManagementRelations," Administrative Science Quarterly(18:3), September 1973, pp. 311-327.

Balzer, W., Doherty, M., and O'Connor, R."Effects of Cognitive Feedback on Perfor-mance," Psychological Bulletin (106:3),November 1989, pp. 410-433.

Boland, R., Tenkasi, R., and Te'eni, D. "Design-,ing Information Technology to SupportDistributed Cognition," Organization Science,forthcoming.

Bowman. E. "Consistency and Optimality inManagerial Decision Making," ManagementScience (9:2), January 1963, pp. 310-321,

Brehmer, B. "Policy Conflict as a Function ofPolicy Similarity and Policy Complexity,"Scandinavian Journal of Psychology (13),1972, pp. 208-221.

Brehmer, B. "Social Judgment Theory and theAnalysts of Interpersonal Conflict," Psycho-logical Bulletin (83:6), November 1976, pp.985-1003.

Brehmer, B. "Preliminaries to a Psychology ofInference," Scandinavian Journal ofPsychology (20:4), 1979, pp. 193-210.

Brehmer, B. "In One Word: Not from Ex-perience," >̂ c(a Psychologica (45), 1980, pp.223-241.

Brehmer, B, "The Role of Judgment in Small-Group Conflict and Decision Making." in Pro-gress in Applied Social Psychology, Vol. 2,G.M. Stephenson (ed.), Wiley, New York, NY,1984, pp. 163-184.

Brehmer B. and Hagafors, R. "Use of Expertsin Complex Decision Making: A Paradigm forthe Study of Staff Work," OrganizationalBehavior and Human Decision Processes(38:2), October 1986, pp. 181-195.

Carroll, J., Bazerman, M., and Maury, R."Negotiator Cognitions: A Descriptive Ap-proach to Negotiators' Understanding of theirOpponents," Organizational Behavior andHuman Decision Processes (41:3), June 1988,pp. 352-370.

Castore, C. and Murnighan, J. "Determinants ofSupport for Group Decisions," Organizational

Behavior and Human Decision Processes(22:1), August 1978, pp. 75-92.

Chaios, P. and Pickard, S. "Information Choiceand Cue Use: An Experiment in Group Infor-mation Processing." Journal of AppliedPsychology (70:4), November 1985, pp.634-641.

Cohen. J. "A Coefficient of Agreement forNominal Scales," Educational andPsychological Measurement (20:^), 1960, pp.37-46.

Cvetkovich, G. "Small Group Dynamics in Ex-tended Judgment Situations," in HumanJudgment and Social tnteraction, L. Rap-poport and D. Summers (eds.). Holt, Reinhartand Winston, New York, NY, 1973, pp.315-335.

Davis, J. "Group Decision and Social Interaction:A Theory of Social Decision Schemes,"Psychological Review {80:2), March 1973, pp.97-125.

Dawes, R. and Corrigan, B. "Linear Models inDecision Making," Psychological Bulletin(81:2), February 1974, pp. 95-106.

De Keyser, V. "Communications Sociales etChange Mentale dans les PostesAutomatises," Psychologie Franpaise (28),1983, pp. 239-253.

DeSanctis, G. and Gallupe, R. "A Foundation forthe Study of Group Decision SupportSystems," Management Sc/ence (33:5), May1987, pp. 589-609.

Doherty, M. and Balzer, W. "Cognitive Feed-back," in Human Judgment: The SJT View,B. Brehmer and C. Joyce (eds.). North-Holland, Amsterdam, 1988, pp. 163-197.

Dougherty, T., Ebert, R., and Callender, J."Policy Capturing in the Employment Inter-view," Journal of Applied Psychology (71:1),February 1986, pp. 9-15.

Dudycha, L. and Naylor, J. "Characteristics ofthe Human Inference Process in ComplexChoice Behavior Situations," OrganizationatBehavior and Human Performance (1:1),1966, pp. 110-128.

Edinger, J. and Patterson, M. "Nonverbal In-volvement and Social Control," PsychologicalBulletin (93:1), January 1983, pp. 30-56.

Einhorn, H., Kleinmuntz, D., and Kleinmuntz, B."Linear Regression and Process-Tracing

Models of Judgment," Psychological Review(86:5), September 1979, pp. 465-485.

Ellis, C . Gibbs, S.J., and Rein, G. "Groupware:



Some Issues and Experiences," Communica-tions of the AOf^ (34:1), January 1991, pp.38-58.

Festinger, L. "An Analysis of CompliantBehavior," in Group Relations at theCrossroads. M. Sherif (ed.), Harper andBrothers. New York, NY, 1953.

Goldberg, L. "Man Versus Model of Man: A Ra-tionale Plus Some Evidence for a Method ofImproving on Clinical Inferences,"Psychological Bulletin (73:6), June 1970, pp.422-432.

Hammond, K. "New Directions in Research onConflict Resolution," Journa/of Soc/a//ssL/es(21), 1965, pp. 44-66.

Hammond. K. and Summers, D.A. "CognitiveControl," Psychological Review (79:1),January 1972, pp. 58-67.

Hammond, K., McCleland, G., and Mumpower,J. Human Judgment and Decision Making:Theories. Methods and Procedures. Praeger,New York, NY, 1980.

Hirokawa, R. and Pace, R. "A Descriptive In-vestigation of the Possible Communications-Based Reasons for Effective and IneffectiveGroup Decision Making," CommunicationMonographs (50), 1983, pp, 363-379.

Hoffman, P. "The Paramorphic Representationof Clinical Judgment," Psychological Bulletin(57:2), March 1960, pp. 116-131.

Hoffman, P., Earle, T., and Slovic, P."Multidimensional Functional Learning (MFL)and Some New Conceptions of Feedback."Organizational Behavior and Human Perfor-mance (27:1), February 1981, pp. 75-102.

Hogarth, R. "On the Surprise and Delight of In-consistent Responses," in Ouestion Framingand Response Consistency, R. Hogarth (ed.),Josey Bass, San Francisco, CA, 1982.

Hogarth, R. Judgment and Choice, 2nd edition,John Wiley and Sons, Chichester, England,1987.

Hursch, C.J., Hammond, K., and Hursch, J."Some Methodological Considerations inMultiple-Cue Probability Studies," Psycho-logical Review (71:1), January 1964, pp.42-60.

Jacoby, J., Mazursky, D., Troutman, T., andKuss, A. "When Feedback is Ignored: Disutili-ty of Outcome feedback," Journal of AppliedPsychology (69:3), August 1984, pp. 531- 545.

Janis., I. Victims of Groupthink, Houghton Mif-flin, Boston, MA, 1972.

Janis, I. and Mann, L. Decision Making, FreePress. New York, NY. 1977.

Johnson, C. and Davis, J. "An EquiprobabilityModel of Risk Taking," OrganizationalBehavior and Human Performance (8:1).August 1972, pp. 159-175.

Kleinmuntz. D. "Cognitive Heuristics and Feed-back in a Dynamic Decision Environment,"Management Science (31:6), June 1985, pp.680-702.

Lane, D., Murphy, R., and Marques, T. "Measur-ing the Importance of Cues in Policy Captur-ing," Organizational Behavior and HumanPerformance (30:2), October 1982, pp.231-240.

Laughlin, P. and Adamapoulos, J. "Social Deci-sion Schemes in Intellective Tasks," in GroupDecision Making, H. Brandstatter, J. Davis,and G. Stocker-Kreichgauer (eds.). AcademicPress, London, 1982.

McGrath, J. Groups: Interaction and Perfor-mance. Prentice-Hall, Englewood Cliffs, NJ,1984.

Nadler, D.A. "The Effects of Feedback on TaskGroup Behavior: A Review of the Experimen-tal Research," Organizational Behavior andHuman Performance (23:3). June 1979, pp.309-338.

Payne, J. "Contingent Decision Behavior,"Psychological Bulletin (92:2), September1982, pp. 382-402.

Piaget, J. The Construction of Reality in the Child,Basic Books, New York, NY, 1954.

Savoyant, A. "Definition et Voies d'Analyse deI'Activit^ Collective des Equipes de Travail,"Cahiers de Psychologie Cognitive (22:1-2),1984, pp. 17-28.

Schmitt, N., Coyle, B., and King, L. "Feedbackand Task Predictability as Determinants ofPerformance in Multiple-Cue ProbabilityLearnng Tasks," Organizational Behavior andHuman Performance (16:2), August 1976, pp.388-402.

Shneiderman, B. Designing the User Interface.2nd edition, Addison Wesley, Reading, MA,1992.

Stasser, G., Kerr, N.L., and Davis, J. "InfluenceProcesses and Consensus Models in DecisionMaking Groups," \n Psychology of Group In-fluence. 2nd edition, P. Paulus (ed.),Lawrence Eribaum Associates, Hillsdale, NJ.1989, pp. 279-322.

Stefik, M., Foster, G., Bobrow. D.G., Kahn, K..


Cognitive Feedback In GDSS

Lanning, S.. and Suchman, L. "Beyond theChalkboard: Computer Support for Collabora-tion and Problem-Solving in Meetings," Com-munications of the ACM (30:1), January 1987,pp. 32-47.

Steiner, J.D. Group Process and Productivity,Academic Press, New York, NY, 1972.

Svenson, 0 . "Illustrating Verbai ProtocolAnalysis: Individual Decisions and DialoguesPreceding a Joint Decision," in Process andStructure in Human Decision Making, H.Montgomery and O. Svenson (eds.). JohnWiley and Sons, Chichester. England, 1989,pp. 83-90.

Te'eni. D. "Feedback in DSS: Experiments Withthe Timing of Feedback," Decision Sciences(22:3), July/August 1991, pp. 597-609.

Te'eni, D. "Analysis and Design of Process Feed-back in Information Systems," Accounting,Management and Information Technology(2:1), January-March 1992, pp. 1-18.

Tindale, R.S. "Group vs Individual InformationProcessing: The Effects of Outcome Feed-back on Decision Making," OrganizationalBehavior and Human Decision Processes(44:3), December 1989, pp. 454-473.

Tucker. LR. "A Suggested Alternative Formula-tion in the Developments by Hursch, Ham-mond, and Hursch, and by Hammond,Hursch, and Todd," Psychological Review(71:6), November 1964, pp. 528-530.

Vallacher, R. and Wenger, D. "What Do PeopleThink They're Doing?" Psychological Review(94:1), January 1987, pp. 3-15.

Von Cranach, M., Ochsenbein. G., and Valach,L. "The Group as a Self-Active System:Outline of a Theory of Group Action," Euro-pean Journal of Social Psychology (16), 1986,

pp. 193-229.Zigurs, I,, Poole, M., and DeSanctis, G. "A Study

of Influence in Computer Mediated DecisionMaking," MIS Quarterly (12:4), December1988, pp. 645-666.

About the AuthorsKishore Sengupta is assistant professor of in-formation systems at the Naval PostgraduateSchool in Monterey, California. He received hisPh.D. in management information and decisionsystems from Case Western Reserve Universi-ty. Dr. Sengupta's research interests are in de-cision support for dynamic tasks, learning andintelligent tutoring, and computer supportedcooperative work. His research has appeared inIEEE Transactions on Software Engineering.Management Science, and other journals. He iscurrently working on a book on cognitive aspectsof decision support for disfributed decisionenvironments.

Dov Te'eni is associate professor of manage-ment information systems at Case Western Re-serve University. He received his Ph.D. inmanagerial studies from Tel Aviv University. Dr.Te'eni's current research interests include thedesign of decision support systems, human-computer interaction, and behavioral aspects ofinformation systems. He has published inBehaviour and tnformation Technology, DecisionSciences, IEEE Transactions on Systems, Manand Cybernetics. International Journal of Man-Machine Studies, Omega. Organization Science,and others. He is currently writing a book onhuman-computer interaction for decision making.

MIS Ouarterly/March 1993 109


Appendix

Social Judgment Theory—A Regression FormulationThis appendix provides a regression formulation of group decision making based on social judgmenttheory.^ We begin with a two-system view of the theory and extend it to capture interpersonal decisionmaking in tasks where there are no optimal solutions.

A Two-System ViewFigure Al presents a regression formulation of the social judgment theory (Dudycha and Naylor, 1966).The formulation has three components: the task structure, the decision maker, and the environment.The task structure is constituted by a set of cues (Xp X2..... X^). In the personnel selection task describedearlier, the cues constituting the task are: work experience, test scores, and education. The decisionmaker's decision, V^, is defined in the right hand side of the model. The consistency score for eachdecision indicates the score that would be given by a decision maker if the person had been entireiyconsistent with his or her decision strategy. Consistency scores are calculated by computing beta weightsfrom a multiple regression of cue values and the decision maker's estimates, and then multiplying thebeta weights with the cue values. Thus, we get:

and

Yg = Yg + Zg. where

Yg denotes the consistency score for a decision Yg, and Zg is the residual.

The cognitive control exercised by a decision maker is measured by the index of cognitive control (Rg),and is operationalized as the multiple correlation between the cues and judgments (Hursch, et al., 1964).From the propositions of multiple regression theory (Tucker. 1964), we get:

variance (Yg) = Rg^ and

variance (Zg) = 1 - Rg^.

Analogously, the left hand side of the model in Figure Al denotes the environment (represented by theactual event, Vg). The extent to which the actual event is similar to the model of the environment isknown as task predictability (Rg), similarly operationalized as the multiple correlation between the cuesand actual events. The predictability of a task depends on the nature of the task. Some tasks (suchas credit risk of a loan applicant) are more predictable than others (such as stock prices).

The relationship between the decision maker and the task is captured through two measures: accuracyand knowledge. A decision maker's performance in a task is summarized through the accuracy index(r^), which indicates the correspondence between the subject's response and the environmental event.Tucker (1964) has shown that:

r- = Covariance lYaYc) = Covariance (Y^YJi + Covariance

formuiations descritwd here are derived from Hammond and Summers (1972), i-iufsch, et ai. (1964). and Tucker (1964).



/

ActualEvent

EnvironmentalPredictability

PredictedEvent

\

/

-——-

•"1

Accuracy

M

Cue Set

—

\ ,\

X2

X3

Xn

\

- — •

—

\ KnowledgeIndex

G

Individual'sDecision

CognitiveControl

Individual'sPredictedDecision

Figure A l . Lens Model: A Two-System View



Now, if the correlation of (YgYg) is denoted by G, and the correlation of (Z^Zg) is denoted by C, thenthe above can be written as:

fg = GRgRg + C[(1- fl/) (1 - Rg^)] (1)

G is known as the knowledge index and represents the similarity of the decision maker's use of cueswith the environmental relationships. The knowledge index is interpreted as the extent to which a modelof the decision maker's strategy is similar to that of the model of the environment. C is the correlationbetween the variance in the task system and the subject's judgmental system that is unaccounted forby G. When the systematic variance in the criterion can be accounted for by a linear function of thecue values, then the contribution of the second term on the right-hand side of Equation 1 is negligible(Hammond and Summers, 1972). Thus, Equation 1 can now be written as:

where:

r̂ , or accuracy, is the correlation between the judgment and the actual event;

G, or knowledge, is the correlation between the linearly predictable variance in the judgment andthe event;

Rg, or cognitive controt, is the multiple correlation between the cues and judgments; and

RQ, or task predictability, is the multiple correlation between the cues and actual events.

An Extension of the Two-System View to Capture InterpersonalInteractionThe two-system view can be modified to capture interaction between multiple decision makers in groupsettings (Brehmer, 1979). We use the extended view to operationalize cognitive conflict and strategyconvergence. To do this, we substitute the environment on the left hand side with another decision maker(illustrated in Figure A2). With this substitution. Equation 2 can be written as:

where

7^. or cognitive conflict, is(1 - the correlation between the judgments made by persons? and thosemade by person S2).G, or strategy convergence, is the correlation between the linearly predictable variance in S1's judg-ment and S2's judgment.

and RQ2 (cognitive control of the respective decision makers) are the multiple correlationsbetween the cues and judgments made by S1 and S2, respectively.

The index of strategy convergence is interpreted as the extent to which the decision strategies of twodecision makers are similar. Equation 3 can be extended to represent situations of three or more deci-sion makers (Cvetkovich, 1973).



• /

CognitiveConflict

\

/ Cue Set N.

/

Subject 2's

S2

CognitiveControl ofSubject 2

Subject 2'sPredictedDecision

\

--•

Xi

Xp

\

^—-^

\

^3

Xn

^ - "

\ StrategyConvergence

G

y

\Subject 1's

Decision

CognitiveControl ofSubject 1

Subject 1'sPredictedDecision

Figure A2. Lens Model: A Modified View

MIS Ouarterly/March 1993 113

Cognitive Feedback in GDSS: Improving Control and Convergence · (GDSS), little research has...

Documents

Transcript of Cognitive Feedback in GDSS: Improving Control and Convergence · (GDSS), little research has...