The role of prior experience and task characteristics in object-oriented modeling: an empirical...

Int . J . Human – Computer Studies (1996) 45 , 639 – 667

The role of prior experience and task characteristics in object-oriented modeling : an empirical study

R ITU A GARWAL AND A TISH P . S INHA

Uni y ersity of Dayton , Dayton , OH 4 5 4 6 9 - 2 1 3 0 , USA

M OHAN T ANNIRU

Syracuse Uni y ersity , Syracuse , NY 1 3 2 4 4 – 2 1 3 0 , USA

( Recei y ed 2 9 No y ember 1 9 9 5 and accepted in re y ised form 8 July 1 9 9 6 )

The object-oriented methodology for systems analysis and design has generated considerable interest recently . Object-orientation represents a fundamental shift in focus from the traditional process-oriented approaches that have dominated software development for over two decades . Although there is anecdotal evidence to suggest that systems analysts experienced in process-oriented modeling approaches will find it dif ficult to apply objective-oriented methodologies , there is no empirical work investigating the relationship between a procedural mindset and an ability to learn and apply object-oriented concepts . Prior research in human problem solving , however , suggests that the ef ficacy of a systems analysis and design methodology should be judged in the context of the task to which it is applied . To explore the ef fects of prior experience and task characteristics on performance in systems analysis and design , we conducted an experiment in which two groups of subjects applied the object-oriented methodology to two types of tasks , one process-oriented and the other object-oriented . One group had significant prior experience in process-oriented methodologies , while the other group had no formal experience . Both groups were provided identical training in object-oriented analysis and design prior to the experiment . The results of the study suggest that both prior experience and task characteristics play a role in determining performance . The implications that follow for research and practice are discussed .

÷ 1996 Academic Press Limited

1 . Introduction

Systems analysis and design constitute two major tasks in the systems development life cycle . Systems analysis involves achieving an understanding of the application domain and defining the user requirements . The analyst specifies the functional requirements of the target system in terms of elements such as inputs , processes , stored data , and outputs (Whitten , Bentley & Barlow , 1989) . In the design phase , the requirements statement from the analysis phase is used first to generate a high-level design . The high-level design is then translated into a detailed design , which is subsequently used for constructing the new system . Successful system implementation requires a high-quality design , which , in turn , is dependent on a thorough and sound analysis of the application problem . Detecting and rectifying mistakes during the early stages of the systems development life-cycle could save the development ef fort from incurring huge additional costs during the later stages (Boehm , 1981) .

639

1071-5819 / 96 / 120639 1 29$25 . 00 / 0 ÷ 1996 Academic Press Limited

R . AGARWAL ET AL . 640

To facilitate the systems development process , several methodologies have been proposed . The use of a methodology is considered critical for the successful completion of a project because of the complexities inherent in the system development process . Analysis and design methodologies include the structured methodologies , such as the structured techniques (DeMarco , 1979 ; Gane & Sarson , 1979) , Jackson System Development (Jackson , 1983) , and the relatively new object-oriented techniques (Rumbaugh , Blaha , Premerlani , Eddy & Lorensen , 1991 ; Jacobson , Christerson , Jonsson & Overgaard , 1992 ; Booch , 1994) Among these , the object-oriented methodology for systems development has generated significant interest recently . The methodology has been proposed for systems analysis (e . g . Coad & Yourdon , 1990) , as well as systems design (e . g . Coad & Yourdon , 1991) . Widely claimed benefits of the object-oriented approach include its expressive power , the data abstraction and encapsulation it provides , the reusability of the resultant software , and the inherently modular nature of the modeling technique , which facilitates extensibility and maintainability . Desiring those benefits , several organizations are actively exploring ways of employing object-oriented tools and techniques for systems development .

As with any new technology , however , the learning curve associated with the object-oriented methodology may be fairly steep . When compared with the structured techniques , which have dominated software development for over two decades , the object-oriented (OO) approach represents a fundamental shift in focus . It requires an analyst or a designer to conceptualize a target problem and use modeling concepts in ways radically dif ferent from those used in traditional process modeling (Fichman & Kemerer , 1992) .

For firms with a significant staf f of information systems analysts and designers , a potential hurdle in incorporating the OO methodology is the procedural or process - oriented mindset of the analysts and designers . To become widely accep- table , the OO approach has to overcome the legacy of ‘‘investment in people whose experience and expertise are in other ways of doing things’’ (Kozaczynski & Kuntzmann-Combelles , 1993 : p . 23) . While there is anecdotal evidence to suggest that overcoming this mindset is not an easy task , we are not aware of any empirical work that examines the influence of the relationship between a particular orientation (e . g . procedural) of experience analysts and the use of a specific methodology on performance in systems analysis and design tasks .

Given the importance of systems analysis and design , it is surprising that so far very little empirical research has been conducted to study the ef fectiveness of alternative methodologies for analysis and design pruposes . The only studies we are aware of are the ones by Yadav , Bravocco , Chatfield and Rajkumar (1988) , and Vessey and Conger (1993 , 1994) , which investigated the degree to which some of the methodologies assisted analysts in specifying information requirements . Further- more , those studies examined the performance of novice or inexperienced systems analysts , usually with students enrolled in academic courses . However , the results were not generalized to include experienced modelers .

There is evidence to suggest that the relationship between experience in a particular methodology and performance using a new methodology may be quite complex . In the domain of programming , Liu , Goetze and Glynn (1992) found that though , in general , more experienced programmers performed better than less

PRIOR EXPERIENCE AND TASK CHARACTERISTICS IN OBJECT-ORIENTED MODELING 641

experienced ones in learning an object-oriented methodology , strong familiarity with certain programming languages such as BASIC and COBOL had a negati y e influence on the programmers’ learning ability . On the other hand , strong familiarity with languages such as C tended to facilitate the learning process . Thus , certain types of prior experience appeared to be more relevant in the learning of new skills . Research into the ef fectiveness of dif ferent systems development methodologies should , therefore , examine whether a predisposition toward a certain methodology facilitates or inhibits the application of a dif ferent methodology .

Research examining the ef fectiveness of systems development methodologies has also not investigated whether the nature of the task itself has any ef fects on performance in systems analysis tasks , though such ef fects have been studied for other phases of the development life cycle , such as design and coding (Vessey & Weber , 1986 ; Sinha & Vessey , 1992) . There is also substantial evidence outside the systems development area suggesting that problem solvers employ dif ferent processing strategies for tasks with dif ferent characteristics (Einhorn & Hogarth , 1981 ; Slovic & Lichtenstein , 1983 ; Tversky , Sattath & Slovic , 1988) . A methodology that facilitates solving one type of task may inhibit solving another type . The ef ficacy of a systems development methodology should , therefore , be judged in the context of the task to which it is applied and the type of experience of the analysts using it . As Vessey and Conger (1993 : p . 199) rightly point out , ‘‘although a range of methodologies exist , little attention has been paid to characterizing applications such that methodology can be matched to application . ’’

We report the results of a study investigating the ef fects of prior process-oriented modeling expereince on performance in object-orientated modeling . Specifically , we examine whether prior experience in process-oriented modeling facilitates or inhibits systems professionals , vis-a ̀ -vis inexperienced modelers , from applying the object- oriented methodology to two types of tasks : one emphasizing processes and sequencing of processes , and the other emphasizing data and structural relationships . We describe the results of an experiment in which experienced process-oriented modelers and inexperienced modelers used the object-oriented methodology to solve both process-oriented and object-oriented tasks .

The paper is organized as follows . Section 2 provides a theoretical background for the study , presents the research model , and states the research hypotheses . Sections 3 and 4 describe the research methodology and the measurement of dependent variables , respectively . Section 5 presents the analysis and results of the study , while the concluding section discusses the implications of the results and presents directions for future research .

2 . Theoretical background and research model

2 . 1 . THEORETICAL BACKGROUND

Researchers in cognitive psychology have explored the complex cognitive processes involved in human problem solving . In their seminal work , Newell and Simon (1972) proposed a theory in which humans were considered to be information processing systems . The theory posits a set of processes or mechanisms that produce the behavior of a human problem solver . Problem solving takes place within a problem


space , which is the internal representation of the task environment used by the human subject . According to the theory , task instructions and previous experience in solving similar tasks contribute significantly to the determination of the problem space . The task instructions not only define the task , but they also provide a specific representation that helps define an initial problem space .

Researchers in human factors pursued exploring the influence of the nature of a task and its representation on problem solving performance . An important notion generated from this stream of work is that of ‘‘cognitive fit’’ (Vessey , 1991) . The basic model of cognitive fit views problem solving as the outcome of the relationship between the problem (or external) representation and the problem solving task , which are both characterized by the type of information they emphasize . Processes act on the information in the problem representation and the problem solving task to produce a mental representation , which is the internal representation of the problem in human working memory (Gentner & Stevens , 1983) and , therefore , a subset of the total problem space . Processes then act on the mental representation to produce the solution to the problem . According to the cognitive fit theory , when the types of information emphasized in the problem solving elements (problem representation and task) match , the problem can employ processes (and formulate a mental representation) that also emphasize the same type of information . Cognitive fit exists because the processes the problem solver uses to act on the problem representation and to complete the task both match ; the synergy results in superior problem solving performance . Conversely , when a mismatch occurs between problem representation and task , cognitive fit will not result , resulting in a deterioration in problem-solving performance .

2 . 2 . DETERMINANTS OF PROBLEM SOLVING PERFORMANCE

Researchers have also examined factors other than task and representation that could af fect problem solving performance . Sinha and Vessey (1992) tested an extended model of cognitive fit that included problem sol y ing tool as an additional determinant of problem solving performance . They argued that matching the type of information provided by the tool to that in the task and the problem representation would lead to ef fective and ef ficient problem solving performance (see also Vessey & Weber , 1986) . The study found that the ef fects of a match between task and problem solving tool (programming language) outweighted the match between problem representation and task . Significant ef fects were also observed when subjects used their preferred programming constructs , although these ef fects were weaker than those due to cognitive fit .

In the domain of programming , Soloway and his colleagues (Soloway , Bonar & Ehlrich , 1983 ; Soloway & Ehlrich , 1984 ; Soloway , 1986 ; Adelson & Soloway , 1988 ; Soloway , Adelson & Ehlrich , 1988) focused on the cognitive fit between the most naturally chosen strategies and the looping constructs provided by the programming language in use . They found that subjects , while planning looping solutions independent of any programming language , overwhelmingly preferred a READ / PROCESS strategy to a PROCESS / READ strategy . They also observed that program accuracy improved when it was written in a language that facilitates the ‘‘cognitively preferred’’ looping strategy for the problem . Those results were


corroborated by Wu and Anderson (1993) , who found that most PASCAL programmers were quite sensitive to the nature of the problems being solved . Also , when subjects were forced to use a non-preferred strategy , their performance deteriorated .

Newell and Simon (1972) conjecture that pre y ious experience with similar tasks plays an important role in structuring the problem space . ‘‘If the given task bears a similarity to another task with which the problem solver is already familiar , then he may attempt to use the problem space and programs from the familiar one’’ (p . 851) . The influence of experience on problem solving is evident from research in expert and novice problem solving in various technical domains (De Groot , 1965 ; Chase & Simon , 1973 ; Egan & Schwartz , 1979 ; Larkin , McDermott , Simon & Simon , 1980) . The studies indicate that experts possess chunks representing functional units in their domains , while novices do not . Chase and Simon (1973) found that experts , in recalling configurations of chessboards , availed of chunks , which consisted of chess pieces forming attack or defense configurations . By identifying the functional relationships and using them to create internal representations of typical configurations , experts could recall real game boards better than novices ; however , there was no dif ference in performance on randomly composed boards . The chunking phenomenon was also observed in the programming domain (Shneiderman , 1976 ; Adelson , 1981 ; McKeithen , Reitman , Rueter & Hirtle , 1981) . A similar notion , that of schemas , emerged independently from research on text comprehension in artificial intelligence and psychology (see , for example , Schank & Abelson , 1977 ; Graesser , 1981) . Schemas are generic knowledge structures that guide the interpretations , inferences , expectations , and attention of a person trying to comprehend passages .

The role of experience in problem solving was further investigated by Soloway et al . (1988) . They examined the knowledge and processing strategies programmers employ in attempting to understand computer programs . Designing an experimental modeled after the Chase and Simon study , they presented both plan - like programs and runnable unplan - like programs to advanced and novice programmers . The plans , which are program fragments representing stereotypical action sequences in programming , and the discourse rules , which specify the conventions in programming , correspond to the notion of schemas and chunks . Because of their knowledge about programming plans and rules of programming discourse , the advanced programmers performed much better in understanding the plan-like programs than the unplan-like programs . Also , they performed significantly better than the novices on the plan-like programs ; however , there was no significant dif ference between the two groups on the unplan-like programs .

2 . 3 . THE RESEARCH MODEL

In this study we compare the performance of experienced process-oriented modelers and inexperienced modelers on two types of systems analysis and design tasks— process-oriented (PO) and object-oriented (OO)—using the object-oriented modeling tool . Figure 1(a) graphically depicts the interactions between schema , task , and tool for experienced PO modelers . Processes involved in problem solving are represented by the directed flows linking pairs of elements in the model . We expect


Schemas

Problem solvingtask

Mentalrepresentation

Problem solvingtool

Problem solvingperformance

Mentalrepresentation

Problem solvingtool

Problem solvingperformance

Problem solvingtask

F IGURE 1 . (a) Problem solving by experienced modelers , (b) problem solving by inexperienced modelers .

that experienced modelers , after years of applying the ‘‘rules of discourse’’ (Soloway et al . 1988) of the process-oriented methodology , would have developed schemas that are also process-oriented in nature . They would construct a mental representation based on internal cognitive processes acting on the schemas , task and tool ; processes then act on the mental representation to produce the solution to the problem . Inexperienced modelers would not have recourse to schemas ; they would therefore construct a mental representation based on processes acting only on task and tool [see Figure 1(b)] .

2 . 4 . THE PROBLEM SOLVING TOOL DIMENSION : OBJECT- AND PROCESS-ORIENTED

METHODOLOGIES

The design of a business information processing application typically requires the specification of two components : the data underlying the application and the processes that are used to manipulate the data . As an example , consider the case of a payroll system . A complete design for a payroll system would include a description of the data needed to support the system (such as employee attributes , time cards , etc . ) , a specification of the processes underlying the payroll system (such as the creation of paychecks , the creation of income reporting forms , etc . ) , and an order representing the sequence of those processes . A complete description of both data and processing constitutes a model of the target application .

In general , systems analysis and design methodologies prescribe the use of a set of procedures and documentation tools to understand and represent a complex target


system (DeMarco , 1979 ; Gane & Sarson , 1979) . Analysis and design methodologies typically use the principle of decomposition as the primary method of reducing complexity . Although several methodologies are now in existence , structured systems techniques have been the predominant choice for over two decades (Fichman & Kemerer , 1993) . The fundamental unit of decomposition in the structured techniques is a process ; the system is represented as a series of processes that transform data . Processes are linked together via data flows , which help establish sequencing of those processes . A complete structured requirements specification consists of a data flow diagram representing data flows and processes , a data dictionary , and a set of detailed process specifications . As Vessey and Conger (1994) note , the emphasis of the structured techniques is on processes ; such techniques constitute a ‘‘process-oriented methodology . ’’

The object-oriented methodology , on the other hand , uses the concept of an object as the primary unit of decomposition (Booch , 1986 ; Pressman , 1987 , 1988) . An object is a data structure that contains data (its attributes) , as well as the operations (services) that manipulate the data . It embodies a key concept in systems development , that of encapsulation and information hiding (Fichman & Kemerer , 1992 , 1993) . Services are invoked by sending messages to objects . Though represented dif ferently , services are similar to processes used in the structured techniques . In addition to object and service specifications , an object-oriented model includes structural relationships that exist among objects , such as a classification hierarchy , which allows lower level objects to inherit the attributes and services of higher level objects .

When comparing the process-oriented methodology with the object-oriented methodology , beyond the surface dif ferences between the constructs and terms used by the techniques , there exist other fundamental dif ferences . The distinction that we focus on is process - vs . data - orientation . As noted by many researchers (see , for example , Rumbaugh et al . , 1991 ; Fichman & Kemerer , 1992 , 1993 ; Vessey & Conger , 1994 ; Fayad , Tsai & Fulghum , 1996) , the process-oriented methodology places a strong emphasis on processes , whereas the object-oriented methodology tends to focus more on the data and structural relationships . Rumbaugh et al . (1991 : p . 9) state that ‘‘Object-oriented development places a greater emphasis on data structure and a lesser emphasis on procedure structure than traditional functional- decomposition methodologies . ’’ Fichman and Kemerer (1992) , in their comparison of process- , data- , and object-oriented methodologies , reiterate this point by observing that ‘‘Process-oriented methodologies focus attention away from the inherent properties of objects during the modeling process and lead to a model of the problem domain that is orthogonal to the three essential principles of object orientation : encapsulation , classification of objects , and inheritance’’ (p . 23) . Therefore , one would expect experience in one technique to not transfer easily to the other , a theme elaborated upon below .

2 . 5 . THE EXPERIENCE DIMENSION

Prior research on expert and novice performance in various domains (see , for example , Chase & Simon , 1973 ; Moran , 1981 ; Adelson , 1984 ; Schweiger , Anderson & Locke , 1985 ; Adelson & Soloway , 1988 ; Soloway et al . , 1988) underscores the role of experience in problem solving . Experience leads to the acquisition of organized ,


hierarchical knowledge structures or schemas (Chi , Glaser & Rees , 1982) . The superior performance of experts is attributed to their ability to identify and activate the right solution schema .

In this study , the two groups of subjects dif fer from one another with respect to their experience in process-oriented modeling . We postulate , based on the literature , that experienced PO modelers , after years of experience in applying the PO tool , would have developed schemas or chunks that are also process-oriented , because of the process-oriented emphasis of the tool . Those schemas can then be readily applied to PO tasks . However , for OO tasks , there would be a mismatch between the types of information emphasized in the schemas and the task . Chi et al . (1982 : p . 70) describe the schema-task interaction ef fectively :

‘‘The declarative knowledge contained in the schema generates potential configurations and conditions of applicabilty for procedures , which are then tested against the information in the problem statement . The procedural knowledge in the schema generates potential solution methods that can be used on the problem . ’’

The interaction between schemas and tool could also create dif ficulties for the experienced modelers , for whom there is always a mismatch . Prior studies on skill transfer indicate that prior knowledge and experience with a particular formalism may inhibit an individual’s ability to form a mental representation of a task using some other formalism . For example , in the domain of user interfaces , Whiteside , Jones , Levy and Wixon (1985) found that the performance of transfer users (those experienced in a particular type of interface) deteriorated as they moved from a command line to an iconic interface , even though the latter interface was theoretically claimed to be easier to use . Studies focused on the transfer of skills between text editors (Singley & Anderson , 1985 , 1988 ; Polson , 1987 ; Polson , Bovair & Kieras , 1987) emphasize that positive transfer ef fects will be noted only when the two editors share elements in common . Similar results were obtained in programming studies of recursion and iteration , where students found it dif ficult to learn recursive techniques once they had learned about iteration (Anderson , Farrell & Sauers , 1984 ; Kessler & Anderson , 1986) . Soloway et al ’s (1988) study on program comprehension found that the performance of experts deteriorated when the programs did not conform to the plans and programming conventions .

Considering that the OO methodology is very dif ferent from the PO methodology , we expect the ‘‘violations in expectations’’ (Soloway et al . , 1988) of the experienced PO modellers , when employing the OO methodology , to lead to a deterioration in performance . The sentiment is echoed by Kozaczynski and Kuntzmann-Combelles (1993) , who speculate that a key hurdle in dif fusing object-oriented methodologies is the investment in people ‘‘familiar with other ways of solving problems’’ (p . 23) . As Scholtz and Wiedenbeck (1990) observe in their study of transfer between programming languages , instruction in the new tool must ‘‘take into account the experience of the programmer , anticipate possible biases acquired from this experience ? ? ? ’’ (p . 70) .

Systems professionals experienced in process-oriented modeling can be characterized as having a procedural mindset . Ideally , the schemas that they have developed over years of experience in process-oriented modeling will lead to


synergistic problem solving when both the task and the tool are process-oriented . However , given that the tool used for the study is object-oriented , we would expect better performance when there is a match between task and experience than when there is not . Thus , any loss in performance attributable to transferring to the OO methodology will be moderated by the nature of the problem solving task .

2 . 6 . THE PROBLEM SOLVING TASK DIMENSION

As established by prior research , the nature of the problem solving task is a key determinant of problem solving performance . According to Vessey (1991) , the formation of the mental representation is facilitated when the types of information emphasized by the task and the problem representation match . Several other researchers (e . g . Soloway et al . , 1988 ; Vessey & Galletta , 1991 ; Sinha & Vessey , 1992 ; Wu & Anderson , 1993) have also explicitly taken into account the nature of the task being solved .

Consistent with the modeling constructs used by process- and object-oriented methodologies , we distinguish between systems analysis and design tasks on the basis of their inherent traits . They are classified as ‘‘process-oriented’’ if their emphasis is on processes and sequencing of those processes , rather than on data and structural relationships . Alternatively , tasks are classified as ‘‘object-oriented’’ if their description highlights data and structural relationships (such as classification) among the data .

At a high level of analysis , tasks may be characterized as process- or object-oriented depending on their dominant features . However , this characterization only focuses on the dominant features . A systems analysis and design task is never fully process-oriented or object-oriented as both processes and data must be modeled . We alluded to this characterization earlier in describing a hypothetical payroll system . Solving a task entails solving the structure sub-task (the representation and organization of data) and the beha y ior sub-task (specification and control of processes) (Coad & Yourdon , 1990 , 1991) . Thus , in our taxonomy of tasks , a task is considered to be inherently process-oriented (object-oriented) if its behavioral (structural) features dominate its structural (behavioral) features . Classification of tasks is done through a feature analysis , as explained subsequently . Each task , however , contains sub-tasks of both structure and behavior , albeit to dif fering degrees .

2 . 7 . HYPOTHESES

We compared the performance of experienced and inexperienced modelers at two levels of task granularity : (i) the task level (OO and PO) and (ii) the sub - task level (structure and behavior) . Because these two levels dif fer only in granularity , we expect each task-related hypothesis to have a corresponding sub-task-related hypothesis . More specifically , hypotheses dealing with the PO (OO) task would have matching hypotheses for the behavior (structure) sub-task . The sub-task- related hypotheses are applicable to both the PO and the OO task , because both of them are composed of structure and behavior sub-tasks .


When the object-oriented tool is applied to model a process-oriented task , one would expect that the lack of cognitive fit would result in deterioration of performance . Consider , however , the experience dimension . Given that both groups face the same dif ficulty , prior research examining the ef fects of experience on problem solving performance (e . g . Adelson & Soloway , 1988 ; Soloway et al . , 1988) suggests that experienced analysts will perfrom better than inepxerienced analysts as they have available schemas or chunks for solving PO tasks . Thus , while both groups contend with the task-tool mismatch , the experienced group is expected to have an edge over the inexperienced group . We therefore state the following hypotheses :

H1a : Experienced PO modelers generate higher quality solutions than inexperienced modelers for the process-oriented task .

H1b : Experienced PO modelers generate higher quality solutions than inexperienced modelers for the behavior sub-task .

For the object-oriented task , on the other hand , there is a match between task and tool for both novice and experienced modelers . There is , however , a mismatch between schemas and task and between schemas and tool for the experienced modelers . Therefore , despite the fact that one would ordinarily expect experienced modelers to generate better solutions than inexperienced modelers , we do not expect any significant dif ference in solution quality between the two groups on the object-oriented task (cf . Soloway et al . , 1988) . Hence , we state the following hypotheses :

H2a : There is no significant dif ference in the quality of solutions generated by experienced PO modelers and inexperienced modelers for the object-oriented task .

H2b : There is no significant dif ference in the quality of solutions generated by experienced PO modelers and inexperienced modelers for the structure sub-task .

3 . Research design and methodology

3 . 1 . RESEARCH DESIGN

The experiment employed a 2 3 2 factorial design , with the two factors being the experience level of the subjects and the nature of the task . One group of subjects (G1) consisted of experienced systems analysts and designers , all having more than 2 years of experience in process-oriented analysis and design . These subjects had been involved in PO modeling and procedural software development , but had no prior exposure to or experience in OO analysis and design . The second group of subjects (G2) consisted of graduate and undergraduate business students . These subjects had limited prior knowledge of PO modeling (through a course in systems analysis and design using data flow diagrams) , no experience in on-the-job


n = 22 n = 24

PO n = 22 n = 24

yes no

Problem solving tool (OO)

Process-oriented experience

Problem solvingtask

OO

F IGURE 2 . Research design .

application of these concepts , and no prior exposure to or experience in OO analysis and design . There were 22 subjects in G1 and 24 subjects in G2 (see Figure 2) .

Both groups were provided identical training sessions (a total of 6 h split into two 3 h sessions) on OO analysis and design . The purpose of these sessions was to teach the subjects the fundamental concepts underlying the OO approach and to give them some practical experience in applying those concepts . Training for the experienced and inexperienced groups was conducted separately . The same instruc- tor and training materials were utilized for both groups to eliminate bias . At the end of the training session , both groups were provided with three example problems to solve ; those problems helped reinforce the concepts learned earlier and provided the students with an opportunity for hands-on experience . The methodology used for training was adapted from the one suggested by Coad and Yourdon (1990 , 1991) .

3 . 2 . EXPERIMENTAL TASK

Both groups were given two experimental tasks—narrative descriptions of business information processing problems . Subjects were required to develop object-oriented models for both tasks ; the problem solving tool , therefore , was constrained . Task T1 was an application that was inherently process-oriented in nature while task T2 was one that was inherently object-oriented . Tasks were selected in the following way . Eight business information processing applications were scored independently by two evaluators on the basis of their inherent features . The applications came from two sources : (i) a collection of cases developed from actual systems analysis and design projects conducted by students with industry , and (ii) applications that were created by the researchers .

The inherent features of a task were evaluated along two dimensions : structure and beha y ior . The specific facets considered for the structure dimension were data intensity (the relative number of objects and attributes identified in the problem


description and the extent to which they dominated the narrative) , structural relationships (such as classification and aggregation) , and polymorphism / inheritance . Thus , in addition to being data-oriented , we characterized an object-oriented task as one that emphasizes structural relationships , polymorphism , and inheritance , all recognized as essential principles of object-orientation (Fichman & Kemerer , 1992) . The facets considered for the behavior dimension were process intensity (the relative number of processes in the problem description and the extent to which they dominate the narrative) , and sequencing / control of the processes . These facets constituted a list of characteristics to look for in a given problem description ; each problem was scored by two independent evaluators on each facet on a scale of 1 – 5 . A score of 5 for a particular facet indicated a strong presence of the facet in the problem description , while a score of 1 indicated that the facet was present to a very small degree . A task was classified as inherently process - oriented ( object - oriented ) if the score on its behavior (structure) dimension greatly exceeded the score on its structure (behavior) dimension .

The average scores on the structure and behavior dimensions were determined by the evaluators . The problem selected as the inherently PO task had a much higher behavior score (4 . 75) than the structure score (2 . 00) . Similarly , the problem selected as the inherently OO task had a much higher structure score (4 . 34) than the behavior score (2 . 50) . The reported scores are averages of scores assigned by the two raters ; the Pearson correlation coef ficient for the evaluator scores on the eight cases was 0 . 83 , significant at p , 0 . 01 .

The selected PO task was an ‘‘Accounts Payable System’’ (T1) at a fictional corporation , and the OO task was an ‘‘Employee Benefits System’’ (T2) at a fictional university (see Appendix A) . Each problem description occupied approximately one page . T1 contained a significant use of the word ‘‘then’’ (suggesting sequencing of activities) , while T2 was characterized by a natural generalization / specialization hierarchy among employees of the university . The order of task presentation was counter-balanced for the experiment so as to eliminate any confounding learning ef fects . To simulate real-world pressure , subjects were provided with an incentive to complete the task as quickly as possible ; a maximum time limit of 1 . 5 h was specified for each problem . A pilot test conducted before the actual experiment indicated that both problems could be reasonably solved within the specified time limit .

3 . 3 . TRAINING MATERIAL AND EXPECTED OUTCOMES FROM SUBJECT SOLUTIONS

The training material closely followed the terminology and methodology prescribed in the texts by Coad and Yourdon (1990 , 1991) . Prior to training the subjects on the methodology itself , some background knowledge on OO concepts was provided . The subjects had to identify the objects and their attributes , identify the structures (classification and assembly) , define the services of each object (i . e . the object’s functionality) , and specify the messages that need to be passed between objects . Subjects were asked to conform to the format of the examples provided in the textbooks .

In addition to providing training on using OO modeling techniques , subjects were provided with some general background on modeling information systems applications . In particular , both PO and OO models were developed for one example . The


Riverside universityprograms

EmployeeStudentstu_idstu_name

Ugrad. Grad.

emp_idnamesalaryschool_contribemp_contribemp_agecalc_pencalc_lifecalc_health

salary*school_contrib*emp_contrib*stipend

GA

calc_pen*calc_life*

Faculty

Asst. Assoc. Prof.

Staff

Secy. Clerk Officer

Department

HRB Payrollissue_penissue_lifeissue_health

tax_rep

salaryextra_inc

calc. pen –

F IGURE 3 . Sample solution to the OO experimental task .

cases used in the training sessions included both an inherently object-oriented and an inherently process-oriented problem ; these cases were selected from the remaining six cases from the original set of eight cases considered for the experimental tasks .

A model solution to task T2 (see Appendix A) , one of the experimental tasks , is shown in Figure 3 . Several facets are considered in assessing the quality of subject solutions , including objects , attributes , structural relationships , services , and sequencing . A rectangle corresponds to an object class , the top section of the rectangle contains the object name , the middle section contains the attributes associated with the object , while the bottom section lists the services the object provides . For example EMPLOYEE is an object class with attributes emp – id , name , salary , etc ., and with services calc – pen , calc – life , and calc – health . The solution exhibits two types of structural relationships—a Part - of or Assembly structure , such as the one existing between RIVERSIDE UNIVERSITY and STUDENT , EMPLOYEE , and DEPARTMENT , and an Is - a or Classification structure , such as the one existing between EMPLOYEE and GA , FACULTY , and STAFF . The asterisks next to the salary , school – contrib , and emp – contrib attributes of the GA object class indicate that these attributes are not inherited from the EMPLOYEE superclass ; however , all other EMPLOYEE and STUDENT attributes are inherited by GA . In addition to the inherited attributes , GA has a new attribute called stipend . Notice also the existence of polymorphism in the example—the service calc – pen is included in both the EMPLOYEE and FACULTY objects , indicating that the service specifications are dif ferent for the two objects .


Although sequencing of processes is an integral construct of the PO modeling paradigm and is represented explicitly through directed data flows , there is no straightforward way of implementing such control in the OO modeling paradigm . In their comprehensive review of several OO methodologies , Fichman and Kemerer (1992) point out that none of those methodologies provides an explicit model of end-to-end processing sequences , although individual parts of a global process are modeled piecemeal . † In our study , therefore , we trained the subjects to enumerate the operations within a service to indicate the sequence in which the operations are carried out . Subjects were also trained in the message passing metaphor , because the operations within a service often invoke other services by sending messages . Since task T2 hardly requires any sequencing and control , task T1 (see Appendix A) is used for illustration . The specifications for the RECON – MNTH – STMT (reconcile monthly statement) service , which is stored inside the BOOKKEEPER object , illustrate both sequencing and message passing . The service first compares the monthly statements with the outstanding invoice and paid invoice files . It then sends a message called file – trbl – stmt to the TRBL – STMT (troublesome statement file) object , which files the statements that are not fully reconciled . Finally , a message called recon – trbl – stmt is sent to the BRANCH (branch store) object , which reconciles those statements .

Service Specification Service : RECON – MNTH – STMT

1 . send – msg compare – files 2 . send – msg file – trbl – stmt TRBL – STMT 3 . send – msg recon – trbl – stmt BRANCH

4 . Dependent variables and their measurement The primary outcome of interest was the performance of subjects for each task and sub-task . Performance was measured using three dif ferent dependent variables , as described below .

4 . 1 . PERFORMANCE

Performance on the tasks was evaluated on the basis of a scoring scheme , which specified rules for determining the scores on each facet (such as as objects , attributes , and services) . The facet scores were then combined into scores on two dimensions : structure and behavior . A model ‘‘correct’’ solution was developed for each task by the researchers , providing a benchmark for evaluating the subjects’ solutions . Each subject’s response was evaluated by two independent evaluators and rated using the scoring scheme . Multiple correct solutions were allowed as long as the relevant facets were recognized and included in the solution . For example , the problem statement for task T2 described a faculty member of a University as an Assistant , Associate , or Full professor . Two alternative and correct representations

† The OMT methodology (see Rumbaugh et al . , 1991) uses three kinds of models : the object model , the dynamic model , and the functional model . The dynamic model is used to specify the control aspects of a system , i . e . how the operations are sequenced . But , as Jacobson et al . (1992 : p . 495) point out , ‘‘It is not very easy to see how all the techniques in OMT can be used to form consistent models , and the relations between the three models are not obvious at all stages . ’’


are possible . One is to model FACULTY as an object class with a rank attribute ; the second option is to model ASST – PROF , ASSOC – PROF , and FULL – PROF as subclasses of the FACULTY CLASS . This type of variation in subject solution was allowed and explicitly recognized in the scoring scheme . Quality of a solution was determined on the basis of a composite score obtained from the individual facet scores .

The problem of achieving consensus and objectivity in scoring has persisted in the research literature for studies of this nature (e . g . Batra , Hof fer & Bostrom , 1990) . Nevertheless , every attempt was made in the development of the scoring scheme to ensure that no systematic bias was introduced by the evaluators . A score on the structure sub-task is comprised of scores on three features : objects , structural relationships , and the degree to which attributes are idenitfied and associated with appropriate objects (attribute presence and match) . Objects and structural relationships were further classified as major and minor , depending on their relative importance in characterizing the problem . Scores for objects and relationships were assigned based on whether they were present or absent in the solution , while ‘‘attribute presence and match’’ was assigned a discrete score between 0 and 10 . Details of the scoring scheme are provided in Table 1 . The features used for scoring solutions to the experimental tasks are presented in Appendix B . As further

T ABLE 1 . Dependent y ariables and scoring scheme

Variable Task features Points Weight Weighted

score Comments

Structure Major object 1 2 2 0 if absent , 1 if present . Minor object 1 1 1 0 if absent , 1 if present . Major

classification / assembly 1 3 3 0 if absent , 1 if present .

Minor classification / assembly

1 1 1 0 if absent , 1 if present .

Attribute presence & match

10 0 . 4 4 0 if presence / match is poor , 5 if ok , and 10 if very good .

Behavior Major service 1 2 2 0 if absent , 1 if present . Minor service 1 1 1 0 if absent , 1 if present . Processing 10 0 . 5 5 0 if quality is poor , 5 if

ok , and 10 if very good . For T1 , the degree of sequencing present was considered . For T2 , detailed computations were considered .

Quality of sequencing 10 0 . 5 5 0 if quality is poor , 5 if ok , and 10 if very good .

Service-object match 10 0 . 5 5 0 if match is poor , 5 if ok , and 10 if very good .

Structure- behavior

Weighted average of structure and behavior scores

10 Relative weights are contribution of structure and behavior to maximum possible score .


validation of the intrinsic characteristics of a task , notice that task T2 has more ‘‘structure’’ components than task T1 .

A score on the beha y ior sub-task is comprosed of scores on the following features : services , processing , quality of sequencing , and service-object match (see Table 1) . Services were classified as major and minor , and were scored based on whether they were present or absent in the solution . Processing scores were given based on the degree of sequencing and the required computations present in the solution . The ‘‘service-object match’’ score depended on the association (match) of services with appropriate objects . For processing , quality of sequencing , and service – object match , discrete scores between 0 and 10 were assigned . Again , as is evident from Appendix B , the inherently process-oriented task (T1) had more behavior components than the inherently object-oriented task (T2) .

Inter-rater agreement for the identification of objects and services , as measured by Cohen’s Kappa (Bishop , Fienberg & Holland , 1975) , was 0 . 84 for T1 and 0 . 92 for T2 . Both the Kappa scores were within the 99% confidence intervals . Disagreements were resolved through discussion ; in most instances , disagreements arose from an oversight on the part of an evaluator .

Finally , an aggregate measure , ‘‘structure-behavior’’ , was used to evaluate a complete solution . The ‘‘structure-behavior’’ score was derived by taking a weighted average of the ‘‘structure’’ and ‘‘behavior’’ scores , where the weights were proportional to the presence of the ‘’structure’’ and ‘‘behavior’’ features in the task . For example , for task T1 (the PO task) , the maximum possible ‘‘structure’’ score is 24 , whereas the maximum possible ‘‘behavior’’ score is 41 . The ‘‘structure’’ and ‘‘behavior’’ scores , therefore , contribute 37% and 63% , respectively , toward the composite ‘‘structure-behavior’’ score .

In summary , three dependent variables (see Table 1) were used to measure subject performance on the analysis and design tasks . They included an aggregate measure (‘‘structure-behavior’’ score) for assessing performance at the task level , and two disaggregated measures that evaluated solution quality independently for ‘‘structure’’ and ‘‘behavior’’ at the sub-task level . Note that while for task categorization we determined if certain general features such as data intensity and process intensity were present or not , for scoring the solutions we had to go one step further . For example , we identified the specific objects , relationships , services , etc . Also , some features used for scoring , such as ‘‘attribute presence and match’’ , make sense only in the context of a specific solution and , therefore , cannot be used to categorize a task . Apart from such dif ferences in detail , the two schemes are consistent with one another .

4 . 2 THE PILOT

A pilot test of the training program , the OO methodology , and the instruments was conducted with 17 subjects . The following insights were obtained from the pilot : subjects had dif ficulty distinguishing between objects and attributes ; they were not able to identify which object a particular service ought to be attached to ; and they were unable to specify message and service sequencing adequately . These results were used to modify the training program substantially . Instead of of fering two 2-h training sessions with two examples , as was our original intent , we extended the training to two 3-h sessions with three examples .


T ABLE 2 . Between group subject demographics

Attribute Experienced

( n 5 22) Inexperienced

( n 5 24)

Mean (S . D . ) Mean (S . D . ) Average age*** Traditional analysis and design experience*** General computer programming experience***

38 . 27 (6 . 92) 3 . 55 (1 . 06) 4 . 32 (0 . 66)

25 . 13 (4 . 50) 2 . 11 (1 . 02) 2 . 48 (1 . 08)

*** indicates significant dif ferences between groups at p , 0 . 01 . Age is measured in years ; both other variables on a 1 – 5 scale .

5 . Analysis and results

5 . 1 . GROUP PROFILES

The operationalization of the experiment required that the two groups of subjects dif fered from one another with respect to their levels of process-oriented modeling experience . Table 2 captures the salient demographic data for both groups—the two sets of subjects exhibited a significant dif ference in the two major attributes of interest : (i) experience in traditional (PO) systems analysis and design and (ii) experience in computer programming . The scores shown in the table are self- reported experience on a 1 – 5 Likert scale , with 1 representing ‘‘no experience’’ and 5 representing ‘‘a lot of experience’’ . There was also a considerable dif ference in age between the two groups . Subjects were asked to indicate if they had heard of OO analysis and design . If they answered in the af firmative , follow-on questions asked if they had any experience in OO analysis and design , and , if so , their level of experience was elicited on a five-point Likert scale , with ‘‘low experience’’ and ‘‘high experience’’ as the end-points . Only three subjects in Group 1 and one subject in Group 2 indicated that they had prior experience in OO analysis and design ; all three Group 1 subjects rated their experience as 1 , while the Group 2 subject rated his experience as 2 . Additional descriptive statistics for all research variables are provided in Table 3 .

T ABLE 3 . Descripti y e statistics for performance

Task Dependent

variable

Overall sample ( n 5 46)

mean (S . D . )

Experienced group ( n 5 22)

mean (S . D . )

Inexperienced group ( n 5 24)

mean (S . D . )

T1 Structure Behavior Structure-behavior

6 . 34 (2 . 79) 6 . 54 (2 . 27) 6 . 46 (1 . 86)

6 . 21 (2 . 68) 7 . 33 (1 . 80) 6 . 92 (1 . 69)

6 . 46 (2 . 94) 5 . 81 (2 . 45) 6 . 05 (1 . 99)

T2 Structure Behavior Structure-behavior

6 . 73 (1 . 83) 6 . 35 (2 . 61) 6 . 56 (1 . 80)

6 . 78 (1 . 85) 7 . 64 (1 . 47) 7 . 16 (1 . 30)

6 . 69 (1 . 85) 5 . 17 (2 . 88) 6 . 02 (2 . 05)

All scores are scaled to a maximum of 10 .


T ABLE 4 . ANOVA results for task performance

Dependent variable Source of variation Sum of squares F -value p -value

Structure-behavior Experience Task Experience 3 task

23 . 017 0 . 226 0 . 424

3 . 676 0 . 071 0 . 134

0 . 008*** 0 . 790 0 . 715

*** Significant at p , 0 . 01 .

5 . 2 . TASK-RELATED HYPOTHESES

An analysis of variance (ANOVA) procedure was used to determine the interaction ef fects of experience and task characteristics on problem solving performance . If the overall F -test was significant for the ANOVA , post hoc comparisons were made to isolate where the dif ferences lay (Kerlinger , 1986) . A test with a family level of significance such as Schef fe’s (Schef fe , 1953) was not utilized because the direc- tionality of the comparisons was derived from theoretical considerations and specified prior to the analysis (Kerlinger , 1986) . Table 4 presents the results of the ANOVA ; plots of major dependent variables are provided in Figure 4 . The interaction ef fect for the structure-behavior dependent variable is not significant ; the only significant ef fect for this dependent variable is the main ef fect of experience .

The hypotheses related to expected dif ferences between the groups with respect to the two tasks . H1a stated that experienced modelers would perform better than inexperienced modelers on the PO task , while H2a cliamed that there would be no dif ference in performance of the two groups on the OO task . For the experienced group , the mean score for structure-behavior on the PO task is 6 . 92 while , for the

8

7

6

5T1 T2

(a)8

7

6

5T1 T2

(b)

8

7

6

5T1 T2

(c)

F IGURE 4 . Plots of mean score for dependent variables . (a) Structure ; (b) behavior ; (c) structure- behavior . G1 : Experienced modelers ( j ) ; G2 : inexperienced modelers ( r ) ; T1 : process-oriented task ; T2 :

object-oriented task .


inexperienced group , the corresponding score is 6 . 05 . Results of the post - hoc t -tests indicate a non-significant trend toward supporting hypothesis H1a ( p 5 0 . 057) . H2a , however , is not supported ; the experienced group performed significantly better than the inexperienced group on the OO task .

5 . 3 . SUB-TASK-RELATED HYPOTHESES

The sub-task-related hypotheses were constructed to examine the ef fects of experience and task characteristics at a finer level of granularity—that of the sub-task . The two sub-tasks were structure and behavior . The analysis procedures used to test hypotheses 1b and 2b were similar to those described above , except that two dif ferent ANOVA procedures were used , one for each task . Each ANOVA was a 2 3 2 analysis , with the two factors being experience and sub-task . The dependent variable used for the procedure was the relevant score on the structure or the behavior sub-task . These results are presented in Table 5 . For the PO task , neither the interaction ef fects ( p 5 0 . 094) nor the main ef fects are significant . For the OO task , both the interaction ef fect and the main ef fect of experience are significant ( p 5 0 . 008 and p 5 0 . 004 , respectively) .

Hypothesis H1b hypothesized that the performance on the behavior sub-task would vary , depending on the modeler’s level of experience . For both PO and OO tasks , we expected experienced modelers to generate higher quality behavior solutions than inexperienced modelers . Results of t -tests confirm the hypothesis ; the experienced group performed significantly better than the inexperienced group on the behavior sub-task ( p -values are 0 . 011 and 0 . 000 for the PO and the OO task , respectively) .

Hypothesis H2b proposed that there would be no significant dif ference in performance of the two groups for the structure sub-task . The t -test results provide support for this hypothesis . The mean scores are very close . For the PO task , the scores for the experienced group and inexperienced group are 6 . 21 and 6 . 46 , respectively , and for the OO task , the corresponding scores are 6 . 78 and 6 . 69 (see Figure 4) .

T ABLE 5 . ANOVA results for sub - task performance

Dependent variable Source of variation Sum of squares F -value p -value

Process - oriented task Score Experience

Sub-task Experience 3 sub-task

9 . 181 0 . 880

18 . 049

1 . 458 0 . 140 2 . 866

0 . 231 0 . 709 0 . 094

Object - oriented task Score Experience

Sub-task Experience 3 sub-task

37 . 565 3 . 394

32 . 467

8 . 545 0 . 772 7 . 385

0 . 004*** 0 . 382 0 . 008***

*** Significant at p , 0 . 01 .


In summary , of the four hypotheses tested , two were supported , while two were not . The implications of the results are discussed in Section 6 .

5 . 4 . LIMITATIONS

The limitations of our experiment hinge around the choice of experimental tasks , the measurement procedures , the generalizability of the results , and , to a limited extent , the nature and size of the sample . We selected our experimental tasks on the basis of features indicative of process-orientation or object-orientation . Insofar as this was a subjective assessment , there is a chance of misclassification . On the other hand , the use of multiple raters and an objective classification scheme for each feature helped address this limitation . Comments gathered from the experienced subjects through open-ended questions on the debriefing survey provide further validation of the intrinsic nature of each task . When asked to identify which methodology they would have preferred to use for each task , experienced modelers generally preferred the OO approach for the OO task and the PO approach for the PO task . For the OO task , nine stated they would prefer an OO approach while only three subjects said they would use a PO methodology . For the PO task , 11 subjects preferred to use a PO approach while only five preferred the OO approach . For the OO task , nine subjects expressed indif ference between the modeling approaches , while five subjects expressed a similar sentiment for the PO task (one subject did not respond to the question) .

Given that modeling in the real world is a slow , interactive , and iterative process , one would expect the real results to be evident at later stages of the development cycle . However , in experimental work , we must necessarily constrain the time limit for investigating phenomena we are interested in to allow for an appropriate level of experimental control . We believe that those phenomena could also be explored in the field utilizing alternative research strategies to yield interesting insights into the relative ef ficacy of a particular systems analysis and design methodology .

In our study we examined subject performance on only two tasks . The generalizability of our results across a wider range of analysis and design applications would require an examination of performance across a larger set of tasks . Nevertheless , the study does provide insights and a foundation that could be expanded through future research . Our measurement procedures required an assessment of subject performance on various tasks . Again , the potential for error was minimized through the following precautions : the use of a comprehensive scoring scheme , the use of multiple evaluators , and significantly higher levels of inter-evaluator agreement .

The two groups of subjects used for the experiment were intended to be representative of individuals with and without experience in traditional systems analysis and design techniques . Although the samples were not randomly constructed , subject demographics indicate that the desired dif ferences between groups had indeed been achieved . We used self-reported experience as a basis for assigning subjects to groups ; however , we do not believe that the subjects had any incentive to misrepresent their experience . This belief is further corroborated by the fact that all of the experienced subjects were practising systems analysts and designers employed by local firms in the city where the study was conducted . In as much as our subjects are representative of experienced and inexperienced systems analysts , the results of


this study must be interpreted in the context of our sample characteristics . There is a significant dif ference in age across the two groups . While we believe that this dif ference is a natural outcome of the desired profile of the two groups (i . e . experienced vs . inexperienced) , others might wish to take the dif ference into account while interpreting the results . The sample size was adequate for the research questions being investigated .

Problem representation was used as a control variable in our study , i . e . we utilized the same narrative form for both tasks . While controlling for the representation , however , there is a possibility that certain types of tasks are more amenable to translation from a narrative to an object-oriented model than others . Prior research has manipulated problem representation by comparing dif ferent representations of the same information , say , graphs vs . tables (Vessey & Galletta , 1991) . Prior human factors research in systems analysis and design , however , has focused on narrative descriptions of problems , with the representation considered to be fixed (e . g . Vessey & Conger , 1993 , 1994) . Future research might be directed at investigating the ef fects of interactions between task and representation for systems analysis and design by utilizing alternative representations of information , such as existing reports , views , forms , screens , etc .

6 . Discussion and implications

As discussed before , no systems modeling task is fully process-oriented or fully object-oriented : it has both dimensions . That is also true for the two experimental tasks used in this study . Looking only at aggregate performance can sometimes result in the obfuscation of important relationships . The performance on such tasks should therefore be assessed by partitioning them along the structure and behavior sub-tasks and studying sub-task-level performances . If the results for the sub-tasks are in concordance with one another and with the results for the aggregate task , then the results can be interpreted unequivocally ; either all the hypotheses are supported or none are . If the results for the sub-tasks agree with one another but are in discord with those for the aggregate task , then the results should be interpreted carefully . The sub-task scores provide a truer picture of the match between schemas , task characteristics , and problem-solving tool , because a sub-task is exclusively structure-oriented or behavior-oriented . Thus if the results for a hypothesis at the task level conflict with those at the sub-task level , we should consider the latter as being more indicative .

H1a suggested that experience modelers would generate better solutions than inexperienced modelers for the PO task . The non-significant trend toward support for H1a , coupled with the significant support for H1b , indicates the dominating influence of the match between schemas and task . Therefore , although there is a mismatch between schemas and tool , the process-oriented schemas that the systems professionals possess gave them an edge over inexperienced modelers in solving process (behavior)-oriented tasks .

Ceteris paribus , experience in systems analysis and design should result in a higher level of performance . However , as hypothesized through H2a and H2b , the improvement in performance can be of fset by the counter-balancing negative ef fects of a process-oriented mindset . Absence of support for H2a indicates that , on the


whole , the performance level of experienced modelers and inexperienced modelers on the object-oriented task was not the same ; in fact , the experienced modelers performed better than the inexperienced modelers at the level of the aggregate task . However , if we probe further and examine performance at a finer level of granularity , support for H2b indicates the negative ef fects of process-oriented schemas of experienced modelers . Prior experience in PO modeling did not help the professionals in generating better structure solutions than the inexperienced modelers . One can interpret this result as a way of leveling the playing field—when it comes to solving the structure sub-task of a given task , both experienced and inexperienced modelers are operating from the same field . Although experience in systems analysis and design ought to result in better solutions to both the ‘‘data’’ and the ‘‘processing’’ aspects of a model , this was not observed . The positive ef fects associated with experience were probably of fset by the bias experienced modelers had developed for focusing attention on processes as emphasized by process- oriented tools . At the aggregate task level , however , it is possible that the ability to synthesize dif ferent elements of a problem (viz . structure and behavior) is acquired through experience .

The experimental results have several implications for both researchers and practitioners . Future research could be directed at evaluating the ef ficacy of dif ferent systems development methodologies for modeling dif ferent types of tasks . Since our study dealt only with the object-oriented methodology , future research could examine how ef fective the process-oriented methodology is in handling the two types of tasks . Also , our study dealt only with experienced PO modelers ; at present , there is a dearth of experienced OO modelers in the real world . However , in the next few years , we expect more and more systems analysts to be trained in the OO methodology . Future research could then explore the performance of experienced OO modelers . If our study could be extended to encompass the PO methodology and experienced OO modelers as well , then the ef fects of the full spectrum of match / mismatch could be explored . That would help clarify the relative importance of each of the possible combinations of match / mismatch between task , problem- solving tool , and experience .

In this study , we did not manipulate problem representation , because systems analysis and design tasks are usually stated in a narrative form . There is , however , a possibility of narratively representing the same task in dif ferent ways . For example , the same task could be stated so as to emphasize its structural features or its behavioral features . Studies examining the ef fects of alternative representations , in conjunction with the other problem solving elements , would provide additional insights into the match – mismatch phenomenon .

Our study examined the process of translating a narrative description of a business information processing task into a structured model . One of the benefits claimed of the OO approach is that it facilitates end-user understanding . An interesting extension of this study would be to investigate which approach (object-oriented or process-oriented) is more powerful at facilitating end-user comprehension . Such a study could be operationalized by re y erse engineering , i . e . having end-users translate a given model of reality into a narrative problem description .

Research in systems development could be directed at devising methodologies that are ef fective for solving a wide variety of systems analysis and design tasks . That could lead to either developing a hybrid methodology , which combines the best


features of PO and OO modeling , or developing an entirely new methodology . In fact , the OMT methodology (see Rumbaugh et al . , 1991) combines three models— the object model , the dynamic model , and the functional model —each of which captures dif ferent aspects of a system . It is interesting to note that the functional model consists of dataflow diagrams , a standard process-oriented technique used to show the transformation of input values to output values through processes . Research could also be focused on examining how process- and object-oriented tools may co-exist within an organizational context .

For practitioners , this study has important implications . Before jumping onto the object-oriented bandwagon , serious consideration should be given to the nature of tasks in the domain . As the results indicate , the OO tool is not universally powerful . The decision on selecting a tool should therefore be made in the context of the tasks to which it is going to be applied . For instance , though the OO tool seems to be ef fective for solving structure-oriented tasks , it does not seem to be that ef fective for process-oriented tasks , which may be better solved using traditional PO tools . If tasks in the domain encompass dif ferent types , then a portfolio of tools could be maintained and the tool that best matches the given task could be selected . A better option still might be to use a hybrid tool such as OMT , which allows modelers to capture a given part of the reality using the model—object , dynamic or functional— that is relevant for the purpose .

We are grateful to Professor Iris Vessey for her comments on an earlier version of the paper .

References A DELSON , B . (1981) . Problem solving and the development of abstract categories in

programming languages . Memory and Cognition , 9 , 422 – 433 . A DELSON , B . (1984) . When novices surpass experts : the dif ficulty of a task may increase with

expertise . Journal of Experimental Psychology : Learning , Memory and Cognition , 10 , 483 – 495 .

A DELSON , R . L . & S OLOWAY , E . (1988) . A model of software design . In M . T . H . C HI , R . G LASER , & M . J . F ARR Eds . The Nature of Expertise . Hillsdale , NJ : Erlbaum .

A NDERSON , J . R ., F ARRELL , R . & S AUERS , R . (1984) . Learning to program in LISP . Cogniti y e Science , 8 , 87 – 129 .

B ATRA , D ., H OFFER , J . A . & B OSTROM , R . P . (1990) . Comparing representations with relational and EER models . Communications of the ACM . 33 , 126 – 139 .

B ISHOP , Y . M . M ., F IENBERG , S . E . & H OLLAND , P . W . (1975) . Discrete Multi y ariate Analysis : Theory and Fractice . Cambridge , MA : MIT Press .

B OEHM , B . W . (1981) . Software Engineering Economics . Englewood Clif fs , NJ : Prentice-Hall . B OOCH , G . (1986) . Object-oriented development . IEEE Transactions on Software

Engineering , 12 , 211 – 221 . B OOCH , G . (1994) . Object - Oriented Analysis and Design with Applications , 2nd Edition .

Redwood City , CA : Benjamin / Cummings . C HASE , W . C . & S IMON , H . (1973) . Perception in chess . Cogniti y e Psychology , 4 , 55 – 81 . C HI , M . T . H ., G LASER , R . & R EES , E . (1982) . Expertise in problem solving . In R . J .

S TERNBERG , Ed . Ad y ances in the Psychology of Human Intelligence , vol . 1 . Hillsdale , NJ : Erlbaum .

C OAD , P . & Y OURDON , E . (1990) . Object - Oriented Analysis . Englewood Clif fs , NJ : Yourdon Press / Prentice Hall .

C OAD , P . & Y OURDON , E . (1991) . Object - Oriented Design . Englewood Clif fs , NJ : Yourdon Press / Prentice Hall .


D E G ROOT , A . D . (1965) . Thought and Choice in Chess . The Hauge : Mouton . D E M ARCO , T . (1979) . Structured Analysis and System Specification . Englewood Clif fs , NJ :

Prentice-Hall . E GAN , D . E . & S CHWARTZ , B . J . (1979) . Chunking in recall of symbolic drawings . Memory

and Cognition , 7 , 149 – 158 . E INHORN , H . J . & H OGARTH , R . M . (1981) . Behavioral decision theory : processes of

judgment and choice . Annual Re y iew of Psychology , 32 , 52 – 88 . F AYAD , M . E ., T SAI , W . & F ULGHUM , M . L . (1996) . Transition to object-orientated software

development . Communications of the ACM , 39 , 108 – 121 . F ICHMAN , R . G . & K EMERER , C . F . (1992) . Object-oriented and conventional analysis and

design methodologies : comparison and critique . IEEE Computer , 25 , 22 – 39 . F ICHMAN , R . G . & K EMERER , C . F . (1993) . Adoption of software engineering process

innovations : the case of object orientation . Sloan Management Re y iew , Winter , 7 – 22 . G ANE , C . & S ARSON , T . (1979) . Structured Systems Analysis : Tools and Techniques .

Englewood Clif fs , NJ : Prentice-Hall . G ENTNER , D . & S TEVENS , A . L . (1983) . Mental models . Hillsdale , NJ ; Lawrence Erlbaum . G RAESSER , A . C . (1981) . Prose Comprehension Beyond the Word . New York , NY :

Springer-Verlag . J ACKSON , M . (1983) . System De y elopment . Englewood Clif fs , NJ : Prentice-Hall . J ACOBSON , I ., C HRISTERSON , M ., J ONSSON , P . & O VERGAARD , G . (1992) . Object - Oriented

Software Engineering : A Use Case Dri y en Approach . New York , NY : ACM Press / Addison-Wesley .

K ERLINGER , F . N . (1986) . Foundations of Beha y ioral Research , 3rd Edition . Chicago , IL : Holt , Rinehart and Winston .

K ESSLER , C . M . & A NDERSON , J . R . (1986) . Learning flow of control : recursive and iterative procedures . Human – Computer Interaction , 2 , 135 – 166 .

K OZACZYNSKI , W . & K UNTZMANN -C OMBELLES , A . (1993) . What it takes to make OO work . IEEE Software , January , 20 – 23 .

L ARKIN , J ., M C D ERMOTT , J ., S IMON , D . & S IMON , H . (1980) . Expert and novice performance in solving physics problems . Science , 208 , 140 – 156 .

L IU , C ., G OETZE , S . & G LYNN , B . (1992) . What contributes to successful object-oriented learning? OOPSLA ’ 9 2 , Vancouver , Canada , pp . 77 – 86 .

M C K EITHEN , K . B ., R EITMAN , J . S ., R UETER , H . H . & H IRTLE , S . C . (1981) . Knowledge organization and skill dif ferences in computer programmers . Cogniti y e Psychology , 13 , 307 – 325 .

M ORAN , T . P ., (1981) . An applied psychology of the user . Computing Sur y eys . 13 , 121 – 141 . N EWELL , A . & S IMON , H . A . (1972) . Human Problem Sol y ing . Englewood Clif fs , NJ :

Prentice-Hall . P OLSON , P . G . (1987) . A quantitative theory of human – computer interaction . In J . M .

Carroll , Ed . Interfacing Thought . Cambridge , MA : MIT Press . P OLSON , P . G ., B OVAIR , S . & K IERAS , D . (1987) . Transfer between text editors . CHI ‘87

Proceedings , Toronto , Canada , pp . 27 – 32 . P RESSMAN , R . (1987) . Software Engineering : A Practitioner ’ s Guide . New York , NY :

McGraw-Hill . P RESSMAN , R . (1988) . Software Engineering : A Beginner ’ s Guide . New York , NY : McGraw-

Hill . R UMBAUGH , J ., B LAHA , M ., P REMERLANI , W ., E DDY , F . & L ORENSEN , W . (1991) .

Object - Oriented Modeling and Design . Englewood Clif fs , NJ : Prentice-Hall . S CHANK , R . C . & A BELSON , R . P . (1977) . Scripts , Plans , Goals , and Understanding . Hillsdale

NJ : Lawrence Erlbaum . S CHEFFE , H . (1953) A method for judging all contrasts in the analysis of variance . Biometrika ,

40 , 87 – 104 . S CHOLTZ , J . & W IEDENBECK , S . (1990) . Learning second and subsequent programming

languages : a problem of transfer . International Journal of Human – Computer Interaction , 2 , 51 – 72 .

S CHWEIGER , D . M ., A NDERSON , C . R . & L OCKE , E . A . (1985) . Complex decision making : a longitudinal study of process and performance . Organizational Beha y ior and Human Performance , 36 , 245 – 272 .


S HNEIDERMAN , B . (1976) . Exploratory experiments in programmer behavior . International Journal of Computer and Information Science , 5 , 123 – 143 .

S INGLEY , M . K . & A NDERSON , J . R . (1985) . The transfer of text-editing skill . International Journal of Man – Machine Studies , 22 , 403 – 423 .

S INGLEY , M . K . & A NDERSON , J . R . (1988) . A keystroke analysis of learning and transfer in text editing . Human – Computer Interaction , 3 , 223 – 274 .

S INHA , A . P . & V ESSEY , I . (1992) . Cognitive fit : an empirical study of recursion and iteration . IEEE Transactions on Software Engineering , 18 , 368 – 379 .

S LOVIC , P . & L ICHTENSTEIN , S . (1983) . Preference reversals : a broader perspective . American Economic Re y iew , 73 , 596 – 605 .

S OLOWAY , E . (1986) . Learning to program 5 learning to construct mechanisms and explana- tions . Communications of the ACM , 29 , 850 – 858 .

S OLOWAY , E ., A DELSON , B . & E HRLICH , K . (1988) . Knowledge and processes in the comprehension of computer programs . In M . T . H . C HI , R . G LASER & M . J . F ARR , Eds . The Nature of Expertise . Hillsdale , NJ : Lawrence Erlbaum .

S OLOWAY , E ., B ONAR , J . & E HLRICH , K . (1983) . Cognitive strategies and looping constructs : an empirical study . Communications of the ACM , 26 , 485 – 492 .

S OLOWAY , E . & E HLRICH , K . (1984) . Empirical studies of programming knowledge . IEEE Transactions on Software Engineering , 5 , 595 – 609 .

T VERSKY , A ., S ATTATH , S . & S LOVIC , P . (1988) . Contingent weighting in judgment and choice . Psychological Re y iew , 95 , 371 – 384 .

V ESSEY , I . (1991) . Cognitive fit : a theory-based analysis of the graphs versus tables literature . Decision Sciences , 22 , 219 – 240 .

V ESSEY , I . & G ALLETTA , D . (1991) . Cognitive fit ; an empirical study of information acquisition . Information Systems Research , 2 , 63 – 84 .

V ESSEY , I . & C ONGER , S . (1993) . Learning to specify information requirements : the relationship between application and methodology . Journal of Management Information Systems , 10 , 177 – 201 .

V ESSEY , I . & C ONGER , S . (1994) . Requirements specification : learning object , process , and data methodologies . Communications of the ACM , 37 , 102 – 113 .

V ESSEY , I . & W EBER , R . (1986) . Structured Tools and Conditional Logic : An Empirical Investigation‘‘ , Communications of the ACM , 29 , 48 – 57 .

W HITESIDE , J ., J ONES , S ., L EVY , P . S . & W IXON , D . (1985) . User performance with command , menu , and iconic interfaces . CHI ’85 Proceedings : Human Factors in Computing Systems , San Francisco , CA , USA , pp . 185 – 191 .

W HITTEN , J ., B ENTLEY , L . & B ARLOW , V . M . (1989) . Systems Analysis and Design Methods , 2nd Edn . Homewood , IL : Irwin .

W U , Q . & A NDERSON , J . R . (1993) . Strategy choice and change in programming . International Journal of Man – Machine Studies , 39 , 579 – 598 .

Y ADAV , S . B ., B RAVOCCO , R . R ., C HATFIELD , A . T . & R AJKUMAR , T . M . (1988) . Comparison of analysis techniques for information requirement determination . Communications of the ACM , 31 , 1090 – 1097 .

Paper accepted for publication by Associate Editor , Professor M . Harrison .

Appendix A : Experimental tasks

T1 : PROCESS-ORIENTED TASK

XYZ company — accounts payable system XYZ company is a retail jewelry organization consisting of a main store and three branch stores . The company sells products such as watches , gold chains , diamond rings , and other precious stones . The company deals with over 120 vendors to supply its four store outlets . The financial record keeping and check payment activities of this company are the responsibility of two individuals that reside in the main of fice .


Merchandise from vendors is received at the branch stores and all associated vendor invoices are forwarded to the main store . The bookkeeper checks all invoices for the sales person’s signature to verify that the shipment was received . The invoice information is then entered simultaneously (using carbon paper) on to the purchase journal and vendor ledger card . The purchase journal is organized by branch store and the ledger cards by vendor . The invoices are then filed into an outstanding invoice file .

Approximately every 10 days , the bookkeeper searches the outstanding invoice file and removes any invoices with pending due dates within the next 10 days . These invoices are placed in the pending payment file . Checks are prepared and recorded in the check book register for those that need payment . The matching paid invoice numbers are written on the checks and the check numbers are written on to the vendor ledger . The bookkeeper submits the prepared checks to Mr . Wilson to be signed . Upon receiving the signed checks the payment amount is recorded in the vendor ledger and the checks are mailed . The corresponding paid invoices are attached to the carbon copy of the check and filed in the paid invoice file .

When monthly statements are received from a vendor , the bookkeeper compares them with the outstanding invoice file and paid invoice file . Completely reconciled statements are discarded . Statements that are not completely reconciled by this comparison are placed in the ‘‘troublesome statement’’ file . These statements are then reconciled by contacting the branch stores to locate missing invoices or verify receipt of merchandise .

T2 : OBJECT-ORIENTED TASK

Ri y erside Uni y ersity : benefits systems Several persons are associated with Riverside University , which of fers undergraduate and graduate programs in dif ferent areas . The faculty and staf f members constitute the employees of the University . The faculty body is comprised of Assistant Professors , Associate Professors and Professors . Secretaries , clerks and of ficers constitute the staf f . The student body is comprised of undergraduate and graduate students . Some graduate students are employed as Graduate Assistants ; they get a monthly stipend and a full tuition waiver in return for their duties .

The Human Resources and Benefits (HRB) department maintains pertinent information for all employees of the school . Periodically , it issues dif ferent reports to employees , keeping them informed on pension , health insurance and life insurance benefits . All employees are eligible for health insurance benefits . Graduate assistants , however , are not eligible for pension and life insurance benefits .

Reports on retirement pension funds are issued to employees quarterly ; such a report contains information about the employee’s contribution , the school’s contribution , and the total accumulation of funds in the employee account . Employees are required to contribute a fixed percentage of their salaries ; the school contributes an equal amount . While the 12-month salary is used as the basis for computing pension contributions for a staf f member , the 9-month salary is used as the basis for faculty members . Any extra money earned by a faculty during the 3-month summer term is not considered for computing pension contributions . Since employee contributions toward pension are tax-deferred , the HRB department issues a report to the payroll of fice annually , listing the names and contributions of all employees .


The HRB department issues a life insurance report annually to each employee ; this report contains information on the employee’s life-insurance amount and insurance premium . The life insurance amount is determined from the employee’s salary ; premiums , which depend on this amount and the employee’s age , are paid in full by the school .

During the Open Enrollment period , held once a year , employees have the option of making changes to their current health insurance benefits by filling out a form at the HRB of fice . If an employee does not submit this form , his / her benefits continue to be the same as the current benefits . Health insurance premiums are shared by the employee and the school . The HRB department issues a report to the payroll department listing the names of all employees signed up for health insurance , along with the amounts to be withheld from their salaries for making premium payments .

Appendix B : task features used for scoring

TASK T1

Structure :

Major Objects : PAID-INV-FILE , PEND-PAY-FILE , OUTST-INV-FILE , CHECK-BK-RGR , PURCH-JNL , VEND-LGR , TRBL-STMT-FILE

Minor Objects : MAIN / BRANCH , BKEEPER / WILSON , INVOICE

Attribute presence and Match : check 4 , check – amt (within CHECK-BK-REGR)

Minor Classification / Assembly Relationships : BKEEPER and WILSON are subclasses of EMPLOYEE . PAID-INV-FILE and PEND-PAY-FILE are subclasses of FILE . MAIN and BRANCH are part of XYZ .

Behavior :

Major Ser y ices : forward – inv , check – inv , enter – inv – info , file – inv , id – outst – inv , file – pending – pay , prep – check , record – paymt , file – paid – inv , comp – files , file – trbl – stmt , recon – trbl – smt

Minor Ser y ices : sign – check , mail – check

Ser y ice - Object Match : Placement of ser y ices in appropriate objects .

Sequencing of Ser y ices : The PROCESS – INVOICE operation requests 5 services by sending the following messages in sequence :

1 . send – msg forward – inv INVOICE 2 . send – msg check – inv INVOICE


3 . send – msg enter – inv – inf PURCH – JRNL 4 . send – msg enter – inv – inf VENDOR – LGR 5 . send – msg file – outst – inv OUTST – INV

The PROCESS – PAYMENT operation requests 8 services by sending the following messages in sequence :

1 . send – msg id – oust – inv OUTST – INV 2 . send – msg file – pend – pay PENDING – PAY 3 . send – msg prepare – checks 4 . send – msg record – pymt CHK – BK – RGR 5 . send – msg sign – checks WILSON 6 . send – msg record – pymt VENDOR – LGR 7 . send – msg mail – check 8 . send – msg file – paid – inv PAID – INV

The RECON – MNTHLY – STMT operation requests 3 services by sending the following messages in sequence :

1 . send – msg compare – files 2 . send – msg file – trbl – stmt TRBL – STMT 3 . send – msg recon – trbl – stmt BRANCH

Total number of messages passed : 16

TASK T2

Structure :

Major Objects : EMPLOYEE , FACULTY , STAFF , STUDENT , GA , HRB , PAYROLL , REPORT , GA

Minor Objects : ASST – PROF , ASSOC – PROF , FULL – PROF , SECRETARY , CLERK , OFFICER , GRAD , UGRAD

Attribute Presence and Match : emp – id , name , salary , school – contrib , emp – contrib , emp – age (within EMPLOYEE) salary , extra – inc (within FACULTY) salary* , school – contrib* , emp – contrib* , stipend (within GA) stu – id , stu – name (within STUDENT)

Major Classification / Assembly Relationships : FACULTY , STAFF , and GA are subclasses of EMPLOYEE . GRAD and UGRAD are subclasses of STUDENT .

Minor Classification / Assembly Relationships : ASST – PROF , ASSOC – PROF , and FULL – PROF are subclasses of FACULTY . SECRETARY , CLERK , and OFFICER are subclasses of STAFF . GA is a subclass of GRAD . STUDENT , EMPLOYEE , and DEPARTMENT are part of RIVERSIDE UNIV .


Behavior :

Major Ser y ices : issue – pen , issue – life , issue – health , calc – pen , calc – life , calc – health

Minor Ser y ices : Other services such as issue – payroll – report .

( Detailed Computations ) : Details of computations , such as those for calculating pension contributions .

Ser y ice - Object Match : Placement of ser y ices in appropriate objects .

The role of prior experience and task characteristics in object-oriented modeling: an empirical...

Documents

Transcript of The role of prior experience and task characteristics in object-oriented modeling: an empirical...