HighBeam Research

Title: Investigating interrelationships among sales training evaluation methods.

Date: 9/22/2003; Publication: Journal of Personal Selling & Sales Management; Author: Liu, Annie H.

Today numerous training options are available to sales organizations, and sales training teams use various means to report training effectiveness. This study utilizes Kirkpatrick's (1959; 1960) training evaluation model and examines the interrelationships among its four levels of sales training evaluation (i.e., reactions, knowledge acquisition, behavior change, and organizational outcomes). Empirical results indicate that sales trainees' use of training materials at work is positively related to achieving sales training outcomes, including improving (1) organizational commitment, (2) sales effectiveness, and (3) customer relations. Furthermore, trainees who had positive reactions to training were more likely to learn the material, and trainees with higher levels of knowledge retention were more likely to apply the material in the work environment. Implications are discussed that may aid sales firms to better evaluate training solutions provided by vendors and to develop more effective and accountable sales training efforts.

U.S. companies spend $7.1 billion annually on training salespeople, and an average salesperson spends more than 33 hours per year in training (Lorge 1998)--73.4 days for an average entry-level rep (Kaydo 1998). In technical markets (e.g., computers, imaging systems, chemicals, etc.), the costs associated with the development of just one salesperson is often in excess of $100,000 and, because of high initial costs, it may take as long as two years before any profit is realized from a new sales hire (Johnston and Marshall 2003).

Companies spending significant amounts on sales training are increasingly required to formally justify these expenditures (Phillips 1998). Most sales practitioners agree that the evaluation of training is extremely important and that firms lack understanding regarding how to measure and evaluate training efforts (Erffmeyer, Russ, and Hair 1991; Honeycutt, Howe, and Ingram 1993; Lupton, Weiss, and Peterson 1999). As such, perhaps the most critical issue facing sales force training is how to consistently assess training programs and their outcomes.

In fact, companies use a wide variety of evaluation procedures that range from self-administered reports completed by the trainees, to informal debriefing sessions, to more elaborate calculations of enhanced sales revenue. The diversity among the sales training evaluation procedures has lead researchers to develop more systematic taxonomies to help classify evaluation procedures and identify their more critical elements. Among these, the most widely adopted framework by practitioners (Kirkpatrick 1996) and researchers (Alliger et al. 1997) is the four-stage training evaluation model proposed by Kirkpatrick (1959). Kirkpatrick's framework classifies training evaluations into four categories: (1) participants' reactions to the training program; (2) changes in attitude,

knowledge, or skill levels; (3) changes in work behaviors; and (4) changes in organizationally desired outcomes.

Organizations that are spending resources on sales training want to see that their training expenditures actually aid the firm in reaching its objectives. However, training organizations rarely provide assessments of how training has helped their customers reach these objectives (Honeycutt and Stevenson 1989). In fact, sales training organizations most often provide evaluations at one of the two lowest stages in Kirkpatrick's hierarchy. The American Society of Training and Development (ASTD) recently reported that 78 percent of organizations assess reaction measures, 32 percent assess learning, 9 percent assess behavioral change, and 7 percent assess organizational results (Van Buren and Erskine 2002). This leaves most sales organizations with imprecise information to evaluate whether their investments in training are well spent or largely wasted.

In order to be able to critically evaluate training investments, organizations need to know how reactions, knowledge acquisition, and behavior change impact outcomes. Similarly, organizations need to know the value of measuring training at multiple stages. When firms assess training at higher levels, past research provides little insight into whether efforts to evaluate training at lower stages can provide additional insight or if these efforts are redundant (cf. Warr, Allan, and Birdi 1999).

Prior sales literature has used Kirkpatrick's model to report descriptive statistics of in-house initial sales training evaluation procedures (Honeycutt and Stevenson 1989), and as a base for further conceptual models (Lupton, Weiss, and Peterson 1999). However, these studies have fallen short of assessing the usefulness of, and interrelations among, various levels of evaluation. As such, the purpose of this study is to empirically examine the relationships among these four levels of sales training evaluations, specifically focusing on the interrelationships between early stage evaluations (e.g., Levels 1 and 2) and later stage evaluations (e.g., Levels 3 and 4). In today's ever more competitive market, it is critical to understand how specific measures of sales training effectiveness are or are not interrelated and predictive of outcomes. Armed with this information, firms may be better able to evaluate training solutions provided by vendors and to develop more effective and accountable sales training efforts. To this end, we will review the relevant literature on training evaluation and Kirkpatrick's taxonomy from which we will propose four hypotheses. These hypotheses will be empirically examined and findings discussed along with managerial implications and suggestions for future research.

SALES TRAINING EVALUATION

For over 40 years, Kirkpatrick's framework for classifying training evaluations has guided the efforts of sales-trainers interested in critically evaluating training efforts (Kirkpatrick 1959). When asked why his model is so widely used, Kirkpatrick answers, "It's simple and practical" (1996, p. 55). However, over the years, Kirkpatrick's taxonomy has been criticized for its simplicity (Alliger and Janak 1989) and its limited use in developing propositions (Holton 1996). Recently, researchers have attempted to improve

the original model by identifying more detailed subcategories (Alliger et al. 1997) or adding additional stages (Burrow and Berardinelli 2003). Specifically in sales, Lupton, Weiss, and Peterson (1999) have adopted Kirkpatrick's framework in their five-stage model of sales training evaluation. Likewise, Honeycutt et al. (2001) suggest evaluating the economic return of sales training.

Some researchers have advocated moving away from Kirkpatrick's taxonomic approach and toward the development of more testable models (e.g., Holton 1996; Kraiger, Ford, and Salas 1993). Subsequently, models have been developed that incorporate motivation, work environment, trainee ability (Holton 1996), and cognitive knowledge structures (Day, Arthur, and Gettman 2001). However, Kirkpatrick's model continues to be the most popular method of representing training evaluation criteria (Arthur et al. 2003; Salas and Canon-Bowers 2001; Van Buren and Erskine 2002), and, thereby, still serves as a critical typology for communicating understandings about training evaluation.

Level 1--Reactions

Reactions are measures of how participant salespeople feel about various aspects of a sales training program. As such, reactions are similar to traditional measures of customer satisfaction. It is assumed that when salespeople dislike a training program, little effort will be put forth to learn and use material. Likewise, if sales trainees enjoy the training, they will learn more, and be more motivated to use the material and participate in future training sessions (Warr, Allan, and Birdi 1999).

Although reaction measures are the most widely applied evaluation criteria, reaction measures may not be a strong indicator of effective training (Tannenbaum and Yukl 1992). In sales, Lupton, Weiss, and Peterson (1999) provide several criticisms of the use of reactions to evaluate sales training. They state that reactions to training can be unduly influenced through extraneous factors, such as (1) training venue (e.g., the beach), (2) trainer personality, and (3) whether or not sales trainees value being away from work. They also note that reactions are often measured in the form of observation, discussion, and debriefing meetings. Each of these has questionable validity given the ease in which an instructor, sales manager, or other stakeholder might be able to influence the reaction measure. However, past management research has found that reactions do lead to learning or knowledge acquisition, which is Level 2 (Alliger et al. 1997; Warr, Allan, and Birdi 1999).

Level 2--Knowledge

Training evaluation at the second level assesses the acquisition and retention of declarative knowledge, procedural knowledge, or attitude change depending upon the objectives of the training program (Lupton, Weiss, and Peterson 1999). When training objectives focus on providing information to salespeople (e.g., product information, competitor information, market dynamics, and economic conditions), trainee learning is often assessed using pencil and paper examinations. When training objectives involve the teaching of selling skills or other forms of procedural knowledge (e.g., handling

objections, negotiation tactics), measurements may use a combination of pencil and paper examination instruments along with behavioral evaluations (e.g., judged role-plays). However, many forms of knowledge assessment could fit under this label (Alliger et al. 1997). For example, an assessment of the accessibility and accuracy of a salesperson's mental models (i.e., scripts and schemas) is a type of knowledge assessment (Leigh and McGraw 1989; Szymanski and Churchill 1990), albeit relatively newer and much less common than typical measures of knowledge (Kraiger, Ford, and Salas 1993). In all cases, however, Level 2 assessments attempt to evaluate whether or not sales trainees acquired and retained the training material.

To measure knowledge level in management studies, primarily two approaches have been taken. Researchers either (1) assess a difference score, such as changes from pre-training to post-training (e.g., Bretz and Thompsett 1992; Warr, Allan, and Birdi 1999), or (2) they look merely at post-training assessments (e.g., Davis and Mount 1984; Warr and Bunce 1995). In many sales organizations, the firm is only concerned with the extent to which salespeople have reached certain levels of attainment, rather than being interested in the degree of learning that has taken place during a sales training program. However, prior differences often exist among trainees with respect to the level of competence each brings to training (Dubinsky et al. 1986). Thus, when investigating the relationships among knowledge, behavior change, and organizational outcomes, it is desirable to assess knowledge as a change or gain score, or as an assessment of the knowledge acquired (Warr, Allan, and Birdi 1999).

Level 3--Behavior Change/Learning Transfer

Behavior change assessments measure the extent to which salespeople modify their job-related behavior due to training. This level of evaluation is also referred to as the "transfer of learning" (Kirkpatrick 1994). Training expenses are most often believed to be wasted because of inadequate transfer (Lorge 1998; May and Kahnweiler 2000). Past frameworks suggest that transfer is a function of three factors: work environment, trainee characteristics, and learning retention (Baldwin and Ford 1988). Specifically, Lupton, Weiss, and Peterson (1999) state that, in order for a "transfer of learning" to take place: (1) trainees must be motivated to apply the material, (2) trainees must have the knowledge of how and what to do, (3) sales management must provide a salutary environment, (4) sales management must encourage and help the trainees to apply the sales training material, and (5) sales management must reward the desired behavior change.

Learning transfer assessments most often involve the direct observation of the sales trainee in the workplace. Thereby, behavior change assessments are rarely measured by outsourced training firms, as sales trainers typically do not have access to observe trainees' sales calls. When behavior change assessments are taken, they are usually measured by (1) sales managers observing calls during "ride-alongs," (2) using self-reported behavior change measures, or (3) content analyzing salesperson diaries and logs. This latter technique may become more prevalent as sales organizations adopt commercial customer relationship management (CRM) software packages that make

access to salespeople's field logs more easily available. Because behavioral outcomes are often susceptible to extraneous environmental influences (e.g., supervisors' attitude, work environment), past studies have had difficulties linking training to desired behavioral changes (Attia, Honeycutt, and Atria 2002).

Level 4--Organizational Outcomes

Level 4 evaluations measure the extent to which a training program has aided the attainment of objectives set forth by the organization. Firm-level training outcomes are generally perceived as the most tangible and objective measures of training effectiveness; however, most researchers consider level four evaluations to be the most difficult to attain (Kirkpatrick 1994; Lupton, Weiss, and Peterson 1999). In fact, measurement difficulties are cited as the primary reason for the lack of research in this area (Warr, Allan, and Birdi 1999). Three factors may account for potential problems. First of all, the autonomy of salespeople has made evaluating their behaviors in the field notoriously difficult. Second, it is often impossible to single out extraneous influences when measuring the effects of a training program on final outcomes (e.g., changing economic conditions, marketing programs, and competitive actions). Third, bottom-line tangible measures and actual dollar contributions of a sales training program are difficult to calculate. For example, evaluators find it problematic to assign dollar values to measured improvements in customer satisfaction or salesperson loyalty.

Training objectives can be very specific to individual training programs (Lupton, Weiss, and Peterson 1999), and specific training objectives have been found to be useful at guiding training efforts (Johnston and Marshall 2003). However, most sales training efforts have similar broad general organizational objectives (Honeycutt, Attia, and Maurer 2001; Honeycutt, Howe, and Ingram 1993). Honeycutt, Howe, and Ingram (1993) systematically developed a list of common sales training objectives by soliciting key objectives from sales managers, sales trainers, and salespeople. Six organizational sales training objectives were identified: decrease turnover, improve customer relations, decrease selling costs, improve control of the sales force, increase sales volume, and improve use of time. Of these six, we have adapted three to represent general training outcomes that are directly under a salesperson's locus of control. These three general organizational outcomes will be investigated in this study and include: (1) to improve organizational commitment and decrease turnover, (2) to improve selling effectiveness, and (3) to improve customer relations. These outcomes are relevant to a variety of sales organizations and applicable whether a training program aims to focus on selling skills, knowledge components, or self-management skills.

MODEL AND HYPOTHESES DEVELOPMENT

One of the strengths of Kirkpatrick's (1959) taxonomy is the implicit hierarchical nature of his four stages. That is, variables representing one level of evaluation directly impact the next level in the hierarchy. As such, it is expected that reactions lead to knowledge acquisition, knowledge acquisition leads to behavior change, and behavior change leads to training outcomes. Anecdotal evidence suggests that "edu-tainment" is valuable and

people are more apt to pay attention and learn when they enjoy the process (Level 1 [right arrow] 2). Models of learning transfer (Baldwin and Ford 1988; Lupton, Weiss, and Peterson 1999) incorporate knowledge as a key predictor (Level 2 [right arrow] 3). And, frameworks for training development (Goldstein 1986) advocate that programs be designed so that learning transfer facilitates the attainment of organizational objectives (Level 3 [right arrow] 4).

Alliger et al. (1997) conducted a meta-analectic investigation of the relationships among the first three stages of Kirkpatrick's framework in the management literature (lack of published studies reporting correlations among outcomes and other stages prevented them from investigating Level 4). Their findings suggest that modest correlations exist between the various types of training criteria. Specifically, reactions were found to correlate positively with knowledge acquisition (Levels 1 and 2), whereas both reactions and knowledge acquisition correlated with transfer (Levels 1 and 3 and Levels 2 and 3). However, given that the strength of the correlations were small, the authors cautioned that measures from one level should not be used as surrogates of other measures. More recently, Warr, Allan, and Birdi (1999) examined the relations among the first three levels of Kirkpatrick's framework for automobile technician training. Their study found a direct positive relationship between training reactions and knowledge acquisition (Level 1 [right arrow] 2), and between knowledge acquisition and behavior change (Level 2 [right arrow] 3).

Although similarities exist across all forms of employee training, sales training and technical training differ in numerous ways. This is particularly true of cognitively based interpersonal skill development training, such as active listening, negotiation skills, and selling skills training (Broad and Newstrom 1992; Georges 1996; Gist, Bavetta, and Stevens 1990). Thus, although prior management literature tends to support the hierarchical nature of Kirkpatrick's framework, an investigation of these relationships in the context of sales training would be valuable to marketers, sales trainers, and sales managers. We therefore hypothesize hierarchical linear relationships among the four levels of Kirkpatrick's framework as follows:

H1: A direct positive relationship exists between trainees' reactions to a training program and the level of knowledge retained from that program (Level 1 [right arrow] 2).

H2: A direct positive relationship exists between the knowledge retained from a training program and level of behavior change resulting from the transfer of learning (Level 2 [right arrow] 3).

H3a; H3b; H3c: A direct positive relationship exists between trainees' behavior change and each of the three general organizational objectives of training: (a) organizational commitment, (b) selling effectiveness, and (c) customer relations (Level 3 [right arrow] 4).

Aside from the linear relationships among the levels of training evaluation, researchers have examined the direct effect of trainees' reactions on subsequent behavior change

(Level 1 [right arrow] 3) (e.g., Clark, Dobbins, and Ladd 1993; Tannenbaum et al. 1991). Alliger et al. (1997) and Morgan and Casper (2000) suggest that, by measuring several aspects of a trainee's level of satisfaction with a training program, reactions often assess the perceived utility of a program. When trainees equate utility with the ability to help them attain their goals, reactions may capture a trainee's motivation to transfer learning. Clark, Dobbins, and Ladd (1993) find that trainees who perceived training to have more job and career utility were more motivated. Likewise, Tannenbaum et al. (1991) provide evidence that, when training meets or fulfills expectations and desires, trainees are more motivated to transfer learning. These findings are consistent with expectancy theory and goal-setting theory (Yamnill and McLean 2001). Trainees who (1) perceive that their efforts will lead to rewards they value, and (2) perceive what they have learned is relevant to the attainment of their goals, will be more motivated to transfer learning to the work environment. Subsequently, they will be more likely to favorably alter their work behaviors. As such, we propose that reactions should have a direct effect on the level of behavior change in addition to the indirect effect through knowledge acquisition. In other words, the relationship between reaction and behavior change (Level 1 [right arrow] 3) is only partially mediated by sales trainees' knowledge (Level 2).

H4: A direct positive relationship exists between trainees' reactions to a training program and the level of behavior change resulting from training (Level 1 [right arrow] 3), and this relationship is in addition to the indirect relationship through the level of trainees' knowledge acquisition (Level 2).

The relationships expressed in our hypotheses are portrayed in Figure 1.

[FIGURE 1 OMITTED]

RESEARCH DESIGN

Sample Selection

Salespeople within the life insurance underwriting industry were selected to examine perceptions of training effectiveness in this study, and a national mail survey was conducted. A firm specializing in the training of life insurance sales personnel provided the mailing list. A sample of 2,000 was randomly generated from its client list of 47,807 salespeople who participated in one of its training programs during the past year.

Respondents were asked to answer all questions as they pertain to their latest sales training course. Respondents were guaranteed confidentiality and assured that their responses would be used for academic purposes. No incentive was given; however, an executive summary of research findings was offered if requested. Among the 2,000 surveys, 61 questionnaires were returned undeliverable, and 445 useful responses were returned, for a response rate of 23 percent.

Responding salespeople had an average 14 years of sales experience (59 percent having between 5 and 20 years). Respondents were salespeople who handle both consumer and

business accounts (only 5 percent deal exclusively with business clients and only 4 percent deal exclusively with consumers). On average, 68 percent of clients were consumers and 32 percent of clients were businesses accounts. Respondents were 79 percent male and 21 percent female. All reported attending a training program in the previous year indicating that their most recent training program covered selling skills (80 percent), selling knowledge (81 percent), or self-management skills (45 percent). Mean comparisons using independent sample t-tests show no significant differences between early and late respondents on any of the sample characteristics. By using late respondents as a proxy for nonresponding salespeople, a level of certainty can be attained that the level of nonresponse bias is low (Armstrong and Overton 1977).

Scale Development

Items were developed to measure salesperson perceptions of training effectiveness. Both Kirkpatrick's (1959; 1960) four-stage training evaluation model and Honeycutt, Howe, and Ingram's (1993) classification of sales training objectives were utilized to construct measurement items. Multi-item scales were developed for each construct (Anderson and Gerbing 1988). All constructs were measured with three-item scales except for the "behavior change" construct that was measured with four items. All items were constructed using seven-point agree/disagree scales, and items were general in nature in order to assess useful information across respondents participating in various types of training programs. Table 1 provides a complete list of items and scale properties. The measurement of each construct is described below.

Measures

Reactions to sales training was conceptualized as a post-training evaluation of the training program. Items for this scale were developed by adapting traditional measures of customer satisfaction (Oliver and Swan 1989). Items focused on value-adding aspects of the training. Therefore, items were developed to assess satisfaction with instruction, satisfaction with course content, and general course satisfaction (Morgan and Casper 2000).

Perceived knowledge retained from sales training was conceptualized as a salesperson's perception that the material covered in the training course was learned and remembered (Warr, Allan, and Birdi 1999). Items assessed trainee perceptions of how much was learned, how much was remembered, and how much was forgotten.

Perceived behavior change (learning transfer) was conceptualized as a salesperson's perception that the material covered in the training course was used or applied in the workplace. Generally speaking, for sales organizations interested in measuring training-related changes in behavior, two approaches are available and each brings with it a set of methodological concerns. First, salespeople can self-report the usage of training material on the current job. Second, a sales manager or evaluator can record key behaviors before and after training to calculate the job-related behavior change (Warr, Allan, and Birdi

1999). This study employs the former approach, in which items assessed the salesperson's perceived level of the transfer of learning to the workplace.

Firm-level training outcomes were assessed by asking respondents to report the perceived efficacy of a training program at reaching each of three general organizational outcomes. Heavily commissioned salespeople have been shown to be extremely value-oriented toward time spent out of the field and in training (Johnston and Marshall 2003). As such, they are uniquely capable of providing accurate assessments for a variety of training outcomes. For this study, items were developed to measure three organizational outcomes. Organizational commitment was conceptualized as the perceived efficacy of the training to enhance the level of organizational commitment the salesperson feels for his or her employer. Selling effectiveness was conceptualized as the perceived efficacy of the training at improving the salesperson's selling. This included enhanced sales volume and a reduction in selling costs. Customer relations was conceptualized as the perceived efficacy of the training program at improving customer relations.

Analytical Procedures and Results

Structural equation modeling was used to validate the measurement of theoretical constructs and to test the hypothesized relationships among constructs. Prior to estimating the structural model, a measurement model (i.e., confirmatory factor analysis) was estimated to evaluate the appropriateness of item indicators (Anderson and Gerbing 1988). The measurement model was specified so items loaded on only the construct they were developed to represent. To specify the scale of each latent construct, one item indicator per construct was set to unity (Joreskog and Sorbom 1979).

Initial confirmatory factor analysis (CFA) results for the measurement model indicated adequate fit ([chi square] = 432.19, degrees of freedom [df] = 137, comparative fit index [CFI] = 0.95, Tucker-Lewis index [TLI] = 0.94) providing support for the appropriateness of the relationship between manifest indicators and their intended latent constructs. To ensure a high degree of homogeneity and unidimensionality of items measuring each construct in the measurement model, modification indices as well as the standardized residual matrix were examined. Judicious re-specification of measurement constraints was made in order to achieve an acceptable fit for the measurement model (Anderson and Gerbing 1988; James, Mulaik, and Brett 1982). As a result, one item from the set of items representing the construct "perceived retained knowledge" was removed. The resulting measurement model fit statistics were: [chi square] = 286.58, df = 120, CFI = 0.97, TLI = 0.96.

Table 2 provides descriptive statistics of the measurement scales and the correlation matrix for the seven latent constructs. To evaluate the discriminant validity among latent constructs, additional models were estimated fixing the correlation between each pair of constructs to unity, one pair at a time. The resulting significant chi-square difference tests between each of these models and the final measurement model provided evidence of discriminant validity among latent constructs (e.g., Burnkrant and Page 1982; Ping 1994).

After evaluating the measurement model, the theorized structural model was estimated. In order to impose the hypothesized pattern of relationships among the latent constructs, structural parameters were systematically allowed to vary freely or were fixed to zero. To ensure that relationships among the four training outcome variables are not inappropriately constrained to zero, latent error terms of these constructs were allowed to freely covary. The theorized structural model was found to have good fit ([chi square] = 354.07, df = 126, CFI = 0.96, TLI = 0.95).

To evaluate common method bias we included a "method factor" following the procedures described by Podsakoff et al. (1990) and previously used in marketing studies by Netemeyer et al. (1997) and Brashear et al. (2003). This extra factor was specified as having all 18 self-report measures as indicators, and thus captures all additional systematic variance common to the measurement items (i.e., common methods, social desirability, acquiescence, etc.). The overall fit of a model with freely estimated "method factor" parameters ([chi square] = 184.2, df = 108, CFI = 0.99, TLI = 0.98) was significantly better than one with 14 method parameters fixed to zero (all "method factor" parameters except those with the four behavior change items), suggesting that a method factor is evident ([DELTA][chi square] = 100.2, [DELTA]df = 14, p < 0.01). Therefore, further hypothesis testing was conducted using a model that incorporates the common method factor, thereby statistically controlling for same-source variance (Podsakoff et al. 1990). Table 3 provides model fit statistics and path coefficients.

To statistically test the hypotheses that freed parameters were not zero, parameter z-values were evaluated. The results provide support for each of the hypothesized paths except for the parameter from reactions to behavior change (H4; [gamma] = 0.11, p = 0.13). Findings support the hypothesized positive relationship between a salesperson's training reactions and his or her level of knowledge acquisition (H1; [gamma] = 0.42, p < 0.01). Similarly, the level of knowledge acquisition was positively related to perceived behavior change (H2; [beta] = 0.50,p < 0.01). Behavior change was related to each of the four training outcomes (i.e., organizational commitment [H3a; [beta] = 0.20, p < 0.01], selling effectiveness [H3b; [beta] = 0.32, p < 0.01], and customer relations [H3c; [beta] = 0.22,p < 0.01]). Therefore, H1, H2, H3a, H3b, and H3c are supported.

To further investigate the relationship between salespeople's reactions to their training and changes in work behaviors (H4; level 1 [right arrow] 3), nested structural models were examined to determine whether this relationship was an indirect relationship or if it was completely mediated by the salesperson's level of learning. Three conditions must be in place in order for a relationship between two variables to be completely mediated by a third variable. First, there must be a relationship between the antecedent variable and the outcome variable (i.e., reactions and behavior change). Second, there must be a relationship between the antecedent variable and the mediating variable (i.e., reactions and knowledge retention). Third, once the variance in the outcome variable associated with the mediating variable is accounted for, the antecedent variable must explain no further variance in the outcome variable (Baron and Kenny 1986). Findings from the hypothesized structural model provide evidence fulfilling two of these conditions; they indicate that there is a relationship between reactions and knowledge retention (Level 1

[right arrow] 2), and that there is not a relationship between reactions and behavior change (Level 1 [right arrow] 3) when knowledge retention (Level 2) is accounted for. Thus, to examine the relationship between reactions and behavior change without controlling for the level of knowledge retained by the sales trainee, a model constraining the parameter between knowledge and behavior change to zero was estimated. As expected, this model fit the data significantly worse than the hypothesized model ([DELTA][chi square] = 45.2, [DELTA]df = 1, p < 0.01). However, the parameter between reactions and behavior change was significantly different from zero ([gamma] = 0.24,p < 0.01). This suggests that there is a relationship between reactions and behavior change (Level 1 [right arrow] 3), however, this relationship is completely mediated by knowledge retention (Level 2). As such, H4 is not supported.

A similar procedure was conducted to evaluate the mediating role of behavior change (Level 3) on the relationship between knowledge retention and organizational outcomes (Level 2 [right arrow] 4). The hypothesized model provides evidence of a relationship between the level of knowledge retention and behavior change. However, to examine the direct relationships between knowledge and organizational outcomes, an additional model was estimated. This model was identical to the hypothesized structural model except that the parameters between knowledge retention and the three organizational outcome variables were freely estimated, and the three parameters between behavior change and the organizational outcome variables were fixed to zero. This model indicated that knowledge retention was directly related to organizational commitment ([beta] = 0.16, p < 0.01), but not selling effectiveness ([beta] = 0.16, p = 0.13) or customer relationship development ([beta] = 0.12, p = 0.28). Moreover, a less constrained model was specified to allow the three parameters between behavior change and the organizational outcome variables to be freely estimated. Findings from this model indicated that the relationship between knowledge retention and organizational commitment was nonsignificant ([beta] = 0.08, p = 0.36) when accounting for behavior change. Thus, the positive relationship between a salesperson's level of knowledge retention and commitment to the firm (Level 3 [right arrow] 4) was found to be completely mediated by the level of behavior change (Level 2).

DISCUSSION

Companies invest in the training of their sales force because they believe that the outcome of this training will help attain organizational objectives. Furthermore, marketers want to be

able to determine if training dollars are well spent (Honeycutt et al. 2001). Thus, the current study is the first to empirically examine the interrelationships among the four levels of sales training evaluations, and findings provide initial empirical evidence of the relative importance of each stage at explaining objectives.

Specifically, we find empirical evidence that trainees who had positive reactions to a program were more likely to learn the material (H1; Level 1 [right arrow] 2; see Figure 2). Moreover, trainees whose level of knowledge acquisition was higher were more likely

to transfer learned material to the workplace (H2; Level 2 [right arrow] 3). Furthermore, each of the organizational outcomes that were measured (i.e., improve organizational commitment, improve selling effectiveness, and improve customer relations) was found to be related to the measurement of a salesperson's transfer of learning from a training program to the workplace (H3a, 3b, 3c; Level 3 [right arrow] 4). Thus, findings from this research provide further support that the stages of Kirkpatrick's taxonomy are hierarchical, where higher levels are predicted by lower levels (Alliger and Janak 1989; Alliger et al. 1997; Warr, Allan, and Birdi 1999).

Surprisingly, we find that reactions provide little additional information to explain learning transfer (H4; Level 1 [right arrow] 3) when knowledge retention (Level 2) is accounted for. Although we did find a direct relationship between reactions and learning transfer, this relationship was completely mediated by knowledge. This suggests that when knowledge is assessed, reactions will not provide additional insight into the level of learning transfer; the value of reaction assessments will be largely redundant. However, reactions may be valuable when no other information is available.

Our findings also provide evidence that assessments of learning transfer (Level 3) are the only type of evaluation that can explain organizational outcomes (Level 4). Specifically, knowledge assessments (Level 2) were found to be unrelated to two of the investigated outcome variables (selling effectiveness and customer relationship development), whereas the relationship between knowledge and a salesperson's level of organizational commitment was completely mediated by learning transfer. Thus, findings support the growing body of management literature advocating the critical importance of learning transfer in training design (Arthur et al. 2003; Colquitt, LePine, and Noe 2000; Tracey, Tannenbaum, and Kavanagh 1995).

MANAGERIAL IMPLICATIONS

Because reaction measures are the easiest to collect, it would be highly useful if they could be used as surrogate measures of salesperson learning and transfer. It would also be helpful if reaction measures were highly predictive of desired organizational outcomes. Not surprisingly, our findings suggest a less critical role for reaction measures. In fact, our findings suggest that, for sales organizations wanting to understand training effectiveness in terms of firm-level organizational outcomes, learning transfer measures may be the only type of evaluation to provide valuable insight. This is not to say that trainee reactions are wholly unimportant. Although positive reactions do not guarantee any level of organizational success, negative reactions can often have adverse effects on trainees' motivation to participate in future programs, as well as sales and training department moral (Alliger et al. 1997; Kirkpatrick 1996).

Our findings suggest that transfer is a function of the level of knowledge retained from a training program. In fact, the primary objective of most sales trainers is to effectively provide knowledge and skills to trainees. However, this point of view may impede training effectiveness as trainee learning appears to be a necessary, but not sufficient, prerequisite for behavior change (Tannenbaum and Yukl 1992). Past frameworks suggest

that transfer is a function of knowledge and several additional factors (Baldwin and Ford 1998; Lupton, Weiss, and Peterson 1999).

Anderson and Robertson (1995) find support that sales training can achieve desirable outcomes (cf. Arthur et al. 2003). However, past research on sales training suggests that training objectives are not always formulated by selling organizations (Dubinsky and Hansen 1981; Honeycutt, Howe, and Ingram 1993). When formulated, objectives are often "platitudes rather than real plans for action" (Erffmeyer, Russ, and Hair 1991), which may lead to the development of inefficient training programs. Thus, the effective measurement of the organizational outcomes of training should begin by developing appropriate and specific training objectives.

The findings of this research suggest that assessments of the transfer of learning (Level 3) are relatively diagnostic measures of a training program's success. Thus, once appropriate and specific objectives are set, sales training may become more valuable to the firm when trainers focus their efforts on developing and delivering programs in ways that facilitate learning transfer. Salespeople do not like to look clumsy and artificial to their bosses, peers, or customers, therefore new skills that are inadequately learned in training classes (Level 2) tend to quickly vanish from lack of use (Georges 1996). Thus, for training to be effective at learning transfer (Level 3), trainers should strive to develop procedural knowledge and skills that are learned to mastery and, thereby, more likely to be used in the workplace (May and Kahnweiler 2000). Only when skills are applied on the job will sales trainees meet an organization's goals and increase selling effectiveness, improve customer relations, and increase their organizational commitment.

In sum, the current study suggests that measures of learning transfer are best at helping assess the impact of training on training objectives and organizational goals. Assessments at Levels 1 and 2 are valuable to sales managers in that they may help to explain sales trainee behavior change, but they have limited value when assessing training effectiveness in terms of organizational outcomes. As such, the findings of our study suggest that these lower-level assessments should be evaluated with the end goal of transfer in mind. For example, did the knowledge exam test if a trainee has learned the material to mastery so that he or she will be more apt to use this knowledge in the workplace? Did the reaction assessment instrument measure the trainee's expectations, valences, self-efficacy, or other variables that may enhance their motivation to transfer their learning? Perhaps by assessing sales training efforts with respect to their transfer potential, marketers can more rapidly develop high performance salespeople and see higher returns on training investments.

LIMITATIONS AND FUTURE RESEARCH DIRECTIONS

Researchers are often faced with decisions pertaining to the trade-off between internal and external validity. Sampling multiple organizations across many industries provides greater external validity and may provide a larger variance on the training variables than using a single industry or organization. However, the use of multiple organizations and industries introduces many confounding complexities to the model. Therefore, given the

limited amount of research devoted to concepts included in this study, the use of multiple firms in one industry is thought to be most appropriate. Future research could also aid our understanding of training evaluation and provide insight into the generalizablility of our findings by investigating various industries.

Another limitation is the use of cross-sectional data. The relationships among the multiple levels of training evaluation are proposed to be causal. Although the rationale was provided for the causal direction of the hypotheses, testing the model using cross-sectional data can limit the validity of the findings. Thus, caution is warranted in drawing causal conclusions and future research would benefit from studies utilizing experimental design and longitudinal data.

This study develops scales to measure salesperson perceptions of training effectiveness at each of the four levels. However, measures used in this study are all self-reported perceptions from salespeople. Thus, even though common method effects have been controlled for in this study, findings are susceptible to the reliability and validity of salespeople as sources of information pertaining to how well training is learned, learning transfer, and how training aids organizational objectives. Therefore, further inquiry is justified to further evaluate potential biases and the validity of these scales. Because these scales are specifically relevant to the assessment of sales training, with further psychometric testing and refinement they may become particularly useful tools for sales organizations. One strength of Kirkpatrick's model is that it traditionally moves beyond perception measures. Therefore, comparing self-reported measures to objective measures, measures taken from additional judges, and difference scores (i.e., gain scores) would be particularly useful.

Similarly, how incentive structures and perceived time constraints impact questionnaire response rate and quality is an interesting question for future research. In this study, we use responses from commissioned salespeople. We assume that they would be highly critical of time spent away from the field and in training; thus, particularly good at providing information on training effectiveness. Conversely, however, salespeople who covet their time may be less likely to respond and provide lower-quality responses to questionnaires.

Based on our findings of the importance of learning transfer, future research should examine the efficacy of evaluations that are more "transfer-focused." For example, measures of reactions and knowledge acquisition could be adapted to include items that directly pertain to transfer. We suggest that these "transfer-focused" scales may be particularly diagnostic at predicting learning transfer and organizational outcomes. However, this is an empirical question that deserves further examination. Various reaction-type questions exist and other researchers have classified reaction measures in valuable ways. For example, Alliger et al. (1997) found it helpful to distinguish between affective and utility reactions; however, both were found to aid learning (see also Morgan and Casper 2000). Future research should distinguish among these and other types of reaction measures.

Likewise, future sales training research would benefit by exploring learning transfer. Holton (1996) and Baldwin and Ford (1988) have proposed models that stipulate that learning is only one antecedent to the transfer of learning. Trainees' motivation to transfer is another important variable along with the transfer climate, transfer design, and trainee characteristics.

Table 1 Measurement Items and Properties [alpha] AVE (1) l z Reaction to Training 0.95 0.91 I was very satisfied with the instruction in this class. 0.98 ~ (2) I was very satisfied with the course content. 0.42 9.38 Overall, I was very satisfied with this class. 0.91 30.31 Perceived Knowledge Retained 0.68 0.75 I learned a lot in this class. (3) I have already forgotten almost everything that this class covered. (r) (4) 0.71 ~ I remember almost everything covered in the class. 0.69 10.13 Perceived Behavior Change 0.87 0.72 I have applied the things covered into my work. 0.75 ~ I use almost everything that was covered in my work. 0.76 15.70 I use very little of what was covered in this class. (r) 0.82 17.04 I use the things covered in this class almost every day. 0.85 17.71 Training Outcome--Organizational Commitment 0.92 0.86 This training class has ... made me feel more committed to my company. 0.95 ~ given me a sense of loyalty to my company. 0.91 32.27 made me feel like I will stay with my company for many years. 0.82 25.48 Training Outcome--Selling Effectiveness 0.83 0.74 This training class has ... allowed me to considerably reduce my selling costs. 0.78 ~ helped me to attain a higher sales volume. 0.89 18.75 helped me to become more efficient with my sales expenses. 0.66 13.87

Training Outcome--Customer Relations 0.90 0.84 This training class has ... helped me get closer to my customers. 0.85 ~ helped me improve relations with my customers. 0.90 24.60 allowed me to form tight partnerships with clients. 0.85 22.52 Notes: Measurement model fit statistics were [chi square] = 286.58, df = 120, CFI = 0.97, TLI = 0.96, GFI = 0.93. (1) The average variance extracted by a single principle components factor. (2) Indicates initial loading fixed to one to set the scale of the construct. (3) Item eliminated through re-specification of the confirmatory factor analysis measurement model. (4) Indicates reverse coded items. Table 2 Scale Means, Standard Deviations, and Correlations Among Latent Constructs Mean S.D. 1 2 1. Reaction to Training 16.07 4.24 2. Perceived Knowledge Retained 15.33 3.55 0.57 3. Perceived Behavior Change 16.87 3.07 0.58 0.67 4. Training Outcome--Organizational Commitment 12.24 5.12 0.44 0.34 5. Training Outcome--Selling Effectiveness 11.07 4.36 0.60 0.44 6. Training Outcome--Customer Relations 13.42 4.60 0.59 0.43 3 4 5 1. Reaction to Training 2. Perceived Knowledge Retained 3. Perceived Behavior Change 4. Training Outcome--Organizational Commitment 0.43 5. Training Outcome--Selling Effectiveness 0.62 0.63 6. Training Outcome--Customer Relations 0.57 0.69 0.77 Table 3 Structural Model Coefficients and Fit Statistics Hypothesized Model

z- Est. (2) value Structural Parameters Training Reactions [right Knowledge arrow] 0.58 9.61 Training Reactions [right Perceived arrow] Behavior Change 0.34 5.57 Knowledge [right Perceived arrow] Behavior Change 0.48 6.07 Knowledge [right Organizational arrow] Commitment Knowledge [right Selling arrow] Effectiveness Knowledge [right Customer arrow] Relations Perceived Behavior [right Organizational Change arrow] Commitment 0.46 8.91 Perceived Behavior [right Selling Change arrow] Effectiveness 0.65 11.04 Perceived Behavior [right Customer Change arrow] Relations 0.60 11.15 Latent Error Correlations Organizational [left and Selling Commitment right arrow] Effectiveness 0.49 7.29 Organizational [left and Customer Commitment right arrow] Relations 0.58 8.91 Selling [left and Customer Effectiveness right arrow] Relations 0.62 8.17 Model Fit Statistics [chi square] 354.07 df 126 CFI 0.96 TLI 0.95 Hypothesized Model Controlling for Common Methods

z- Est. value Structural Parameters Training Reactions [right Knowledge arrow] 0.42 5.23 Training Reactions [right Perceived arrow] Behavior Change 0.11 1.52 * Knowledge [right Perceived arrow] Behavior Change 0.50 5.49 Knowledge [right Organizational arrow] Commitment Knowledge [right Selling arrow] Effectiveness Knowledge [right Customer arrow] Relations Perceived Behavior [right Organizational Change arrow] Commitment 0.20 2.91 Perceived Behavior [right Selling Change arrow] Effectiveness 0.32 3.49 Perceived Behavior [right Customer Change arrow] Relations 0.22 2.25 Latent Error Correlations Organizational [left and Selling Commitment right arrow] Effectiveness 0.38 4.23 Organizational [left and Customer Commitment right arrow] Relations 0.52 5.52 Selling [left and Customer Effectiveness right arrow] Relations 0.33 2.79 Model Fit Statistics [chi square] 184.21 df 108 CFI 0.99 TLI 0.98 Mediation Test (1): Levels 1 and 3 by Level (2)

z- Est. value Structural Parameters Training Reactions [right Knowledge arrow] 0.35 3.90 Training Reactions [right Perceived arrow] Behavior Change 0.24 3.09 Knowledge [right Perceived arrow] Behavior Change Knowledge [right Organizational arrow] Commitment Knowledge [right Selling arrow] Effectiveness Knowledge [right Customer arrow] Relations Perceived Behavior [right Organizational Change arrow] Commitment 0.13 1.82 * Perceived Behavior [right Selling Change arrow] Effectiveness 0.22 2.38 Perceived Behavior [right Customer Change arrow] Relations 0.08 0.75 * Latent Error Correlations Organizational [left and Selling Commitment right arrow] Effectiveness 0.37 4.22 Organizational [left and Customer Commitment right arrow] Relations 0.52 5.21 Selling [left and Customer Effectiveness right arrow] Relations 0.33 2.62 Model Fit Statistics [chi square] 229.42 df 109 CFI 0.98 TLI 0.97 Mediation Test: Levels 2 and 4 by Level 3 (A)

z- Est. value Structural Parameters Training Reactions [right Knowledge arrow] 0.43 4.93 Training Reactions [right Perceived arrow] Behavior Change 0.03 0.35 * Knowledge [right Perceived arrow] Behavior Change 0.54 5.37 Knowledge [right Organizational arrow] Commitment 0.16 2.05 Knowledge [right Selling arrow] Effectiveness 0.16 1.49 * Knowledge [right Customer arrow] Relations 0.12 1.09 * Perceived Behavior [right Organizational Change arrow] Commitment Perceived Behavior [right Selling Change arrow] Effectiveness Perceived Behavior [right Customer Change arrow] Relations Latent Error Correlations Organizational [left and Selling Commitment right arrow] Effectiveness 0.35 3.70 Organizational [left and Customer Commitment right arrow] Relations 0.51 5.04 Selling [left and Customer Effectiveness right arrow] Relations 0.24 1.84 * Model Fit Statistics [chi square] 192.80 df 108 CFI 0.98 TLI 0.98 Mediation Test: Levels 2 and 4 by Level 3 (B)

z- Est. value Structural Parameters Training Reactions [right Knowledge arrow] 0.43 5.04 Training Reactions [right Perceived arrow] Behavior Change 0.11 1.43 * Knowledge [right Perceived arrow] Behavior Change 0.50 5.49 Knowledge [right Organizational arrow] Commitment 0.08 0.91 * Knowledge [right Selling arrow] Effectiveness 0.01 0.09 * Knowledge [right Customer arrow] Relations 0.04 0.31 * Perceived Behavior [right Organizational Change arrow] Commitment 0.16 2.03 Perceived Behavior [right Selling Change arrow] Effectiveness 0.33 3.13 Perceived Behavior [right Customer Change arrow] Relations 0.20 1.87 * Latent Error Correlations Organizational [left and Selling Commitment right arrow] Effectiveness 0.39 4.31 Organizational [left and Customer Commitment right arrow] Relations 0.53 5.59 Selling [left and Customer Effectiveness right arrow] Relations 0.34 2.77 Model Fit Statistics [chi square] 183.11 df 105 CFI 0.99 TLI 0.98 Notes: * Nonsignificant parameter. (1) All mediation tests were conducted while controlling for common methods. (2) By "Est." we mean completely standardized estimates.

The authors thank the editor and three anonymous JPSSM reviewers for their helpful comments and suggestions.

REFERENCES

Alliger, George M., and Elizabeth A. Janak (1989), "Kirkpatrick's Levels of Training Criteria: Thirty Years Later," Personnel Psychology, 42, 2 (Summer), 331-342.

--, Scott I. Tannenbaum, Winston Bennett, Jr., Holly Traver, and Allison Shotland (1997), "A Meta-Analysis of the Relations Among Training Criteria," Personnel Psychology, 50, 2 (Summer), 341-358.

Anderson, Erin, and Thomas S. Robertson (1995), "Inducing Multiline Salespeople to Adopt House Brands," Journal of Marketing, 59, 2 (April), 16-31.

Anderson, James C., and David W. Gerbing (1988), "Structural Equation Modeling in Practice: A Review and Recommended Two-Step Approach," Psychological Bulletin, 103, 3 (May), 411-423.

Armstrong, J. Scott, and Terry S. Overton (1977), "Estimating Nonresponse Bias in Mail Surveys," Journal of Marketing Research, 14, 3 (August), 396-402.

Arthur, Winfred, Jr., Winston Bennett, Jr., Pamela S. Edens, and Suzanne T. Bell (2003), "Effectiveness of Training in Organizations: A Meta-Analysis of Design and Evaluation Features," Journal of Applied Psychology, 88, 2 (April), 234-245.

Atria, Ashraf M., Earl D. Honeycutt, and Magdy Mohamed Attia (2002), "The Difficulties of Evaluating Sales Training," Industrial Marketing Management, 31, 3 (April), 253-259.

Baldwin, Timothy T., and J. Kevin Ford (1988), "Transfer of Training: A Review and Directions for Future Research," Personnel Psychology, 41, 1 (Spring), 63-105.

Baron, Reuben M., and David A. Kenny (1986), "The Moderator-Mediator Variable Distinction in Social Psychological Research: Conceptual, Strategic and Statistical Considerations," Journal of Personality and Social Psychology, 51, 6 (December), 1173-1182.

Brashear, Thomas G., James S. Boles, Danny N. Bellenger, and Charles M. Brooks (2003), "An Empirical Test of Trust-Building Processes and Outcomes in Sales Manager-Salesperson Relationships," Journal of the Academy of Marketing Science, 31, 2 (Spring), 189-200.

Bretz, Robert D., and Robert E. Thompsett (1992), "Comparing Traditional and Integrative Learning Methods in Organizational Training Programs," Journal of Applied Psychology, 77, 6 (December), 941-951.

Broad, Mary L., and John W. Newstrom (1992), Transfer of Training: Action Paced Strategies to Ensure High Payoff from Training Investments, Reading, MA: Addison-Wesley.

Burnkrant, Robert E., and Thomas J. Page, Jr. (1982), "An Examination of the Convergent, Discriminant, and Predictive Validity of Fishbein's Behavioral Intention model," Journal of Marketing Research, 19, 4 (November), 550-561.

Burrow, Jim, and Paula Berardinelli (2003), "Systematic Performance Improvement--Refining the Space Between Learning and Results," Journal of Workplace Learning, 15 (1), 6-13.

Clark, Catherine S., Gregory H. Dobbins, and Robert T. Ladd (1993), "Exploratory Field Study of Training Motivation," Group and Organization Management, 18, 3 (September), 292-307.

Colquitt, Jason A., Jeffery A. LePine, and Raymond A. Noe (2000), "Toward an Integrative Theory of Training Motivation: A Meta-Analytic Path Analysis of 20 Years of Research," Journal of Applied Psychology, 85, 5 (October), 678-707.

Davis, Brian L., and Michael K. Mount (1984), "Effectiveness of Performance Appraisal Training Using Computer Assisted Instruction and Behavior Modeling," Personnel Psychology, 37, 3 (Autumn), 439-452.

Day, Eric A., Winfred Arthur, Jr., and Dennis Gettman (2001), "Knowledge Structures and the Acquisition of a Complex Skill," Journal of Applied Psychology, 86, 5 (October), 1022-1033.

Dubinsky, Alan J., and Richard W. Hansen (1981), "The Sales Force Audit," California Management Review, 24, 2 (Winter), 86-95.

--, Roy D. Howell, Thomas N. Ingram, and Danny N. Bellenger (1986), "Salesforce Socialization," Journal of Marketing, 50, 4 (October), 192-207.

Erffmeyer, Robert C., K. Randall Russ, and Joe F. Hair, Jr. (1991), "Needs Assessment and Evaluation in Sales-Training Programs," Journal of Personal Selling and Sales Management, 11, 1 (Winter), 18-30.

Georges, James C. (1996), "The Myth of Soft-Skills Training," Training, 33, 1 (January), 48-54.

Gist, Marilyn E., Anna G. Bavetta, and Cynthia Kay Stevens (1990), "Transfer Training Method: Its Influence on Skill Generalization, Skill Repetition, and Performance Level," Personnel Psychology, 43, 3 (Autumn), 501-523.

Goldstein, Irwin L. (1986), Training in Organizations: Needs Assessment, Development, and Evaluation, Pacific Grove, CA: Brooks/Cole.

Holton, Elwood F., III (1996), "The Flawed Four-Level Evaluation Model," Human Resource Development Quarterly, 7, 1 (Spring), 5-25.

Honeycutt, Earl D., Jr., and Thomas H. Stevenson (1989), "Evaluating Sales Training Programs," Industrial Marketing Management, 18, 3 (August), 215-222.

--, Vince Howe, and Thomas N. Ingram (1993), "Shortcomings of Sales Training Programs," Industrial Marketing Management, 22, 2 (May), 117-123.

--, Kiran Karande, Ashraf Attia, and Steven D. Maurer (2001), "An Utility Based Framework for Evaluating the Financial Impact of Sales Force Training Programs," Journal of Personal Selling and Sales Management, 21, 3 (Summer), 229-238.

James, Lawrence R., Stanley A. Mulaik, and Jeane M. Brett (1982), Causal Analysis: Assumptions, Models, and Data, Beverly Hills, CA: Sage.

Johnston, Mark W., and Greg W. Marshall (2003), Churchill, Ford, and Walker's Sales Force Management, 7th ed., New York: McGraw-Hill.

Joreskog, Karl G., and Dag Sorbom (1979), Advances in Factor Analysis and Structural Equation Modeling, Jag Magidson, ed., Cambridge, MA: Abt Books.

Kaydo, Chad (1998), "Should You Speed Up Sales Training?" Sales & Marketing Management, 150, 10 (December), 102.

Kirkpatrick, Donald L. (1959), "Techniques for Evaluating Training Programs," Journal of the American Society for Training and Development, 13 (11), 3-9.

-- (1960), "Techniques for Evaluating Training Programs," Journal of the American Society for Training and Development, 14 (1), 13-18.

-- (1994), Evaluation Training Programs: The Four Levels, San Francisco: Berrett-Koehler.

-- (1996), "Great Ideas Revisited," Training and Development, 50, 1 (January), 55-57.

Kraiger, Kurt, J., Kevin Ford, and Eduardo Salas (1993), "Application of Cognitive, Skill-Based, and Affective Theories of Learning Outcomes to New Methods of Training Evaluation," Journal of Applied Psychology, 78, 2 (April), 311-328.

Leigh, Thomas W., and Patrick E McGraw (1989), "Mapping the Procedural Knowledge of Industrial Sales Personnel: A Script-Theoretic Investigation," Journal of Marketing, 53, 1 (January), 16-34.

Lorge, Sarah (1998), "Getting Into Their Heads," Sales & Marketing Management, 150, 2 (February), 58-64.

Lupton, Robert A., John E. Weiss, and Robin T. Peterson (1999), "Sales Training Evaluation Model (STEM): A Conceptual Framework," Industrial Marketing Management, 28, 1 (January), 73-86.

May, Gary L., and William M. Kahnweiler (2000), "The Effect of a Mastery Practice Design on Learning and Transfer in Behavior Modeling Training," Personnel Psychology, 53, 2 (Summer), 353-373.

Morgan, Ronald B., and Wendy J. Casper (2000), "Examining the Factor Structure of Participant Reactions to Training: A Multidimensional Approach," Human Resource Development Quarterly, 11, 3 (Fall), 301-317.

Netemeyer, Richard G., James S. Boles, Daryl O. McKee, and Robert McMurrian (1997), "An Investigation into the Antecedents of Organizational Citizenship Behaviors in a Personal Selling Context," Journal of Marketing, 61, 3 (July), 85-98.

Oliver, Richard L., and John E. Swan (1989), "Consumer Perceptions of Interpersonal Equity and Satisfaction in Transactions: A Field Survey Approach," Journal of Marketing, 53, 2 (April), 21-35.

Phillips, Jack J. (1998), "Level Four and Beyond: An ROI Model," in Evaluating Corporate Training: Models and Issues, Stephen M. Brown and Constance J. Seidner, eds., Boston: Kluwer Academic Publishers, 113-140.

Ping, Robert A. (1994), "Does Satisfaction Moderate the Association Between Alternative Attractiveness and Exit Intention in a Marketing Channel?" Journal of Academy of Marketing Science, 22, 4 (Fall), 364-371.

Podsakoff, Philip M., Scott B. MacKenzie, Robert H. Moorman, and Richard Fetter (1990), "Transformational Leader Behaviors and Their Effects on Followers' Trust in Leader, Satisfaction, and Organizational Citizenship Behaviors," Leadership Quarterly, 1 (2), 107-142.

Salas, Eduardo, and Janis A. Cannon-Bowers (2001), "The Science of Training: A Decade of Progress," Annual Review of Psychology, 52,471-499.

Szymanski, David M., and Gilbert A. Churchill, Jr. (1990), "Client Evaluation Cues: A Comparison of Successful and Unsuccessful Salespeople," Journal of Marketing" Research, 27, 2 (May), 163-174.

Tannenbaum, Scott I., and Gary Yukl (1992), "Training and Development in Work Organizations," Annual Review of Psychology, 43, 399-441.

--, John E. Mathieu, Eduardo Salas, and Janis A. Cannon-Bowers (1991), "Meeting Trainees' Expectations: The Influence of Training Fulfillment on the Development of Commitment, Self-Efficacy, and Motivation," Journal of Applied Psychology, 76, 6 (December), 759-69.

Tracey, J. Bruce, Scott I. Tannenbaum, and Michael J. Kavanagh (1995), "Applying Trained Skills on the Job: The Importance of the Work Environment," Journal of Applied Psychology, 80, 2 (April), 239-252.

Van Buren, Mark E., and William Erskine (2002), The 2002 ASTD State of the Industry Report, Alexandria, VA: American Society of Training and Development.

Warr, Peter B., and David Bunce (1995), "Trainee Characteristics and the Outcomes of Open Learning," Personnel Psychology, 48, 2 (Summer), 347-375.

--, Catriona Allan, and Kamal Birdi (1999), "Predicting Three Levels of Training Outcome," Journal of Occupational and Organizational Psychology, 72, 3 (September), 351-375.

Yamnill, Siriporn, and Gary N. McLean (2001), "Theories Supporting Transfer of Training," Human Resource Development Quarterly, 12, 2 (Summer), 195-208.

Mark P. Leach (Ph.D., Georgia State University), Assistant Professor of Marketing, College of Business Administration, Loyola Marymount University, Los Angeles, mleach@lmu.edu.

Annie H. Liu (Ph.D., Georgia State University), Assistant Professor of Marketing, College of Business Administration, Loyola Marymount University, Los Angeles, aliu@lmu.edu.

This document provided by HighBeam Research at http://www.highbeam.com