A Comparison between Peregrine Academic Services’ CPC ......Inbound/Outbound Results No Yes...

Citation:

Peregrine Academic Services. (2012, revised 2017). A comparison between Peregrine Academic

Services’ CPC-based COMP Exam and the Educational Testing Service’s MFT (Report

No. 2012-1). Gillette, WY: Author.

Peregrine Academic Services, LLC

A Comparison between Peregrine Academic Services’ CPC-based COMP Exam and the

Educational Testing Service’s MFT

The purpose of this report is to describe the differences and similarities of two business

education assessment exams, the Common Professional Component (CPC)-based

Comprehensive (COMP) Exam offered by Peregrine Academic Services (PAS) and the Major

Field Test (MFT) offered by Educational Testing Service (ETS).

Major sections in this report include:

1. Exam Purpose and Application

2. Summary Table of the CPC-based COMP Exam and ETS MFT Comparisons

3. Discussion of Comparison Areas

4. Change Considerations and Data Comparability

5. Four Common Reasons for Faculty Resistance to Change

6. Exam Administration Comparison

7. Testimonials Regarding the Switch to the CPC-based COMP Exam

Exam Purpose and Application

ETS MFT

The purpose of the ETS MFT is to measure the critical knowledge and understanding obtained

by students in a major field of study, in this case business education. The focus of the exam is on

assessing student knowledge. Results can be used for outcomes assessment. The exams have to

be pre-ordered with separate pricing for reports beyond the base report included with the exam

order.

The proctored, non-customizable exam includes 120 questions from nine topical areas:

1. Accounting (~15%)

2. Economics (~13%)

3. Management (~15%)

4. Quantitative Business Analysis (~11%)

5. Information Systems (~10%)

A Comparison between Peregrine Academic Services’ CPC-based COMP Exam and the Educational Testing Service’s MFT

2

6. Finance (~13%)

7. Marketing (~13%)

8. Legal and Social Environment (~10%)

9. International Issues (overlapping and drawn from other content areas)

The exam is fixed, meaning the identical exam is given to all students. Therefore, a proctor is

required to administer the exam.

The total score is reported for each student and summarized for the group. Assessment indicators

are reported for the group only. A minimum five (5) students is required for assessment

indicators to be reported. Assessment indicators correspond to the nine topical areas.

CPC-based COMP Exam

The purpose of the CPC-based COMP exam is to assess learning outcomes at the program level

of the business program by measuring retained knowledge. The exam is based on the Common

Professional Component (CPC) topics as defined by the Accreditation Council for Business

Schools and Programs (ACBSP) and the International Assembly for Collegiate Business

Education (IACBE). The exam can be customized with topic selection for alignment with

learning outcomes. Exams are paid for after the testing period is complete and pricing includes

all reports.

The base exam (before any topic customization) consists of 120 questions, 10 each from 12 topic

areas:

1. Marketing

2. Business Finance

3. Accounting

4. Management (Operations/Production Management, Human Resource Management,

Organizational Behavior)

5. Legal Environment of Business

6. Economics (Macroeconomics and Microeconomics)

7. Business Ethics

8. Global Dimensions of Business

9. Information Management Systems

10. Quantitative Techniques/Statistics

11. Leadership

12. Business Integration and Strategic and Management

Each exam is randomly generated from a test bank of questions and no two exams are identical.

Random question selection for each exam, along with other security measures in place, means no

proctor requirement for the exam.


3

The total score and score for each topical area are reported to the student immediately upon

completion of the exam. Individual exam scores (total, topic, and subject), the group total score,

topic and sub-topic scores, and subject-level scores (4-8 subjects/topic) are available and

reported to the institution immediately upon completion of the assessment. There is no minimum

group size limitation for summarized scoring.

Summary Table of the CPC-based COMP Exam and ETS MFT Comparisons

Comparison Area ETS MFT CPC-Based COMP Exam

Topical Coverage 9 Topics 12 Topics (All CPC Topics)

Exam Pricing $24-$31 with One Price Point

for volume discounting

$32-$40 with Four Price Points

for volume discounting

Report(s) Pricing Separate Charges Included

Exam Customization Not Offered Offered, No Additional Charge

Student Groups Limited to 2 with 10 options

per group

Unlimited user defined fields

with unlimited options per field

Exams for Non-MBA

Programs None Offered

Customized Exams for MS and

MA Programs

Minimum Group Size for

Reporting 5 No Minimums

Proctor Requirements Yes No

Access to Raw Data Limited to only total Score

Raw data provided in an Excel

Spreadsheet with all data

provided

Exam Pre-ordering and Pre-

Pay Yes No, pay after the exam period

External Comparisons One provided based on pre-

selection of 10 schools

Multiple aggregate pools

provided for direct

comparisons with a variety

school types

Exam Customization No Yes with Topic Selection

Longitudinal Reporting No Yes


4


Ability to do

Inbound/Outbound (Pre-

test/Post) Testing

No Yes

Pairwise Reporting of

Individual Student

Inbound/Outbound Results

No Yes

Cohort/Program Reporting Limited to 2 groups and one

report Unlimited

Ability to re-run reports

using different intervals of

time

No Yes

Admin Access to Data and

Reports

Limited, but must purchase an

account Yes, no additional charges

Access to Results After exam period 24/7

Self-registration for the

Exam No Yes, use a micro-site

Learning Platform

Integration (e. g. e-College) No Yes

Sell the exam through the

Bookstore No Yes

Different exams for

different programs in the

same CAPSTONE course

No Yes

ACBSP and IACBE

Acceptance Yes Yes

Data storage for future

reporting No Yes

Subject-level Analysis No Yes

Use for both Summative

and Formative Assessment No Yes


5


Ability to add new, client-

school defined Topics

One topic, 50 questions Unlimited topics and questions

Include a Student Survey

with the Exam No Yes

Content Focus Research-based Institutions Teaching and Research-based

Institutions

Access to Supplemental e-

learning Modules No Yes

Ability to embed within a

course and grade the exam Limited to online testing Yes

Scoring Scale Varies by academic degree

level

100-point Scale based on

Percent Correct

Availability of Support with

Data Analysis and

Application of the Results

for Program Improvement

Limited Yes

Available for use outside of

the US No

Yes, with translated exams and

with the base exam


6

Discussion of Comparison Areas

1. Topical Coverage

ETS MFT: Nine topic areas included.

CPC-based COMP Exam: Up to 12 topic areas with the base business exam; 13 topics

for the Accounting/Finance exam along with 47 supplemental accounting/finance topics.

Topic areas correspond to the accreditation standards.

2. Exam Pricing

ETS MFT: Prices range from $24-$31/exam based on academic degree level

(undergraduate and MBA) and based on the number of exams ordered (price break point

is 100 exams). Separate charges for shipping if paper tests are used.

CPC-based COMP Exam: Prices range from $32-$40/exam based on annual exam

requirements (price break points are 50, 100, 250, and 500). No shipping charges.

3. Report(s) Pricing

ETS MFT: Base report included with the exam price. Special reports range from $150 to

a $700 1-year subscription.

CPC-based COMP Exam: No additional charges for reports.

4. Exam Customization/Set-up Pricing

ETS MFT: N/A – no customization of the exam is offered.

CPC-based COMP Exam: Customization and set-up with unlimited access to all the

data and reports is included with the exam price.

5. Ability to segregate the data based on Student Grouping

ETS MFT: Two choices for groups and up to 10 options within a group.

CPC-based COMP Exam: Unlimited options for groups and unlimited options within a

group.

6. Exams for Non-MBA Masters Programs

ETS MFT: Only an MBA exam offered

CPC-based COMP Exam: Can build a customized exam based on topic selection for

MS and MA programs.


7

7. Minimum reported sample size (ETS limits minimum to 5 for any one report)

ETS MFT: Minimum group sample size of 5 for any one report.

CPC-based COMP Exam: No group size minimum restrictions

8. Proctor Requirements

ETS MFT: Exam proctor required

CPC-based COMP Exam: No exam proctor requirement.

9. Access to Raw Data and Individual Student Results

ETS MFT: Departmental Roster report only shows the total score for the student.

CPC-based COMP Exam: Raw data provided in Excel format for each student. The file

can be linked with the institution’s database for any subsequent demographical analysis.

The Excel file lists the total score, topic/subtopic scores, subject-level results, and mean

completion times (in minutes) for each student.

10. Exam Pre-Ordering

ETS MFT: Exams must be pre-ordered and paid for in advance of testing.

CPC-based COMP Exam: Exams are paid for after the exam period based only on

exams completed. No need to pre-order exams.

11. External comparisons

ETS MFT: Pre-select 10 schools in their database for comparison reporting. One

comparison shown with the report.

CPC-based COMP Exam: Comparison reporting based on fixed aggregate pools based

on delivery modality (online, on-campus, blended/hybrid), affiliation (faith-based,

HBCU), ownership (private, public, large private), and master’s degree program (MBA,

MS, MA). Comparisons based on IACBE and ACBSP regions coming in 2013. No limit

on the number of external comparisons displayed and no need to pre-select which

comparison pool to use.

12. Exam Customization with Topic Selection

ETS MFT: None offered

CPC-based COMP Exam: Can select which topics to have on the exam. Associate

degree programs typically select 4-6 topics; bachelors programs 10-12; masters programs


8

6-10; and doctoral programs 8-12 topics. Unlimited topic selection with the

accounting/finance exams.

13. Longitudinal Reporting


CPC-based COMP Exam: The Longitudinal Report allows up to four distinct test

periods to be compared side-by-side in one report.

14. Ability to use as with an Inbound/Outbound Exam Construct (Program-level Pre-

test/Post-test)

ETS MFT: Not available

CPC-based COMP Exam: Inbound/outbound testing is an optional method for internal

benchmarking and summative assessment.

15. Pairwise reporting

ETS MFT: N/A

CPC-based COMP Exam: When inbound and outbound exams are giving to the same

students, report can show both group comparisons as well as individual student

comparisons.

16. Cohort/Program reporting

ETS MFT: Two grouping levels allowed with up to 10 choices within a group.

CPC-based COMP Exam: Unlimited groupings allowed with unlimited choices within

a group. Subsequent reporting can compare group results.

17. Ability to re-run reports using different periods of time and with different

comparisons

ETS MFT: Not offered

CPC-based COMP Exam: Ability to select any period of time for subsequent reporting

and unlimited access to reports.

18. Admin access to the data and reporting

ETS MFT: Limited, but first must purchase and create an account.

CPC-based COMP Exam: 24/7 access allowed for program managers to see a

dashboard view of exam results, access raw data, and generate all reports.


9

19. Access to Scores

ETS MFT: After exam period is complete and paper copies mailed, scored, and results

returned. More immediate with online testing.

CPC-based COMP Exam: Immediately for the student with a downloadable PDF

completion certificate that shows both the total score and the topical score. Access to

group scores and raw data available anytime to the school after the first exam record is

created.

20. Self-registration

ETS MFT: No option for self-registration for the exam

CPC-based COMP Exam: Student can self-register for the exam using a customized

client-school micro-site.

21. Opportunities for Learning Platform Integration


CPC-based COMP Exam: Integration options for e-College and Blackboard (and others

upon request provided that they allow third party API/SSO integration with the platform).

Registration records can be created automatically along with automatic posting of results.

22. Selling the Exam through the Bookstore

ETS MFT: Not an option

CPC-based COMP Exam: The exam can be individually purchased by the student

through an online or physical bookstore when the student purchases their course

materials.

23. Different exams for different programs when the students are in the same

CAPSTONE (or final) course


CPC-based COMP Exam: Multiple exams can be used within a course if students from

more than one degree program are taking the same final course in a program of study.

Each program’s exam can be customized to align with the program’s learning outcomes.


10

24. ACBSP and IACBE approvals/acceptance

ETS MFT: Commonly used for the requirements of the ACBSP Standard #4 and the

IACBE Principle #1.

CPC-based COMP Exam: Accepted by the ACBSP for Standard #4 (Assessment) and

approved by the IACBE for Principle #1 (Outcomes Assessment).

25. Storage of data for future reporting


CPC-based COMP Exam: Unlimited storage of exam results for future reporting. Can

go back multiple years to generate new comparison reports with either/both pooled and/or

segregated reporting.

26. Number of exams administered/year adequate for statistically valid comparisons

ETS MFT: Yes

CPC-based COMP Exam: Yes

27. Subject-level analysis along with topic level analysis

ETS MFT: Analysis areas only include Total Score, Sub-scores, and Assessment

Indicators

CPC-based COMP Exam: Analysis provided down to the subject level of a topic or

subtopic. Each topic and subtopic has 4-8 subject areas.

28. Can be used for both summative and formative assessments

ETS MFT: Summative only

CPC-based COMP Exam: Within inbound, outbound, and an optional mid-point exam,

results are used for both summative and formative assessment.

29. Ability to add client-school specific topics on an exam

ETS MFT: Institutions can add up to 50 locally authored questions

CPC-based COMP Exam: New topic areas based on client-specific topical areas can be

created with the assistance of the client school. Such topics can be used for internal

benchmarking and comparisons. No question limit and no limit on the number of client-

specific topics. Such customization is offered at no additional charge.


11

30. Ability to also include a student survey with the exam (satisfaction or a learning-

outcomes based survey)


CPC-based COMP Exam: Yes, the school can require a student survey embedded into

the CPC-based COMP exam, given after the exam. The survey can have both open-ended

as well as Likert-type survey items. The client can write the survey. A summary report of

the survey results is provided. There are no additional charges for this service.

31. Content focus - applied business knowledge OR theoretical business knowledge.

ETS MFT: Exam developed primarily for research-focused institutions.

CPC-based COMP Exam: Exam developed primarily for teaching institutions with a

focus on applied business knowledge, the member schools of the ACBSP and the

IACBE.

32. Access to additional learning modules should students not do as well as

expected/desired.


CPC-based COMP Exam: To close any gaps between desired knowledge levels and

exam-expressed knowledge levels, Peregrine Academic Services also provides online

educational modules based on the same CPC topics.

33. Ability to use the exam within a course and make the exam part of the course grade

ETS MFT: Limited based on the time required to obtain the results (dependent upon if it

is a paper exam or online exam).

CPC-based COMP Exam: Embedding the exam within a course and assigning a grade

to the exam is the recommended strategy so that the course professors can help hold the

students accountable for the exam and to offer their best effort.

34. Scoring Scale

ETS MFT: Scoring on a 120-200 point grading scale (MBA Exam on a 220-300 scale

and the Associate Exam on a 520-600 scale). Sub-scores reported only if minimal sample

sizes are met.

CPC-based COMP Exam: Scoring based on a 100-point scale for ease of understanding

by both the students and the program managers. All scores reported regardless of the

sample size.


12

35. Availability to the PAS team to help you understand and apply the results of the

exam.

ETS MFT: Website support only.

CPC-based COMP Exam: Dedicated professionals with accreditation and assessment

experience to help the institution understand and apply the results.

36. Available for use outside of the US.

ETS MFT: No

CPC-based COMP Exam: Yes, with both translated exams and with the core English

language exam. External comparisons available by region.

Change Considerations and Data Comparability

In 2012, a team at Alabama State University conducted an analysis comparing the results from

the ETS MFT exam and the PAS CPC-based COMP exam over a 3-year period. The exam

results include those that were pre/during/post change from ETS MFT and the CPC-based

COMP Exam.

The findings are available at: http://www.peregrineacademics.com/resources.

Alabama State University used the ETS MFT Mean Percent Correct scores to compare with the

CPC-based COMP Exam scores that are also based on percent correct for their analysis.

Scoring Reports

ETS (*) PAS (**)

Individual Overall Score. ETS’s score

range is scaled at 120 to 200 (i.e. 120 =

0.

Individual Overall Score. PAS’s score range

is 0 to 100.

Individual CPC Scores: Available as a

separate paid report

Individual CPC Scores: Included as part of the

service

Assessment Indicators: Reported only

for groups of students. Assessment

Indicators report the average percent of

correct answers, in nine CPC topics

CPC score is the same as average percent

correct; reported for all CPC topics. Reported

for both groups and by individual students

for all topic areas down to the subject level of

the topic.

* Source: http://www.ets.org/mft/scores/

** Source: http://www.peregrineacademics.com/

http://www.peregrineacademics.com/resources/


13

The direct comparison of scores allowed for trends over time even though two different

instruments were used. The Alabama State University researchers concluded that the PAS CPC-

based COMP Exam was in better alignment with the Alabama State University business program

curriculum and teaching methods. The Alabama State University College of Business

Administration’s teaching method is more of a hands-on, practical and applied approach to

business education. Both faculty and students felt that the CPC-based COMP exam was more

reliable and valid for assessing student retained knowledge based on the ASU curriculum.

Four Common Reasons for Faculty Resistance to Change (And Strategies to Overcome the

Resistance)

1. If the institution has been using the MFT for many years, then the faculty typically knows the

exam.

Allow the faculty to take the CPC-based COMP exam at no charge so that they are

comfortable with the content. The CPC-based COMP Exam draws from content

included when teaching the foundations of any topical area. Preliminary research

conducted at three separate universities (results to be presented in 2013) indicates that

the CPC-based COMP exam scores correlate closely with the student GPA. The

conclusion drawn from these preliminary analyses suggests a good match between the

exam content and the institution’s curriculum.

2. The institution may be scoring well and the fear with change is that the results will not be as

positive.

Examine the comparison report. This report allows the institution to draw its own

conclusions regarding score results based on multiple aggregate pools and analysis

methods. The institution “owns” the learning outcomes and therefore it is up to the

institution to determine if they are at, above, or below desire levels.

3. More detailed analysis provided by the PAS reporting may mean that the institution is doing

worse than previously believed.

The comparison report has two major sections: an internal benchmarking section and

an external benchmarking section. These two sections allow the institution to analyze

the results down to the subject (or learning outcomes) level. The total score and topic

scores must be taken in the context of the learning outcomes, which the subject-level

analysis provides.

4. Concerned that the institution will lose its historic information from ETS MFT with a change

in exams.


14

Based upon the findings from research conducted at Alabama State University

(http://www.peregrineacademics.com/resources/), the institution can retain its historic

ETS MFT data and display these data side-by-side with the CPC-based COMP exam

results for trend analysis purposes. Some adjustment (scaling) may be needed

depending upon the institution’s goals.

An Exam Administration Process Comparison

Administering an ETS MFT Exam to

Student Group

Administering a CPC-based COMP Exam

to a Student Group

1. Determine the number of needed

exams, determine the type of exam

(paper or online), and order the exams

from ETS.

N/A

2. Designate classroom time and space to

administer the exam

N/A

3. Select an exam proctor N/A

4. Give the exam the Students 1. Give the students the passcode to the

micro-site for self-registration

5. Close the exam N/A

6. Obtain the exam scores (immediately

if administered in the online form;

several weeks if using a paper form)

2. Students submit their exam

completion certificate

N/A 3. Review all of the student data in an

Excel file

7. Review the base report 4. Review all of the analysis reports

8. Order additional reports and pay

associated report fees

N/A

N/A 5. Re-run reports as required using

different student groups, different

dates, and/or different exam periods

http://www.peregrineacademics.com/resources/


15

Testimonials Regarding the Switch to the CPC-based COMP Exam

“Having used the Major Field Test for 8 years, we are committed to external assessment. What

Peregrine Academics has done is extend the reach of external assessment so that we can now

capture data from 100% of our student population. They also provide a range of relevant

solutions that help us ensure the quality of our academic programs. Peregrine obviously has

listened to the evolving needs of business schools globally.”

William S. Lightfoot, Ph D

Dean

College of Business & Mass Communication

Brenau University

“Peregrine Academic Services has been outstanding to work with! We use their CPC-Based

testing services for our inbound as well as our outbound test in our MBA and MS Management

programs. They have tailored an exam that fits the CPC for each degree. This has helped our

assessment program tremendously, giving us a direct measure that learning is indeed occurring

in our programs. We also use the CPC-Based outbound exam for our BBA degrees and plan to

use them for an inbound BBA exam as well. Previously, we had used the ETS Major Field Exam

in Business and the ETS MBA Exam. The results in terms of details we receive from Peregrine

Academic Services are far superior.”

Dr. Les Crall

Associate Dean, School of Business & Technology

Southwestern Oklahoma State University

“We have found Peregrine Academic Services to be responsive, not only in the timeliness of the

reporting but in the ability to produce customized reports that allow us to drill down and analyze

student learning at various levels, across programs and geographic regions. When you become a

client of Peregrine, you have a thought partner in the assessment arena. Olin and his staff not

only provide you with the data, but they also are available to help you make sense of your

student learning data.”

Peter J. Holbrook, Ph.D.

Dean

College of Business and Management

Cardinal Stritch University

“We appreciate the flexibility of these exams by your allowing us to develop a specialized

section, entrepreneurship, to be incorporated into the exam.”

Timothy A. O. Redmer, Ph. D.

Professor of Accounting

School of Global Leadership & Entrepreneurship

Regent University


16

“We at the Heuther School of Business, College of Saint Rose have been looking for some time

for a way to measure student progress and performance utilizing an outcome based rubric. We

have used other firms in the past, but where not happy with their approach or service. All that

has changed since we have been working with Peregrine Academic Services! Their testing

instruments are excellent and easy to administer. Outcomes assessment is immediate and can

easily be adapted to our particular programs. Most important, the staff at Peregrine is readily

available and happy to help with questions or concerns. Peregrine is an excellent partner for

us.”

Charles Finn Ph.D. MA

Management Professor

Heuther School of Business

The College of Saint Rose

“We have many good things to say about the comprehensive exam. Your customer service is

excellent. The department faculty uses the results to help identify weak areas in our curriculum.

The exams are fair and rigorous and we find it an excellent way for business faculty to gauge

specific outcomes related to the areas they teach. We receive monthly reports in a timely fashion

and whenever I have contacted the office your staff is professional, helpful, courteous, and

informative. The department is ever so happy with your program. It has saved us many hours of

the development, administration, and correction of this type of an exam.”

Joe Adamo, Ph.D.

Professor, Business & Management

Cazenovia College

“In fall 2011, Northwood University piloted the Peregrine Assessments for both BBA and MBA

graduates and based on the pilot, implemented this across our system in 2012. The primary

reason we chose this assessment was because it could be implemented online, in the eight states

we operate, and internationally. We found no other assessment that could meet our needs at this

level. The web-based assessment allowed us to incorporate the exam into our capstone class

without using valuable classroom time. The results were quickly available and we were able to

segment students by major, location, and other key data points. Using these data, we quickly

saw where our strengths were and areas where we could improve.”

Rhonda C. Anderson

Associate Dean, Adult Degree Program

Northwood University

A Comparison between Peregrine Academic Services’ CPC ......Inbound/Outbound Results No Yes...

Documents

Transcript of A Comparison between Peregrine Academic Services’ CPC ......Inbound/Outbound Results No Yes...