Classroom Techniques

Knowledge Assessment of Food SafetyManagers in Utah and Its Implications

on the Exam and InstructionBrian A. Nummer, Stanley M. Guy, and Joanne P.H. Bentley

ABSTRACT: Food Safety Manager’sCertification is offered through a state-localExtension partnership in Utah using an onlinecourse management system. Exams and coursematerials were created by an ExtensionSpecialist at Utah State Univ. Extension Agentsprovide exam and curriculum facilitation in eachcounty. This form of distance education enablesaccess for students in all 29 Utah counties. All454 students (Jan 2008–Mar 2009) successfullypassed the food safety managers exam in 1 or 2attempts and received certification. Twenty-ninepercent of those taking the exam receivedcertification without accessing any of the coursematerials. The overall passing exam score meanwas 82%. When combining both passing andfailing exam attempts, the students’ mean was79%. The students’ mean of the lower 25% wasonly 65%. When evaluating the 7 food safetyknowledge domains, students’ means werehighest in personal hygiene and lowest infacility, equipment, and layout. The lower 25%of students failed to score 70% in 6 of 7 foodsafety knowledge domains. The English coursematerials were provided in 3 options wherestudents could select 1 or more: online,textbook, and DVD. For students who chose toaccess educational materials 67% used onlinelearning, 40% textbooks, and 10% DVDs as partof their study options. Extension Agent feedbackon Hispanic students enrolled in the programindicates a need for Spanish language studymaterials. Overall, the Utah Food SafetyManagers program demonstrates a successfulstate-local Food Science Extension approach todistance education, online curricula, and the useof computers to facilitate testing and learning.

IntroductionThere were approximately 5000 food service establishments in Utah,

employing nearly 105000 people, and generating an estimated gross incomeover 3 billion dollars in 2000 (NRA 2000). This translates into the preparation ofapproximately 82 million dollars’ worth of food daily in Utah foodserviceoperations. These numbers do not include not-for-profit foodservice, such asschools and hospitals. A recent Food and Drug Administration (FDA) reportindicates as many as 40% of foodservice operations are out-of-compliance withcritical regulatory guidelines for food safety (FDA 2004). The report cited5 critical out-of-compliance risk factors: food from unsafe sources, inadequatecooking, improper holding temperature, contaminated equipment, and poorpersonal hygiene (FDA 2004).

Several studies have shown a positive impact on health inspectioncompliance when managers have been certified in food safety (Kneller andBierma 1990; Raval-Nelson and Smith 1999; Binkley and others 2008). UtahState Univ. (U.S.U.) Cooperative Extension certifies foodservice managers infood safety through outreach educational efforts. Exams and curricula are basedon portions of the 1st uniform model Food Code published by the U.S. Food andDrug Administration in 1993 (FDA 2005). The code specifies that food safetymanagers must demonstrate knowledge of food safety and the Food Code bypassing a test that is part of an accredited program (FDA 2005). The Utah Dept.of Health adopted this portion of the model Food Code in 1999 when it createdthe Food Safety Manager Certification rule R392-101 (Utah Dept. of Health2009). The rule defines certification, recertification, and exam content, andspecifies provider accreditation. However, R392-101 does not provide guidanceon learning materials that may be required to assist managers to pass thecertification exam.

The rule mandates that the exam contain 50 or more questions, each with4 multiple choice answers. At least 85% of the questions must come from thefollowing knowledge domains: (1) Identify foodborne illness terms, (2) identifytime/temperature relationship with foodborne illness, (3) describe therelationship between personal hygiene and food safety, (4) describe methods forpreventing food contamination from purchasing to serving, (5) identify correctprocedures for cleaning and sanitizing equipment and utensils, (6) recognizeproblems and potential solutions associated with facility, equipment, and layout,and (7) recognize problems and potential solutions associated with temperaturecontrol, preventing cross contamination, housekeeping, and maintenance. Eachdomain may have between 6% and 20% of the exam devoted to it. Food safety

MS 20090791 Submitted 8/17/2009, Accepted 12/18/2009. Authors Nummer and Guy are withCooperative Extension and author Bentley is with Dept. of Instructional Technology and LearningSciences, Utah State Univ., Logan, UT 84322, U.S.A. Direct inquiries to author Nummer (E-mail:Brian.nummer@usu.edu).

This article features distance education, online curricula, assessment, the use of computers to facil-itate learning, non-English language accommodation, and self-guided mastery learning.

c© 2010 Institute of Food Technologists® Vol. 9, 2010—Journal of Food Science Education 59

JFSE: Journal of Food Science Education

managers must answer 70% or more correctly to “pass” theexam and receive certification. Utah Food Safety ManagerCertificates are valid for 3 y at which time managers mustre-take the exam.

In 1999, U.S.U. Extension documented some of their earlyefforts to bring research-based food safety education tofoodservice workers (Martin and others 1999). The authorsdescribe food safety training in both Utah and Pennsylvania thatbuilt on a successful partnership between cooperative extensionspecialists, county agents, public health officials, and thefoodservice industry to promote safe food handling practices.From 1999 to 2005, the state-certified U.S.U. Food SafetyManager Exam was paper based and administered in Utahcounties by extension agents. During this time, the sole learningoption was a lecture video with textbook. After exams weretaken, they were mailed to U.S.U. and hand graded. Studentswere notified by mail if they passed, and if so, were sent acertificate. Currently, U.S.U uses Blackboard Vista Release 8 toadminister the exams online. Once the exam is completed thesoftware can grade and report to the student the results of theexam immediately. The courseware also allows access tolearning materials such as text, video, audio, and onlinepresentation. The objective of this study was to evaluate theperformance of students taking the Food Safety Managers Examonline. Results will be used to fulfill state mandates and to makeimplications on the quality of the exam and supporting curriculathat can help guide program improvement.

Food safety knowledge is the foundation to safe foodpreparation behavior. Foodservice workers must know whatthey should be doing before they can choose to do it.

Materials and Methods

Online learning management systemFor this study, Utah State Univ. first employed Web CT

software, an online learning management system that wassubsequently integrated into the new Blackboard Vista (ver. 8.0)system (Blackboard, Inc. 2006). Initial course materials wereplaced into Web CT in 2006. Test question metrics (difficultyand discrimination) were determined in Web CT from2006–2007. Utah State Univ. migrated to the Blackboard Vistacourseware January 1, 2008. The remainder of the data in thisstudy refer to data collected from the Blackboard Vista (ver. 8.0)System.

Exam question creation and validationThe certification exam was created using knowledge domains

in the Utah Food Safety Managers Rule R392-101 (Utah Dept.of Health 2009), the FDA Food Code (2005) and foodborneillness risk factors in foodservice facilities (FDA 2004). Thematerial was grouped under 4 topic sections: foodborne illness,risk factors, preventative measures, and food safety programs.Learning objectives were developed for each of the 4 main topicareas. Two or more questions were created for each learningobjective. Questions were reviewed by 2 outside food safetyprofessionals and the Utah Dept. of Health.

Questions were uploaded into the Web CT courseware andmade available to students. As students took the exam prior tothe study period, questions were monitored for difficulty anddiscrimination. The Web CT software provides individualquestion means for the total and upper and lower 25% ofscores. If the question mean for all students or the lower 25%was outside of 50% to 90%, the question was edited orremoved. Student feedback was also used to edit questionwording or to correct misleading questions. After editing or

removing questions that failed to meet criteria, the databasecontained 172 questions. In January 2008, the Utah State FoodSafety Manager materials were transferred into the newBlackboard Vista System.

ExamThe Food Safety Managers Exam is split into 4 sections

corresponding to the 4 topic sections: foodborne illness, riskfactors, preventative measures, and food safety programs. Eachexam section contains 15 questions for a total of 60. Thelearning management system is instructed to randomly select aquestion when there are several alternatives in a learningobjective. Each question has 4 multiple choice answers that arealso randomly sequenced for each student. Food safetymanagers must answer 42 questions (70%) correctly to pass thecombined exams.

The online course management system is available ondemand to students with a username and password provided byU.S.U. and who have access to a computer with an internetconnection. They must take the exam at a county cooperativeextension office in the state. An extension staff member acts as aProctor assuring exam security and that students understandhow to take the computer exam. Students log into theBlackboard Learning System with their unique username andpassword to access the exam sections one at a time. The Proctormust enable each exam section using a password. Students thenanswer the questions and submit the exam section for grading.The number correct is immediately presented along with adetailed list of learning objectives that were answeredincorrectly (exam feedback). In the spirit of mastery learning(Bloom 1968), the students can then study the materials for thequestions that they answered incorrectly. They may retake anyor all exam sections a 2nd time after waiting 24 h. Students whofail to pass after 2 attempts get individualized attention todetermine language or learning difficulties until they pass theexam.

Exam accommodations for English-as-a-Second Languagestudents

The state of Utah has approximately 12% Hispanic and 5%Asian-Pacific Islander residents (EdcUtah 2009). Other minoritygroups, such as African American, represent less than 1%(EdcUtah 2009). Prior to 2008, the U.S.U. Food Safetycertification exam was available separately in either English orSpanish. In 2008, the exam was made bilingual(English–Spanish). All students see both languagessimultaneously in the exam as illustrated below. In 2008, theexam was translated into Chinese (Mandarin), but that version iscurrently not available online. Only 2 students have chosen thisexam form since its creation.

Foodborne illness are: (Select the best answer) Enfermedadesalimentcias son (seleccione la major respuesta):

a. diseases or illnesses carried or transmitted to people fromfoodenfermedades transportadas o transmitidas a personas atravez de alimentos

b. diseases caused by bacteria and parasitesenfermedades provocadas por baterias y parasitos

c. diseases caused by chemical and physical contamination offoodenfermedades provocadas por contaminacion quımica yfısica de alimentos

60 Journal of Food Science Education—Vol. 9, 2010 Available on-line through ift.org

Food safety manager knowledge assessment . . .

d. diseases caused by bacteriaenfermedades provocadas por baterias

Curricula to support certification examsPrior to 2006, the U.S.U. Food Safety Manager’s Certification

curriculum consisted of an English language VHS video thatincluded 6 h of lecture and an accompanying textbook. Studentswere required to borrow the video from a county extensionoffice and purchase the textbook. A frequent complaint fromstudents was that the curricula and the exam did not coordinate.

In 2006, the curriculum was rewritten to start with learningobjectives that directly correlated with exam question learningobjectives. The curriculum was developed into 4 sectionscorresponding to the 4 exam sections. Each section was dividedinto 5 modules. Each module was designed to approximate20 min of learning time for online presentation. Curriculamaterials are only available in English at this time.

Accommodating educational materials and MasteryLearning

Using the capabilities of the Blackboard course managementsoftware each module (total 20) is presented in multiple formats(text with graphics, PowerPoint presentation, and online video)Online students can choose between one or all online methodsthat suits them best. Online self-tests are available that can begraded and reported back to the student by the coursemanagement software, but not recorded. This processincorporates Mastery Learning concepts in that students aregiven feedback on deficiencies along with suggested learningobjectives to review. For students that may not like computers orhave access to computers, the text with graphics version isprinted into a manual and the videos are placed onto a DVD forrental. Self-tests are included in both the textbook and DVD.

Data analysisData to assess the food safety manager’s certificate program

were obtained from student usage within the Blackboard VistaLearning System between January 2008 and March 2009.Student demographics and learning choices were collectedusing course registrations. Hispanic and Asian ethnicbackgrounds are estimates based on name and personal contactinformation from Extension Agents. As other minority groups, forexample, African American, are less than 1% of the populationin Utah, they are not included in this study. User tracking, examscores, and question metrics were performed by the BlackboardLearning System. Additional analyses of data was generatedusing SAS software (SAS Institute Inc., Cary, N.C., U.S.A.).Telephone conference queries of 14 county U.S.U. ExtensionAgents who had more than 10 enrolled students providedinsights on students who needed additional help beyond simpleexam security, such as testing or curriculum assistance.

Results and Discussion

Students enrolledTable 1 lists the total enrolled students (625) between January

2008 and March 2009 by rural and urban county wherestudents wanted to take the exam. Between 2 and 80 studentsenrolled per county during this 15-mo period. The data showthat this distance education program reaches nearly all areas ofUtah. This is important for rural counties where residents wouldnot have easy access to other food safety manager certificationprograms. The lower numbers may be due to a presence of onlya few foodservice operations, or a neighboring county extensionoffice is actually closer or more convenient for them to take thetest.

Table 1—Distribution of students by rural/urban Utah county—January 2008–March 2009.

Rural: Nonmetro countya Urban: OMB metro countya

Beaver – 13 Millard – 11 Cache – 39Box Elder – 12 Piute – 3 Davis – 39Carbon – 31 Rich – 8 Juab – 7Daggett – 0 San Juan – 3 Morgan – 2Duchesne – 9 Sanpete – 7 Salt Lake – 61Emery – 5 Sevier – 13 Summit – 3Garfield – 5 Uintah – 36 Tooele – 18Grand – 21 Wasatch – 42 Utah – 42Iron – 14 Wayne – 5 Washington – 55Kane – 6 Weber – 80

Data indicate county where student will take the exam. Thirty-five applications did not report acounty.aRural/urban definitions based on Office of Management and Budget (OMB) metro counties.

Based on name there were 55 Hispanics (9%) and 34 Asians(5%) enrolled during this study’s 15-mo period. The Asian ethnicgroup matches the demographics for Utah; however, there are12% Hispanics in the overall population (EdcUtah 2009). Onepotential explanation for lower than expected participation byHispanics is that while the exam is bilingual, there are nocurriculum materials in Spanish.

Student testing assistanceIn a telephone conference extension agents report

approximately 10% of students require testing or curriculumassistance. Testing assistance included providing a quick review,providing an oral exam, clarifying question wording, assistingwith calming test anxiety, and reviewing incorrect answers.Curriculum assistance included locating native language studymaterials such as Spanish and Chinese language textbooks.Agents approximated that 20 to 30 students requested Spanishlanguage learning materials and 2 requested Chinese(Mandarin) learning materials. The reports indicate that agentsplay a valuable role in facilitating the Food Safety ManagerCertification Program, especially for students with the mostneeds. It also verifies the need for Spanish language learningmaterials.

Student performance on the internet-based examOf the 625 students enrolled in the program, 454 completed

and passed the exam from January 2008 to March 2009 toreceive a Food Safety Manager’s Certificate. The remainingstudents were enrolled but had not yet taken the exam. Theaverage time required per student to take all 4 exams is 33 min.The overall exam correct answer mean was 81.26% (SD ± 8.7).Passing exam means for individual sections were: Section1—foodborne illness 82.13%; Section 2—foodborne illnessrisks 79.32%; Section 3—preventative measures 82.84%; andSection 4—food safety programs 78.97%. Note that theseaverages are for the higher of 2 exam attempts, if more than oneattempt was required. No students failed to receive certification.Students who did not pass on the first 2 attempts were allowedto re-enroll and were given assistance to enable them tosuccessfully pass the exam.

Student performance based on knowledge domainsStudent performance on exam questions based on knowledge

domains are indicated in Table 2. Results from all student examattempts (failing and passing) show that there were a combined34510 question attempts and 79% of their answers werecorrect. For each of the 7 knowledge areas the upper 25% ofstudents’ means were at least 87%. For all students, the means

Available on-line through ift.org Vol. 9, 2010—Journal of Food Science Education 61

JFSE: Journal of Food Science Education

Table 2—Exam question performance based on knowledge domains.

% Questions correctState of Utah knowledge domains Na All students Upper 25%b Lower 25%c Discriminationd

1. Identify foodborne illness terms 7120 76.81 90.53 60.84 29.692. Identify time/temperature relationship with foodborne illness 5090 80.50 93.91 64.99 28.923. Describe the relationship between personal hygiene and food

safety3045 86.98 96.54 79.08 17.46

4. Describe methods for preventing food contamination frompurchasing to serving

3227 78.08 88.88 66.69 22.19

5. Identify correct procedures for cleaning and sanitizing equipmentand utensils

3454 82.34 93.31 68.73 24.58

6. Recognize problems and potential solutions associated withfacility, equipment, and layout

3227 72.29 87.43 56.28 31.15

7. Recognize problems and potential solutions associated withtemperature control, preventing cross contamination,housekeeping, and maintenance

8347 79.03 94.21 61.15 33.06

Total exam questions 34510 79.43 92.11 65.39 26.72aN = total number of question attempts per knowledge domain, percent questions correct for bupper 25% and clower 25% question scores, and dDiscrimination = difference between upperand lower 25%.

Table 3—Food safety manager exam performance—January 2008–March 2009.

Learning choice n Total score (%) Section 1a (%) Section 2a (%) Section 3a (%) Section 4a (%)

Textbook 76 80.93 ± 6.87a,b 80.21 ± 10.97a,b 80.00 ± 11.10b 82.07 ± 8.88a 81.37 ± 10.68a,b

Textbook + DVD 10 83.19 ± 5.32a,b 81.90 ± 11.22a,b 84.60 ± 13.05a,b 84.10 ± 7.71a 82.10 ± 12.01a,b

Textbook + online 44 83.87 ± 6.90a 80.25 ± 11.22a,b 85.75 ± 9.05a 84.18 ± 8.76a 85.18 ± 10.38a

DVD 21 78.39 ± 6.59b 74.29 ± 11.47b 81.67 ± 9.96a,b 80.23 ± 9.31a 77.28 ± 12.52b

Online 170 82.28 ± 6.54a,b 82.12 ± 12.40a 82.93 ± 9.29a,b 82.61 ± 10.04a 81.37 ± 11.70a,b

No instruction (exam only) 133 79.37 ± 6.53b 76.70 ± 11.38b 81.92 ± 9.54a,b 80.24 ± 9.27a 78.50 ± 11.72b

aSection 1 = foodborne illness; Section 2 = risk factors; Section 3 = preventative measures; and Section 4 = food safety programs. Data represent the mean ± SD. Mean values in the samecolumn with different letters (a,b) are significantly different (P < 0.05). Total n = 454.

on each of the 7 knowledge areas were at least 72%. On theindividual knowledge domains the lower 25% of students’means were between 56.28% and 79.08%.

The lowest mean for correct answers among the lower 25%students is in the knowledge domain (6) facility, equipment, andlayout at 56.28%. This knowledge domain has the lowestaverage even among all students (72.29%). This might be due tothe fact that food safety managers often pay attention to thisknowledge domain only prior to opening of a facility. Anotherpossibility is that perhaps the current food safety manager wasnot involved in the construction and opening of the foodservicekitchen. This knowledge domain is important so food safetymanagers understand how the setup of a facility can affect itssanitation and safety.

The 2nd lowest mean (61%) for the lower 25% of students isknowledge domain (1) identifying foodborne illness terms,including foodborne illness bacteria and viruses. Thisknowledge gap is significant, since food safety managers musthave some understanding of foodborne bacteria and viruses toprevent foodborne illness.

The remaining knowledge domains where the lower 25% ofstudents’ means are below 70% include: (2) identifyingtime-temperature relationships with foodborne illness, (4)preventing contamination from purchasing to receiving, (5)cleaning and sanitizing, and (7) temperature control andcross-contamination. These knowledge domain gaps are critical,since they include most of the 5 major risk factors for foodborneillness in foodservice (FDA 2004). One area where the lower25% of students did quite well is (3) personal hygiene (79.08%).This may be due to the fact that many of the personal hygieneissues, such as hand washing, are well emphasized and areoften common knowledge.

Exam performance and types of educational materialsaccessed

Table 3 shows that using online training exclusively has thelargest number of students, followed by no instruction,textbook, textbook + online, DVD, and DVD + textbook. Themean correct answer scores for students who used a textbook,textbook + DVD, textbook + online, or just online methodswere higher than the mean score for those students whoreceived no instruction. For individual exam sections, studentschoosing online training scored higher on Section 1 (foodborneillness) and students choosing the textbook and online trainingscored higher on Section 4 (food safety programs).

ConclusionsThe online learning management system provides an

excellent tool for both exams and curriculum via distanceeducation. Local county cooperative extension agents provide akey role in this state–local partnership. Agents facilitate examsand assist students using minimal time and resources while stillachieving positive results. Using this form of distance educationprovides Food Safety Manager Certification access to students innearly all rural and urban Utah counties.

Using the mastery learning principle of individualizedinstruction tailored to identify deficiencies (practice testfeedback on incorrect answers tied to corresponding learningobjective materials) all students passed the exam and receivedcertification, indicating a minimum mandated level ofknowledge of food safety practices. Student performance in thedifferent knowledge domains provides feedback for futureinstructional emphases. The use of distance education andself-guided learning is successful in enabling students to achieve

62 Journal of Food Science Education—Vol. 9, 2010 Available on-line through ift.org

Food safety manager knowledge assessment . . .

passing scores on the exams and receive certification. Thespread of students choosing one or more of 3 alternativelearning methods (textbook, online, DVD) demonstrates theirindividual preferences. Out of the number of students whochose to access educational materials, 67% used onlineinstruction and 40% used a textbook. These percentagesindicate an acceptance of online learning by students, yet stillsuggest a comfort level with textbooks. Agent feedback indicatesapproximately 20 to 30 out of 55 Hispanic students requestSpanish language learning materials that are not available at thistime. This program deficiency is currently being addressed asSpanish curriculum for the test is under development.

Overall, this study indicates that food safety managerparticipants meet and exceed minimum knowledge levelsmandated by Utah state law. Self-guided learning optionsprovide students with choices to help them learn materials topass the certificate exams. Student data, question data, andknowledge domain data will be used to improve the exam andcurricula.

ReferencesBinkley M, Nelson D, Almanza B. 2008. Impact of manager certification on

food safety knowledge and restaurant health inspection scores in TippecanoeCounty, Indiana. J Culinary Sci Tech 6:343–50.

Blackboard, Inc. 2006. Blackboard learning system—Vista Enterprise License.Release 8. Designer and instructor reference. Available from: https://behind.blackboard.com/s/faculty/refcenter/docs/details.Bb?DocumentID=3254&pid=200&rid=5765&dt=. Accessed Aug 12, 2009.

Bloom BS. 1968. Learning for mastery. Evaluation Comment 1(2):1–12.EdcUtah. 2009. Utah demographics. Available from: http://www.edcutah.org/

files/Section3_Demographics_09.pdf. Accessed Jul 30, 2009.[FDA] U.S. Food and Drug Administration. National Retail Food Team. 2004.

FDA report on the occurrence of foodborne illness risk factors in selectedinstitutional foodservice, restaurant, and retail food store facility types.Available from: http://www.fda.gov/Food/FoodSafety/RetailFoodProtection/FoodborneIllnessandRiskFactorReduction/RetailFoodRiskFactorStudies/ucm089696.htm. Accessed Jul 30, 2009.

[FDA] U.S. Food and Drug Administration. 2005. Model food code. Availablefrom: http://www.foodsafety.gov/∼dms/foodcode.html. Accessed Jul 30,2009.

Kneller P, Bierma T. 1990. Food service certification-measuring theeffectiveness of a state program. J Environ Health 52:292–4.

Martin KE, Knabel S, Mendenhall V. 1999. A model train-the-trainer programfor HACCP-based food safety training in the retail/food service industry: anevaluation. Journal of Extension 37:3FEA1. Available from: http://www.joe.org/joe/1999june/a1.php. Accessed Aug 12, 2009.

[NRA] Natl. Restaurant Assn. 2000. Utah restaurant industry at a glance.Available from: http://www.restaurant.org/pdfs/research/state/utah.pdf.Accessed Jul 30, 2009.

Raval-Nelson P, Smith P. 1999. Food safety certification and its impacts. JEnviron Health 61:9–12.

Utah Dept. of Health. 2009. Utah Rule R392-101. Food safety managercertification. Available from: http://www.rules.utah.gov/publicat/code/r392/r392-101.htm. Accessed Mar 15, 2009.

Available on-line through ift.org Vol. 9, 2010—Journal of Food Science Education 63