Measuring the Effectiveness of Lagoon Quest: A Case Study...

CIR1528

Measuring the Effectiveness of Lagoon Quest: A Case Study in Environmental Education Program Evaluation1

Judith Chen-Hsuan Cheng, Martha C. Monroe, and Alison Gordon2

1. This document is CIR1528, one of a series of the School of Forest Resources and Conservation Department, UF/IFAS Extension. Original publication date August 2008. Reviewed August 2017. Visit the EDIS Web site at http://edis.ifas.ufl.edu.

2. Judith Chen-Hsuan Cheng, research assistant; Martha C. Monroe, professor, School of Forest Resources and Conservation, UF/IFAS Extension, Gainesville, FL 32611; and Alison Gordon, Brevard Zoo.

The Institute of Food and Agricultural Sciences (IFAS) is an Equal Opportunity Institution authorized to provide research, educational information and other services only to individuals and institutions that function with non-discrimination with respect to race, creed, color, religion, age, disability, sex, sexual orientation, marital status, national origin, political opinions or affiliations. For more information on obtaining other UF/IFAS Extension publications, contact your county’s UF/IFAS Extension office.

U.S. Department of Agriculture, UF/IFAS Extension Service, University of Florida, IFAS, Florida A & M University Cooperative Extension Program, and Boards of County Commissioners Cooperating. Nick T. Place, dean for UF/IFAS Extension.

Judith Chen-Hsuan Cheng

Martha C. MonroeSchool of foreSt reSourceS

and conServationuniverSity of florida

Alison GordonBrevard Zoo

Measuring the Effectiveness of Lagoon QuestA case study in environmental education program evaluation

CIR 1528

Acknowledgments

This is both a report of the evaluation of the second year of Lagoon Quest’s county-wide implementation and a case study of program evaluation written to accompany professional development programs for environmental educators. The format parallels the process described in the North American Association for Environmental Education’s workbook, Evaluating Your Environmental Education Programs. We hope that NAAEE members as well as those from the Association of Zoos and Aquariums, US Fish and Wildlife Service, US Forest Service, and National Park Service will find it helpful as they work to measure and improve their program successes.

We gratefully acknowledge all of the zoo staff and the fourth-grade teachers and students in Brevard County who participated in Lagoon Quest evaluation. Our activity and this publication were made possible by the leadership, support, and special skills of the following individuals:

Keith Winsten, Executive Director, Brevard Zoo

Andrea Aubert, Education Program Coordinator, Brevard Zoo

Ed Short, Elementary Science Coordinator, Brevard Public Schools

Patty Leonard, Sherwood Elementary School, Brevard Public Schools

Kathy Lueck, Sherwood Elementary School, Brevard Public Schools

Linda Garrett, Quest Elementary School, Brevard Public Schools

Neyda Riley, Assistant Director of the Office of Accountability, Brevard Public Schools

Anne Graham, Technology Integrator, Brevard Public Schools

Kathy Hill, IRL Education Coordinator, Indian River Lagoon National Estuary Program

Eleanor Sommer, Editor

Shannon Paulin, Designer, JS Design Studio

Funding for this work was provided by:National Fish and Wildlife Foundation School of Forest Resources and Conservation UF/IFAS

Printed by Storter Childs on FCS-Certified paper

Credited Photos courtesy of the Indian River Lagoon National Estuary Program

© 2008 University of Florida

Pho

to b

y G

. Co

hen

Located in East Central Florida, Brevard County is home to approximately 531,250 people,1 most of whom live near the Atlantic Ocean in the largest cities,

Palm Bay and Melbourne. A large number of tourists visit the ocean and NASA’s Kennedy Space Center at Cape Canaveral. Paralleling the coast is the Indian River Lagoon, a linear, 156-mile estuary, which greatly enhances Brevard County’s agricultural, fishing, economic, and recreational opportunities.2 Like other subtropical estuaries, the Indian River Lagoon has very high biological productivity and is the home of many endangered and threatened species. To help protect this important treasure, Brevard Zoo developed an environmental education program, called Lagoon Quest, to enhance environmental stewardship.

Lagoon Quest was originally developed to in-crease knowledge of local environmental issues and estuarine ecology for students in fourth grade through eighth grade classes. Because of the success of this voluntary program, Brevard Public Schools included the program in their fourth-grade science curriculum beginning in 2005. Now all fourth-grade students and teachers are required to participate in Lagoon Quest. The purpose of the program’s expan-sion is to encourage all students to respect and be responsible for the Indian River Lagoon.

In the summer prior to the required expansion of the Lagoon Quest program, zoo staff and six fourth-grade teachers developed the Teacher Guide and Student Guide (Lagoonie Logbook). The Teacher Guide provides an introduction to the study trip and instructions for the Lagoon Quest in-class activi-ties, while the Student Guide provides worksheets for students and their parents to complete. Twelve in-class activities were designed to bracket the one-day study trip to the Indian River Lagoon, which is staffed by Brevard Zoo. During the study trip students learn about the Indian River Lagoon, collect aquatic organisms while wading in the water, test water quality, discuss water pollution with an Enviroscape model, and hunt for litter and evidence of animals on a beach scavenger hunt using GPS technology.

Brevard Zoo asked students and faculty at the University of Florida’s School of Forest Resources and Conservation to assist with the program evaluation. When the National Fish and Wildlife Foundation (NFWF) agreed to provide funding for the evaluation, the process was enlarged to accommodate their interests and the graduate student’s research agenda. This training case study discusses the development of the program evaluation and provides the results of the evaluation for Lagoon Quest.

I feel the program helps our students understand more of the world around them. —Teacher

m e A s u r i n g t h e e f f e c t i v e n e s s o f l A g o o n Q u e s t 3

1. U.S. Census Bureau (Retrieved, 2006, 07, 04) http://www.census.gov/Press-Release/www/releases/archives/facts_for_features_special_editions/006838.html

2. South Florida Water Management District (Retrieved, 2006, 07, 04). http://www.sfwmd.gov/site/index.php?id=67

Focusing the EvaluationZoo staff and Brevard school district administrators helped articulate the purpose of the evaluation: to understand if the program is increasing students’ knowledge about the Indian River Lagoon and helping to develop a conservation ethic for taking care of the Lagoon and its watershed. Additional interests were exploring the impacts of expanding a voluntary environmental educational program to the entire fourth grade, identifying changes to improve the program, and identifying indicators that could be used over time to track the development of a conservation ethic in the county. The results obtained from this study will be used for program improvement and new program development. The program stakeholders include the zoo staff, school district administrators, and fourth-grade teachers (including those who helped write the program materials). The program audience includes about 5500 fourth-grade students, about 260 fourth-grade teachers, and about 900 parent chaperones that accompany the study trips.

To become familiar with the program, the evaluators created a logic model for Lagoon Quest. The logic model was presented to zoo staff to confirm that it included all of the elements of the program and to help identify those elements that should be included in the evaluation (Table 1).

n situAtion: The Indian River Lagoon is a biologically diverse estuary that is threatened by rapid population growth, development in the

watershed, and increased boat traffic.3 The long-term health of the Lagoon is dependent upon the knowledge and actions of local citizens, their willingness to support regulations and policies that protect the Lagoon, and their investment of resources. Brevard Zoo is a local environmental education facility that developed Lagoon Quest. Brevard Public Schools has adopted the program for all fourth-grade classes to support science education and help strengthen a conservation ethic in the county.

n inputs: The Lagoon Quest inputs are funding (from NFWF and Brevard Schools), written materials (Teacher Guide, Lagoonie Logbook from Brevard Zoo, and Indian River Lagoon Activity Book from St. Johns River Water Management District), personnel (zoo staff, teachers, chaperones), and program equipment (from schools and zoo).

n outputs: An in-service teacher workshop, in-class activities, study trips, family activities, and family festival are outputs. The workshop introduced the Lagoon Quest teaching materials, in-class activities, study trip, and evaluation tools. The family activities are part of the Lagoonie Logbook, and the family festival is a zoo-sponsored exploration and celebration of the Indian River Lagoon for families of fourth-grade students. The program participants are fourth grade public school students, teachers, and chaperones. The festival participants are fourth grade public school students and their families.

The purpose of the evaluation is to understand if and how the program increases students’ knowl-edge about the Indian River Lagoon and attitudes toward caring for the Lagoon and its watershed.

4 m e A s u r i n g t h e e f f e c t i v e n e s s o f l A g o o n Q u e s t m e A s u r i n g t h e e f f e c t i v e n e s s o f l A g o o n Q u e s t 5

3. St. Johns River Water Management District (Retrieved, 2007, 08, 16) http://www.sjrwmd.com/programs/index.html

n outcomes: Lagoon Quest has three types of outcomes: short-term, medium-term, and long-term. Short-term outcomes are students’ knowledge and teachers’ knowledge, experience, and skill enhancement. Medium-term outcomes are students’ environmental attitudes and pride in the Indian River Lagoon. The long-term outcomes are enhancement of conservation behaviors, interest in conservation careers, and the improvement of the biophysical environment.

n Assumptions: A number of assumptions about education and learning are built into this program: Hands-on, real-world learning experiences are meaningful and memorable. Teacher-led classroom activities help introduce and reinforce study trip concepts. One school-based experience can help motivate students to engage their parents in additional lagoon-related activities. The activities of collecting organisms, picking up litter, and being in the lagoon help motivate students to care for the lagoon.

situAtion inputs outputs outcomes

(What we invest) Activities (What we do)

Participation (Who we reach)

Learning (Short-term)

Action (Medium-term)

Impact (Long-Term)

Like other subtropical estuaries, the Indian River Lagoon has very high biological produc-tivity and is home to many endangered and threatened species. To protect this impor-tant treasure, Brevard Zoo developed an environmental educa-tion program that enhances environ-mental stewardship: Lagoon Quest.

funding• school board• NFWF

materials• SJRWMD• student/

family guide• teacher guide

personnel• teachers• zoo staff• chaperones

equipment• school• field

In-service teacher workshop

Activities• pre-trip • study trip• post-trip

activities• family

activities• family

festival

• fourth grade students

• fourth grade teachers

• parents• chaperones• families

student knowledge: • Indian River Lagoon (IRL)

ecology• geography • watershed issues• water quality

Teachers’ and chaperone’s impression of program and environmental attitudes becomes more positive

teaching skills• comfort and confidence to

teach about IRL• address state standards through IRL• address real world issues• address critical thinking skills

students’ attitudes:• more positive

toward environment

• increased connection to nature

• feelings of pride regarding IRL

Behavioral intentions and conservation behaviors (personal and family):• Conservation careers

explored• Improved

environmental health of IRL

Table 1. Logic Model for Lagoon Quest Program


Box 2. Who Could Be in a Control Group?

Evaluations typically compare the results of people who have participated in a program to people who have not to measure the value of the program. Since the Lagoon Quest program involved all fourth grade teachers and students, a control group would have to be from another county. That raised questions of whether the control group would really be similar to the treatment group on the characteristics that matter—access to the Lagoon, socio-economic status, awareness of the estuary, interest in aquatic ecology. Because the program is site specific, i.e., the Indian River Lagoon, we decided a control group was not feasible.

Logistics and TimelineThe Brevard Zoo asked students and a professor from University of Florida to lead the program evaluation process. Funding for the evaluation came from NFWF in a two-year matching grant. Evaluators became familiar with Lagoon Quest through observations and interviews in fall 2005 (the first season of the expanded program) and developed the logic model. In January 2006 a stakeholder meeting was held to review and improve the evaluation plan, logic model, and timeline.

In spring 2006, student and teacher interviews were conducted to develop evaluation tools in the form of student, teacher, and chaperone surveys. Following the pilot-test, the revised evaluation tools were distributed to the stakeholders for review. The data collection process was implemented from fall 2006 to spring 2007 and the data analysis was initiated in spring 2007.

The Brevard school administrators preferred requiring the evaluation activities as part of the program for all students and teachers. The school system posted the surveys and test of knowledge on its Web site and expected full participation (Box 1). As a result, there is no reasonable control group (Box 2) for this evaluation. A completed evaluation focus form is provided in Table 2.

Evaluations depend on obtaining good data from participants. In general, evaluators can do a better job of obtaining, managing, and working with data from a smaller group of people. If the population can be sampled, and if the sample will be representative of the whole, this is often the preferred strategy. In the Lagoon Quest program, however, school administra-tors wanted everyone included in the evaluation to build ownership and treat everyone equally. Because the administration made the evaluation a required part of the curriculum, they were confident that every teacher would participate.

Box 1. Conduct a Sample or a Census?


A. PURPOSE FOR EvALUATION

who is the intended user of the evaluation results? n Brevard Zoo and Brevard Public Schools n Training courses in environmental education evaluation n National Fish and Wildlife Foundation

what is the intended use for the evaluation results? n To better understand if program outcomes are being achieved. n To identify potential areas for further program improvement. n To provide an example for other agencies that want to expand their

programs.

evaluation purpose statement: n To measure the program’s effects on students’ knowledge about the

Indian River Lagoon.

n To explore the impacts and consequences of expanding a voluntary environmental educational program to the entire fourth grade in Brevard County.

n To suggest changes to improve the program.

n To identify indicators that could be used over time to track the development of a conservation ethic in Brevard County.

Are you able to reach consensus among major stakeholders on the purpose of the evaluation? Yes, we met with all the stakeholders and reviewed the logic model and evaluation tools.

is the intended use and user of the evaluation clear, specific, and well-defined? Yes, please see the answers on the left.

is what is at stake in this evaluation appropriate for an “in-house” evaluation? No, Brevard Zoo hired evaluators from University of Florida to evaluate the program.

can the evaluation purpose be addressed in a way that respects the rights and dignity of those involved? Yes, confidentiality was maintained and the participants had right to withdraw their consent.

will evaluation results be used? Yes, please see the answers on left.

will decisions be made be based on the data that are collected? Yes, Brevard Zoo and Brevard Public Schools will use the evaluation results for further program improvement.

B. DESCRIPTION OF PROGRAM TO BE EvALUATED:

see the logic model in table 1 Are the program objectives well-defined? Yes.

is it possible for the program objectives to be achieved with the intended audience? Yes.

is the program as carried out similar to the way it was intended? Yes.

is the program grounded in sound assumptions? Yes: experiential education, science education, environmental education.

is the program likely to achieve the stated goals/objectives based on the program’s inputs, outputs, and/or assumptions? The program objectives intend to achieve short-term and medium-term outcomes. The long-term outcomes will be explored in the future evaluation opportunities.

does the program have the potential for sufficient impact, thus warranting the time and expense of evaluation? Yes, since the program is required of all students. We would like to understand the long-term impacts from a future longitudinal study.

c. logisticAl considerAtions:

Available staff for the evaluation: Evaluators from University of Florida, Zoo staff

information needed by: Brevard Zoo, Brevard Public Schools, National Fish and Wildlife Foundation

resources available for the evaluation: n Funding and staff from Brevard Zoo and NFWFn Evaluators from University of Florida

political context/external factors: n Because full participation is expected, there is no reasonable

control group for this evaluation.n Extreme weather conditions could delay the program

or create negative attitudes.n Mandatory participation might create negative attitudes.

is the desired evaluation purpose feasible given available staff, time, and resources? Yes, two years were available and needed.

given logistical constraints, can an evaluation be carried out that would yield useful and relevant information? Yes, and the evaluation report will be produced for AZA, NAAEE, and FWS training courses.

Table 2. An Evaluation Focus for Lagoon Quest


evAluAtion Questions

indicAtors sources of informAtion

methods / tools

sAmpling/ design

A. how was lagoon

Quest implemented and perceived?

• Response to teacher surveys• Response to student surveys• Staff observations

• Fourth grade public

school teachers• Fourth grade public

school students• Staff

• Teacher surveys• Student surveys• Staff observations

• Post only• All fourth grade public school teachers and students in Brevard County, Florida, who participated in Lagoon Quest program

B. does the lagoon Quest program influence students’ knowledge of indian river lagoon?

• Responses to knowledge test • Fourth grade public school students

• Student test of knowledge

• Pretest/posttest• All fourth grade public school students in Brevard County, Florida, who participated in Lagoon Quest program

c. do students enjoy their experience with the lagoon?

• Response to student surveys • Fourth grade public school students

• Parents who chaperoned the study trip

• Student surveys• Chaperone survey

• Post only• All fourth grade public school students in Brevard County, Florida, who participated in Lagoon Quest program

• Parents who chaperoned the study trip

d. do teachers’ previous experiences in environ-mental education affect their use of the lagoon Quest unit?

• Teachers’ previous experience in environmental education

• Students’ knowledge test scores

• Fourth grade public school teachers

• Fourth grade public school students

• Teacher surveys• Student test of

knowledge

• Post only• All fourth grade public school teachers and students in Brevard County, Florida, who participated in Lagoon Quest program

e. how can lagoon Quest program be improved?

• Suggestions from teacher and chaperone surveys

• Comments to staff• Observations by staff

• Chaperones • Staff• Fourth grade public

school teachers

• Chaperone survey• Teacher survey • Staff records

• Post only• Parents who went on the Lagoon Quest

study trips• The zoo staff observation forms filled out during the study trips

Table 3. Evaluation Plan Matrix

Developing an Evaluation PlanThe evaluation plan began with brainstorming evaluation questions that could be asked about the program. Because of the various aspects such as the in-service training, in-class activities, study trip, and family festival, many potential questions were suggested and discarded. Although the program discussed conservation behaviors, evaluators initially believed that

the fourth-grade program did not emphasize conservation behaviors that youth could undertake in a manner that would create a measurable change. Stakeholders, however, suggested that students’ intention regarding picking up litter and returning to the Lagoon would be appropriate indicators of program success, so these conservation behaviors were included. Because the family festival was not initiated until spring 2007, it was removed from the evaluation plan. The final evaluation questions are explained later and are summarized in Table 3.


Tool Development and ModificationThe initial student knowledge test was developed by the Lagoon Quest team of teachers and zoo staff in 2005. Evaluators in the University of Florida analyzed the results after the first year of the program and offered some suggestions for vocabulary and additional questions, based on student interviews.

The other evaluation tools were initially designed by graduate students from University of Florida in fall 2005 after observing the Lagoon Quest study trip and interviewing teachers. In spring 2006, stakeholders reviewed these evaluation tools and offered suggestions. School district staff confirmed their interest in the answers to all five evaluation questions and suggested that evaluators develop an additional tool, the chaperone survey. The student attitude survey was significantly revised after pilot testing and interviews with students. The teacher attitude survey was tested with teachers from the planning committee. The following tools were used in the Lagoon Quest evaluation:• Teacher attitude survey• Student knowledge test • Student attitude survey• Staff observation form• Chaperone survey

Additional details about these tools are provided in the methods and tools portions of each question.

? evAluAtion Question A:

how was lagoon Quest implemented and perceived?

n indicAtors: Responses to student attitude surveys, teacher surveys, and staff observations.

n sources of informAtion: Fourth-grade public school teachers, fourth-grade public school students, and staff.

n methods And tools: The tools used to collect this information were a student survey, a teacher survey, and staff observation forms. Before the student survey was developed, an interview guide was created to explore students’ interest in the natural environment and their daily outdoor activities. Fifteen group interviews were conducted in four fourth-grade classes. After the survey was developed, two pilot tests were conducted in fourth-grade classes to check the vocabulary and language. The survey included questions that explored students’ in-class and study trip experiences.

The teacher survey was developed to gauge teachers’ previous experience in using Lagoon Quest and other environmental education materials; to determine teacher’s attitude toward Lagoon Quest; and for teachers to report what portion of the program they completed, their perception of support from their principals, other teachers, and parents,


Pho

to b

y A.

hu

mPh

rey

and their level of knowledge of and comfort with teaching about the lagoon.

Zoo staff completed an observation form at the end of each study trip. The form was created after the evaluators observed study trips and included questions about the weather conditions; species diversity; and the preparation of teachers, chaperones, and students.

n sAmpling And design: Post program only, with all fourth-grade students and teachers who participated in Lagoon Quest and zoo staff who implemented the Lagoon Quest study trips. Fourth-grade teachers helped implement the student survey. Teachers were asked to have students complete the Web-based or paper-and-pencil surveys after completing Lagoon Quest. The teacher survey was made available to teachers through the school district’s Web site, in the Teacher Guide, and through the school mail. The staff observation form was distributed to the zoo staff who conducted the study trips.

? evAluAtion Question B:

does lagoon Quest influence students’ knowledge of indian river lagoon?

n indicAtor: Students’ knowledge of the Indian River Lagoon, as measured by a test of knowledge.

n sources of informAtion: Fourth-grade students who participated in Lagoon Quest.

n methods And tools: An initial knowledge quiz was developed by teachers and zoo staff and based on the Lagoonie Logbook and Indian River Lagoon Activity Book. It was tested in the first program season and revised to increase variability. The second version of the test consisted of eighteen questions ranging from simple to difficult. The test was developed to explore students’ ecological concepts of Indian River Lagoon and the basic concepts of ecology. Teachers reviewed and approved of the test.

n sAmpling And design: Evaluation participants consisted of all fourth-grade public school students. The design was pretest/posttest. The test was distributed through the school system Web site and for those who requested a paper version, the mail.

? evAluAtion Question c:

do students enjoy their experience with the lagoon?

n indicAtor: Responses to a student attitude survey.


n methods And tools: The tool used to collect the information was a student attitude survey described in evaluation question A.


n sAmpling And design: All fourth-grade public school teachers were asked to use the Lagoon Quest materials, including the student survey. The survey design was posttest only, because attitudes are not likely to change very quickly and some questions required the program experience to answer. Teachers were asked to have students complete the Web-based or paper-and-pencil surveys after completing the Lagoon Quest unit.

? evAluAtion Question d:

do teachers’ previous experiences in environmental education affect their use of the lagoon Quest unit?

n indicAtors: Responses to teacher survey and students’ knowledge scores from their Lagoon Quest knowledge test.


n methods And tools: The development of the teacher survey was described in evaluation question A, and the student test of knowledge was described in evaluation question B.

n sAmpling And design: Post program only, with all fourth-grade teachers who participated in Lagoon Quest. The survey was

distributed to teachers by the school district on their Web site, in the Teacher Guide, and by paper through the school mail.

? evAluAtion Question e:

how can the lagoon Quest be improved?

n indicAtors: Teachers’ and chaperones’ suggestions and staff observation records.

n sources of informAtion: Fourth-grade public school teachers, students’ parents who attended the study trips as chaperones, and zoo staff who implemented the study trip activities.

n methods And tools: Surveys and observations were used to obtain the information. The survey for chaperones consisted of several open-ended questions to understand their attitudes toward Lagoon Quest and their idea about how the program affected their children.

n sAmpling And design: Teachers and chaperones who accompanied the study trips, and the staff, were the study population. The surveys were distributed to the chaperones by the zoo staff upon the completion of each study trip. The chaperones sent the surveys back to the evaluators by business reply envelopes enclosed with the surveys. The staff completed the observation form at the end of each day. Teachers completed the teacher survey on the Web or on paper.

In the end of the year, we ask students to pick their favorite activity and most of them choose Lagoon Quest. —Teacher


Pho

to b

y K.

bu

rKe

Data CollectionIn keeping with the technology requirements of state education reform efforts, the school administration wished to encourage the use of technology with the Lagoon Quest evaluation. In addition, the program developers saw this as a good opportunity to develop a long-term tracking system for students’ environmental knowledge, attitudes, and intentions. As a result, they requested that all students and all teachers be involved in the program evaluation and that the data collection be conducted in the schools’ technology class, using the school system’s Web site (Box 3).

Instructions for the evaluation implementation were provided in the Teacher Guide to the Lagoon Quest program, reinforced during an in-service workshop, and mentioned in teacher updates sent by the teacher coordinators of the program.

In August 2006, three forms were posted on the school system’s Web site: student test of knowledge (to be completed before and after the program), student attitude and experience survey (post program), and teacher attitude and experience survey (post program). Students and teachers were asked to use their school-assigned identification number to maintain confidentiality, allow evaluators to group students into the proper classroom, and link the classes to the teachers.

Some teachers voiced concerns that they either could not or did not wish to use the school’s Web site or computer facility. Those teachers were sent packets of student tests and surveys on paper. The act of requesting a paper survey may have increased a teachers’ commitment to using it. Chaperones were handed their survey and a return envelope by zoo staff during the study trip. Zoo staff completed their observation forms daily and submitted them at the end of October, November, and April.


Box 3. Using Computers to Collect Data

We discovered several challenges to using computers to collect data. Unfortunately, all schools do not have a computer lab and although all classrooms have a computer, it is not always easy for teachers to enable every student to enter his or her own data. Teach-ers may not have been aware that mistakes made while entering information on the Web form can be corrected before submitting it. After reviewing the results, evaluators felt the paper version was more accurate and accessible.

Lagoon Quest affected my child in a very positive way by teaching him about the animals in our Lagoon as well as how important it is to preserve their habitat. —Chaperone

Pho

to b

y S.

Ch

Alm

erS

As the paper versions of the tests and surveys were sent to the evaluators, they were entered into a spreadsheet. The school system compiled the computer-based Web responses into parallel databases and gave the evaluators access to their system. By September 29, 2235 pretests of knowledge had been submitted. By November 27 this number had risen to 2772 but only 1470 posttests were completed. Reminders to complete the surveys were distributed to teachers in November 2006 and March 2007. By April 18 a total of 3463 pretests, 2369 posttests, 1084 student surveys, and 99 teacher surveys were received.

Data AnalysisStudent knowledge was measured by eighteen questions regarding Indian River Lagoon, estuary ecology, and water quality. Because the test was administered before and after the program, one can assume that something in the intervening time produced any changes observed in the answers. It could have been a TV program, increased parental interest in the Lagoon, or teacher-led activities and instruction. Although any one student could have watched a TV program on the Lagoon, it was not likely that thousands of students did. Given the large sample size, an increase in knowledge was deemed likely to have been a function of the program. Knowledge of the Indian River Lagoon

was defined as understanding the concepts presented in Lagoon Quest: watershed, local biodiversity, estuarine ecology, and water quality.

Several other variables were used to interpret test results, including Title I status (Box 4) and teacher experience with environmental education. Teacher experience and familiarity with the lagoon could have led to a difference in instructional quality, amount of time spent on the material, and therefore student knowledge.

Data were entered, student numbers were matched, and the student tests were compared against the correct answers, providing a student score for correct responses. After data cleaning, 3398 pretests and 2093 posttests were useable, but only 1616 were matched pairs. This enabled evaluators to determine the difference between using matched


Box 4. Determining Impacts on Students

Originally, we planned to correlate students’ Lagoon Quest knowledge test with their FCAT scores, the state achievement test. However, fourth grade students take the FCAT in reading and writing but not science. Not having science achievement data to work with, we used Title I school classification. These are schools with at least 40 percent of the students from families whose income is below the poverty level. Because of correlations between income, health care, home environment and achievement,4,5 we could determine how the Lagoon Quest program works for disadvantaged students by comparing Title I and non-Title I student scores.

4. Borman, G. and J. D’Agostino. (1996). Title I and student achievement: A meta analysis of federal evaluation results. Educational Evaluation and Policy Analysis. 18(4), 309-326.

5. Duncan, G and J. Brooks-Gunn. (1997). The effects of poverty on children. The Future of Children. 7(2), 55-71.

and unmatched data. Both the matched pairs and unmatched data were analyzed with t-tests. The students whose entire class had perfect scores in posttest were excluded in data analysis (Box 5). The random missing data were replaced by the mean value of each variable (Box 6). Descriptive statistics were conducted to describe the materials that were used in classes, the teachers’ perception of Lagoon Quest and study trips. ANOVA was used to assess the difference between student scores from Title I and non-Title I schools. Correlation analysis was used to assess the relationships between teacher experience and student scores. Descriptive statistics were used to describe student’s enjoyment of Lagoon Quest. Independent t-test was used to determine if teachers’ previous

experiences with environmental education affected the number of Lagoon Quest activities they completed. And finally, chaperones and teachers’ suggestions were categorized into different themes, and direct quotes were provided.

Box 5. Perfect Posttest Scores?

Six classrooms submitted perfect posttests for every student in the class. Because schools with strong records in science achievement were not scoring as well, evaluators wondered what occurred with a few classrooms in a few schools. The teachers might have effectively taught the content that the knowledge test covered. On the other hand, the teachers might have reviewed the correct answers with students before the post-tests were submitted. From talking to teachers, evaluators learned that some teachers asked students to complete the posttest on paper, and then enter their own responses on the computer. Some teachers might use the opportunity of the quiz to con-tinue teaching as they review the correct responses. It may have been the case that some teachers asked students to input the corrected version of their posttest rather than the initial version. Not knowing exactly how these high scores were obtained, we excluded these classes from the data analysis.

Missing data is a problem when conducting data analysis because some statistical tests are sensitive to the number of cases in each variable. Evaluators need to have their own standards to address missing data. We coded missing values when the data were entered and then dealt them in two ways. Surveys in which an entire section was missed were considered incomplete, and the entire sur-vey was deleted. For the surveys that contained random single missed questions, the mean for those variables were used to replace the missing values.

Box 6. Dealing with Missing Data


Results? Question A

how was lagoon Quest implemented and perceived?

One hundred and thirty-one teachers responded to the teacher survey. Among these teachers, nearly all com-pleted the first two activities and about 80% completed activities 3, 4, and 6. About 75% of teachers completed activities 5 and 9 while about 65% of teachers complet-ed activities 7 and 12. However, less than 65% of the teachers completed activities 8, 10 and 11 (Table 4).

Zoo staff reported in the observation forms that more than 80% of the classes seemed prepared for their study trip, suggesting that most teachers conducted

the pre-trip activities and followed the directions in the Teacher Guide.

More than 50% of the teachers indicated that Lagoon Quest was very helpful for increasing students’ awareness of Indian River Lagoon, while more than 45% of teachers said that Lagoon Quest was very helpful for increasing environmental responsibility. About 38% of teachers indicated that Lagoon Quest was very helpful for promoting conservation ethic. Regarding educational standards, most of the respondents indicated that Lagoon Quest was only somewhat helpful for addressing state standards in language arts, social studies, and math, while 40% of the respondents stated that Lagoon Quest was very helpful for addressing science standards (Table 5).

Activity used Before trip (%) used After trip (%) totAl used (%)

1. Brainstorming the lagoon! 87 3 90

2 . the irl Activity Book 88 9 97

3. can i get there from here? 76 2 78

4 . what’s for supper? 75 6 81

5. taking a closer look (i) 63 11 74

6. how healthy is the lagoon (i) 59 21 80

7. who am i? 47 20 67

8. wetland in a pan 42 20 62

9. indian river lagoon reflection 12 60 72

10. changing waters 18 36 54

11. taking a closer look (ii) 17 40 57

12. how healthy is the lagoon (ii) 21 44 65

Table 4. Lagoon Quest Activities Used by Teachers


I learned more about the lagoon and its inhabitants than I knew before. —Chaperone

About 70% of the teachers indicated that the zoo instructors were very knowledgeable, and only 9% of the respondents strongly agreed that the zoo instructors could improve their facilitation skills. About 65% of the teachers stated that it was easy to get parent chaperones, and 80% of them indicated that students were comfortable while in the water. Most teachers agreed that the study trip experiences were connected with classroom activities. Indeed, the teachers also specified that the study trips encouraged them to learn more about the environment. More than 90% of the teachers agreed or strongly agreed that the study trip was an essential element of the Lagoon Quest program (Table 6).

About 50% of the teachers indicated that the Lagoon Quest Teacher Guide, Indian River Lagoon Activity Book, and Lagoonie Logbook had very high educational value. They also indicated that students were better able to use the Indian River Lagoon Activity Book than Lagoonie Logbook (Table 7).

There were six family activities in the Lagoonie Logbook. About 42% of the students indicated that they did some family activities. Among 1392 students who completed the student attitude survey, about 10% reported that they completed family activity 1. About 19% of them completed family activity 6, and 14 % of them completed activity 3. However, less than 10% of the students

completed activity 2, 4 and 5 (Table 8). The Brevard Zoo received 61 certificates of completion from families that enabled 112 adults and 117 children to take advantage of the opportunity for free admission to the zoo.

These various data suggest that a large majority, but not all, of the responding teachers implemented Lagoon Quest as intended. The post-trip activities on water quality were not used as often as the other activities. The teachers are generally quite pleased with the program and its impact on student awareness and attitudes, even admitting that they learned more about the Lagoon from the program. As designed, the program is recognized as supplementing the science standards. For the most part, the family activities were not implemented. It is not possible to know if the teachers did not encourage students to take them home, if parents were too busy, or both.

? Question B

does lagoon Quest influence students’ knowledge of indian river lagoon?

There was a significant increase in knowledge about the Indian River Lagoon because posttest scores were significantly higher than pretest scores in the matched pairs (i.e., the p value is less than .01, assuming a 99% confidence level) (Table 9). The results of unmatched data (Table 10) are nearly identical.


how helpful wAs this progrAm for…. very little A little somewhAt much very much

Increasing your students’ awareness of Indian River Lagoon 0 1.6 7.1 34.1 57.1

Enhancing your students’ sense of environmental responsibility 0.8 0.8 14.3 38.9 45.2

Promoting your students’ conservation ethic 0.8 1.6 24.8 35.2 37.6

Addressing Language Art Sunshine State Standards 8.0 7.2 43.2 24.8 16.8

Addressing Social Studies Sunshine State Standards 5.6 9.6 37.6 24.8 22.4

Addressing Science Sunshine State Standards 0.8 1.6 19.2 38.4 40.0

Addressing Math Sunshine State Standards 13.7 17.7 41.1 18.5 8.9

Table 5. Teachers’ Perceptions of the Lagoon Quest Program (percent responding)

stAtements strongly disAgree

disAgree neutrAl Agree stronglyAgree

Zoo instructors had sufficient knowledge of program content 0 0.8 4.0 24.0 71.2

Zoo instructors’ could improve their facilitation skills 16.1 33.9 25.8 15.3 8.9

It was easy to get parent chaperones for the Lagoon Quest field trip 8.0 11.2 16.8 38.4 25.6

Students were comfortable in the water 0.8 5.6 13.6 58.4 21.6

Field trip experiences were not connected with the classroom activities 42.7 41.9 9.7 3.2 2.4

Field trip experience encouraged me to learn more about environment 0 3.2 29.8 47.6 19.4

Overall, field trip was an essential element of the Lagoon Quest program 0.8 0.8 6.4 20.8 71.2

Table 6. Teachers’ Perception of the Lagoon Quest Study Trip (percent responding)

stAtements very low low ok high very high

Your perceived educational value of the Lagoon Quest Teacher Guide 1.7 2.6 14.7 29.3 51.7

Your perceived educational value of the Indian River Lagoon Activity Book 1.0 1.0 14.4 35.1 48.5

Your perceived educational value of Lagoonie Logbook 3.0 3.0 19.8 27.7 46.5

The students’ ability to use the Lagoonie Logbook 5.0 4.0 26.0 32.0 33.0

The students’ ability to use the Indian River Lagoon Activity Book 0 3.2 18.1 34.0 44.7

Table 7. Perceptions of the Lagoon Quest Materials (percent responding)

fAmily Activity completed (%)

1. Build a plankton net 10.1

2. What happened to all that rain? 8.7

3. Who am I 13.6

4. Monofilament activity 6.2

5. Salt matters! 9.4

6. Clean it up! 18.7

Table 8. Lagoonie Family Activities Reported by Students

RESULTS

pretest meAn (n)

sd posttestmeAn (n)

sd t p

15.73 (1616) 3.32 21.06 (1616) 4.23 47.89 0.00

Table 9. Matched Cases of Pre and Post Test Scores


pretest meAn (n) sd postpost meAn (n) sd

15.70 (3398) 3.39 21.03 (2094) 4.28

Table 10. Unmatched Cases of Pre and Post Test Scores

An analysis can be conducted on each of the eighteen questions in the knowledge test to indicate where initial misunderstandings were changed and where learning had not yet occurred (Table 11). In this case, the greatest increase in learning is with questions 11, 13, 15, and 16, which covered topics of lagoon ecology, habitats, endangered species, and water quality. Over 75% of the students already knew the correct answers to questions 2, 4, 5, and 9, which included information about conservation ethics and the roles of plants and animals. Less than 50% of the students were able to answer questions 14, 15, 16, and 17 correctly after the program. These questions covered more challenging aspects of the topics of wetlands, endangered species, and water quality.

Thirty-one schools in Brevard County qualify for Title I status, represented by 677 matched pretests and posttests from respondents. The remaining 939 students with matched pretests and posttests attend non-Title I schools. There was a significant difference in the pretest scores between Title I and non-Title I schools, with the Title I schools scoring significantly lower than the non-Title I schools. After the program, there was still a significant difference between the posttest scores at the 95% confidence level (Table 12). However, the improvement in Title I schools is significantly greater than the improvement in non-Title I schools. Because Title I students

averaged lower pretest scores than non-Title I students, they had more potential to improve (Table 13).

? Question c

do students enjoy their experience with the lagoon?

The students rated how much they liked the study trip activities and classroom activities from one (did not like at all) to five (liked very much). Some data were missing, perhaps due to errors in using the computer system. To avoid the bias caused by missing data, the missing data were replaced by the mean value for that item.

Regarding Lagoon Quest study trip, a majority of students indicated that they very much liked catching fish in the lagoon and spending time in the water. Between 3% and 14% of students did not like any of the activities. Seeing birds and plants, finding things around Lagoon, and testing water quality in the Lagoon were less popular (Table 14).

With regard to Lagoon Quest classroom activities, the majority of students liked learning about animals and the Indian River Lagoon while learning about salinity and turbidity were not popular (Table 15). All teachers; however, did not use the classroom activity that covered water quality. Approximately 36% of teachers used all twelve lessons.


Pho

to b

y P.

Co

rAP

i

Table 12. Knowledge Scores for Title I and Non-Title I Schools

Table 11. Frequencies of Correct Answers in Pretest and Posttest

pretest posttest increAse

Question # freQuencies percentAge Question # freQuencies percentAge percentAge

1 1106 68.4 1 1476 91.3 22.9

2 1225 75.8 2 1403 86.8 11

3 815 50.4 3 1163 72.0 22.4

4 1487 92 4 1564 96.8 4.8

5 1345 83.2 5 1531 94.7 11.5

6 658 40.7 6 1041 64.4 23.7

7 976 60.4 7 1351 83.6 23.2

8 1118 69.2 8 1494 92.5 23.3

9 1494 92.5 9 1553 96.1 3.6

10 958 59.3 10 1157 71.6 12.3

11 485 30.0 11 979 60.6 30.6

12 653 40.4 12 1021 63.2 22.8

13 417 25.8 13 888 55.0 29.2

14 157 9.7 14 550 34.0 24.3

15 63 3.9 15 472 29.2 25.3

16 79 4.9 16 691 42.8 37.9

17 385 23.8 17 759 47.0 23.2

18 860 53.2 18 1204 74.5 21.3

title i schools non-title i schools

test meAn (n) sd meAn (n) sd F Sig.

pretest 15.05 (677) 3.22 16.22 (939) 3.31 50.39 0.00

posttest 20.80 (677) 4.35 21.24 (939) 4.14 4.31 0.04

title i schools non-title i schools

meAn (n) difference

sd meAn (n)difference

sd F Sig.

5.75 (677) 4.76 5.02 (939) 4.23 12.85 0.00

Table 13. Improvement between Pretest and Posttest Scores

RESULTS

not At All A little somewhAt much very much meAn*

Spending time in the water 3.2 2.9 5.4 21.3 67.2 4.51

Catching fish in the lagoon 3.7 4.2 5.6 23.1 63.4 4.42

Finding things around the lagoon 10.9 9.6 27.1 19.5 32.8 3.62

Testing water quality 14.0 19.0 32.3 19.7 15.0 3.03

Seeing birds at the lagoon 12.9 25.4 28.7 13.5 19.4 3.01

Seeing plants in the lagoon 12.8 35.4 20 16.4 15.4 2.96

Table 14. Student Enjoyment of Study Trip Activities (percent responding)

* scale of one to five in which one is “not at all” and five is “very much”


To investigate students’ overall enjoyment of Lagoon Quest experience, two composite variables were created. The first variable explored the overall enjoyment of Lagoon Quest study trip while the second variable measured the overall enjoyment of Lagoon Quest classroom experience. Both variables indicated that students liked their Lagoon Quest experience (Table 16).

? Question d

do teachers’ previous experiences in environmental education affect their use of the lagoon Quest materials?

Because Lagoon Quest is now required of all fourth-grade teachers, we wanted to explore teachers’ attitudes toward Lagoon Quest. We wondered if their previous experience with Lagoon Quest or their experience with other environmental education programs could predict their willingness to use these materials. We received 131 teacher surveys by the end of May; however, only 96 were completed because of a computer glitch. With

the term ending, school administrators asked that we not bother these teachers with a request to complete the survey. Among the 96 teachers, 62 did not have previous experience in environmental education while 34 had prior environmental education experience. Independent t-test was used to determine if teachers’ previous experience with environmental education affected the number of Lagoon Quest activities they completed. The results suggested that there was no significant difference in the number of Lagoon Quest activities used between teachers with and without environmental education experience. Among the 131 teachers, we received student data for only 35 of the classrooms. Of these 35 teachers, only 20 of them provided their environmental education experience (Box 7). Of these 20 teachers, only 6 teachers had experience in environmental education. Among the 20 teachers, there was no significant difference in students’ knowledge based on teacher experience with environmental education (Table 17). Similarly, there was no difference between teachers who had participated in Lagoon Quest

Box 7. Increasing Response and Completion Rates

A successful evaluation requires not only high response rate, but also a high completion rate. Twenty-four percent of the teacher respondents to the Lagoon Quest survey did not complete the entire survey due to the following: 1) a computer glitch failed to bring up additional pages, 2) the perception that some questions were not relevant, 3) mistaken skips. In addition to issues of confidentiality and anonymity, the Institutional Review Board that ap-proves research with human subjects expects that participants will be able to skip a question if they choose. Most respondents, however, will answer an entire survey if they believe their responses will be helpful. Providing clear and frequent information about how to complete questions and why each section will be useful can reduce the number of incomplete surveys. Paper-based instruments have been shown to have fewer skipped question than computerized instruments.6 Paper-based surveys are also easier to read and can easily to be noticed if people keep the surveys instead of throwing them away.


6. Paolo, A. and Gibson, C. and Partridge, T. and Kallail, K. (2000). Response rate comparisons of E-mail and mail-distributed student evaluations. Teaching and Learning in Medicine. 12(2), 81-84.

RESULTS

clAssroom topics not At All A little somewhAt much very much meAn*

Indian River Lagoon 1.8 2.9 5.4 16.6 73.4 4.6

Animals 1.6 4.5 8.4 26.9 58.6 4.38

The watershed 14.8 15.3 26.3 18.0 25.7 3.27

pH 13.0 18.1 27.7 18.2 23.0 3.22

Plants 10.9 24.3 26.4 19.1 19.3 3.12

Turbidity 16.8 28.3 22.7 14.7 17.5 2.96

Salinity 15.3 30.7 22.3 16.4 15.4 2.93

Table 15. Student Enjoyment of Classroom Activities (percent responding)

* scores

1 = did not like at all

2 = like a little

3 = somewhat like

4 = like much

5 = like very much

enjoyment scores*

Study Trip Experience 3.59

Classroom Experience 3.50

Table 16. Students’ Enjoyment of Lagoon Quest Program

test experienced teAchers meAn (n)

sd less experienced teAchers meAn (n)

sd F Sig.

pretest 15.63(6) 1.27 15.90(14) 1.15 0.32 0.58

Posttest 21.31(6) 3.58 21.20(14) 2.90 0.45 0.51

Table 17. Association between Environmental Education Experience and Pretests and Posttests

Box 8. Contacting Non-Respondents

If the response rate is not high (more than 70%) an evaluator should wonder if the respondents are truly representative of the population. The best way to find out is by contacting those who did not respond. Because of confidentiality concerns with the school system, however, surveys were coded with identification numbers, but the evaluators were not allowed to match those numbers to people to contact them. Other strategies to address the low response rate would be to compare the respondents’ demographic data to census data and to compare early respondents to late respondents. Neither strat-egy was feasible with the Lagoon Quest evaluation because fourth grade teachers and students may not be representative of the county population and response time was a function of the study trip schedule, not necessarily respondent interest. To achieve higher response rate, sending multiple reminder letters may help increase the response rate as long as recipi-ents are tolerant of them.

*scale of one to five in which one is “not at all” and five is “very much”

before and those who had not. The nonsignificant results might be due to the small sample size, but it might also suggest that the program was designed and the curriculum was written well enough for less experienced teachers to use the Lagoon Quest program as well as those with more experience. It could also mean that other

variables are more indicative of success than previous experience (Box 8).

Because of the small number of teachers included in this analysis, additional data were collected through focus groups at several schools where we did not receive many surveys from teachers. We visited 5 schools and spoke to 13


teachers, 11 of whom had participated in Lagoon Quest. Included were teachers with a great deal of experience from the days of voluntary participation and teachers who were brand new to the program. Interest in using Lagoon Quest ranged from very strong to rather weak. Teachers described the way they used the material and some of the barriers associated with completing the post-trip activities. With regard to Question D, we believe that prior voluntary Lagoon Quest experience is not an indicator of interest and commitment to using Lagoon Quest because some teachers were told to sign up for the study trip by their principals. However, it appeared that teachers who had prior experience in teaching science and had stronger commitment to Lagoon Quest may be more likely to spend more time using Lagoon Quest activities, to use additional teaching materials that supplement Lagoon Quest, and to reinforce conservation ethics.

? Question e

how can lagoon Quest be improved?

n chAperones’ suggestions for the progrAm

After the Lagoon Quest study trips were completed in 2007, 171 surveys from parents who attended the study trips as chaperones were reviewed. These surveys provided useful suggestions to Brevard Public Schools and Brevard Zoo for further program improvement. Suggestions included modification of program components and improvement of the environment around the trip areas. The suggestions were categorized into several perspectives, and some direct quotes are provided in the following paragraphs.

A majority of the chaperones enjoyed the study trip, appreciated the program, and credited the passionate and knowledgeable staff.

The two young people who spoke and helped were very knowledgeable, educated, very well spoken. Full of energy and you could tell they loved what they were doing! They also liked kids. Thank you for the fun day. Sincerely!

It was great, the presenters were well informed and in touch with the kids and my daughter and I both learned a lot.

Regarding program components, the chaperones recommended including more explanation on the threats to the health of the Lagoon and the strategies to conserve the Lagoon and its wildlife. Moreover, the chaperones suggested that more reflection could be included after each activity, especially after students got out of water. More information on how and why to help the lagoon might move the discussion toward conservation actions.

Some kind of summary discussion would have helped tie the day together. The introduction was great, the water experience was great, the pollution demonstration was well worth it, but can we see evidence of this pollution right here in the lagoon? Do the tests prove it? What can your school do, what can you do at home? Why is the diversity so important? We were left with these questions and I am sure the kids thought they had a great day of fun, but now what? (Six similar comments.)

Let kids see they can make a positive difference in ecology with care and forethought. (Two similar comments.)

Chaperones also suggested reducing the use of handheld devices such as GPS because they were not age appropriate and it was difficult for chaperones to assist their children. Other chaperones suggested that more resources such as nets could be provided, and that more instructions could improve the experience.

Do not use handheld device on treasure hunt. Children could not work them and neither could the adults. (Two similar comments.)

Seining was very difficult for some of the children. Maybe a little more instruction by Lagoon Quest instructors. The children were shown in school but it is not the same when you enter the water.

Extending the length of the study trip was suggested. Some chaperones indicated that more time should be allowed for discussion of findings and sharing of experiences.


Follow up on scavenger hunt. I’m not sure, but a lack of time apparently kept us from discussing our findings. It might be better to have each group focus on one part of scavenger hunt then allow each group to share their finds and comment about them.

Many chaperones also raised a concern about weather conditions. They mentioned that picking a warmer day will be helpful for children, teachers, and chaperones.

The field trip was excellent information was great. The temperature of the water was terrible; it was too cold to stand in. It was the worst time to go. I could not stand the burning in my legs. The other adults would not go in the water. This trip should be planned in warmer weather. Go during warmer weather not in the winter.

Schedule during warmer months as sea life in the shallows was too limited. (Six similar comments.)

Regarding the facilities in the trip area, the chaperones raised some concerns about the nets, the need for more trashcans, more bathrooms, and the possibility of changing the trip site.

I think that the group should have had more green nets for the kids. One net wasn’t enough. I would suggest at least three per group. The kids caught more interesting things with the green net than the large net. Great field trip. (Two similar comments.)

Have better bathroom facilities and more trashcans around. (Two similar comments.)

The chaperones also proposed having more staff for trips with larger numbers of students. The chaperones indicated that they were not familiar with the procedures, so more helpers would be an advantage for children to learn better.

Needed more knowledgeable assistants to help the kids in the river. Although the chaperones tried to help the kids, we were unable to catch anything. (Four other similar comments.)

n teAchers’ suggestions for the progrAm

One hundred and thirty-one teacher surveys were received by the end of May and only 96 were completed. Because we were unable to identify the nonrespondents, we cannot generalize to the entire teacher population. We can only summarize the comments that were submitted. More than 10 teachers stated their appreciation for the program while some teachers indicated that their students were enjoying the program. Following are some examples.

My students thoroughly enjoyed the program and our visit to the IRL. The IRL staff was well organized, knowledgeable and helpful.

I really love the curriculum and the trip and appreciate that the background work is so well prepared and thorough. There might even have been too much between both books we used.


I was very pleased with the educational knowledge and experience the Lagoon Quest program provided for me and my students.

A few comments indicated that some respondents were not thrilled with the program. They felt the requirement took too much time from their teaching.

This is a great optional program, but when it becomes another requirement it’s not as much fun for the teachers as it could be. We have so many things we are required to do, and this becomes just one more thing. I completed my responsibilities and tried to make it fun for the students, but I know many teachers in the county who didn’t. My curriculum is already mapped out for the year, and this had to be worked into my classroom theme.

Due to the FCAT reality, Lagoon Quest was not a priority in fourth grade.

I have enjoyed this program for many years. I do not like being scheduled to go on the trip. There are often other things happening which I should prefer not to change for the trip. I am also concerned for the ecology of the areas to which the thousand of students tramp through for weeks on end.

Many teachers provided suggestions for program improvement, specifically on the subject of the schedule, teaching materials, facilities at the trip sites, and evaluation tools.

Regarding the teaching schedule, several teachers suggested that the study trips be scheduled after FCAT

so the focus can be on teaching Lagoon Quest after students have concluded the other requirements.

The ideal time to take Florida’s fourth-grade students is in the spring after FCAT testing. (Five similar comments.)

Some teachers did not receive sufficient Lagoon Quest materials or they received the materials too late to prepare for the program. Some teachers wanted to be able to link these materials to the school science requirement so that they could correlate the program to their teaching schedule. Teachers also stated that there was a need to make the teacher guide and Lagoonie book coordinated.

Please make sure that the water management district sends enough student books to every school. (Three similar comments.)

We needed to get the materials earlier in the year. I felt rushed to get it done. I would like to see this correlate a bit more with the county required science test so that I don’t have to go back and reteach from the Lagoonie to get ready for the test.

The teacher manual needs further explanation of some activities. It also needs to reflect what is in the Lagoonie Logbook.

Teachers also commented on the facilities at the trip sites. Some areas did not have sufficient restrooms and trash cans to accommodate large groups of students. They suggested looking for a more appropriate place that could accommodate large groups.


Find a location that has enough bathrooms to accommodate sixty children and twenty plus adults. Kelly Park patrons were angry and verbally abusive because they had to wait outside the bathroom while sixty kids changed one at a time in two stalls. (Three similar comments.)

Some teachers suggested reducing the test and surveys for Lagoon Quest program evaluation. Additionally, they would like to have a paper version of the evaluation instead of a computer version.

Reduce teachers’ paperwork regarding lagoon quest activities (online surveys). Make sure teachers and students receive appropriate materials needed for this field trip in a timely manner.

The online test was awful. Please stick to the hard copy. Computer access/dependability can be a problem.

Besides program improvement, the teachers also provided the subjects that they want to learn more about in a teacher in-service training day. These topics are listed in Table 18.

The focus group discussions also revealed suggestions for program improvement. Because these focus group participants were selected from

schools that did not submit many teacher surveys, it is likely that their voices were not included in the previous summary of themes from the survey. Although we initially believed these teachers might be less enthusiastic about Lagoon Quest, some focus group comments were extremely positive. Teachers said their students count the Lagoon Quest study trip as one of the best things of the entire semester. Teachers found the Indian River Lagoon Activity Book easy for students to use and informative and suggested the Lagoonie Logbook be written in the same style. They also suggested making the Lagoonie activities more relevant to the study trip to enhance student interest.

Not all teachers completed the program, however. Some were stymied by the need to obtain a variety of equipment or materials to conduct the science experiments and would appreciate receiving kits for the activities. Others were severely limited by the lack of time to teach science and requested more activities that integrate the Indian River Lagoon into reading and writing exercises. Some requested that the zoo staff demonstrate all the experiments during the in-service training so the teachers will be more confident to conduct these experiments in class.

topics Activities

teaching method 4 How to fit in all of the pre-field trip activities before the actual field trip?4 How to incorporate more inquiry methods into the experiments?4 How to incorporate activities in the classroom to impact all students

program 4 I would like to know the agenda for the day at the lagoon, so I can let my students know what to expect4 How can this be related to FCAT4 I would like to do some of the family activities such as making a plankton net and demonstrating storm water runoff4 Water and program safety

water 4 Water experiments4 What does water quality indicate?4 Water quality and its effect

ecology 4 Identify species4 Animals and plant life around the lagoon4 Integrated Florida history and geography of IRL and animals identification into the field trip

environmental issues 4 Recycling4 Development along the IRL4 Temperature and weather in general, and as it relates to the health and activity in the lagoon

Table 18. Topics of Interest to Teachers


DiscussionThe results of the Lagoon Quest knowledge test suggested that students significantly improved their knowledge of estuarine ecology after they participated in Lagoon Quest. This is true for students in both Title I and non-Title I schools. According to the student surveys, students enjoyed Lagoon Quest. Some activities successfully attracted students’ attention, for example, the water activities and learning about animals. Activities such as salinity and turbidity were less popular, which might be because those concepts were new to the students and might have been difficult for them to understand. Some chaperones mentioned that their children expressed interest in returning to the Indian River Lagoon.

We believe that this program provides many benefits for students, especially for those who do not have previous experience with the Lagoon. It is unrealistic to say that one single program creates change in all students, but it may bring some environmental awareness for many students. For those students who want to participate in additional activities, more information is needed. Opportunities for additional activities and resources would be useful for students, teachers, and families who wish to learn about and experience the Indian River Lagoon.

According to teacher and chaperone surveys, there are a few challenges for future implementation of Lagoon Quest.

n chAllenges for lAgoon Quest

Teachers prefer to use Lagoon Quest during warm weather and after FCAT. However, on the current schedule, there are not enough weeks in the school year to accommodate everyone after FCAT. If it is possible to hire more Lagoon Quest staff, the four trip sites could be filled at the same time. All the trips could be finished earlier to avoid cold weather, and the staff would have time to make up the trips that are canceled due to bad weather.

Teachers and chaperones suggested having sites that can accommodate large groups of students


The program has a great affect on my children. They learn about the health of nature around them while having fun.

—Chaperone

Pho

to b

y C

. FiF

er

and avoid encounters with other visitors. Also, chaperones suggested having more staff to instruct children and more equipment (dip nets) for students to use during the experience. Are other sites available? Could a site accommodate two groups instead of one? If other sites are not available, it would be helpful to recognize these concerns and explain the limitations to teachers and chaperones. Additionally, an analysis of student knowledge by trip location suggested that the locations with less biodiversity are just as effective as other locations.

After the study trips, some chaperones asked questions such as, “So what now?” “How can people help the lagoon?” “What can kids do at home?” The curriculum focused on estuary ecology and science, but one of the program’s long-term intentions was to support students’ conservation behavior. To increase conservation behavior, science education could be connected with conservation education. Some of this can be done with discussion questions at the end of the day. The zoo could also design a fact sheet to provide tips for students, teachers, and chaperones to help the lagoon and the environment around them. The staff could consider modifying the scavenger hunt to allow more time for discussion of students’ findings and reinforcement of conservation practices. An additional follow-up activity in the Lagoonie Guide might also be useful.

We recognize that more than half of the teachers did not complete surveys, and that many responding teachers did not complete the entire unit. Encouraging greater participation will take a concerted effort from the schools and the zoo. Teacher in-service time, required all-grade meetings, and follow-up support may be needed. Additional research to better understand the needs of nonrespondents would be helpful.

It appears that few families completed the family activities in Lagoonie Logbook and took advantage of their free admission to the Zoo. The primary purpose of the family activities was to encourage parents to learn about Indian River Lagoon with their children. However, some parents did not know about these activities because the students did not bring the Lagoonie Logbook home. Perhaps multiple reminders could be sent to the fourth-grade teachers to encourage fourth-grade parents’ participation in Lagoonie family activities.

This evaluation suggests that Lagoon Quest is an effective and well-received program. It provides a foundation for future study of an important local ecosystem. By offering this program every year, teachers will increase their knowledge and improve their abilities to lead the in-class activities. This report provides ideas for the in-service training and other strategies that zoo staff might use to support fourth-grade teachers.


Pho

to b

y r

. Du

bin

28 m e A s u r i n g t h e e f f e c t i v e n e s s o f l A g o o n Q u e s t

Recommendations for EvaluationIn reflection, we would do some things differently and therefore recommend that future evaluation efforts consider the following suggestions:

1. Data from the Web-based surveys were much easier to analyze, but for young students answering questions accurately with a computer may be a barrier. Access to computers may also be a barrier. Some teachers requested paper versions of the surveys and others duplicated forms from the Teacher Guide. Until computers are more accessible in all schools, and youngsters and their teachers are more computer savvy, it may be best to use paper versions of a survey.

2. There are several ways to imagine future evaluation efforts, depending on what is important. We did a census instead of sample because the school system wanted to treat students and teachers equally. However, we were not able to obtain 100 percent response. If evaluating the program is paramount, it would be possible to randomly sample the study population and work more diligently to collect all of the forms from a fewer number of participants. If program ownership by all

participants is most important, shorter forms with fewer questions and more reminders to complete them may prove successful.

3. We requested students’ and teachers’ identification numbers for matching and research purposes. Although we emphasized confidentiality, some participants were more comfortable providing their suggestions without their identification numbers. If program evaluation is the primary purpose, emphasizing anonymity and the usefulness of responses are recommended, but collecting identification numbers will be essential to track respondents over time.

4. The focus groups were added because of the poor response rate, but they provided such valuable insights that we recommend that future evaluators consider ways to collect in-depth data from a small sample of teachers and youth.

5. Providing the answers to the knowledge quiz enables teachers to score the quiz. It also enables teachers to teach to the test and undermines its validity. A different posttest, if it reliably measures the same concepts, may be considered. Correct answers could be provided after tests were submitted.

This is the study trip the kids always talk about and remember. Kids still come back to me and tell me that was the best trip they have ever had. —Teacher

Pho

to b

y C

. FiF

er

m e A s u r i n g t h e e f f e c t i v e n e s s o f l A g o o n Q u e s t 29

Conclusion

Lagoon Quest is an extremely beneficial addition to the Brevard County curriculum. Most teachers, chaperones, and students who participated in the evaluation were very pleased with the program and committed to making it even better in future years. Because the program involves all fourth graders, the long-term commitments from Brevard Public Schools and Brevard Zoo to this program promise to bring significant opportunities for shaping the development of conservation attitudes, ethics, and career interests among the entire population of students. There are many possibilities for launching supplementary units and activities that reinforce the concepts that Lagoon Quest introduces and for monitoring and assessing changes over time to both the residents and the lagoon. The potential for long-term tracking of the entire school-aged population suggests an important opportunity for research and evaluation.

The program requires a significant amount of instruction, equipment, and time. Responding teachers have demonstrated that they can teach the concepts, though there is variation in program success. Providing the teachers who are less comfortable with the program with supportive material, relevant training, and assistance will be challenging. Fortunately, the thrill that fourth graders feel when catching organisms in the Indian River Lagoon should help bolster and sustain teacher interest and participation in Lagoon Quest.

Pho

to b

y e.

mo

on

Pho

to b

y D

. Ste

wAr

t

Lagoon Quest gave me a greater understanding of how special the Indian River Lagoon is and how important it is to protect it. —Chaperone

Follow-Up

The development of this evaluation report has involved several meetings with zoo and school staff and has resulted in their commitment to work toward the following types of changes:

1. Require that only new teachers attend the fall In-service Training.

2. Provide equipment and materials for in-class science experiments.

3. Provide a To Do List and Sunshine State Standard checklist in the Teacher Guide.

4. Include information about good environmental practices to help the Lagoon in the Lagoonie Logbook.

5. Use the study trip to reinforce concepts introduced in the pre-trip lessons.

6. Simplify the Lagoonie Logbook.

7. Create a package for parents that includes information about Lagoon Quest family activities, preparation for the study trip, Lagoon Quest family festival, and environmentally friendly practices that can help the Lagoon.

8. Design the knowledge test to better reflect the information covered in the study trip and pre- and post-trip activities.

30 m e A s u r i n g t h e e f f e c t i v e n e s s o f l A g o o n Q u e s t

this document is cir 1528, UF/IFAS school of forest resources and conservation .

original publication date: April 2008 visit the edis website:http://edis.ifas.ufl.edu

The Institute of Food and Agricultural Sciences (IFAS) is an Equal Opportunity Institution authorized to provide research, educational information and other services only to individuals and institutions that function with non-discrimination with respect to race, creed, color, religion, age, disability, sex, sexual orientation, marital status, national origin, political opinions or affiliations. For more information on obtaining other UF/IFAS Extension publications, contact your county’s UF/IFAS Extension office.

U.S. Department of Agriculture, UF/IFAS Extension Service, University of Florida, IFAS, Florida A & M University Cooperative Extension Program, and Boards of County Commissioners Cooperating. Nick T. Place, dean for UF/IFAS Extension.

copyright informationThis document is copyrighted by the University of Florida, Institute of Food and Agricultural Sciences (UF/IFAS) for the people of the State of Florida. UF/IFAS retains all rights under all conventions, but permits free reproduction by all agents and offices of the Cooperative Extension Service and the people of the State of Florida. Permission is granted to others to use these materials in part or in full for educational purposes, provided that full credit is given to the UF/IFAS, citing the publication, its source, and date of publication.

Measuring the Effectiveness of Lagoon Quest: A Case Study...

Documents

Transcript of Measuring the Effectiveness of Lagoon Quest: A Case Study...