A Twenty-Year Survey of Science LiteracyAmong College UndergraduatesBy Chris Impey, Sanlyn Buxner, Jessie Antonellis, Elizabeth Johnson, and Courtney King

First results from a 20-yearsurvey of science knowledge andattitudes toward science amongundergraduates are presented.Nearly 10, 000 students takingastronomy as part of a generaleducation requirement answered"a set of questions that overlap

"a science literacy instrumentadministered to the general publicby the National Science Foundation.The research questions addressed

are: What is the level of scienceliteracy among undergraduates, andwhat variables or attributes predictscience literacy? Their attitudestoward science and pseudo-sciencewere probed by a set of 22 statementscoded on a Likert scale. On theknowledge items, freshmen performonly marginally higher than thegeneral public, with the exceptionof large positive differences in theirknowledge of evolution and theBig Bang. Gains on any particularitem through the time that studentsgraduate are only 10%-15%, despitethe fact that they have taken twoor three science courses. Belief inpseudoscience runs high, and the factthat the level ofpseudoscience beliefdoes not correlate well with the levelof science knowledge is particularlynoteworthy. In addition, no variable

in the analysis is predictive ofscience literacy. Over the interval

1988-2008, there s no detectableimprovement in undergraduatescientific literacy.

A nyone who teaches under-

graduate science plays an

important role in our society.If they teach science majors,

they fulfill the need for a technicallytrained workforce by preparing stu-dents who are numerate and who haveinquiring and analytic habits of mind.If they teach nonscience majors, theyoften provide the last formal exposureto the evidence-based reasoning andideas that have transfontned our under-standing of the natural world.

Advanced math and science arevoluntary in most high schools; amajority of students do take biology,but less than a half take chemistry, aquarter take physics, and only I in 20take calculus (Blank, Langesen, andPetersen 2007). At a university, stu-dents face a smorgasbord of choices,and despite the popularity of generaleducation science requirements, sci-ence is being taken less frequently

overall. The National Academy ofScholars surveyed science curriculaused in bachelor of arts degrees fromthe top 50 institutions ranked by theU.S. News and World Report; thepercentage having science require-ments dropped from 90% in 1964 to34% in 1993, and the percentage withboth math and science requirementsdropped over the same period from36% to 12% (Balch and Zurcher 1996).A U.S. Department of Educationanalysis of the "empirical curriculum"found that science accounted forjust 7of the 100 course categories with themost undergraduates, whereas a thirdof all future school teachers do nottake any college-level math (Adelman2004). In this paper, we consider thescience knowledge and attitudes of acohort that is typical of the 80% of thestudents at a major Land Grant publicuniversity who graduate with degreesin nontechnical subjects.

Vol. 40, No. 4, 2011 31

Scientific literacy views on climate change, stem cell of scientific ingenuity without any

Science and technology form an research, nuclear power, and the real knowledge of how they work."amniotic fluid" around all who live space program. They compete for For good or for ill, science is pro-in the industrialized world and, in- jobs in technology-driven sectors foundly shaping human culture, yetcreasingly, in any part of the world. of the economy that didn't exist a any ability to assimilate the insightsAmericans encounter science in this generation ago. They are consumers of scientists into the natural worldcountry in their roles as citizens, of ubiquitous high-technology de- depends on a foundational under-workers, and consumers. They vote vices and are mostly content to use standing of how scientific knowl-for political candidates with diverse these tools and other manifestations edge is gained, how the scientific

The first part of the two-sided questionnaire administered to students in introductory astronomy classes atthe University of Arizona in Tucson.The knowledge-based objective response questions include nine that arepart of the instrument administered by the National Science Foundation and reported biannually in the Sci-ence and Engineering Indicators series.The four open-ended questions were transcribed and coded by newlyconstructed schema.

"* Are you male _ or female _ ?"• Are you a freshman -, sophomore _-,junior - or senior_ ?"• What is your major _ and estimated GPA ?"* How many science courses have you taken at the U of A _ ?"* What does it mean to study something scientifically?"• Would you say that astrology is very - sort of , or not at all scientific ?"* The oxygen that we breathe comes from plants.True or False?"• Lasers work by focusing sound waves.True or False?"* Electrons are smaller than atoms.True or False?"* The universe began with a huge explosion.True or False?"* The continents on which we live have been moving apart for millions of years, and will continue to move in the future.

True or False?"* Humans, as we know them today, evolved from earlier species of animals.

True or False?"• The earliest humans lived at the same time as the dinosaurs.True or False?"* Which travels faster, light __ or sound ___?

"• Does the Earth go around the Sun - ,or does the Sun go around the Earth - ?" Does the orbit in the previous question take one day _,one month _,or one year__ ?"* What is DNA?"* What is radiation?"* If the rate of inflation is falling, are prices decreasing -, level -,or increasing _ ?"* Radioactive milk can be made safe by boiling it.True or False?"• Antibiotics kill viruses as well as bacteria.True or False?"• Is all radioactivity manmade __ or does some occur naturally __ ?

"* A doctor tells a couple that they have a one in four chance of having a child with an inherited illness. Does this mean that,a. If they have only three children, none will have the illness.Yes or No?b. If their first child has the illness, the next three will not.Yes or No?c. Each of the couple's children will have the same risk of suffering the illness. Yes or No?d. If the first three children are healthy, the fourth will have the illness.Yes or No?

"* Briefly, define computer software."* Which is the largest contributor to heart disease: smoking -, eating a lot of animal fat -, stress , not getting enough

exercise _,or lack of vitamins _?

32 Journal of College Science Teaching

enterprise proceeds, and how to dis-tinguish scientific facts from otherkinds of information-a collectionof skills and knowledge commonlyreferred to as science literacy.

Scientists in different fields mightnot agree on a core set of principlesand facts that every citizen shouldknow, but they are probably in accordon the fact that science literacy goes

Survey of Science Literacy

beyond their self-interest. To makeinformed decisions, Americans needto have assimilated enough fromtheir education to use evidence-basedreasoning to separate substance from

The second part of the questionnaire has 22 statements, with responses on a Likert scale.The items were newlycreated for this survey.

Overall, the progress of science and technology has been beneficial toour Civili7;tifnn I

I There are phenomena that physical science and the laws of naturecannot exolain.

Sui-us are real anc snloulic ie investigatecl.

I Some people possess psychic powers.

There are some circumstances when medical science should not beused to prolong life.

El EII

o1 i I

0l El 1

l 1 El 1

E I

El 1

I L I LiI L I Li I Li I

1 I E 1 i El E l1

El El El El El

The Unives ws created in a nro' explo ion blin of years I 0 11 11111

Some ancient civilizations were visited by extraterrestrials.. El 0 E 11 E

Compteswl evenually be nelgn nuht thn l~ikeumans. 11 11 1 lETechnology has too much control over our lives. 0 E El El E

Scientists should tak repni itfothe bad effect oftheir theorie 0> 13 011an netons. L

The government should strongly support the manned space program. El [] El El

Gen engiens god . i p 0

We should devote more of our money and scientific resources to repair El El El El Edamage done to the environment.

Pure siece shoudbe une regarles of it akoImmdaeT1

Science will come up with a way to dispose of toxic waste. El El El El

We houd xert morecotrol ove thpula epn evlpdb 3 00

Faith healing is a valid alternative to conventional medicine. El El El 11 E

We shoud makea cnerted ffottserhfrlfonterpat. 1 0ED0Scientists should be allowed to do research that causes pain to animals, El E] El o1 Elif it helps solve human health problems.

Vol. 40, No. 4, 2011 33

spin and cull corroborated fact fromunsubstantiated assertion. The healthof civic society depends in part onscience literacy.

August bodies have weighed in.The Organization for EconomicCooperation (OECD 2002) definesit as understanding key scientificconcepts and frameworks, the meth-ods by which science builds ex-planations based on evidence, andhow to critically assess scientificclaims and make decisions basedon this knowledge. The AmericanAssociation for the Advancement ofScience (AAAS; 1993) states thatscientifically literate citizens shouldbe aware that science, mathematics,and technology have strengths andlimitations and are interdependenthuman enterprises; should recog-nize the unity and diversity of thenatural world; and should be ableto use scientific ways of thinkingand knowledge for individual andsocial purposes (AAAS 1993). TheNational Research Council (NRC)has even more lofty goals for theschools: Education should promotescience as one of the pinnacles ofhuman thinking capacity; providea laboratory of common experiencefor developing language, logic, andproblem-solving skills; and preparestudents for a democracy, which de-mands that its citizens make personaland community decisions about sci-entific issues (NRC 2007).

The goal of this paper is to describea large survey of undergraduate sci-ence literacy at one public university,with homogeneous data spanning 20years. The data is then used to mea-sure the level of science literacy ofthis population and discover which, ifany, variables or attributes are predic-tive of science literacy.

Previous measurements

There are many ways to frame thegeneral principles described aboveinto a definition and measurement;therefore, this paper focuses on asurvey that tethers to the best Ion-

34 Joural ot College Science Teaching

gitudinal data available. The Na-tional Science Foundation (NSF)has commissioned a survey of sci-ence knowledge and attitudes for20 years as part of the Science andEngineering Indicators series, thebiannual report to the National Sci-ence Board that is used to shape na-tional research and education policyand guide workforce developmentin technical fields. Jon Miller, whoconducted the surveys on behalf ofthe NSF for many years, set the cri-teria of literacy to be (1) a vocabu-

lary of basic scientific constructs,(2) an understanding of the processor nature of science inquiry, and (3)a level of understanding the impactof science and technology on indi-viduals and on society. He estab-lished a threshold based on (1) cod-ed responses to the question "Whatdoes it mean to study somethingscientifically?" (2) the recognitionthat astrology is not at all scientific,and (3) correct answers to six ormore out of nine knowledge ques-tions (Miller 1987).

Responses to subset of items on the science literacy instrument.These items form the basis for NSF knowledge-based metric ofpublic sdence literacy. UA freshmen have not taken any scienceclasses, whereas UA general education students have taken two tothree science classes. Sample sizes are 1,864 for NSF, 1,275 for UAfreshmen, and 828 for UA general education; data is from 2006 forNSF and 2004-2008 for UA.The anomalous result on astrology-thevery low performance of the college age cohort-seems real, becauseresponses to the separate Likert scale measurement are consistent.UA = University of Arizona; Gen Ed = general education; NSF =National Science Foundation.

Antibiotics kill viruses as well as bactena.(false)

Humans evolved from earlier speciesof animals.

(true)

The continents have been moving apartfor milions of years.

(true)

All radioactivity is manmade.(false)

Electrons are smaller than atoms.(true)

Lasers work by focusing sound waves.(false)

The earth goes around the sun.(true)

The orbit takes one year.(true)

Astrology is not at all scientific.(true)

The universe began with a huge explosion.(true)

0 20 40 60 80 100

Percent Correct

o UA Gen Ed

"* UA Freshman

"* NSF

MethodologyThe first author of this paper (CI) hasbeen interested in science literacy for23 years as an astronomy professorat the University of Arizona (UA) inTucson. Since 1988, he has adminis-tered a survey instrument to the stu-dents in general education astrono-my lecture classes, taught by himselfand his colleagues. The introductoryastronomy course counts for a thirdof the science requirement. Aboutone in five of all undergraduates atthe university take the course, whichis primarily composed of freshmenand sophomores. Only 5% of the en-rollment consists of science majors.The survey is given in the first weekof class; it is anonymous and volun-tary and doesn't count for any parta student's grade. Typically 10 to 15minutes are allowed for its comple-tion. Only 3% of the students choosenot to complete the survey. Over 20years, almost 10,000 surveys havebeen collected and used for analysisin this study.

The rich data set enables us toaddress a wide range of researchquestions; this paper summarizesthe survey and provides preliminaryresults to the overarching questions:What is undergraduates' level of sci-

ence literacy? and What variables arepredictive of undergraduates' scienceliteracy? The purpose of this initiallook is to establish a baseline of sci-ence literacy for this population ofundergraduates compared with NSF'snational sample and, more specifically,to investigate Miller's (2007) conclu-sions about the changes in the nationalsample. This analysis is a precursor tosuggestions for the improvement ofteaching science to nonscience majorsat the college level.

Although the survey is anony-mous, the following demographicinformation is collected: gender,major, self-reported GPA, class stand-ing, and number of science coursestaken at the university. The instru-ment is administered on paper, andsubsequent data entry and coding aredone by hand. Figure 1 presents theknowledge-based part of the question-naire. Most of the science contentquestions are true/false, yes/no, ormultiple choice. Four are open re-sponse. Nine of the questions overlapwith the much more extensive NSFScience Indicators instrument. Figure2 shows the attitudinal, second halfof the questionnaire, comprised of 22statements about science, technology,and society designed to probe under-

Survey of Science Literacy

graduates' opinions toward scienceand their belief in pseudoscience.Responses are coded on a 5-pointLikert scale, with no opinion insteadof the more traditional neither agreenor disagree as a middle item. Thispart of the survey was devised by oneof the authors (CI) when the surveybegan in 1988.

The survey instrument has beenstable for over 20 years and has beenadministered in the same fashionthroughout that span. It should be not-ed that although there is a substantialoverlap of knowledge questions withthe NSF instrument, NSF calls respon-dents by telephone at their homes, asopposed to obtaining written responsesin a classroom setting. Questionnaireswere typically given out with classesof 100 to 150 students, using two tothree classes per semester, givingan average of about 500 completedquestionnaires per year across the 20years of the project so far, with fairlyuniform time coverage.

Data and analysis

The enormous job of data entry wascarried out by two of the authors (EJand CK) over 18 months from 2007to 2009. The rate of incomplete data,spoiled or bad data, and illegible an-

Overall responses, by nearly 10,000 undergraduates over 20 years, to 6 out of 24 statements about scienceand technology.This is the group of items relating to pseudo-science or nonscientific beliefs. These beliefscoexist with solid performance on science knowledge questions.

There are phenomenathat physical scence The position of the planets

and the laws of have an influence on the Some people possessnature cannot explain events of everyday life psychic powers

6o

.1so

CD40

30

% 20

20 'd2. z 5e 005a

Vol. 40, No. 4, 2011 35

iu•0

i2-1

5

•n

swers or facetious or obviously frivo-lous responses amounted to around3% overall. An independent cross-check of a sample of the data showedthat error rate for data entry was below0.1%. Although the survey includedboth forced-choice and open-responsequestions, analysis of the forced-choice questions is included here.

Descriptive analysis began bylooking at responses to each question,both knowledge as well as belief andattitude questions, by each demo-graphic variable. Next, compositescores were generated on the basisof both overlapping questions withthe NSF survey and scores based onall knowledge questions for each in-dividual. Additionally, the belief andattitude questions were grouped intosets using a combination of expertgrouping by scientists and scienceeducators, exploratory factor analysis,and cross-validating through confirma-tory factor analysis. Both descriptiveand regression analysis were used todetermine the relationship between thedemographic variables and belief andattitude indicators on the one hand, andstudents' performance on individual(and aggregated) science knowledgequestions on the other hand.

Initial resultsUA students got an average of 7.2(SD = 1.5) items correct out of the9 items shared on the NSF survey(80%) and an average of 11.2 (SD =2.3) correct overall out of 15 (75%;the astrology and the inflation ques-tions were left out).

There was an overall stability ofquestions that were answered cor-rectly over the 20 years. Figure 3compares the UA sample of bothpre-general-education requirement(students equivalent to high schoolgraduates in terms of science courses),post-general-education requirement(students having completed at leasttwo university science courses) to thelatest published NSF sample (NationalScience Board 2008). The UA sampleis truncated (2004-2008) to be morecomparable to the NSF sample. Figure3 shows that entering freshmen scoremarginally higher than the generalpublic, with the exception of largepositive differences in their knowledgeof evolution and the Big Bang.

The gains in knowledge on anyparticular item through the time thatstudents graduate are only I 0%-l 5%,despite the fact that they have takentwo or three science courses by then.

The generally high level of scienceknowledge is reassuring, but the gainduring college amounts to only one ad-ditionally correct item among the coreset of nine from the NSF instrument,and the gaps that remain are disconcert-ing. The data show an important gendereffect, with slightly better performanceby men on the scientific knowledgequestions. However, this is the onlydemographic factor or variable that cor-relates with performance; it is strikingthat there seem to be no good predictorsof science literacy. Results from thistime-limited sample of the UA data arerepresentative of the full data in whichUA students perform better than thosein the NSF sample but do not showa marked improvement after takinguniversity science courses. Using a fullbaseline of 1988 to 2008, there's no de-tectable improvement in undergraduatescientific literacy over 20 years. Thiscontrasts with Miller's (2007) findingfor the general public and will be thesubject of future analysis.

On the attitudinal part of the survey,students are generally well-disposedtoward science and technology and93% agree or strongly agree with thestatement that "Overall, the progressof science and technology has been

The analysis of 9,200 questionnaires for true/false reactions to a statement about antibiotics and 9,300 truefalse reactions to a statement about astrology shows a modest but statistically significant gender effect (fewerstudents answered the question about antiblotics). here% a larger knprovement with the number of sciencecourses taken on astrology (from a low base) than on antibiotics. Students who have taken four or more sciencecourses are mostly science majors. Soc Beh Sci = Social Behavioral Science Gen Ed = general education.

Antibiotics kill viruses as well as bacteria Astrology is not at all scientific100 100

90 90so 80

70 70 _ _ _ _ _ __ _ _ _ _

60 ~60-~4040

(L 20 2

36 Journal of College Science Teaching

I;[ll F ;IGR 5

beneficial to our civilization." Belief inpseudoscience runs high. About 40%believe that the positions of the planetsaffect everyday life, and the same per-centage think some people have psy-chic powers. About one in six believethat aliens visited ancient civilizations,one in four think that faith healing is alegitimate alternative to conventionalmedicine, and a quarter think that somenumbers are lucky for some people(Figure 4). One striking aspect of theanalysis is a null result: None of thesebeliefs is strongly correlated with levelof science literacy.

The results for students who havefinished all of their science require-ments are disconcerting. One in threethink that antibiotics kill viruses aswell as bacteria, one in four think thatlasers work by focusing sounds waves,one in five think that atoms are smallerthan electrons, and one in five eitherdo not believe or are unaware thathumans evolved from earlier speciesof animals and that the Earth goesaround the Sun in a year. Only one infive undergraduates say that astrologyis "not at all" scientific, although thatfraction increases from 17% to 34% asthey move through the university. TheLikert scale item on the position of theplanets is an important cross-check onthe astrology question because it doesnot use the word astrology and studentsmight plausibly confuse astrology andastronomy. Equally troubling, half ofall science majors say that astrology iseither "sort of' or "very" scientific (seeFigure 5). In this case, it is possible thatthey are aware of astrology's roots inobservational astronomy, which meansthe astrology item must be interpretedwith caution. Education majors-thecohort of future teachers-performworse than the average on almost allthe individual questions and in termsof overall scientific literacy.

SummaryOverall, it appears that high schooleducation and students' exposureto media and popular culture con-vey a basic knowledge of science,

although it is piecemeal and barelyadequate to make students familiarwith the major achievements of sci-ence in the past century. After that,college science instruction producesincremental gains, but students stillreach the end of their formal educa-tion with substantial holes in theirknowledge and understanding ofscience. It is up to science educa-tors and higher education leaders todecide whether it is an acceptableoutcome when a significant fractionof college graduates are unaware ofmajor tenets of life and physical sci-ence and also hold persistent pseu-do-scienctific belief systems.

Future papers will closely exam-ine (1) the relationship between stu-dents' beliefs and attitudes and howthese attitudes can correlate withscience knowledge, (2) the patternsof student belief in pseudoscience ingeneral and belief in astrology in par-ticular, (3) a study of the responsesto the open-ended questions and across-calibration of undergraduatescience literacy with science lit-eracy in the general public, and (4)separate analysis of the differencesbetween the way science is viewedby students and by professional sci-entists. Another paper will present anew coding scheme for responses tothe open-ended question "What doesit mean to study something scien-tifically?" in order to derive a moredetailed view of the way studentsconceive of scientific informationand the process of science. E

AcknowledgmentsWe acknowledge many conversations oneducational issues with Ed Prather andGina Brissenden. The first author (CI)

is grateful.for the assistance of his col-leagues in the Astronomy Department

for over 20 years, who gave up some oftheir class time to administer the ques-tionnaire in their classes.

ReferencesAdelman, C. 2004. The empirical cur-

riculum: Changes in post-secondary

Survey of Science Literacy

course-taking, 1972-2000. Washing-ton, DC: U.S Department of Educa-tion. Unpublished report availableonline at www.ed.gov/rschstat/re-

search/pubs1empircurr/index. html.

American Association for the Advance-ment of Science (AAAS). 1993.Benchmarks for science literacy.

New York: Oxford University Press.Balch, S.H., and R.C. Zurcher. 1996.

The dissolution of general educa-tion: 1914-1993. Princeton, NJ:National Academy of Scholars. Un-published report available online atwww. nas. orglpolReports. cfm ?Doc

Id= 113.

Blank, R., D. Langesen, and A. Pe-tersen. 2007. State indicators of

science and mathematics education.

Washington, DC: Council of ChiefState School Officers.

Miller, J.D. 1987. Scientific literacy inthe United States. In Communicatingscience to the public, eds. D. Everedand M. O'Connor, 19-40. London:Wiley.

Miller, J.D. 2007. Civic literacy acrossthe life cycle. Unpublished paperpresented at the annual meeting ofthe American Association for theAdvancement of Science, San Fran-cisco, CA.

National Research Council (NRC).2007. Taking science to school:

Teaching and learning science in

grades K-8. Washington, DC: Na-tional Academies Press.

National Science Board. 2008. Sci-ence and engineering indicators

2008. Vols. I and 2. Arlington, VA:National Science Foundation.

Organization for Economic Cooperationand Development (OECD). 2002.Education at a glance. Paris: OECD.

Chris Impey (cimpey@as.arizona.edu)

is a University Distinguished Professor inthe Department of Astronomy, SanlynBuxner and Jessie Antonellis are grad-uate students in the College of Education,and Elizabeth Johnson and CourtneyKing are undergraduate students in theCollege of Letters, Arts, and Sciences, all atthe University of Arizona in Tucson.

Vol. 40, No. 4, 2011 37

COPYRIGHT INFORMATION

TITLE: A Twenty-Year Survey of Science Literacy AmongCollege Undergraduates

SOURCE: J Coll Sci Teach 40 no4 Mr/Ap 2011

The magazine publisher is the copyright holder of this article and itis reproduced with permission. Further reproduction of this article inviolation of the copyright is prohibited. To contact the publisher:http://www.nsta.org/college