From Rubric to Assessment
Fluency with Information Technology
2/18/2005
Gail Matthews-DeNatale, Ph.D.Senior Instructional Designer, Simmons College
Bruce Tis, Ph.D.Computer Science and Information TechnologyFIT Task Force Chair, Simmons College
A Liberal Arts Approach to FIT
TECHNOLOGY• Computer Science
• Telecommunications
OriginalFIT domain as
defined by NAS*
* National Academy of Sciences
A Liberal Arts Approach to FIT
TECHNOLOGY• Computer Science
• Telecommunications
INFORMATION LITERACY• Library Science
• Research
Yet more and more information is “born/found
digital”
A Liberal Arts Approach to FIT
TECHNOLOGY• Computer Science
• Telecommunications
INFORMATION LITERACY• Library Science
• Research
MEDIA LITERACY• Semiotics / Linguistics
• Art / Media Studies• Communications
Another layer: understanding
multimedia
A Liberal Arts Approach to FIT
TECHNOLOGY• Computer Science
• Telecommunications
INFORMATION LITERACY• Library Science
• Research
MEDIA LITERACY• Semiotics / Linguistics
• Art / Media Studies• Communications
• Philosophy / Ethics• Cultural Studies
• Anthropology• Sociology • Education
… and much more
A Liberal Arts Approach to FIT
TECHNOLOGY• Computer Science
• Telecommunications
INFORMATION LITERACY• Library Science
• Research
MEDIA LITERACY• Semiotics / Linguistics
• Art / Media Studies• Communications
• Philosophy / Ethics• Cultural Studies
• Anthropology• Sociology • Education
Implications for Assessment
Need to assess three dimensions (technology, information, and media literacy)
No off-the-shelf assessment filled the bill
Composed questions and drew from:
IT Fluency: NAS / Washington State University Information Literacy: California State University Media Literacy: Media Awareness Network (CA)
The Baseline Assessment
Part I: Background Information Demographics Access to computers Experience with computers Technology skills (self-assessed, scale of 1-10)
Part II: Fluency (Developed out of Rubric) Emphasis on critical thinking, problem-solving,
and ethical reasoning
The Results
Ample Access at Home …
Mean = 2.5
SD = 1.5
How many computers in household?
How many computers in household?
876543210
Perc
ent
40%
30%
20%
10%
0% 35
11
25
29
25
… and at / for School
Do you have a computer with you at school?
Do you have a computer with you at school?
noyes
Perc
ent
120%
100%
80%
60%
40%
20%
0%
96
96% of incoming students own a computer that can be used for schoolwork
This includes those who live off campus
Type of internet access at home
Type of internet access at home
otherdon't knowDSLcabledial-up
Perc
ent
40%
30%
20%
10%
0%
12
2727
34
97% of students have internet access at home
How often do you check email?
How often check email?
several times a daydailyw eeklymonthly or less
Perc
ent
50%
40%
30%
20%
10%
0%
22
43
29
5
98% of incoming students had email before Simmons
Access and Familiarity ≠ Fluency
Despite considerable access to computers and the Internet, most incoming students are poorly prepared to live, learn, and work in a technology-rich,multimedia saturated society.
Why?
Social vs. Intellectual / Academic
Prior use of technologyPrior experience with technical problem-solvingHas been largely social and/or informal
Example:
Q: What do you do when you have a computer Problem?
A: 96% “Ask a family member or friend for help”
What’s wrong with that? Basic technical understanding
Example: How email works
Only 41% the students surveyed knew how email messages get sent
Lays a foundation for more sophisticated thinking Example: Distinguishing between valid and fraudulent email
Only 13% of the students surveyed knew how to best deal with a fraudulent email message requesting personal bank account information
Discrepancy Between Self-Assessment ...
Rate your technology skills
Rate your technology skills (1=Poor, 10=Excellent)
10.09.08.07.06.05.04.03.02.01.0
Perc
ent
30%
20%
10%
0%
13
18
21
23
12
65
Mean = 5.6
SD = 1.8
and Actual Insight, Understanding,
Example:
Students who believed they were highly proficient were less likely to say that there were "legal and ethical issues to consider" in:
Copying and pasting extended passages from a web site into a term paper; and
Buying software and letting your friends install it on their computers.
Ability, and …
Example:
Q: "You’re interested in career opportunities in nursing. What would you type into the Web search engine to find the most information?“
A: Only 11% selected the correct response
Awareness
Example:
33% of the students surveyed did not know that computer viruses spread through files on disk and on CDs
Their Greatest Difficulty?
Integrating Multiple Dimensions of Analysis
56
34
2926
0
10
20
30
40
50
60
Selected Fake Site (incorrect) Selected NSF (correct) Selected NSF for AppropriateReasons
Not Sure
You are doing background research for a science project. In addition to perusing library sources, you decide to go online to see what's available.Only 24% selected the most credible site for appropriate reasons
24%22%
49%
29%
Incoming Student Misconceptions
has a .org domain name (more credible than .gov)
has lots of “information” / links
has a “shorter,” “easy,” “straightforward” URL
has a “professional “look”
includes information on how to “send feedback” and/or contact the webmaster/designer
“comes from the United States”
“claims to be unbiased”
is “official,” or has a name that “sounds official” (such as “Centers” and “Organizations”)
has an online store
links to other “well-known organizations”
includes advertising
includes a search option
includes press releases
A web site is more likely to be credible if it
In a Nutshell
Incoming students need the most help with:
problem diagnosis and solving; media literacy (as consumers, researchers, and
authors); and grappling with ethical challenges.
In the same way that incoming students need to learn systematic processes for writing and research,
they also need to learn how to work, think, create, and problem-solve systematically with technology.
FIT Breakout GroupsDivide into groups (4-5 faculty members per group). Please take notes – you will report back to all. Each group has the FIT rubric, baseline assessment with explanations, and tabulated results.
Given the volume of information before you, you may decide that you need to focus on one issue or theme. That’s okay. Considering the rubric and assessment information, discuss the following questions:
What do you think are the most critical learning needs of first year students?
What are you already doing to address these learning needs? How could you be more explicit in connecting what you're already doing with the rubric?
What's one thing you'd like to add or change in your courses?
Top Related