Post on 13-Aug-2015
Development and Initial Validation of a Student Assessment Toolkit for Networked Learning in Higher Education Settings
Laura GogiaDecember 2014
The Digital Age
Period in human history in which traditional industry of the industrial revolution has been replaced by an economy based on digital information and processes.
Photo: “Where is the Internet?” Flickr user Lea, https://www.flickr.com/photos/violinha/
The Digital Age is Networked.
People and information sources are linked together in a global network, allowing for unprecedented and disruptive forms of communication.
Photo: “Virtual Marketing, Shirley Chan http://09820006s.blogspot.com/
The Digital Age is Participatory.
Consume but also actively contribute. Affiliation
Formation Creative
Expression Collaborative
Problem Solving Contributing to
Information Circulation
Photo: “Michael Wesch Keynote” by Flickr user Guilia Forsythe https://www.flickr.com/photos/gforsythe/
Networked & Participatory
The new economy and culture have sparked calls for educational reform among a variety of
stakeholders. 21st Century skills New Media skills
Digital skills
Photo: “What’s in my bag?” by Flickr user Brandy Shaul, https://www.flickr.com/photos/zoologist/
Networked Learning
Takes advantage of: The open, distributed
spaces of the web The participatory,
creative nature of digital tools
The power (reach) of networked communication
The power (potency, innovation) of networked thinking
Photo: “Network” by Simon Cockell https://www.flickr.com/photos/sjcockell/
The Problem
Networked Learning emerged from informal learning environments, where
formal assessments of student learning are not common and not necessarily valued.
The Problem
Traditional formalized learning assessments are not consistent with networked learning
pedagogies.Networked Learning Traditional Assessments
Values learning as a process Values knowledge as acquired object
Values collaborative work Value individual work
Values remixing, synthesis Values “original” work
Study Purpose
To create and validate an assessment toolkit for grading student performance in networked learning courses in higher education environments.
Aligned
Agile
Valid
Digital
Study Significance
Networked Learning Is Becoming More Popular In Higher Education.
Higher Education Requires Formalized Student Assessments.
Techniques Or Protocols Must Be Developed That Provide Formalized Assessment Without Compromising Pedagogical Alignment.
What Needs to be Assessed?
Networked Learning (noun):
Connectivist-based learningthat takes place on
distributed, openly networked, &
digital platforms
Networked Learning
Connectivist-based
learningthat takes
place on distributed,
openly networked,
&digital
platforms
Distributed: Consists of more than one platform, supporting different types of activities.
Openly networked: Occurs on the open web, which allows students to interact with a larger community & provides more opportunities for students to make connections between the “spheres of learning.”
Digital: Internet-based
“Connectivist-Based”
Knowledge: a set of connections formed by actions and experience distributed across networks of human and non-human nodes. It is dynamic, changing quickly based on context. It is interpreted by the individuals who are making the connections. It rests in the diversity of opinions.
Learning: the process of making decisions through the connection of information sources.
(Downes, 2006)
“Connectivist-Based”
Core skills include the ability to:▪Choose what to learn & interpret it based on a lens of shifting reality ▪Nurture & maintain personal connections for continuous learning. ▪See connections across fields, ideas, and concepts.
Ability to learn through this process is more critical than current knowledge base.
(Siemens, 2004)
Networked Learning Principles
Developing & Maintaining
Personal Learning Networks
Curating, Critiquing, &
Organizing Data and Data Sources
Connecting Concepts & People Across Spheres of
Learning
Transforming Concepts into a
New Product
Sharing with Personal Learning
Network
How Should It Be Assessed?
Computer-Supported Collaborative Learning Literature (Strijbos, 2011)
Participation
Social
Content Acquisiti
onIndividua
l
Assessment Principles
Participation
Individual Extent to which students contribute to the social stream
Transactivity
Extent to which students refer to and build on each other’s contributions
Transliteracy
Extent to which students move across types of digital media
Content Engageme
nt
Sharing Extent to which students share resources or information
Construction
Extent to which students use information to solve problems within the same situation in which the information was presented
Transformation
Extent to which students collaboratively remix or repurpose information to create new knowledge
How are these operationalized?
(Fact: Digital platforms record the actions of its users and store them in ways amenable to analysis and visualization)
**Preliminary Research of Social Media Analytics from a Networked Learning Course provided potential factors for student assessment.
OverviewNetworked Learning Principle
Assessment Principle
Operationalization
Blogs Twitter
Establishing and Maintaining a Personal Learning Network (PLN)
Participation (Individual)
#Posts, Comments
#TweetsNetwork Degree Centrality
Curating, Critiquing, Organize Data and Data Sources
Content Engagement (Sharing)
Keywords#Links
Keywords#Links
Connecting or coordinating people and concepts over space, time, and spheres of learning a
Participation (Transliteracy) Ratio: Posts, Comments, Tweets
Participation (Transactivity)
#LinksClassmate Mentions
#Retweets, Mentions, Replies, LinksNetwork Betweeness Centrality
Content Engagement (Construction)
KeywordsLinks (Content)
KeywordsLinks (Content)
Transforming data into new products
Content Engagement(Transformation)
Content Content
Sharing new product with PLN
Participation (Individual) #Posts
#TweetsNetwork Degree Centrality
Participation (Transactivity)
Links (Content)Classmate Mentions
#Mentions, Replies, LinksNetwork Betweeness Centrality
Research Questions
How can networked learning be documented for the purposes of student assessment?
How can social media analytics be used to inform student assessment in networked learning environments?
How does the use of social media analytics relate to a qualitative content analysis of the same data?
How can social media analytics be integrated into a larger student assessment toolkit for higher education faculty?
Construction Phase
Exploration of Student Data in Network Learning Environments▪ Descriptive Statistics▪ Quantities of student participation in the form of posts,
comments, retweets, replies, mentions, etc.
▪ Social Network Analysis▪ Characterization of interactions between students within
the community
▪ Content Analysis▪ Identification of key themes within the discourse
▪ Discourse Analysis ▪ Characterization of interactions between keywords, posts
containing keywords, and students using keywords.
Student Data Sources
Secondary Data Analysis #Thoughtvectors MOOC
VCU undergraduate 2014 summer course 95 Student Participants and 30 Open Participants Students were required to blog daily and comment on each
others’ posts – 98% of activity took place on WordPress Over 2309 student posts and 419 open participant posts Twitter discourse was encouraged but with limited student
participation – approximately 4000 tweets total
#ConnectedCourses MOOC Digital Media Literacy Research Hub course on teaching
connected courses with over 250+ active participants Weekly blogging encouraged (over 1880 posts) Active Twitter discourse among students
Tools for Analysis
Application Programming Interface (API)
WordPress Plug-In
Microsoft Excel
KBDeX
Descriptive StatisticsKeyword Searches
NodeXL
Content AnalysisDiscourse Analysis
Social Network Analysis
OverviewNetworked Learning Principle
Assessment Principle
Operationalized (Variables) Measurement Tool
Blog Twitter
Establishing and Maintaining a Personal Learning Network (PLN)
Participation (Individual)
Number: Posts, Comments
Number: Tweets,People Followed
TAGS-WP/Excel
Curating, Critiquing, Organize Data and Data Sources
Content Engagement (Sharing)
KeywordsNumber: Links
KeywordsNumber: Links
KBDexTAGS-WP/ExcelProduct Assessment b
Connecting or coordinating people and concepts over space, time, and spheres of learning a
Participation (Transliteracy)
Ratio: Posts, Comments, Tweets TAGS-WP/Excel
Participation (Transactivity)
Number: LinksClassmate Mentions
Number: Retweets, Mentions, Replies, Links
TAGS-WP/Excel/NodeXL
Content Engagement (Construction)
KeywordsContent: Links
KeywordsContent: Links
KBDexProduct Assessment b
Transforming data into new products
Content Engagement(Transformation)
Content ContentKBDexProduct Assessment b
Sharing new product with PLN
Participation (Individual)
Number: Posts Number: Tweets TAGS-WP/Excel
Participation (Transactivity)
Content: LinksClassmate Mentions
MentionsRepliesLinks
TAGS-WP/Excel
Validation Phase
Construct Validity:▪ Qualitative Content Analysis of post and comment
text▪ How do descriptive statistics, social network analysis, and
discourse analysis metrics relate to a thematic analysis of student posts and comments – with an emphasis on networked learning principles?
Internal Validity:▪ Statistical Analysis▪ Looking for opportunities to assess internal validity through
factor analyses and Chronbach alpha testing▪ Potential correlation with post-course surveys meant to
measure “connectedness”▪ Potential correlation with final course grades
Limitations
This is an exploratory process which will need continued study, replication, and refinement.
Some student data obtained from courses not consistent with formal higher education settings.
Results and usefulness of toolkit are impacted by instructor’s devotion to making networked learning a priority in the learning environment.
References
Downes, S. (2006). Learning networks and connective knowledge: Discussion paper #92. Instructional Technology Forum. Retrieved from: http://it.coe.uga.edu/itforum/paper92/paper92.html
Oshima, J., Oshima, R., & Matsuzawa, Y. (2012). Knowledge building discourse explorer: A social network analysis application for knowledge building discourse. Educational Technology Research and Development, 60(5), 903–921. doi: 10.1007/s11423-012-9265-2
Siemens, G. (2004, December 12). Connectivism: A learning theory for the digital age. Retrieved from: http://www.elearnspace.org/Articles/connectivism.htm
Strijbos, J.W.(2011). Assessment of (computer-supported) collaborative learning. IEEE Transactions on Learning Technologies, 4(1), 59-73.