Int. Conf. e-Learning 2011 Project Management and Assessment of Distributed Teams in an Online...
-
Upload
austen-lane -
Category
Documents
-
view
213 -
download
1
Transcript of Int. Conf. e-Learning 2011 Project Management and Assessment of Distributed Teams in an Online...
Int. Conf. e-Learning 2011
Project Management and Assessment of Distributed Teams in an Online Capstone
Masters-Level Information Technology Course
Charles Tappert and Allen StixPace University, New York
Int. Conf. e-Learning 2011
Five Previous Conference Papers on Real-World Student
Projects
1. Integrating Real-World Projects for Actual Customers into Capstone Courses (E-Learn 2002)
2. Security-Related Real-World Projects (ISECON 2004)3. Interplay of Student Projects and Student-Faculty
Research (E-Learn 2007)4. Pedagogical Issues in Managing Information
Technology Projects Conducted by Geographically Distributed Student Teams (SITE 2009)
5. Assessment of Student Work on Geographically Distributed Information Technology Project Teams (E-Learn 2009)
Int. Conf. e-Learning 2011
Real-World Student Projects
Conducted in capstone courses for 10 years
Student teams build real-world computer information systems for actual customers
Project systems serve the community internal university community at Pace greater university community external non-profit local community
Real-World Student Projects (cont)
Real-world projects are a stellar learning experience for students
Win-win situation for all Students Customers Instructors and other involved faculty School of CSIS University
Int. Conf. e-Learning 2011
Migrate to Online Format
Migrated from traditional face-to-face format to online format in Fall 2006
To be progressive Technology for online courses adequate Online preferred by employed students –
no scheduling conflicts & no commuting To expand the population of students
beyond the greater NYC area
Int. Conf. e-Learning 2011
Challenges of Online Format
Uncertainties of how traditional course methods port to the online environment and what new methods might be required
Teams lacking co-presence require higher level of organizational and process skills
No weekly classroom meetings as safety net for teams’ interaction and functioning
Int. Conf. e-Learning 2011
Team Projects – Categories
Int. Conf. e-Learning 2011
Project CategoryNumberProjects
ProjectSemesters
ProjectRelated
Pubs
OffshootPubs
Web Applications 21 25 21
Pervasive Systems 15 25 18
PC Applications 11 18 13
Artificial Intelligence 9 11 12
Pattern Recognition 9 12 34 19
Biometric Systems 32 35 39 19
Quality Assurance 5 9 5 5
Totals 102 135 142 43
Table 1. Summary of projects and publications.
Team Projects – Sources
Int. Conf. e-Learning 2011
Project Source Number
Faculty Ideas or Research 42
Student Ideas or Research 36
External Community 13
Internal University Needs 11
Totals 102
Table 2. Project sources.
Team Projects – Publication Types
Int. Conf. e-Learning 2011
Publication Type Number
External Conference Papers 53
Journal Articles 7
Book Chapters 2
Doctoral Dissertations 17
Masters Theses 4
Internal Conference Papers 98
Internal Technical Reports 4
Totals 185
Table 3. Publication types.
Journal 2010
Team Projects – Spring 2011
Course website “Projects” page Fall 2009
Int. Conf. e-Learning 2011
Team Website
Project title and description Project members and customers All deliverables posted
Weekly status reports Midterm & final presentation slides User manual Technical paper
Int. Conf. e-Learning 2011
Team Project – Example Website
Personality Assessment from Handwriting Project
Journal 2010
Team Project – Example Website
Team Projects – Example Systems
Handwriting Forgery Quiz System Rare Coin Grading System Biometric Systems Projects (2010-2011)
Keystroke biometric system Stylometry system Biometric product investigation
Freeware – face & voice Licensed products – face & fingerprint
Int. Conf. e-Learning 2011
Keystroke & Stylometry Systems
The 2008 U.S. federal Higher Education Opportunity Act (HEOA) requires institutions of higher learning to make greater access control efforts to assure that students of record are those actually accessing the systems and taking exams in online courses by adopting identification technologies as they become more ubiquitous
To meet these needs, keystroke and stylometry biometrics were investigated towards developing a robust system to authenticate online test takers
Performance statistics on keystroke, stylometry, and combined keystroke-stylometry systems were obtained on data from 40 test-taking university students
Int. Conf. e-Learning 2011
Keystroke Biometric System
Keystroke biometric systems measure typing characteristics believed to be unique to an individual and difficult to duplicate
Most previous work on passwords – two commercial products Keystroke biometric is appealing for online student verification
not intrusive to computer users inexpensive – only hardware required is a computer with keyboard text continues to be entered for potential repeated checking after an
initial authentication phase (referred to as dynamic verification) Our keystroke system developed over last 6-7 years is unique
collects raw keystroke data over the Internet focuses on long-text input where sufficient keystroke data are available to
permit the use of powerful statistical feature measurements
Int. Conf. e-Learning 2011
Keystroke Biometric SystemSpring 2011 Results
The best equal-error-rate performance on the keystroke system was 0.55% which is an improvement over earlier reported results on this system
Receiver Operating Characteristic (ROC) curves: False Reject Rate (FRR) vs False Accept Rate (FAR)
Experiments 1 and 2:Open systems – trained and tested on different students
Experiment 3:Closed system – trained and tested on all studentsInt. Conf. e-Learning 2011
Stylometry Biometric System
Stylometry is the study of determining authorship from the linguistic styles of the authors
Traditionally, it has been used to attribute authorship to anonymous or disputed literary documents
More recently, computer-based communication and digital documents have been the focus of research, sometimes with the goal of identifying perpetrators or other malicious behavior
We are just beginning to apply stylometry to verify online student test takers, early results poor
Int. Conf. e-Learning 2011
Stylometry Biometric System
The performance of the stylometry system a poor 33% EER, indicating that stylometry is not suitable for text lengths of short-answer tests (50-100 words per question)
Receiver Operating Characteristic (ROC) curves: False Reject Rate (FRR) vs False Accept Rate (FAR)
Experiments 1 and 2:Open systems – trained and tested on different students
Experiment 3:Closed system – trained and tested on all studentsInt. Conf. e-Learning 2011
Biometric Product Investigation
Freeware systems (usually 30-day trial) – fall 2010 project Face recognition
KeyLemon (KeyLemon) & FaceSDK (Luxand) Voice recognition
VoiceCipher (VoiceLatch) & bio-iVault/iWallet (myBiodentity)
Inexpensive licensed products – spring 2011 project Face recognition SDK (Neurotechnology) Fingerprint recognition SDK (Neurotechnology)
Results - freeware and cheap licensed systems equally poor Eigenface algorithm (coded in 2003 project) worked better Recent info exposes poor accuracy of biometric products
Int. Conf. e-Learning 2011
Simple Eigenface Algorithm coded in 2003 Project
Int. Conf. e-Learning 2011
?Query
Face DB
Face Recognition: SystemFace Recognition: System
Int. Conf. e-Learning 2011
Biometric IdentificationBiometric Identification
Tom Cruise in Minority Report, 2002Int. Conf. e-Learning 2011
Inspirational Portrait of IndividualityInspirational Portrait of Individuality
Int. Conf. e-Learning 2011
Face Recognition: National SecurityFace Recognition: National Security
Int. Conf. e-Learning 2011
Speaker Individuality: “My name is …”Speaker Individuality: “My name is …”
Int. Conf. e-Learning 2011
“My name is” from Two Different Speakers
Speaker IndividualitySpeaker Individuality
Int. Conf. e-Learning 2011
“My name is” divided into seven sound units.
Speaker IndividualitySpeaker Individuality
Int. Conf. e-Learning 2011
biomouse Fingerprint scanner
Digital
Camera
LCD Pen
tabletMicrophone
Multi-modality Biometric AuthenticationMulti-modality Biometric Authentication
Embeded & Hybrid User Verification system
System that requires user verification
Int. Conf. e-Learning 2011
Issues/Solutions Stemmingfrom Scattered Teams
Project stakeholder communication Issue – communication gets difficult
For example, scattered team members more likely to feel isolated and want to communicate directly with instructor or customer
Solution Communication between team and instructor/customer
must be through team leader Email distribution lists for whole class and for each team Project team leaders must be local to facilitate
communication/meetings with instructor and customers Course website provides central source of course
information Blackboard discussion forum for each project (see below)
Int. Conf. e-Learning 2011
Issues/Solutions Stemmingfrom Scattered Teams (cont)
How to handle quizzes, deliverables, etc. Issue – classroom meetings not available Solution – use Blackboard educational software
Quizzes Collecting digital deliverables Discussion forums
Forum for archiving instructor email Forum for student introductions Forum for textbook and other course material Forum for each team project
Int. Conf. e-Learning 2011
Issues/Solutions Stemmingfrom Scattered Teams (cont)
Provide some face-to-face interaction Issue – no weekly classroom meetings Solution – three classroom meetings for
local students/customers1. Near beginning of course
1. Face-to-face introductions, nature of course, specifics of course, student team project meetings
2. Midterm1. Project status presentations
3. End of semester1. Final project presentations
Int. Conf. e-Learning 2011
Current Assessment of Online Students
Individual quizzes (20%) Blackboard educational software system
Team initial assignment (10%) Students learn to function as a team
Team project midterm checkpoint (20%) Team project final checkpoint (20%) Team technical paper (30%) Strong emphasis on projects
No midterm/final exams (used in two-semester course)
Int. Conf. e-Learning 2011
Team Member Self and Peer Evaluations
Issue – lack of classroom meetings makes it difficult to determine individual team members’ contribution to the project work
Peer evaluations critical for distributed teams
Minimal team member/customer contact Minimal team member/instructor contact
Literature indicates Various granularity levels in peer evaluations Some automated systems reported
Int. Conf. e-Learning 2011
Team Member Self and Peer Evaluations
Three times during the semester After initial assignment to learn the process At the midterm checkpoint At the final end-of-semester checkpoint
Process for a graded team event First assign a team grade Adjust individual grades up/down based on
self/peer, customer, and instructor evaluations
Int. Conf. e-Learning 2011
Example Team Peer Evaluationand Grade Chart (4 member
team)
Team Member
Eval 1 Eval 2 Eval 3 Eval 4 Summary
Grade
1 + = + ++ + + + + 93
2 = = – – – – – – 79
3 – = + – – 83
4 = = – + = 85
Average = = = = = 85
+/- 2% for each summary +/- sign, showing only peer evaluations.
Int. Conf. e-Learning 2011
Pedagogical Course Evaluations
Issue – lack of classroom meetings makes it difficult for instructor to determine relative value of the course methodologies
Solution – semester-end survey (Survey Monkey)
Procedures/methods that worked well, or did not work well, and why
Int. Conf. e-Learning 2011
Pedagogical Customer Evaluations
Issue – instructor is often not aware of the quality of team-customers interactions
Solution – semester-end survey Obtain student feedback on customer
interaction Were customer requirements clear? Was amount of contact/interaction adequate? Was help on the project work appropriate?
Int. Conf. e-Learning 2011
Something New in 2010-2011Compared Projects to Reality Game
Shows
Capitalize on student interest in TV game shows Like Survivor and The Apprentice
Common with project teams & game show groups
Work toward common goals with division of responsibilities Acquire and share new knowledge Harness skills of different teammates Adjust to different personalities Exhibit initiative without disruptiveness
Differences between projects & game shows Survivor “voted out” & The Apprentice “you’re fired” Game show participants compete against each other
Int. Conf. e-Learning 2011
Conclusions
Over five year’s experience in face-to-face mode
Over four year’s experience in online mode
Techniques for managing and assessing distributed teams have been successful and they continue to evolve
Int. Conf. e-Learning 2011