Evaluating the Quality of Online Programs
description
Transcript of Evaluating the Quality of Online Programs
Evaluating the Quality of Online
ProgramsDiane Ruiz Cairns, Ed.S.
Lawrence Technological UniversityeLearning Services
Agenda
• Overview• Why monitor quality?• Methods for monitoring quality of online programs• Lawrence Tech, eLearning Services experience• Next steps
Overview
• Program Evaluation• Making data-driven decisions• Support of performance improvement• Alignment of resources, performance and strategic goals• Add measurable value
Overview
• Quality of online programs impacts student retention, enrollment and graduation rates
• Online environment includes:• Institutional Technology• Course Development• Course Structure• Teaching and Learning• Student and Faculty support• Methods of Evaluation and Assessment
Overview
• Measures of success include:• Enrollment Reports• Assessment of Learning• Student Evaluation Survey• Informal Feedback
Why Monitor Quality?
• Results based• Measurable results• Effective course content• Efficiencies in operation• Teaching effectiveness
Why Monitor Quality?
• Alignment of methods for measuring and assurance of quality• Stability of online programs• Impacting student satisfaction• Value of online programs
Evaluating Quality of Online Programs
• Views and methods for evaluating programs varies• Adoption of a comprehensive tool or methods brings
alignment• A validated tool recognized by industry can assist with
reliability• Requirements before adopting a tool
Evaluating Quality of Online Programs
• Repeatable data collection results in meaningful collection of data• Comprehensive approach• Multiple collection cycles supportive of reasonable and
responsible data
Evaluating Quality of Online Programs
• Requires careful planning• Data collection focus on mega, macro, micro levels• Systems approach• Data collected include:• Effective course content• Efficiencies in operation• Teaching effectiveness
Evaluating Quality of Online Programs• Mega
• Success of online program at meeting university enrollment goals• Support of teaching and learning goals
• Macro• Technological infrastructure• Individual courses support of teaching and learning guidelines• Faculty engagement
• Micro• Instructional design impact• Student, Faculty, Staff use of technology• Student and faculty support services
Evaluating Quality of Online Programs
• Planning• Timeline• Participation• Communication, Communication, Communication• Conduct the evaluation• Plan for interventions
Monitoring of Online Program
• Create Dashboard• Seven to Ten Metrics
• Method for reporting (communicating)• Data collection periods
Example of Data Collection Schedule
Building a Dashboard
• Elements of organizational performance• Enrollment Goals• Teaching and Learning Goals• Graduation rates• Employment outcomes• Technological metrics: uptime, type of support calls• Quality of teaching and learning• Faculty engagement• Faculty training, participation• Student evaluation survey data
Dashboard Examples
Lawrence Tech Experience• Sloan-C
• Evaluation of Online Program organization• Self assessment
• Baldrige Education Performance Excellence• Evaluation of education organization• Assessed by Baldrige evaluators
• Blackboard Exemplary Course Rubric• Evaluation of course development• Self assessment
• Quality Matters• Evaluation of course development• Assessed by qualified evaluators
Lawrence Tech Experience• Operation Quality• Course Quality• Course Delivery Quality• Documenting standards• Identify metric requirements• Adopting industry standards:• Sloan-C• Blackboard Exemplary Course Rubric• QM Course Design• Baldrige - future
Getting Started
• Why do this?• What will you do with the data?• Benchmarking• Building team• Confirming plan• Collecting data, what data• Reporting results• Engagement across campus services
Confirming
• Monitoring schedule• Reinforcement of quality measures• Integration • Policy and practices of monitor, evaluating, assessing• Managing, planning for change• Oversight
Change
• Be an agent of change• Lens of student, employers, accrediting bodies, stakeholders• Define critical success practices
Timeline
Conclusion
• Confirm metrics• Begin program evaluation• Sloan-C
• Develop Dashboard• Report• Refine• Apply intervention
Dashboard Data
Dashboard Data
Discussion
There is nothing wrong with change, if it is in the right direction.-- Winston Churchill
References
• Cokins, G. (2008, April 3). How are balanced scorecards and dashboards different? Information Management.com. Retrieved April 12, 2014, from http://www.information-management.com/news/10001076-1.html?zkPrintable=true
• Cowan, K. (2013, December 15). Higher education’s higher accountability. Accreditation and Standards, Winter(2014). Retrieved from http://www.acenet.edu/the-presidency/columns-and-features/Pages/Higher-Education%27s-Higher-Accountability.aspx
• Dessinger, J. C. & Moseley, J. L. (2004). Confirmative evaluation: Practical strategies for valuing continuous improvement. San Francisco, CA: John Wiley & Sons, Inc.• Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2004). Program Evaluation: Alternative approaches and practical guidelines. Boston, MA: Pearson Education.• Frigo, M. (2012). The balanced scorecard: 20 years and counting. Strategic Finance, p. 49-53.• Griggs, V., Blackburn, M., & Smith, J. (2012). The educational scorecard: The start of our journey. The Electronic Journal of Business Research Methods, 10(2), 121-131.• Guerra-López, I. (2007). Evaluating impact: Evaluation and continual improvement for performance improvement practitioners. Amherst, MA, HRD Press, Inc.• Hell, M., Vidačić, S., & Garača, Ž. (2009). Methodological approach to strategic performance optimization. Management, 14(2), 21-42.• Hughes, K. E., & Pate, G. R. (2013). Moving beyond student ratings: A balanced scorecard approach for evaluating teaching performance. American Accounting
Association, 28(1), 49-75.• Kaufman, R., Gurerra, I., & Platt,W. A. (2006). Practical evaluation for educators: Finding what works and what doesn’t. Thousand Oaks, CA: Corwin Press.• Kaufman, R., Oakley-Browne, H., Watkins, R., & Leigh, D. (2003). Strategic planning for success: Aligning people, performance, and payoffs. San Francisco, CA: Josey-
Bass/Pfeiffer.• Kesler, G., & Kates, A. (2011). Leading organization design: How to make organization design decisions to drive the results you want. San Francisco, CA: Josey-Bass.• Laureate Education, Inc. (Producer). (2011a). Assessment and accountability in education: Dashboards, part 1. Baltimore, MD: Author.• Laureate Education, Inc. (Producer). (2011b). Assessment and accountability in education: Dashboards, part 2. Baltimore, MD: Author.• Popham, W. J. (2008). Transformative assessment. Alexandria, VA: Association for Supervision and Curriculum Development.• Shelton, K. (2010). A quality scorecard for the administration of online education programs: A Delphi study. Journal of Asynchronous Learning Networks, 14(4), 36-62.• Shelton, K., & Saltsman, G. (2005). An administrator’s guide to online education. USDLA Book Series on Distance Learning.• The Sloan Consortium (2012). Changing course: Ten years of tracking online education in the United States (2013). Babson Survey Research Group and Quahog
Research Group. Retrieved from http://sloanconsortium.org/publications/survey/changing_course_2012• U.S. Department of Education, NCES (2011, October 5). Learning at a distance: Undergraduate enrollment in distance education courses and degree programs v. 154.
NCES: Author. Retrieved from http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2012154• United States Government Accountability Office. Higher Education (2011). Use of new data could help improve oversight of distance education. (GAO-12-39).
Retrieved from Retrieved from http://www.gao.gov/assets/590/586340.pdf• U. S. News World Report (2014, January 7). Online education. Retrieved from http://www.usnews.com/education/online-education