Quality Matters TM Overview ©MarylandOnline, Inc. 2013. All rights reserved Updated 02112013.

Applying the QM Rubric

Quality MattersTM OverviewMarylandOnline, Inc. 2013. All rights reserved

Updated 02112013Welcome to the Quality Matters Program Overview.1AgendaProgram IntroductionQM Basics2011-2013 Quality Matters Higher Education RubricCourse Reviews and CertificationProfessional Development and CertificationWhats In It for FacultyWhats In It for InstitutionsThe QM Organization

MarylandOnline, Inc. 2013. All rights reserved

The Agenda for this presentation includes an introduction to the QM Program, an explanation of the basic elements of the program, an overview of what the program has to offer faculty and institutions, and will close with a perspective on the QM organization.

2The Quality Matters ProgramQuality Assurance through Faculty Development and Course DesignMarylandOnline, Inc. 2013. All rights reserved

3Academic & Non-ProfitQuality Matters is a not-for-profit subscription service providing tools and training for quality assurance of online courses

Developed by MarylandOnline with funding from FIPSE, it was designed by faculty for faculty and focused on improving student learning

Adopted by a large and broad user base, QM represents a shared understanding of quality in online course design


Quality Matters is both academic and non-profit. We provide inter-institutional quality assurance in online learning. The widespread adoption of Quality Matters throughout the United States represents the shared understanding of quality in online course design. We operate as a not-for-profit subscription service providing tools and training for quality assurance of online courses. MarylandOnline, a consortium of colleges and universities in Maryland, developed the grant-funded project and continues to provide oversight and governance of the program. 4Quality Matters is . A set of standards (Rubric) for the design of online and blended courses

A peer review process (faculty to faculty) for reviewing and improving online and hybrid courses

A faculty support tool used by instructional development staff

A professional development opportunity


In practice, QM is a set of quality standards for online course design, a faculty-centered peer review process for online courses, a tool used by instructional design staff in their work with faculty, and a professional development opportunity for both faculty and staff.5 The QM ApproachQM is a faculty-driven, peer review process that isCollaborativeCollegialContinuousCentered in academic foundationaround student learning


About The Content: The underlying principles of QM are a primary reason for its wide-spread adoption. Quality Matters provides a faculty-driven, peer review process that is.Collaborative: QM was designed by and for faculty to share expertise and experience relative to the design of a course.Collegial: The course review process is a collegial discussion between faculty peers committed toContinuous quality improvement. It is not an evaluation.Centered in national standards of best practice, the research literature and instructional design principles designed to promote student learning.

6 Factors Affecting Course Quality

QM ONLY Reviews Course DesignMarylandOnline, Inc. 2013. All rights reserved

About the Content: There are many factors that affect the quality of an online course. Among these factors are:

the course design (the forethought and planning that goes into an online course,) the course delivery (i.e. the way the course is taught, also known as faculty performance), the course content, the learning management system and its functionality; technical support the institutional infrastructure (help desk, online library access, online tutoring access, etc), a faculty members training and readiness for online teaching, and the students role with respect to engagement and readiness for an online course.

QM reviews just one aspect of online course quality Course Design.

7QMs Role in Quality Assurance QM looks at course designThe harnessing of technology to deliver instruction and promote student learning

QM provides a process for peer-to-peer feedback for faculty in the continuous improvement of their course and certifies courses as meeting shared standards of best practice

Quality Matters is not the complete answer to quality assurance for online education, but it can be a critical component


QM is focused on the design of the course and it is not meant to resolve all quality assurance issues. For example, although reference is made in the rubric standards to technical support or student support services, QM is not designed to evaluate those aspects of quality assurance, rather QM strictly focuses on course design.8 What QM is NOT AboutNot about an individual instructor (its about the course)

Not about faculty evaluation (its about course quality)

Not about judgment(its about diagnosis and improvement)

Not about win/lose or pass/fail (its about continuous improvement in a supportive environment)


Its important to compare what QM is and is not: about course design, not about the instructor about course quality, not about faculty evaluation score that comes out of a QM course review is Diagnostic rather than Judgmental. It tells us how much revision a course may need. about continuous improvement, no about pass or fail

9QM as a National Standard600+ current subscribers46 states representedQM has trained 22,000+ faculty and instructional design staffCommunity CollaborationLargest community collaboration on quality in online educationAward Winning- Sloan-C , USDLA, WCETMarylandOnline, Inc. 2013. All rights reserved

Quality Matters has become a widely adopted program of quality assurance. It currently has more than 530 subscribers across 44 states, as well as subscribers in Bermuda, Canada, Saudi Arabia, Australia, Singapore, and Guam. Statewide systems also subscribe to provide coordination and more economical access to QM tools for their member institutions. Please visit our website for a list of current subscribers. Quality Matters has trained well over 14, 000 faculty and instructional design staff and has been recently recognized by both the Sloan Consortium and USDLA for excellence. Because of the number of participants in the program and the collaborative development and implementation of our tools and program offerings, QM represents the largest community collaboration on quality in online education.

10MONTANAWYOMINGIDAHOWASHINGTONOREGONNEVADAUTAHCALIFORNIAARIZONANORTH DAKOTASOUTH DAKOTANEBRASKACOLORADONEW MEXICOTEXASOKLAHOMAKANSASARKANSASLOUISIANAMISSOURIIOWAMINNESOTAWISCONSINILLINOISINDIANAKENTUCKYTENNESSEEMISSALABAMAGEORGIAFLORIDASOUTHCAROLINANORTH CAROLINAVIRGINIAWVOHIOMICHIGANNEW YORKPENNMARYLANDDELAWARENEWJERSEYCONNRIMASSMAINEVTNHALASKAHAWAIIPUERTO RICOVIRGIN ISLANDSCurrent SubscribersConsortium Subscribers Quality Matters SubscribersStatewide SubscribersMarylandOnline, Inc. 2013. All rights reserved

The map provides another view of the wide-spread adoption of QM. The color red/rust indicates states where there are QM subscribers, blue shows the presence of large subscribing consortia, and yellow indicates statewide subscriptions. Even though this slide doesnt indicate it, we have a growing number of international subscribers as well.

11QM BasicsMore than the sum of its three partsMarylandOnline, Inc. 2011. All rights reserved

This section of the presentation will introduce the basic elements of the Quality Matters Program: The QM Rubric, professional development workshops and certifications, and course reviews.12Consisting of:8 key areas (general standards) of course quality 41 specific review standards21 essential standardsDetailed annotations and examples of good practice for all 41 standards

The Rubric is the Core of QM


Note that the detailed explanations of the 41 review standards is contained in the annotations to each standard. The annotations are quite extensive and deal with the nuances in each standard, issues of interpretation and examples of good and poor practice.

QM has a pdf on our website of the rubric standards for public view. The fully annotated version of the rubric cannot be distributed outside your subscribing institution, but may be published on your password protected intranet.13 The QM RubricEight General Standards:

Course Overview and IntroductionLearning Objectives (Competencies)Assessment and MeasurementInstructional MaterialsLearner Interaction & EngagementCourse TechnologyLearner SupportAccessibility

Key components must align.

Alignment: Critical course elements work together to ensure that students achieve the desired learning outcomes.


Of the 8 general standards, 5 contain standards that address key components that must be in alignment.

QM tries to take a holistic view of the course and thats why its so important that the Learning Objectives ALIGN with the Assessments, Resources and Materials, Interaction, and Technology.

14Standards Supported by Research

The rubric is research-based and was developed in conjunction with best practices in the industry. (QM standards come from a detailed review of existing research literature, updated every few years, from existing best practice sets, and from our community of practitioners, that provide feedback on every new rubric version.) QM focuses on the standards needed in course design and to promote student learning.

This is an excerpt from a document, available on the QM website, that shows how the specific review standards are supported by existing literature. A literature review is conducted for each new rubric edition that builds upon this core matrix. 15Course Reviews and CertificationThe Peer Review ProcessMarylandOnline, Inc. 2013. All rights reserved

Another central element of Quality Matters is the Peer Review Process

16The QM Peer Review Process


This circle graphic represents the continuous nature of a Quality Matters course review, which starts with a faculty course developer submitting an instructor worksheet which is review together with the review team before the start of the course review. Reviewers work independently of each, but communicate throughout the review process as necessary. Once the reviewers submit their forms, the online tool aggregates the points as well as the feedback and provides a report for the faculty developer. If the course meets standards at this point, the course will be recognized by QM, If it does not, the faculty developer will have an opportunity take all the feedback, make amendments to the course, and resubmit all as part of the single review process. There are only two possible states for a course submitted for a QM review it is either under review or it has met standards. 17Unofficial ReviewsInternal/Informal subscriber reviews not required to follow QM processOfficial Reviews follows QM policies/protocols QM-Managed review - QM manages and pays review teamSubscriber-Managed review - Subscriber manages and pays review team

Types of Course ReviewsMarylandOnline, Inc. 2013. All rights reserved

There are two general types of course reviews distinguished by whether or not they are recognized by QM. Unofficial reviews are what QM calls internal reviews and these are not required to follow the QM process. Official reviews, recognized by QM, can be done in one of two ways. Either QM, for a fee, manages the review and compensates the review team, or Subscribers (with the appropriate subscription) takes the management role. Most subscribers do both types of reviews.18 QM = Process and RubricProcessRubricOFFICIAL

Outcome: Earn QM recognitionMust follow official QM guidelines and procedures

Outcome: Improve courses, meet institutional goals, demonstrate commitment to quality

Outcome: Earn QM recognitionTool to assess online courses during formal QM review

Outcome: Improve courses, meet institutional goalsGuide to develop new online courses AND review and update online courses

INFORMAL

Outcome: Improve courses, meet institutional goals, demonstrate commitment to qualityInstitutions determine use and procedures

Outcome: Improve courses, meet institutional goals, demonstrate commitment to qualityGuide to develop new online courses AND review and update online courses

Its important to note that QM is both a Process and a Rubric (tool) for course reviews.

There are significant benefits of using QM informally, but official QM recognition comes only if the official process is followed.

19 The Peer Review TeamOn an official review, Review Teams consist of:

Three QM-Certified Peer ReviewersPre-requisite: Current online teaching experience & trainingMaster Reviewer as Team ChairPeer Reviewer with additional experience and trainingOne reviewer must be a subject matter expertAt least one reviewer must be external to the institution sponsoring the course AND Faculty course developer: access to rubric prior to reviewinvolved in pre-review discussionsconsulted during review

Three QM-certified faculty peer reviewers use the Course Review Management System (the fully automated rubric tool) to complete the review. One of these reviewers must be a Subject Matter Expert (SME) and at least one must be external to the courses home institution. The team chair must be a Master Reviewer. A single person could play multiple roles, could be both the SME and the master reviewer, for example.

The Faculty Course Developer (the Instructor) should be consulted by the team as questions or issues arise.

20Peer FeedbackThe peer feedback provided in a course review is collegial in tone and language; however, the reviews are rigorous and even courses that meet standards will benefit from the detailed, specific, and relevant feedback provided.


21QM Certification of CoursesCourse reviews take 4-6 weeks; 20-week window for reviews, including any amendments, to be completed

QM recognition is provided by year recognized QM logo on course and catalog; registry on QM Web site


QM certification is provided by the year the course met standards and is good for 5 years, unless the course undergoes significant change.

Its important to note that while QM certification is the benchmark of quality online course design, a course that meets QM standards is not considered perfect. Continuous improvement, a basic principle of QM, is encouraged and expected for all courses that carry the QM certification mark.22Professional Development and CertificationQM Workshops & CertificationsMarylandOnline, Inc. 2013. All rights reserved

The 3rd basic element of the QM program is professional development.23QM Workshops & CertificationsDelivery Options:Online virtual, asynchronousOnsite face-to-face, synchronousCollaborate virtual, synchronousFocus:QM ImplementationPrepare faculty and staff to hold QM roles Faculty Development WorkshopsPrepare faculty to design and improve courses


QM workshops and certification courses are offered online and onsite. The focus for some are on certification of roles necessary in the implementation of QM and others are more about faculty development.24Faculty Development WorkshopsCourse-focused WorkshopsApplying the QM Rubric*Designing Your Online CourseImproving Your Online CourseDesigning Your Blended Course

*Note: Flagship training is also required for QM ImplementationStandards-Focused WorkshopsDesign that Welcomes Your StudentsCreating a Foundation with Learning ObjectivesChoosing and Using Media EffectivelyAddressing AccessibilityAlignment Standards WorkshopsConnecting Learning Objectives and AssessmentsLinking Instructional Materials and Learner Engagement


Faculty development workshops differ by those focused on working with the whole course, those that drill down to provide a more time working on specific QM standards, and those that focus on helping faculty achieve alignment in their courses.25QM Implementation: CertificationsApplying the QM Rubric (APPQMR)Peer Reviewer Course*Master Reviewer Certification*Certified Facilitator*F2F CF: For face-to-face delivery of APPQMRCOF: For online delivery of APPQMRInstitution Representative Training*

*Offered online only


The certification courses for QM implementation include Peer Reviewer Certification, Master Reviewer Certification, and Certified Trainer courses. The Institution Representative workshop is also required for implementation. The Applying the QM Rubric workshop is a prerequisite to these certification courses.26

Capacity Building: QM Roles & CriteriaMarylandOnline, Inc. 2013. All rights reserved

For a cost effective, robust QM implementation, you will want to develop your internal capacity. To manage your own reviews, you will want to have your own certified peer reviewers and Master Reviewers. (You may always use reviewers and Master Reviewers from QMs database, but you would be responsible for compensating them.) You will also need a QM-trained Institution Representative (IR). All IRs are STRONGLY encourage to take at least the APPQMR training and the PRC is also suggested.

To conduct the training, you will need a certified QM-trainer.

And you would need the licenses that come with the Full Subscription option.Quality Matters Program Overview2/11/201327Quality Matters: By Faculty for FacultyWhats In It for FacultyMarylandOnline, Inc. 2013. All rights reserved

28Whats In It For Faculty? Improvement of online courses

External quality assurance

Participation useful for professional development plan and portfolio

Faculty development trainings

Review other courses and gain new ideas for own course; expand professional community

For faculty, in particular, Quality Matters provides a number of benefits. The direct and primary benefit of an improved course is obvious. But undergoing, and participating in, a peer review process is an enriching activity that provides external quality assurance in the spirit of the academy. Quality Matters has an extensive pool of peer reviewers that provide a professional network connected by the association with Quality Matters program and its emphasize on collaboration to improve student learning outcomes. Participation in Quality Matters is a meaningful professional development tool.

29Roles for Faculty in QM

The primary roles for faculty in QM are as Peer Reviewers and Master Reviewers.30Tools in the Spirit of the AcademyStandards grounded in the research literature and best practicesTools and process developed by faculty and instructional design experts for faculty supportPeer review and other opportunities to collaborate and mentor around quality in online learning

The tools and processes in QM were developed collaboratively by faculty for faculty. Peer review using research-based tools and collegial interactions are hallmarks of QM. 31What to Expect from Course ReviewsConsistency and Rigor

Professionalism and Commitment

Useful and Constructive Feedback

Positive Outcomes for Faculty

Less than 50% of courses in QM-managed reviews meet standards upon initial review (but all can meet after amendment)

A QM course review is a rigorous process, but there are many positive outcomes for faculty as course developers and as reviewers. The next two slides contain faculty comments taken from the exit survey completed at the end of every course review. 32Review Exit Survey CommentsSample Comments from Faculty Developers

I was too close to see what could be improved.

Provides a great way to get an objective view of your course.

It made all of my online courses better.

It provides a view from a more student oriented perspective.

Many elements that might contribute to a student withdrawing can be eliminated.

33Review Exit Survey CommentsSample Comments from Peer Reviewers

I always learn new wonderful approaches from viewing other courses and working with expert peer reviewers.

Whenever I read the annotations, I make changes to my courses.

I always find reviews to be a learning experience.

This experience was very rewarding for me as both an educator and course developer.

Its a great tool for honing your own course.

34Research Opportunities and SupportQM encourages research through

Consultation with QM Director of Research

Collaboration on research and data resources

Participation in faculty surveys of all QM subscribers

Sponsorship of research - see RFP at www.qmprogram.org

QM also provides research opportunities, to include issuing an annual RFP and funding a small number of research projects on QM-related topics.35Quality Matters: A Multi-faceted ResourceWhats In It for the InstitutionMarylandOnline, Inc. 2013. All rights reserved

QM also meets many different needs for the institution36Multiple Uses of QMReported Uses of QM Tools:Guidelines for initial online course developmentQuality assurance of existing coursesOngoing faculty professional developmentInstitutional re-accreditation packagesFocus attention on distance learning policies & steering committeesCreate a campus climate and structure that promotes teaching and learning

QM is used by schools in a number of ways and for different purposes. Many schools use the rubric as a guideline for their initial development of courses. Many use if for quality assurance of existing courses. Often, QM trainings serve as an important piece of a schools faculty development efforts. Schools often specifically reference their use of Quality Matters in their re-accreditation packages. Finally, QM can be a useful tool in a schools institution building efforts as it relates to their distance learning program. Early research on this topic is suggesting that QM can be a very useful tool to create and sustain a campus climate more focused on teaching and learning.

37Reported uses:Educate faculty about quality in online courses Reduce time and labor in designing courses.Provide professional development opportunities for faculty and staffThreshold requirement for adjunct facultyInform teaching practices related to design and structure for BOTH online and F2F coursesQuality process indicators for accreditation self-studies

QM Professional DevelopmentMarylandOnline, Inc. 2013. All rights reserved

In terms of the professional development benefits, the education of faculty on quality in course design provides professional growth opportunities for them in ways that directly benefit the institution.38Subscription BenefitsCurrent QM Rubric and related rubricsFully automated QM Rubric ToolSubscriber DiscountsIR TrainingMyQM & Users GroupInternal Course ReviewsTraining License*Subscriber-Managed Course Reviews*MarylandOnline, Inc. 2013. All rights reserved * Depends on subscription type

39Subscription benefits include access to the QM Rubric, the online rubric tool (CRMS), subscriber discounts for trainings and services, Institution Representative training, and participation in the online users group. The users group, a password-protected area of the QM website, serves as a resource for communication and information sharing, collaboration with other QM Subscribers,access to important documents and web links. The IR training documents may be accessed through the QM Institute site.

In addition, the Full Option includes the site license for training and the right to conduct your own official course reviews.Quality Matters: A Collaboration on QualityThe QM CommunityMarylandOnline, Inc. 2013. All rights reserved

Now a final word on QM as a collaboration on quality. 40The QM CommunityMarylandOnline, Inc. 2013. All rights reserved

The impact QM is having on quality in online education has been achieved only through the cooperative efforts of many and through collaboration with the groups shown on this graphic. The small QM staff provides support but it is the faculty and staff throughout the country, and now beyond, serving as IRs, trainers, and reviewers, that are implementing the program. QMs many volunteer committees bring community-based governance to the process. The many institutions supporting QM, from subscribers to our partners and sponsors, help to disseminate the best practices and standards as broadly as possible. And students, the primary beneficiaries of a quality course, are indicating that a well-designed course truly matters to them. 41QM is Continuous ImprovementImproved coursesEngaged facultyReduced course development timeOngoing faculty developmentQuality benchmarkingAwareness & support for online learningInstitutional improvement


The point of Quality Matters is continuous improvement. The purpose is to provide tool and processes that are adaptable to the different goals institutions want to achieve in pursuit of quality.42Your Questions?Please contact the appropriate QM representative available to you via email, Skype or phone.MarylandOnline, Inc. 2013. All rights reserved

The Quality Matters staff is available and eager to answer your questions.43We are here for you! QM Contacts:Deb Adair, Managing Director [email protected] Hall, Communications Manager [email protected] Ragias, QM Registrar, Higher [email protected]

Marijane Hakun, Subscriber Services Manager [email protected] Poole, Course Review Manager [email protected] Henn, CIO; Director, Technology [email protected]


Thanks to you, Quality Matters.

44

45

Quality Matters TM Overview ©MarylandOnline, Inc. 2013. All rights reserved Updated 02112013.

Documents

Transcript of Quality Matters TM Overview ©MarylandOnline, Inc. 2013. All rights reserved Updated 02112013.