1. Garbage In. Garbage Out.Quality Online Learning Begins with Great Courses Inputs Matter http://bit.ly/GarbageIn

2. Outputs tell us what happened Inputs tell us if what going in is any good. Inputs Matter - Garbage in, Garbage out http://bit.ly/GarbageIn 3. What is CLRN? California Learning Resource Network State-funded Education Technology ServiceInitial charter: Review supplemental electronic learning resources 4. Open Educational Resources6000+ free and commercial-free resources Aligned to CCSS and other CA standards 5. What is CLRN? Online Course Reviews Six subject areasStandards Alignment Common Core State Standards Californias other standards Next Generation Science StandardsiNACOL Standards for Quality Online Courses 6. Why do reviews matter?How do Colorado districts vet online courses? Colorado Graduation Pathways Program Survey 149 of 179 districts (83%) 7. What steps does your school/district take to validate that the rigor and quality of online or blended learning courses meets or exceeds that of traditional courses taught in a brick and mortar classroom? 8. Answers and QuestionsWe examine the course outlines with teachers and administration. Matching up the state standards with course, requiring minimums for passing scores None required 9. More Answers and QuestionsData supplied by curriculum provider. Student input on rigor compared to last course or class taken. Ensure alignment to state standards we need to be doing more 10. What factors did you consider when selecting courseware? Price77%Compared to content standards60%U.C. A-G list60%Examined course outlines55%Data supplied by provider42%Vendor demonstration42%Colleague recommendation Selected CLRN Certified courses. None01020304050 11. California eLearning CensusBetween the Tipping Point and Critical Mass http://clrn.org/census/ 12. 2013 Responses March 1 May 1 516 Responses 29% of 1777263 K-12/9-12 Districts/charters 253 K-5/K-8 Districts/charters 13. Who is eLearning?516 districts & charters 46% Yes No 14. eLearning by Grade Span80% 70%73%60% 50% 40%201230%201320% 10%19%0% K-5/K-8K-12/9-12 15. Who wants to eLearn?26%Yes NoN=273 16. Who wants, by grade spanK-5/K-8K-12/9-12 Yes20%No44% 17. Full and Part-time Online Learning Populations 120000 100000 100.9K80000 60000 40000 2000086.3K23% Increase 19.8K17% Increase24.4K0VirtualBlended2012 2013 18. Blended and Virtual Medians 120Medians Increase by 25%10010080 608070 20124056201320 0Blended MedianVirtual Median 19. Blended Model Breakdown 2012Self-BlendEnriched-VirtualRotationFlex0%10%20%30%40%50%60%70% 20. Blended Model Breakdown: 2013Self-BlendEnriched-VirtualRotationFlex0%10%20%30%40%50% 21. 2013: K-5/K-8 Blended Model Breakdown FlexRotation80% N=40Enriched-VirtualSelf-Blend05101520253035 22. 2012: K-12/9-12 Blended Model Breakdown FlexRotation N=199Enriched VirtualSelf-Blend020406080100 23. 2013: K-12/9-12 Blended Model Breakdown Flex28%Rotation39% N=163Enriched Virtual37%Self-Blend48% 020406080100 24. Course Publisher Distribution 2012 25. Course Publisher Distribution 2013 Apex Learning Aventa (K12, Inc) Cyber High Odysseyware ALEKS K12.com Pearson Digital Learning Plato Learning District Created Education 2020 Compass Learning BYU Independent Study Accellus Advanced Academics Florida Virtual School Middlebury Interactive Languages Currium A+ Learning Class.com Accelerate Education Voyager Learning National University Virtual High Connections Learning University of Nebraska, Lincoln01020304050 26. Between the Tipping Point and Critical Mass 27. Why CLRN Reviews Online Courses Disrupting Class (2008) Digital Textbook Initiative (2009/10) Full courses of study Authorization (2010) 28. Review Process Literature review & initial criteria SREB, iNACOL, TxVSN, QM, and WA Digital LearningCLRN Advisory (Stakeholder) meetings Initially review HS ELA & math courses to CCSS & online course standardsStrength of TxVSN version Criteria rewrite: California Criteria rewrite: national 29. Online Course Reviews Walkthrough 30. CLRN HOme 31. List View 32. Not Met standards 33. How CLRN Reviews Utilized existing CLRN review sites Six physical locations around CA Subject specific Established reviewer cohorts (20-30 educators)Three-person review teams Two-part review. Well, actually 8 partsEdited & normed at CLRN Central 34. CLRN Review Process Database driven Publishers complete submission online Attach standards correlation documentPublishers supply 10 teacher and student accounts 35. Publisher Eight Step Entry 36. Step 2: Content Standards 37. Standards Correlation Algebra I CCSS 38. Step 3: Course Standards 39. Step 4: Instruction Type Length Accreditation 40. Step 5: Technology 41. Step 6: Features 42. Step 7: Abstract 43. CLRN Review SitesSix, subject-specific sites County office based Managed by COE content expert 20-25 active teachers Meet monthly during school year 44. CLRN Review SitesEnglish-language arts San Diego COEHistory-social science San Joaquin COEMathematics Kings COEScience Humboldt COEVisual and Performing Arts Orange DoEWorld Languages Orange DoE 45. Reviewer Application Credentialing Info Teaching Experience Content Professional Development Technology ExperienceShort Response QuestionsConflict of Interest Professional References 46. Reviewer Training & NormingCLRN Review Process Social Content Criteria Online Course Standards 47. CLRN OCR Training Videos 48. OCR Retraining & Norming 49. Place video clip of b3 here 50. CLRN Course Review Process Nine-step process1. Team assignment 2. Content (CCSS) Standards review Develop, Practice, Assessment3. Minimum Requirements Reading levels, spelling/grammar, content accuracy, etc.4. Social Content Review 13 areas, including male/female roles, ethnic/cultural groups, & advertising. 51. CLRN Review Process 5. Online Standards 6. Features7. Profile 8. Abstract 52. Step 1: Team & Purpose 53. Step 2: Content Standards 54. Step 4: Social Content Male/Female Roles Ethnic/Cultural GroupsOlder Persons/Aging Process Disabled PersonsReligion Gay, bisexual, & transgender Brand Names/Advertising 55. iNACOL Standards for Quality Online Courses Online courses participate in the teaching/learning process 56. Online Course Standards Southern Regional Education Board iNACOL Texas Virtual School Network California Learning Resource Network 57. Step 5: Course Standards Reviewers vet 24 of the 52 OCR standards 58. iNACOL Course Standards ContentInstructional Design Student AssessmentTechnology Course Evaluation and Support 59. A. Content Content depth and breadth Information literacy skills Learning resources and materials Communication process between teachers, parents, and students Content accuracy and bias 60. B. Instructional Design Course design and organization Meaningful and authentic learning experiences Multiple learning paths for students to master the content Higher-order thinking skills Instructor-student and student-student 61. OCR Standards/Considerations B3. The course instruction includes activities that engage students in active learning. NOTE: Reading and watching are not activeThe course provides multiple opportunities for students to be actively engaged in the content that includes meaningful and authentic learning experiences such as collaborative learning groups, studentled review sessions, games, analysis or reactions to videos, discussions, concept mapping, analyzing case studies, etc. 62. C. Student AssessmentAlignment between the course goals and activities and its assessment strategies Insure that there are adequate and appropriate methods to assess students Assure that students are constantly 63. D. Technology Course architectureUser interface Rich MediaAccessibility 64. E. Course Evaluation and Support Evaluating course effectiveness Accreditation Teacher and Student preparation and support 65. CLRN Central Review Validation & Norming Work the courseReview/update all 52 course standards Standardize review comments Notes inform reviewer retraining 66. CLRN Central Review Editing & Proofreading Review, modify, and standardize content standards rating & commentsPublisher Feedback Seven-day window New evidence required for Re-Review One Re-Review permitted per course.Publishing Reviews valid for three years 67. Texas H.B 4294eTextbook publishers may submit updated content for review Districts/schools may select a subscription-based electronic textbook 68. Supplemental Update Review For Updated Courses Updated, not new (New=>30% new content) New content and/or course standards. Strengthen alignment to the content standards and/or the course standards. Publishers provide specific evidence.Once per year 69. CLRN OversightStanislaus COE/CLRN Central CDE/Curriculum, Frameworks and Instructional Resources CLRN Management Team Meetings CLRN Advisory Team 70. Individual Course Review 71. Standards Display 2 72. CLRNCertifiedUniversity of California Partnership A-G Approved CoursesCLRN-Certified 80% content standards 80% online course standards 15 power standards 73. Course Criteria 74. CLRN/iNACOL Course Standards 75. OCR Standards & Considerations 76. CLRN Certified Information 77. U.C. Online Learning Home 78. CLRN Certified FAQ 79. The Good NewsCLRN Reviews inform course improvement 80. Re-Reviews 81. Content Re-reviewsAdditional content New activitiesAdditional components to meet course standards Syllabi, policies, resources, etc. 82. AccessibilityYoure only a lawsuit away from making your courses accessible 83. DOJ StoryD10: Course materials and activities are designed to provide appropriate access to all students 84. Closed-Captions/TranscriptsEdgenuity (e2020) BYUCompass Learning K12, Inc.(Word has gotten out) 85. Why Inputs Matter Resubmit an Assignment Story 86. Ed CodeEd Code 49011 states that school districts and schools shall not establish a two-tier educational system through payment of a fee or purchase of additional supplies that the school district or school does not provide; and 87. Analysis of the 398 published 53% certified Up from 46& 27% teach < 80% content standards Median: 63% Range from 4% met to 78% met 12% only missing captions or transcripts Most common problem Content standards alignment 88. Certification by Subject ELA:39% (down from 54%)History: 44% Up from 27%Math: 41% Down from 47%Science: 45% Up from 28%World Languages: 69% Down from100% 89. How do you know if its any good? Garbage inGarbage out 90. Has the course been reviewed/approved by any agencies? CLRN University of California A-G requirementsNCAA College Board 91. Common Core State Standards or state content standardsDemonstrate, Practice, & Assess Quantity, Depth and Degree 92. Selecting a great courseOr How Not to Buy a Car 93. The Need for Multiple Filters CLRN Self-review of content and course standards Customer feedback 94. Have you taken the course for a test drive?Would you buy a car sight unseen? Would you sit in the passenger seat? Then why rely solely on a provider demonstration? 95. Course Test DriveFully functioning demo Student and Teacher accounts 96. Play as a studentParticipate in several units, spread across the course Be a great studentBe a struggling student 97. Be a teacherAccess the LMS Can you add content / projects? Check for student progress Formative & summative assessments 98. Course Test DriveIs the course engaging? Does teach beyond Knowledge and ComprehensiveText based vs. multi-media 99. You dont enter the online revolution with the courses you want. You enter it with the courses you have. 100. What is not a great online course Class 1: Watch lecture. Complete worksheets. Take a multiple choice test. Repeat Class 2: Read. Take a multiple choice test. Repeat Class 3: Read. Print and take quiz. Grade your quiz. Take multiple choice test. Repeat Class 4: Read a physical book. Take quiz from a physical book. Take part in online discussion. Repeat Moving a textbook online is not a foundation for a great course. 101. What is not a great online lesson.Writing an introduction slide show Vocabulary exercises. Students then:READ 12 textbook pages about verbs READ 13 pages of textbook reading about verb agreement 102. Not a great online lesson, continued READ 10 pages of textbook reading about verb tense READ 6 textbook pages about verbs made easy READ a one-page textbook page about writing a first draft, and READ two textbook pages about simple narratives. 103. Your course should be better than the worst teacher at your school 104. What should we expect from great online courses? 105. Expect moreA great online course should not look like a textbook. 106. High-quality course Stimulating lecturesWritingJust-in-time readingStudents create, evaluate, an d analyze.Active learningEngaging activities that go beyond knowledge and comprehensionVariety of assessment types 107. Make use of the mediumRich media Lecture clips, video demonstrations and clips, variety of multi-media, simulations, Ease of use LMS helps inform instruction. Students/teachers always know where they are. 108. Make use of the mediumMultiple learning paths Teacher is informed of problems. Students are provided alternative lessons.Engaging Active learning, online discussions, writing & analyzing, authentic activities 109. Inputs Matter Proficiency cant occur unless courses are high quality Teaching a critical mass of the standards Actively engage students Challenging students beyond knowledge and comprehension 110. Has CLRN Certified it? 111. eLearning Strategies SymposiumDecember 6-7, 2013 Hilton, Costa Mesa elearns.org @elearns 112. Garbage In. Garbage Out.Quality Online Learning Begins with Great Courses Inputs Matter