Nokia/Pearson Foundation Mobile Learning Institute ...

For more information about this report, please contact: Dawn Anderson-Butcher, PhD

Community and Youth Collaborative Institute The Ohio State University, College of Social Work

614-292-8596 [email protected].

Produced by: Dawn Anderson-Butcher, Terry Medalen,

Annahita Ball, Sarah Davis, & Rebecca Wade-Mdivanian

Mobile Learning Institute: Adoption of Innovation

Key Findings and Conclusions

mailto:[email protected]

Introduction Over the past year, our OSU team has enjoyed working with Nokia and Pearson Foundation leaders as Mobile Learning Institute (MLI) leaders examine preliminary outcomes associated with this innovative program. We are impressed with the initial evaluation results, as well as the growing commitment among MLI leaders to use evaluation findings to support MLI program improvements and infrastructure development. It has been an enriching experience to contribute to the growth and development of the MLI. Initial evaluation results point to positive student and teacher participant outcomes associated with participation in the MLI. Student participants indicate that they have improved media technology, innovation and critical thinking, and life skills. Teachers describe improved competencies related to using technology in their classrooms, and incorporating pedagogical practices into their instruction that support the development of 21st century skills. Stakeholders overall report great satisfaction with the MLI experience. In addition, the Pearson Foundation has committed staff to oversee evaluation, and provide formative data back to program sites about progress, successes and areas of improvement. Many program and infrastructure improvements are already underway. Collectively, these are all excellent indicators of success. In the end, the MLI is a novel approach to education that blends innovation, technology, and learning to foster critical 21st Century skills. MLI is an innovative program design, one focused on preparing the global workforce of tomorrow. Future endeavors to improve upon the MLI design will further build upon the initial progress gained from this first year of evaluation. The MLI is poised for scale-up and replication efforts that will allow for even greater impact in the future. I expect the MLI to continue to evolve as a leading innovative program that promotes teaching and learning through the use of media technology. Our OSU team looks forward to continuing its partnership with the MLI as it expands in the U.S. and globally. Respectfully,

Dawn Anderson Butcher, PhD, LISW-S Associate Professor & Principal Investigator Community and Youth Collaborative Institute College of Social Work The Ohio State University

2

Nokia/Pearson Foundation Mobile Learning Institute: Adoption of Innovation

Key Findings and Suggestions

The Ohio State University (OSU), under the leadership and guidance of Dr. Dawn Anderson-Butcher, was funded by the Pearson Foundation to provide consultation and technical assistance in the ongoing internal evaluation of the Mobile Learning Institute (MLI). This project had two main goals that built upon priorities set forth in the Pearson Foundation’s internal evaluation:

1. Assist with the further refinement of the MLI program design and theory of change, especially in support of future scale-up and replicate efforts, and

2. Provide ongoing consultation and technical assistance in support of the comprehensive internal evaluation of the MLI engagements. Overall, the project included several components. More specifically, efforts supported the

development of a detailed description of various programming tiers, the construction of matrices that showcase the overall MLI design strategy and key outcomes, and the development and pilot tests of evaluation tools. Additionally, these tools were implemented, refined, and further developed throughout the project. This report outlines key deliverables and progress in relation to the aforementioned goals. First, the refinement of the MLI program design and theory of change is presented. A discussion of the development of evaluation tools in support of the comprehensive evaluation of the MLI follows1.

Program Design

It was necessary to first understand the MLI program design before developing assessment tools. Initially, a detailed map of programming tiers was co-developed by the OSU team and Pearson Foundation evaluators. Please see the Appendix for this map. Collectively, these tiers summarized three levels of intervention received by each targeted group within the MLI. The first tier was comprised of Innovation sites. That is, these sites had existing, long-term relationships among the Pearson Foundation, Nokia, and the school district. Various MLI strategies were being implemented system-wide at these sites (including those among teachers, students, and the community). Adoption sites also had relationships among the Pearson Foundation, Nokia, and the district but these relationships were less developed than those for innovation sites. Sites in the Introduction tier were those with a developing relationship among the Pearson Foundation, Nokia, and the district. These sites demonstrated little demand from the school system in relationship to the MLI and initial relationships and capacity were still in process.

Program participants were cross-mapped across the three tiers to identify varying levels of programming throughout the MLI. The resulting matrix (located in the Appendix) showcased

1 Please note, a complete Glossary of Terms is available at the end of this report.

3

the level of MLI involvement by targeted districts/communities involved across the broad MLI initiative. This matrix allowed for key insights to be drawn about the overall MLI design strategy, particularly in relation to the connection between program goals, practices, and desired outcomes. Further, this mapping allowed for the refinement and alignment of program strategies and targeted outcomes by each tier area and across the MLI initiative. More specifically, this initial work on the MLI program design supported the creation of logic models that visually depicted MLI activities and expected outcomes.

In consultation with Pearson Foundation staff, the OSU team developed two logic models summarizing the key activities and outcomes of the MLI engagements for students (Figure 1) and teachers (Figure 2). Both models include program activities (i.e. services provided) along the left columns. These activities then “link” to expected outcomes (i.e. benefits for participants). The outcomes along the far right columns of these models are themed in relation to the four Kirkpatrick levels summarized in the middle column: (1) Reaction; (2) Learning; (3) Behavior; and (4) Results. Brief descriptions of these theme areas are outlined in the middle column of the models. In addition, other factors that likely impact the outcomes of the MLI engagements are listed at the bottom of the model.

Figure 1. Student MLI Logic Model

4

Figure 2. Teacher MLI Logic Model

These models summarize the relationship between the MLI program and its intended outcomes. In addition, they highlight the program vision and rationale. Clearly, the program design articulates the connection between activities and outcomes. Supplementary materials (e.g., descriptions of activities and outcomes, curriculum guides, etc.) can elaborate on these key program components and aide in further development of the MLI. Several examples of the MLI curriculum are available in the Appendix. In turn, models such as these will also help “tell the story” of the MLI to key stakeholders or other potential partners.

MLI Assessment

Logic models and other program design tools are useful for guiding the development of MLI assessment procedures, as well (Weiss, 1972). The key outcomes for the MLI engagements are clearly delineated along the far right columns of Figures 1 and 2. Program assessments can be aligned with the models, including assessment of specific indicators associated with these outcomes. During this project year, the OSU team provided consultation and technical assistance in support of a comprehensive internal evaluation of the MLI engagements. This included previously outlined assistance in relation to program design but also involved the development of assessments for the student and teacher MLI engagements.

5

To begin, the OSU team developed processes for data collection and management (e.g., development of data collection procedures, strategies for on-site data collection, tools to assist in data collection, and online survey mechanisms). In addition, various measurement tools were created, including pre-post and post-reflection evaluations. Throughout this project component, Pearson Foundation staff provided ongoing input and increasing assistance in the formulation of student and teacher assessments. Initially, the OSU team “housed” all data collection and management. That is, the OSU team created the online assessments and processed and stored the data. As the Pearson Foundation began collecting data and “housing” this information on their own, two separate pools of data were created. In the end, the Pearson Foundation data were merged with the OSU data to form one large pool of data reflecting both the initial efforts of OSU and the emerging capacity at the Pearson Foundation. The development of these tools and initial results are outlined below.

MLI Student Post-Reflection Assessment

The MLI Student Post-Reflection Assessment (see Appendix) was designed to capture the perceived experiences of students involved in the MLI. Following the development of the initial item pool, assessments were piloted and refined until a final assessment was used to gather data across 13 MLI sites.

Item Pool

Consistent with DeVellis’ (2003) recommendations for scale development, the OSU team first generated an item pool to reflect the underlying assumptions of the program design. Initially, several items were included from existing measurement tools developed during previous projects, such as the Ohio Comprehensive System of Learning Supports Initiative (Anderson-Butcher & Wade-Mdivanian, 2009). These items measured several of the constructs identified as priority outcomes for the MLI (see Figure 2), including those in the categories of Life and Career Skills; Learning and Innovation Skills; and, Information, Media, and Technology. Items were developed for each construct, utilizing existing definitions described by the Partnership for 21st Century Skills (2009) and in other related research (e.g., Anderson-Butcher, Ball, & Drew, 2009; Anderson-Butcher, Boester, & Iachini, 2009; Anderson-Butcher, Iachini, Wade-Mdivanian, & Gezinski, 2008). In sum, 201 items comprising the following scales and subscales were included in this collection of tools:

Scales and Subscales

• Life and Career Skills o Leadership o Accountability o Time management o Adaptability/Flexibility o Initiative & Self-Direction (Goal Setting & Effort)

6o Social Responsibility

o Social Skills (Self control, Empathy, Communication) o Cross-Cultural Skills

• Learning and Innovation Skills o Teamwork o Creativity & Innovation o Conflict Mediation & Problem Solving o Critical Thinking

• Information, Media, & Technology o Media and Information Literacy o Technology Use Competence o Intentions for Future Technology Use

• Learning Transfer • Quality of Experience • Previous History/Experience of Technology Use

These items were drawn upon for the development of both a pre-post assessment and a post-reflection assessment. The post-reflection assessment included items that asked students to reflect on the changes in their learning as a result of the specific programming; while the pre-post assessment measured changes over time. The items were further revised after consultation with the Pearson Foundation and Nokia staff. The resulting assessments are outlined below.

Student Pre-Post Assessment

Using the item pool described above, the pre-post assessment included 43 items comprising the following scales:

• Information, Media, and Technology • Social Responsibility and Group Work • Competence to use Technology

Additional items were presented for potential use in the pre-post assessment. These included items related to leadership, adaptability, time management, personal accountability, initiative, social skills, cross-cultural skills, and 21st Century skills in the classroom. Following additional consultation, these scales were further revised, resulting in a 12-item pilot pre-post assessment in Summer 2009. A copy of the items for the pre-post assessment is available in the Appendix. Due to limitations in capacity, it was determined that preliminary evaluation of the MLI would be limited to post-reflection assessment for this project year. Therefore, there are no available data for the pre-post assessment at this time.

Student Post-Reflection Assessment

Building on the items developed earlier in the project year, the OSU team in consultation with the Pearson Foundation and Nokia staff further developed items for a post-reflection assessment. This measurement tool was specifically designed to assess students’ perceptions of

7

changes in their learning after completing the MLI. Similar to the pre-post assessment, this post-reflection assessment measured key constructs related to MLI programming. Scales were developed to measure specific skill areas highlighted by the Partnership for 21st Century Skills (2009). In addition, other youth development evaluation tools (e.g., Anderson-Butcher, Boester, & Iachini, 2009; Anderson-Butcher, Iachini, Wade-Mdivanian, & Gezinski, 2008) were consulted to further refine the subscales for the post-reflection assessment. In total, twenty-nine items asked participants to reflect on their experience and learning within the MLI. For instance, one item measuring quality of experience stated, “As a result of MLI, I feel more confident presenting information to other people.”

Item analysis results for this post-reflection assessment indicated that this tool contained several strong items, as identified through various statistical analyses (DeVellis, 2003). This assessment was further revised to include additional items related to overall youth development and technology. For instance, items were added to assess participants’ views of mobile technology in their lives (“The more I know about mobile technology, I will be better able to manage multiple tasks at one time”). Scales were revised to better assess participants’ perceived change in specific skills (e.g., “Because of my participation in the MLI, I have improved my ability to share information and ideas with others using mobile devices”). Items measuring changes in self-efficacy also were added to provide information on changes in participants’ confidence related to various 21st Century skills such as working on a team or summarizing information found on the internet. A final item also was added to assess participants’ overall perception of the MLI program. Participants were asked to respond on a 7-point Likert scale to the following question: “Overall, my experience during the MLI program was…” (1=Extremely Poor; 7=Extremely Good). The second iteration of the post-reflection assessment had 39 items and included items measuring skill development, learning transfer, changes in self-efficacy, and intentions to use technology in the future.

To shorten the survey and to further refine the items for evaluation, items were removed that were not conceptually congruent with the operational definitions of the aforementioned constructs and MLI program outcomes. Additionally, four items were added in the beginning of the questionnaire that focused on students’ perceptions of their projects/presentations (included in the construct titled “Core Subjects and Relationship to 21st Century Skills”). Participants responded to these items on a scale ranging from Not at all True (1) to Really True (5). These items included:

8

1. During the MLI Experience, you worked to develop a presentation/project that was focused on a central topic such as parks and recreation, forestry, sports, etc. To what extent did working on your presentation/project help you learn more about the topic of your presentation/project?

2. Being able to use mobile and digital technology helped me to be more involved in my project/presentation.

3. Being able to use mobile and digital technology helped me to learn more about the topic.

4. Being able to use mobile and digital technology would help other young people to be more interested in completing a project/presentation like mine.

Based on feedback from Pearson Foundation staff, the item stems also were changed to better reflect the participants’ work on their projects/presentations. For instance, item stems on the previous survey asked participants to consider the items while reflecting on their “participation in the MLI.” The revised stem states, “Because of my work on this project/presentation…” Items measuring intentions also were revised to assess participants’ intentions, rather than their future access to technology.

In an effort to allow for more exploratory, qualitative analysis, a final open-ended item was added that asked participants to indicate one change they would make to the MLI residency experience. Upon further item analysis and discussion, items were revised or dropped to strengthen the assessment and provide improved reliability and validity. In the end, a revised 24-item post-reflection assessment was piloted in the autumn of 2009 and included the following subscales:

• Core Subjects and Relationship to 21st Century Skills • Learning and Innovation Skills • Information, Media, and Technology Skills • Intentions to Use Technology in the Future • Overall Reactions in Relation to Youth Development

This final assessment was piloted with 1,049 student participants at 13 MLI sites during the Autumn of 2009. Fifty-three percent of participants were male, and 47.1% were female. Thirty percent of participants reported their race/ethnicity as Hispanic, 21.1% reported “Other,” 20.6% reported Asian/Pacific Islander, 14.8% reported African-American, 9% reported Caucasian/White, 3% reported multi-racial, and 1% reported Native American. These results of this data collection process are summarized here. The psychometric properties of the scale held up well, and items were able to pick up perceptual differences among students, as indicated by their standard deviations.

Additionally, the items were conceptually aligned with the scales in the assessment (e.g., Learning and Innovation Skills) and with the MLI program objectives and activities. For instance, five items comprised the subscale measuring Learning and Innovation Skills and each

9

item is aligned with an outcome in the student MLI logic model. Likewise, four items measured Core Subjects and the Relationship to 21st Century Skills; four items measured Information, Media, and Technology Skills; three items measured Technology Intentions and Transfer; and seven items measured Overall Reactions in Relation to Youth Development. Finally, all scales demonstrated adequate internal reliability, as measured by Cronbach’s alpha.

MLI Student Post-Reflection Assessment Unless otherwise noted, all items had a 5-point Likert-type response scale.

1 = Not at all true; 2 = A little true; 3 = Somewhat true; 4 = Pretty true; 5 = Really true Item M SD

Core Subjects & Relationship to 21st Century Skills (α = .82, items 2-4)

1. To what extent did working on your presentation/project help you learn more about the topic of your presentation/project? (1=I didn’t learn anything about the topic; 7=I learned a lot about the topic)

5.06 1.63

2. Being able to use mobile and digital technology helped me to be more involved in my project/presentation.

3.69 1.14

3. Being able to use mobile and digital technology helped me to learn more about the main topic I studied.

3.43 1.19


3.85 1.18

Learning & Innovation Skills (α = .87)

Because of my work on this project/presentation, I have improved my ability to… 5. Express my creativity. 3.55 1.19 6. Work on a team. 3.66 1.25 7. Communicate with others. 3.75 1.20 8. Use digital technology to share information with others. 3.70 1.19 9. Work with others to solve problems when working on a

project/presentation. 3.64 1.24

Information, Media, & Technology Skills (α = .88)

Because of my work on this project/presentation… 10. I have improved my ability to use digital technology. 3.64 1.22 11. I better understand how digital media are created. 3.62 1.18 12. I am better able to teach others how to use mobile devices. 3.38 1.25 13. I understand better how digital technology can be use to find information. 3.70 1.16 14. I am more confident in my ability to use mobile technology to find

information. 3.74 1.22

Intentions to Use Technology in the Future (α = .81)

Because of my experience working on my project/presentation… 15. I would like to use a mobile device for my learning in the future. 3.95 1.21 16. I would like to use a mobile device in my regular classroom work. 3.81 1.26 17. I would like to use a mobile device in my personal life. 3.92 1.21

Overall Reactions in Relation to Youth Development (α = .91, items 18-22)

18. The experience of working on my project/presentation was a good way for me to learn.

3.75 1.16

10

19. My work on this project/presentation has increased my creativity skills. 3.59 1.18 20. My work on this project/presentation has increased my confidence in my

ability to accomplish a goal. 3.56 1.21

21. My experience using mobile technology has made me more interested in learning.

3.63 1.24

22. My experience using mobile technology has increased my desire to be successful in school.

3.64 1.24

23. Overall, my experience during the mobile learning institute program was… (1=extremely poor; 7=extremely good)

5.37 1.36

24. Because of my experience working on my project/presentation, I am more likely to create a digital presentation in the future.

3.68 1.01

Initial results using this post-reflection assessment indicate that students’ perceptions of the MLI program were largely positive. A full synthesis of results is summarized in the compendium report developed by the Pearson Foundation staff (Pearson Foundation, 2010). In summary, responses to all items exceeded a benchmark set at 50%. Key findings include:

• 62.5% of students reported that they improved their abilities to communicate with others.

• 60.3% of students reported improved ability to work on a team. • 61.4% of students reported that they have improved their ability to use mobile and

digital technology to share information with others. • Over half of students felt the MLI experience taught them how to express

creativity.

These strengths are encouraging as MLI continues to evolve and support the integration of technology and education. The results of this assessment, however, also indicated variance in scores across MLI sites. While participants at some sites reported strong positive perceptions of their program experience, other participants at different sites reported less positive experiences. For instance, the site mean scores for the Learning and Innovation Skills scale ranged from 3.08 for one site to 4.36 for another site. Likewise, the site mean scores for the Information, Media, and Technology skills ranged from 3.02 to 4.37. The following figure offers some examples of this variance for a selection of MLI sites. Please note, site names have been de-identified for the purposes of this figure. Site-specific information is available by contacting the Pearson Foundation. Clearly these scores point to significant differences in students’ perceptions of learning across program sites. In turn, one potential area for future improvement could focus on creating a standard experience for all program participants across all program sites, thus ensuring all MLI programs are grounded in a common program design.

11

Figure 3. MLI Site Comparisons

To what extent did working on your presentation/project help you learn more about the main topic you were studying.

MLI Site M (SD) Site A (N = 144) 4.92 (1.57) Site B (N = 97) 5.71 (1.19) Site C (N = 92) 5.30 (1.37) Site D (N = 165) 4.58 (1.74)

Because of my work on this project/presentation, I am better able to teach others how to use mobile devices.


My work on this project/presentation has increased my creativity skills.


MLI Teacher Assessment

The MLI Teacher Assessment measured the perceptions of classroom teachers participating in the MLI Teacher Residency program. The development process for the teacher assessment was similar to that for the student assessment. A pool of 91 items for a pre-post assessment and post-reflection assessment was generated reflecting key 21st Century Skills identified by the Partnership for 21st Century Skills (2009). These items were consistent with those utilized in the development of the Student Post-Reflection assessment. That is, the knowledge and skills evaluated in the student assessment were aligned with the teacher assessment; however, the teacher assessment measured teachers’ perceptions of their abilities to teach the skills assessed in the student assessment (e.g., teach students to use mobile and digital technology). The following scales were included in the initial drafts of the teacher pre-post and post-reflection assessments:

Pre-Post Assessment Scales (14 items)

• Perceptions of Technology • Perceptions of Technology Skills • Use of Technology in the Classroom • Perceptions of Technology & Teaching • Perceptions of Facilitators and Constraints

12

Post-Reflection Assessment Scales (39 items)

• Reflection on Life and Career Skills • Reflection on Learning and Innovation Skills • Reflection on Information, Media, & Technology • Learning Transfer • Intentions of Future Technology Use • Quality of Experience • Previous History/Experience of Technology Use

After consultation with the Pearson Foundation and Nokia staff, it was determined that the focus would be limited to only the post-reflection assessment as an initial starting place for development. In response, a 19-item post-reflection assessment was created that included a combination of pre-post and post-reflection items.

This version of the assessment was piloted with 13 teachers at two sites in July 2009. Using these results, the Pearson Foundation and Nokia staff determined that there was a need to further refine the instrument so that it more closely reflected the activities in the MLI. Additionally, pre-post items were removed to focus only on measuring post-reflection and a subscale was added to measure teachers’ intentions to use technology in the future. Two final items assessed teachers’ current use of technology in their classrooms. The revised 21-item post-reflection assessment measured teachers’ perceptions of the following areas:

• Information, Media, & Technology • Availability of Technology • Technology Intentions • General Residency Reactions • Current Use of Technology in the Classroom

MLI staff surveyed 241 teacher participants from 21 different MLI residencies using this post-reflection questionnaire. Nearly 70% of participants were female, and 31% were male. The majority of the sample reported White, Non-Hispanic (63.10%) as their race/ethnicity, followed by Hispanic (16.20%), Asian/Pacific Islander (10.80%), and African American (5.9%). Over half of the participants reported less than 10 years of teaching experience, and over three-quarters of the participants reported teaching in general education. Ninety-seven percent reported owning a mobile phone for personal use.

Results from this data collection process are summarized below. Results indicated that this version of the assessment had adequate psychometric properties, and each item had adequate variance as indicated by the standard deviation. Additionally, the items were conceptually aligned with the overall scales of interest (e.g., Learning and Innovation Skills). Again, nearly all of the scales demonstrated adequate internal reliability, as measured by Cronbach’s alpha. One scale, titled Availability of Technology, demonstrated moderate internal reliability (α = 0.60),

13

suggesting that the items within this scale could be further developed or refined to measure this construct more reliably.

MLI Teacher Post-Reflection Assessment Unless otherwise noted, all items had a 5-point Likert-type response scale.

1 = Not at all true; 2 = A little true; 3 = Somewhat true; 4 = Pretty true; 5 = Really true Item M SD

Information, Media, & Technology (α = 0.92)

Because of the MLI Teacher Residency, I have improved my ability to… 1. Use mobile and digital technology in my classroom instruction. 3.87 1.09 2. Use mobile and digital technology to find content for my classroom

instruction. 3.66 1.19

3. Use mobile and digital technology to find research on content and best practices.

3.56 1.23

4. Use mobile and digital technology to monitor student progress. 3.53 1.21 5. Solve most problems that may occur when using mobile and digital

technology. 3.25 1.18

6. Use mobile and digital technology, in general. 3.99 1.08 7. Teach students to use mobile and digital technology. 3.68 1.20

Availability of Technology (α = 0.60)

8. My school building is not equipped with mobile and digital technology. 2.59 1.35 9. My school building lacks technology support staff. 2.59 1.43 10. The utilization of technology in instruction is a priority at my school

building. 3.40 1.16

Technology Intentions (α = 0.96)

Because I used mobile and digital technology in the MLI… 11. I would like to use mobile and digital technology more in my classroom. 4.28 0.92 12. I think that mobile and digital technology is a valuable resource for

educators. 4.42 0.86

13. I would like to use mobile and digital technology for my teaching in the future.

4.42 0.86

14. I am more interested in using mobile and digital technology to teach subject content.

4.30 1.01

15. I would recommend that more teachers integrate this technology into their classrooms.

4.42 0.86

General Residency Reactions (α = 0.94, items 18-20)

16. The MLI Teacher Residency was a good way for me to learn. 4.33 0.87 17. I have learned new skills at the MLI Teacher Residency. 4.34 0.88 18. I would recommend the MLI Teacher Residency to my coworkers. 4.44 0.83 19. I would rate the overall MLI Teacher Residency as…

(1=extremely poor; 7=extremely good) 6.17 0.91

Current Technology Use in the Classroom % “YES” 20. Do you use mobile and/or digital technology in your classroom? 66.4 21. Do you use mobile and/or digital technology to teach subject content? 63.8

14

Similar to the student assessment, the results using this post-reflection assessment indicated that teacher participants’ perceptions of the MLI program were generally positive. The results are summarized in the compendium report developed by the Pearson Foundation staff (Pearson Foundation, 2010). In brief, responses to all items exceeded a benchmark set at 50%. To highlight, the following are key indicators of program success:

• 73.3% of teacher participants reported that they improved their abilities to use mobile and digital technology (73.3%).

• 61.8% of teacher participants felt better prepared to teach students how to use mobile and digital technology.

• 86.4% of teacher participants indicated that they would like to use mobile and digital technology for their teaching in the future.

• 96.3% of teacher participants reported that their experience at the MLI was “good” or better.

These findings suggest that MLI teacher participants perceived that they learned new and useful skills as a result of their participation. The results of this assessment, however, also indicated variance in scores across MLI sites. While participants at some sites reported strong positive perceptions of their program experience, other participants at different sites reported less positive experiences. Figure 4 offers some examples of this variance for a selection of MLI sites. Please note, site names have been de-identified for the purposes of this figure. Additionally, scale scores ranged quite a bit across sites. For instance, scale mean scores ranged from 2.62 to 4.28 across the MLI sites for the Information, Media, and Technology Skills scale. Likewise, the range of the site mean scores was from 3.63 to 4.92 for the Technology Intentions scale. One potential area for future improvement could focus on creating a standard experience for all program participants across all MLI program sites.

15

Figure 4. Teacher MLI: Site Comparisons

Because I used mobile and digital technology in the MLI, I would like to use mobile and digital technology for my teaching in the future.

MLI Site M (SD)Site A (N = 20) 4.20 (0.83)Site B (N = 22) 3.81 (1.12)Site C (N = 21) 4.86 (0.48)Site D (N = 14) 4.43 (1.09)

Because I used mobile and digital technology in the MLI, I am more interested in using mobile and digital technology to teach subject content.


I have learned new skills at the MLI Teacher Residency.


MLI Summit Assessment

The Pearson Foundation staff gained considerable knowledge and capacity throughout the development of the student and teacher assessments. With these new capacities, the Pearson Foundation staff developed their own items to assess the MLI Leadership Summit (the MLI Leadership Summit Assessment is available in the Appendix). In addition, the Pearson Foundation staff collected data independently to pilot the assessment. Initial results are presented below. Clearly, this assessment of the Leadership Summit represents increased capacity at the Foundation.

Practitioners surveyed 61 participants from three leadership summits. The items were able to pick up perceptual differences among participants, as indicated by their standard deviations. Additionally, the items were conceptually aligned with the constructs in the subscales (e.g., Expectations and Preferences) and with the MLI program objectives and activities. For instance, six items comprised the subscale measuring Expectations and Preferences Likewise, five items measured Understanding; six items measured Specific Impressions; six items measured Intentions; and three items measured Overall Impressions. Additional scale construction analysis is underway as this assessment is further refined for use in the future.

16

MLI Leadership Summit Post-Reflection Assessment

Unless otherwise noted, all items had a 5-point Likert-type response scale. 1 = Not at all true; 2 = A little true; 3 = Somewhat true; 4 = Pretty true; 5 = Really true

Item M SD

Expectations and Preferences

1. I attended today's session because I am looking for strategies to develop 21st century skills in support of student achievement.

4.38 0.90

2. I attended today's session because I am looking for strategies to apply the latest mobile technologies in support of student achievement.

4.25 1.07

3. Now that the Summit is over, I think the issues discussed and presented today have the potential to help me better meet my students' needs.

4.18 0.93

4. I would have preferred a session in which my district's leadership teams could also have taken part.

3.44 1.46

5. I would have preferred more time to network with colleagues. 3.39 1.08 6. I would have preferred more presentation of information and less

discussion. 1.90 1.21

Understanding

As a result of today's Mobile Learning Institute Leadership summit, I have a better understanding of…

7. 21st Century skills and their role in student learning. 4.03 1.09 8. The potential role mobile devices and related technologies can play in

student learning. 4.16 0.97

9. The ways in which the Mobile Learning Institute is working to support students and teachers.

4.25 0.83

10. Concrete approaches I can take to bring the latest technologies to my teachers.

3.62 1.11

11. Concrete approaches I can provide to help my teachers make better use of these technologies.

3.60 1.12

Specific Impressions

All items in this subscale had a 5-point Likert-type response scale. 1 = Not good at all; 2 = Not good; 3 = So-So; 4 = Good; 5 = Very good

12. I found the opening introductory remarks… 4.25 0.74 13. I found the Summit segment about 21st century skills… 4.37 0.69 14. I found the Summit segment related to mobile technology… 4.25 0.77 15. I found the Summit segment related to school district transformation… 4.06 0.97 16. I found the Summit segment related to teacher quality… 4.17 0.96 17. I found the day's final conversations… 4.13 0.84

Intentions

As a result of today’s MLI Summit… 18. I would recommend attending a future MLI Summit to my peers. 4.41 0.70 19. I would be interested in attending another MLI Summit on another theme

important to my district. 4.57 0.70

20. I would like to explore a Mobile Learning Institute program for teachers in my district.

4.38 0.93

21. I would like to explore a Mobile Learning Institute program for students in 4.38 0.98 17

my district. 22. I would welcome the chance to invite members from my district leadership

teams to a similar summit intended just for them. 4.46 0.97

23. I better understand how to include project-based programs within my curriculum.

3.89 1.24

Overall Impressions

All items in this subscale had a 5-point Likert-type response scale. 1 = Not good at all; 2 = Not good; 3 = So-So; 4 = Good; 5 = Very good

24. I found the way my time was spent today to be… 4.47 0.75 25. I thought the way in which the Summit met my own needs to be… 4.23 0.87 26. Compared to my expectations, I thought today's Summit was… 4.42 0.72

Initial results using this post-reflection assessment indicate that participants’ perceptions

of the MLI program were largely positive. Key findings are summarized in the compendium report developed by the Pearson Foundation staff (Pearson Foundation, 2010). In summary, responses to all items exceeded a benchmark set at 50%. Participants reported that, upon completing MLI Leadership Summits, they appreciated the overall format and related discussions. In addition, they indicated future intent to support MLI initiatives.

Taking a cue from the results of the first delivery of the MLI Leadership Summit assessment instrument, ongoing changes to the MLI Leadership Summits took place to immediately address the needs of the participants. The MLI Leadership Summits now close with a lengthy period of hands-on instruction and demonstration of concrete uses of mobile technology to improve educational outcomes. Similarly, this period of time allows for more specific discussion and networking opportunities in order to provide a forum for participants to address their unique needs and concerns. Finally, Pearson Foundation staff are currently developing a logic model for the Leadership Summit that will further articulate the activities, goals, and expected outcomes.

MLI Implementation Fidelity

In addition to assessing outcomes, it also is helpful to explore the degree to which a program is implemented with fidelity. These types of assessments provide an understanding of how well a program is being implemented in relation to the original program design. More specifically, assessments of implementation fidelity compare the actual implementation of program components to the espoused implementation. Deviations from the espoused implementation could result in unintended outcomes.

Existing literature (e.g., Dusenbury, Brannigan, Falco, & Hansen, 2003; Fagan, Hanson, Hawkins, & Arthur, 2008; Mowbray, Holter, Teague, & Bybee, 2003) emphasizes five critical elements for measuring implementation fidelity:

1) Dosage refers to the amount of the program delivered, such as number of sessions, time spent on each activity, or weeks of implementation.

18

2) Adherence refers to the extent to which implementation of particular activities and methods are consistent with the way the program design.

3) Quality of program delivery considers the extent to which the provider approaches a theoretical idea in program delivery.

4) Participant responsiveness refers to participants’ engagement in the program or activities.

5) Program differentiation includes the identification of critical, distinguishing features of the program.

The OSU team developed several tools to aid the Pearson Foundation assessment team in the evaluation of implementation fidelity. These tools differ from the student and teacher assessments described previously in that they are more focused on measuring the implementation of programming, rather than participant outcomes. That is, these tools address needs to understand how the programming at MLI sites matches the espoused MLI program design. As such, each tool was developed to assess one or more of the five elements outlined above.

As outlined in Figure 5, the OSU team and Pearson Foundation staff developed an engagement log and site observation checklist. Please see the Appendix for examples of each tool. The MLI Engagement Log, a brief log completed by the session instructor after the completion of the MLI, documents the objectives taught during the program and notes modifications in instruction, and session activities. In addition, instructors are asked to reflect on their experiences and report strengths, weaknesses, and potential improvements for the MLI. This log is used to assess adherence, program differentiation, and participant responsiveness.

The MLI Site Observation Checklist can validate information from the MLI Session Log. It assesses dosage, adherence, quality of program delivery, program differentiation, and participant responsiveness. These checklists are completed by trained observers who visit sessions and note whether or not specific components are present (e.g. students had opportunities to work in teams). It is recommended that observations occur periodically in 10 to 15 percent of all program sessions. The MLI Site Observation Checklist is closely aligned with the previously developed logic model; thus, it assesses the specific activities outlined in the model, such as the use of project-based activities and opportunities to express creativity.

19

Figure 5. Measuring Implementation Fidelity of the MLI

Assessment Tool

Description Purpose Provider Periodicity

MLI Engagement Log

MLI‐specific log that includes expected outcomes and notes modifications, and instructor/student activities. Open‐ended questions ask instructors to report on their experiences in the MLI session as well.

Assess:1) Dosage – length of session, how

many sessions total 2) Adherence – objectives

taught/not taught, modifications

3) Participant Responsiveness (activities)

4) Program Differentiation

Session Instructor

After the last session of the MLI

MLI Site Observation Checklist

Trained observers view sessions and complete assessment checklists

• Validate information from Session Logs

• Assess: 1) Dosage 2) Adherence 3) Quality of Program Delivery

(checklists) 4) Program Differentiation

(checklists) 5) Participant Responsiveness

Program leadership and trained research staff

Periodically, in 10‐15% of all program sessions

In sum, these measures of implementation fidelity can assist in the Pearson Foundation’s internal evaluation of the MLI by highlighting areas of programmatic deviance from the original program design and curriculum. These tools may be useful for ensuring comparable methods of implementation across MLI sites. Additionally, when used in conjunction with the MLI program logic models and outcomes assessments, these tools could illuminate areas for improvement within the MLI sessions. For instance, the engagement log and site observations may indicate that students are not offered adequate opportunity to work in teams. As a result, outcomes assessments may reflect students’ perceptions that their skills in relation to teamwork have not improved.

MLI Observation

In addition to consultation and technical assistance, the OSU team and Pearson Foundation staff conducted a site observation of one student school-based MLI residency. This helped the evaluation team gain a more comprehensive understanding of the MLI overall. A summary of this observation and key areas of strength and weakness is available in the Appendix.

Ultimately, this observation was integral in developing a complete picture of the current MLI programming and a broader perspective of future improvements for MLI programming nationally and internationally. Specifically, insights were gleaned in relation to partnerships among MLI staff and local leadership, techniques for instruction, and explicit program goals. For instance, student learning was supported by the facilitators’ interactions with them, efficient use

20

of available equipment, and the students’ interactions with each other. With the guidance of the facilitators, the students shared computers and mobile devices effectively and appropriately.

Areas for potential improvements were noted as well. Among others, an increased and more explicit emphasis on the program’s broader goals focused on 21st Century skills could prove most valuable for the MLI. Beginning with program development, the MLI has established clear goals and objectives. To this end, making these goals and objectives explicit in every step of program development and delivery will likely improve student and teacher outcomes. This could include more explicit emphasis in MLI staff training, program marketing materials, and MLI program delivery.

Concluding Thoughts and Recommendations

In conclusion, OSU’s partnership with the Pearson Foundation resulted in a more refined MLI program design and evaluation tools. Notably, the MLI student and teacher logic models articulate key program activities and outcomes that can serve to guide future scale-up and assessment efforts. Additionally, the development of procedures and tools for the comprehensive internal evaluation of the MLI yielded three measures of program outcomes and several tools to measure implementation fidelity.

Findings and Progress Indicators

Notably, one key finding in relation to the partnership between OSU and the Pearson Foundation is the considerable capacity developed at the Pearson Foundation in relation to the MLI evaluation infrastructure and procedures. The OSU team initially developed the assessments, collected data, and disseminated results. As the assessments were piloted and revised, Pearson Foundation staff provided expanded contributions in the refinement and purpose of the student and teacher assessments. Additionally, the OSU team initiated efforts to map, articulate, and refine various program components but Pearson Foundation staff ultimately guided these project components as they gained an increased understanding of evaluation. The ultimate capacity built at the Pearson Foundation is evidenced by their growing role in assessment development, data collection, data management, and program design modifications. This was exemplified in their leadership in evaluating the MLI Summits without much support from the OSU team.

Initial results of the MLI evaluation suggest the program is meeting several identified outcomes. The results of these initial assessments are presented in more detail in the Pearson Foundation’s compendium report (Pearson Foundation, 2010). Key findings from this report indicate several strengths of the MLI student and teacher engagements. Specifically, the participants in the student MLI engagements indicated growth in their skills related to the use of mobile and digital technology and positive reactions to the overall MLI experience. Moreover, participants in the teacher MLI engagements reported growth in their skills related to the use of mobile and digital technology to engage students. Teachers also noted support for the use of mobile and digital technology in classroom instruction. The developing logic models will further

21

enhance ongoing program improvements, particularly in light of the site observation’s recommendations for increased emphasis on articulated program goals and objectives throughout the Pearson Foundation’s program management and within the MLI engagements.

While these results are promising, areas for improvement still remain. Data suggest that the MLI experience may be quite dependent upon site-specific characteristics, such as the classroom environment or session-specific thematic program. This suggests inconsistency in programming across MLI sites. In turn, the enhanced program design outlined above may be helpful in establishing a more formal program design that explicitly articulates the program activities and outcomes. This will allow for greater articulation and emphasis on commonly expressed goals through the Pearson Foundation’s program management. Additional or modified training of MLI staff may also be necessary to ensure all program leaders adhere to MLI curriculum activities and goals. In addition, the measures of fidelity created here also assist in ongoing assessment and monitoring of program design. Modifications in the MLI program can be informed by these existing tools, ultimately resulting in more consistent program outcomes.

Future Development

In light of these findings and progress, several suggestions follow that can enhance the MLI engagements and potentially improve program outcomes. Significant progress has been made in the further refinement of the MLI program design. As suggested previously, even closer analysis of the MLI logic models, assessment results, and implementation fidelity can aide in a more standardized delivery across MLI sites and lend greater consistency in outcomes.

One potential starting point for this analysis may involve the development of detailed descriptions of several core components of MLI programming. For instance, the MLI logic models outline broad activities and the MLI student curriculum highlights specific activities for each session. A cross-mapping of these broad program activities and specific session activities can highlight gaps in programming or areas of redundancy. Ultimately, clear descriptions of program activities can include specific core components that all MLI sessions must include to achieve the program’s desired outcomes. Steps in this direction are already underway as the Pearson Foundation further develops MLI curriculum.

In that same vein, the site observation and assessment results suggest that the over-arching MLI goals and objectives may not be explicitly stated in the program curriculum or implementation. A potential next step may be the alignment of curriculum to program logic models and cross-program training for instructors that emphasizes a shared understanding of the program’s vision. These steps may allow for a more standardized approach to programming across the MLI sites. In addition, a closer alignment of curriculum and program design can streamline the path to desired outcomes.

Indeed the assessment of outcomes can be modified and expanded to allow for a richer continuous improvement process aligned with the Kirkpatrick Model of Summative Evaluation. Interim assessments could be used in addition to the post-reflection assessments occurring at the

22

end of each MLI program session. The results of such assessments could be delivered in real time to site instructors, allowing them the opportunity to receive ongoing feedback on participants’ learning. In turn, instructors could modify their lessons to better achieve the program’s intended outcomes. Again, initial steps to develop these assessments are underway.

Additionally, the current assessments could be expanded to include focus groups or pre-post assessments as well. These types of evaluation may focus more on Kirkpatrick’s Reaction, Learning, or Behavior Levels. Focus groups could lend a qualitative assessment of program outcomes. For instance, participants could be asked to share their perspectives on the program, their learning, or other program activities. This type of assessment is often useful in gaining a deep understanding of a specific program component or the experience of a particular type of participant. While pre-post assessments were developed for this project, to date, no data have been collected using these instruments. As such, it is still unclear whether participants’ skills change over time. Pre-post assessments are most appropriate at Kirkpatrick’s Learning level of evaluation. These assessments could identify specific areas of strength in which participants improve a great deal over time. Likewise, areas for improvement could be identified in which participants’ learning remains constant or declines over the course of the program.

In the end, however, the findings of the initial MLI assessments emphasize the potential impact of this program for expanding the purview of technology in educational settings. Several progress indicators and preliminary evidence support that the MLI is developing students’ 21st Century skills and empowering teachers to do the same. In summary, this report highlights the following areas of strength and growth within MLI:

• Student and teacher participants in the MLI report growth in the following areas: life and career skills; information, media, and technology skills; and learning and innovation.

• Outcomes for the MLI are now more clearly articulated and connected to program activities.

• Assessment procedures across the MLI sites are more standardized, leading to an increasingly strategic assessment of program quality.

• The Pearson Foundation now has committed staff who oversee evaluation, and provide formative data back to program sites about progress, successes and areas of improvement.

Clearly, significant progress has been made to improve program quality and support student and teacher learning in the MLI. Future evaluative efforts to expand assessment and clarify program design may lead to even more consistent outcomes. The OSU team looks forward to future partnership with the Pearson Foundation and Nokia as the MLI is further refined and replicated in other communities in the future.

23

References

Anderson-Butcher, D. & Wade-Mdivanian, R.L. (2009) Comprehensive System of Learning Supports in Ohio: 2008-2009 Final Report to the Ohio Department of Education. Columbus, OH: The Ohio State University, College of Social Work.

Anderson-Butcher, D. (2009). Middle School Reform Needs Assessment Pilot Project. Ohio State University: College of Social Work.

Anderson-Butcher, D., & Cash, S. (2005). Cap City Kids Evaluation. Columbus, OH: The Ohio State University, College of Social Work.

Anderson-Butcher, D., Ball, A., Drew, H. (2009). Technology Magnet Survey. Columbus, OH: The Ohio State University, College of Social Work.

Anderson-Butcher, D., Boester, A., & Iachini, A. (2009). LiFE Sports Evaluation Survey. Columbus, OH: The Ohio State University, College of Social Work.

Anderson-Butcher, D., Iachini, A., Wade-Mdivanian, R.L., & Gezinski, L. (2008). Ohio Quality Assessment Rubric (O-QAR).Columbus, OH: The Ohio State University, College of Social Work.

DeVellis, R.F. (2003). Scale development: Theory and applications. Thousand Oaks, CA: Sage.

Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W.B. (2003). A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Education Research, 18(2), 237-256.

Fagan, A.A., Hanson, K., Hawkins, J.D., & Arthur, M.W. (2008). Bridging science to practice: Achieving prevention program implementation fidelity in the Community Youth Development Study. American Journal of Community Psychology, 41, 235-249.

Mowbray, C.T., Holter, M.C., Teague, G.B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24(3), 315-340.

Tabachnick, B.G. & Fidell, L.S. (2007). Using multivariate statistics. Boston, MA: Pearson.

Thomas, J.R. & Nelson, J.K. (1990). Research methods in physical activity. Champaign, IL: Human Kinetics Books.

Partnership for 21st Century Skills. (2009). Framework for 21st Century Learning. Retrieved from http://www.p21.org.

Pearson Foundation. (2010). MLI Assessment Summary. CA.

Weiss, C.H. 1972. Evaluation Research: Methods of Assessing Program Effectiveness. Englewood Cliffs, NJ: Prentice Hall.

24

http://www.p21.org/

Glossary of Terms adherence The extent to which implementation of particular activities and methods are consistent with the program design.

construct An abstract idea.

Cronbach’s alpha (α) A widely used measure of reliability. In general, alphas should be above 0.70; sometimes, an alpha between .60-.69 is acceptable as well (Tabachnick & Fidell, 2007).

dosage The amount a program delivered, such as number of sessions, time spent on each activity, or weeks of implementation.

fidelity The extent to which a program is administered in approximation to an espoused design.

focus group A planned discussion among a group of participants designed to elicit perceptions on a defined area of interest.

item analysis The process of evaluating individual measurement items to identify appropriate and inappropriate items for a scale.

item pool A collection of measurement items developed to comprise one scale. These items are reduced or refined upon further statistical analysis.

item stem An introductory component of a measurement item that to focus the respondent’s attention on a specific aspect of the question or situation being measured.

Kirkpatrick Model for Summative Evaluation A four-level model of training evaluation developed by Donald Kirkpatrick to guide an iterative evaluation process. Each level is predicated on completion of the previous level. The four levels are as follows:

Level I. Reaction measures whether the environment was suitable for learning. Level II. Knowledge measures whether the participants learned the content. Level III. Behavior measures whether the participants transfer the skills to other settings. Level IV. Results measures whether the training led to the desired outcomes.

For information on the complete model, see Kirkpatrick, D.L. & Kirkpatrick, J.D. (2006). Evaluating Training Programs: The Four Levels (3rd Ed.). New York, NY: Berrett-Kohler.

Likert scale A set of response options for a declarative measurement item. Each response option indicates varying degrees of agreement with or endorsement of the item (DeVellis, 2003). An example of a Likert scale is: Strongly Disagree – Disagree – Agree – Strongly Agree.

logic model A graphical representation of program activities and outcomes.

mean A measure of the statistical “center” of a distribution of scores. The mean (average) is calculated by summing the scores and dividing the total by the number of items in the scale. 25

operational definition A description of an observable phenomenon that allows for empirical testing.

participant responsiveness Individuals’ engagement in the program or activities.

post-reflection measurement A measurement technique in which data are collected after the completion of a program. Participants are asked to reflect on their experience in the program and respond to measurement items accordingly.

pre-post measurement A measurement technique in which data are collected at one point in time and then collected again at another point in time with the same participants.

program differentiation The identification of critical, distinguishing features of the program.

psychometric properties Characteristics of measurement scales based on commonly established practices of psychometrics. Examples of psychometric properties include measures of reliability and validity.

psychometrics Field of social science study concerned with measuring and interpreting social and psychological phenomena. See psychometric properties, reliability, and validity.

quality of program delivery The extent to which the provider approaches a theoretical idea in program delivery.

reliability The consistency of a measurement instrument (i.e. Does the measure consistently demonstrate the same scores when measured at different times in different ways?). See Cronbach’s alpha.

scale A collection of measurement items designed to reveal a theoretical construct not observable by direct means.

scale development The process of creating and validating a measurement instrument. See reliability and validity.

standard deviation A measure of variability in scores that is in the metric of the original scores. See variance.

subscale A component scale of a larger measurement instrument.

validity The degree to which a measurement instrument measures the construct it purports to measure (Thomas & Nelson, 1990).

variance The distribution of scores. See standard deviation.

26

27

APPENDIX

06‐23‐09 Mapping MLI Program Areas to Expected Outcomes

1

Mobile Learning

Institute (MLI) Program Areas

Descriptions

Implementation Sites

Tiersi

Introduction Adoption Innovation

MLI Teacher Introductory Professional Development: Tier 1

EXAMPLE CONTENT: PowerPoint projects; Digital storytelling; Community Geo‐mapping; Blogging; Podcasting; Polling/Survey projects. TARGET GROUP: Teachers from select education communities where need and partnerships have been identified, and in certain instances where Nokia and Pearson have relationships and/or offices. Some school systems will recruit teacher volunteers, whereas others select certain teachers. TRAINING DESIGN: Training last 2‐3 days; Typically conducted in a local technology center or innovation center. Introductory MLI in First Year provides an introduction to a broad spectrum of technologies adaptable for use in education settings. Specific technologies include: document sharing/collaboration software (Google Docs for peer support and networking); idea generation/brainstorming software (Inspiration); digital storytelling; geo‐mapping; blogging; and podcasting. Sessions will choose one or more larger project based learning theme for a future in class student residency preparation. Ultimately teachers will later implement strategies learned. MLI STAFF: Facilitate the training. The hope is that these will result ultimately in classroom‐based residencies during the school year either conducted with the full participation of Pearson/MLI staff with interested teacher, or, for those teachers eager to adopt autonomously the program, with limited help in hardware/software loan and general support. Next steps depend upon how involved Pearson staff can be in the future. OUTCOMES:

• Student: Theoretically there will be indirect benefits to students as teachers transfer their own learning about technology into their classroom instruction. In those instances when a student residency is the projected outcome, direct student benefits result in acquiring 21st Century skills focused on technology enhanced mobile learning.

• Teacher: Improved skills in certain technologies; Enhanced self‐efficacy in relation to use in classrooms; Improved 21st Century Skill pedagogical strategies; Increased use of technology in classrooms.

• System: Increased readiness to adopt technology and 21st Century Skill development priorities, and a direct result through implementation of curriculums resulting from the professional development.

Chicago

Las Vegas

San Francisco

New York

Chicago

(YouMedia/MacArthur fund)


2

Mobile Learning


Descriptions

Implementation SitesTiers


MLI Youth Summer Residency

EXAMPLE CONTENT: Summer camp focused on digital story telling, with ongoing adoption of related technology features, such as geo‐mapping, polling, blogging, etc.; Typically focuses in on a theme area such as: food is power [sustainable agricultural and consumption], national parks (environmental preservation), earth day challenge, self‐empowerment [Envision 20/20]). TARGET GROUP: Youth in middle school or early high school from select school systems or community where Nokia and Pearson have relationships and/or offices; increasingly targeted at participants from community‐based organizations (CBOs); Some school systems or CBOs will have staff participate with the youth, but not in all cases. TRAINING DESIGN: Typically last 5 or 10 days

• 5 day model: one student group of 25‐30; roughly six hours of contact time each day • 10 day model: two student groups of 25‐30 students each; three hours in a.m. for one of

the groups, three in afternoon for the second group; for field trip experiences the two groups are combined.

Youth participants are formed into filmmaking teams where they explore basic information surrounding the theme; while learning how to put together digital artifact (i.e. video, blogging/ podcasting, geo‐mapping project); Video depends on themes, however almost all of them have digital story telling (film or video or simple PowerPoint, moving video images, voice narration) as a key component; In the end, artifacts (i.e., films, podcasts) are open for others to use and teach from by going to Google doc to share here what is happening in other settings. The degree to which 21st Century Skills are targeted and developed is uncertain (perhaps something to add). MLI STAFF: MLI staff conduct provide knowledge portion through theme focus of digital story telling; Perhaps these summer residencies also set the stage for the extension of their work into the school year OUTCOMES:

• Student: Increased technology skills, Increased use of technology in school work and life; Improved 21st Century Skills

• Teacher: Indirectly through youth being “encouraged to teach teachers and other youth.” Perhaps eventually MLI might engage teachers from the system in the camp to add power

Chicago Las Vegas


3

to the transfer. Mobile Learning


Descriptions



MLI In‐School Residency

EXAMPLE CONTENT: Digital story telling with a critical theme (Example: English class may focus on Greek myths) TARGET GROUP: Both students and teachers. TRAINING DESIGN: Teachers come to 2‐3 day PD session where work in teams, develop shared documents, learn from each other as develop and share curriculum (2 days in one situation; 3 days in the others) where introduced to project and parameters, the editing system, and asset gathering process. Information provided by school system and teachers are either selected or voluntary participation. A sample document/video is created that can be used as an exemplar for their classes. The degree to which 21st Century Skills are targeted and developed is uncertain (perhaps something to add). There then is the in‐class component involving the teachers use of technology and 21st Century pedagogy strategies in the teaching of a core subject area; These may last for 3‐4 days to 2 weeks; teachers create the framework and content for the in‐class MLI; they gather digital story telling assets (still images of story) and content ahead of time with assistance from MLI staff; teachers then implement the curriculum. Perhaps strategies targeting long‐term outcomes such as improved youth engagement in classes and teacher leadership in supporting other teachers’ adoption and implementation efforts should be added. MLI STAFF: MLI staff complete the original 2‐3 day professional development session; In‐class involvement of staff then depends on level of competency of the teachers. For instance, in some cases there is minimal direct engagement of MLI staff in the “in‐class” portion; whereas in others there is more direct instruction. Some of this depends on MLI staff interests/styles; but it also involves determinations about the level of competency of the teachers. OUTCOMES:

• Student: Increased technology skills, Increased use of technology in school work and life; Improved 21st Century Skills; Mastery of content area being covered; Final product of a digital story telling outcome artifact. Perhaps long‐term outcomes such as improved youth engagement in classes is important.

San Francisco

New York


4

• Teacher: Improved skills in certain technologies (learn basic technical skills to use editing software, to create video presentations, as well as podcasting/blogging and spatial literature technology); Increased use of technology in classrooms; Enhanced self‐efficacy in relation to use of technology in classrooms; Improved 21st Century Skill pedagogical strategies (i.e., project‐based learning in instruction; emphasis on learning/innovation and life/career skills); Improved peer support and networking related to technology and 21st Century Skill application among teachers; Perhaps long‐term outcomes teacher leadership in supporting other teachers’ adoption and implementation efforts is important.

• System: Potential of early adopters in changing the system and helping others adopt

Mobile Learning


Descriptions



MLI Centers of Innovation

EXAMPLE CONTENT: School site‐based “center” fully housed with the latest technologies; Center serves as an incubator for advancing the value and use of mobile technology in education across the system; Intentions are to serve the school (and its teachers/students), and ultimately the district and community as whole. TRAINING DESIGN: There are ongoing workshops, trainings, competitions, and experiences offered; the Center is available for teachers as well as other providers (CBOs) to use for certain projects; Teachers and others may use the space independently, but also with guided support and technical assistance from MLI staff; MLI staff also support the infusion of technology in classrooms, and will also build capacities of teachers across the system. The degree to which 21st Century Skills are targeted and developed is uncertain (perhaps something to add). MLI STAFF: MLI staff coordinator is hired within the Center to support daily activities; This person also supports teachers in their classrooms with infusing technology in instruction; The level of involvement of MLI staff somewhat also depends on approach of MLI staff or capacities of students, so they provide varying levels of direct instruction to students. OUTCOMES:

• Student: Increased technology skills, Increased use of technology in school work and life; Improved 21st Century Skills; Mastery of content area being covered; Final product of a digital story telling outcome artifact; Improved engagement in school. Perhaps long‐term outcomes such as improved youth engagement in classes is important.

• Teacher: Improved skills in certain technologies (learn basic technical skills to use editing

San Francisco New York City


5

software, to create video presentations, as well as podcasting/blogging and spatial literature technology); Increased use of technology in classrooms; Enhanced self‐efficacy in relation to use of technology in classrooms; Improved 21st Century Skill pedagogical strategies (i.e., project‐based learning in instruction; emphasis on learning/innovation and life/career skills); Improved peer support and networking related to technology (Google docs) and 21st Century Skill application among teachers. Perhaps long‐term outcomes such as teacher leadership in supporting other teachers’ adoption and implementation efforts is important.

• System: Increased adoption and implementation of technology and 21st Century Skill pedagogy across the system.

i i Innovation Sites: Places where there is a long‐term relationship among Pearson/Nokia/School District; Various MLI strategies are being implemented system‐wide (teachers, students, district, and community are served); primarily involves the infusion of an Innovation Center and a full‐time MLI staffperson to facilitate local efforts; (2)Adoption: Places where there is a long‐term relationship among Pearson/Nokia/School District; Teachers and students who have been involved previously are participating in MLI continuing education experiences; New teachers and students are participating in Introductory stages; Previously MLI where we don't have innovation centers; where we are doing second PD of designing successful 21st skill with teacher PD and then do classroom residencies; (3) Introduction: Places where there is a developing relationship among Pearson/Nokia/School District; There is little demand from the school system in relation to the MLI; Initial relationships/contacts are being built; Perhaps few teacher and/or summer residencies have occurred.

Program Outcomes Map1

Mapping Programs to Outcomes PROGRAM OUTCOMES DOMAINS

Core Subjects and 21st Century Themes

21st Century Interdisciplinary Themes

Learning and Innovation Skills

Information, Media, and Technology

Life & Career Skills

Initiative Intentions Facilitators & Constraints

Youth Development

Quality Practices

MLI PD in Summer Teacher

Student Content Area‐Connection with Class Subjects (i.e. Earth Day) Think of Technology Standards Stick to STEM priority Content standards by grade level

Global awareness Financial, economic, business, entrepreneurial literacy Civic literacy Health literacy

Creativity & innovation Critical thinking & problem solving Communication & collaboration Work with others Solve problems Work in teams to solve problems Know how to resolve differences Mediate conflicts

Information literacy Media literacy Research information and communication technologies literacy Know how to use technology and learn content and skills Know how to use technology and to think critically, solve problems, use information, communicate Improve technology skills in: blogging, podcasting, editing equipment, mobile device, geo mapping, PowerPoint, use an online network where I can share work and learn from others.

Flexibility & adaptability Initiative & self direction Social and cross cultural skills Productivity & accountability (time management) Leadership, ethics, people skills, personal and social responsibility

Able to start complete and demonstrate results of project Solve problems with minimal supervision, direction

To use technology Apply to other settings Transfer of learning What learned help with future schoolwork What I learned can help me achieve academic goals Can do a project on my own Can use the

Staff care about me Good leadership Provide help Give me opportunities to make decisions/have a say Belonging and connections with others Empathy for students Connect and build relationships with peers Opportunities to work in teams


PROGRAM OUTCOMES DOMAINS Core Subjects and 21st Century

Themes 21st Century Interdisciplinary

Themes Learning and Innovation

Skills Information, Media,

and Technology Life & Career

Skills Initiative Intentions Facilitators &

Constraints Youth

Development Quality

Practices

skills in a future career Nokia equipment and Pearson learning materials

Teachers Improve pedagogy Improve content knowledge of area as a result of technology Evidence of student learning due to technology Increase knowledge, skills, values

Shared networking, peer support, collaboration software used Teach others in school Share resources with others Ability to do project based learning instruction Ability to get students to work in teams

Self‐Efficacy Use of mobile technology in classroom instruction Use of technology to get content for instruction Use of technology to get research on content and best practices

Improve youth engagement in class

Motivation to learn

Value Ease of use and application Technology works/ breaks down Know of how to use technology

Ask teachers how facilitators did

Systems Current use of these in classroom Dollars allocated to technology

Current use of technology in classrooms Technology infrastructure in district Build capacities of staff working in districts to help others (technology staff)

Current infusion of life career skills in teaching

Limited support personnel to assist staff Reinforcement Student interest and motivation








Constraints Youth

Development Quality

Practices

Increased readiness and adoption of innovation

Curriculum alignment barriers Standards based Accountability Value

MLI PD Summer Residency

Students Teachers Systems MLI: In School Residency

Students Teachers Systems








Constraints Youth

Development Quality

Practices

Innovation Centers Students Teachers Systems

Pre‐Post Student Measurement Tool: Key Constructs for Mobile Learning Institutes

Information, Media, & Technology

1. I am able to use the internet to find information

2. I am able to determine whether or not information on the internet is true

3. I can summarize information I find on the internet

4. I can communicate information I find on the internet to other people

5. I can use technology to present information to other people

6. I am confident in my ability to use the internet to find information

7. I am confident in my ability to determine whether or not information on the internet is true

8. I am confident in my ability to summarize information I find on the internet

9. I am confident in my ability to communicate information I find on the internet to other people

10. I am confident in my ability to use technology to present information to other people.

Competence to Use Technology

How good do you think you are at…

Not good at all Not

good So‐So Good Very Good

Using Google docs? O O O O O

Digital storytelling? O O O O O

Geo mapping? O O O O O

Blogging? O O O O O

Podcasting? O O O O O

Searching the internet? O O O O O

Demographics

1. What is your gender? 2. What is your age? 3. What race or ethnicity do you consider yourself? 4. On average, what grades do you get in school? 5. How honest were you in filling out this survey

Social Responsibility & Group Work

1. I help others when working on a project

2. I am able to change my school or community for the better

3. I stand up for what I believe in

4. I interrupt others when working on a project

5. I respect others when working on a project

6. I control my temper when working with others on a project

7. I take responsibility for my actions when working with others on a project

8. I am concerned with other people’s feelings when working on a project

9. I listen to others when working on project

These items are adapted from a pilot project to develop a middle school reform focused on Science, Engineering, Math, and Technology in an Ohio middle school. Suggested citation: Anderson‐Butcher, D. (2009). Middle School Reform Needs Assessment Pilot Project. Ohio State University: College of Social Work.

Pre‐Post Student Measurement Tool: 21st Century Skills

These items are adapted from a pilot project to develop a middle school reform focused on Science, Engineering, Math, and Technology in an Ohio middle school. Suggested citation: Anderson‐Butcher, D. (2009). Middle School Reform Needs Assessment Pilot Project. Ohio State University: College of Social Work.

Leadership

1. I am confident in my ability to lead others to accomplish a goal. 2. I am confident in my ability to assign tasks to others.

Adaptability 1. I can cope with situations that don’t go as planned. 2. I am open to change.

Personal Accountability 1. I follow through with my commitments. 2. Team members can count on me.

Time Management: 1. I organize my time. 2. When I have multiple tasks, I plan how to accomplish the tasks ahead of time.

Initiative 1. I start tasks without being asked. 2. I put all my energy into accomplishing my goals.

Social Skills 1. I am confident in my ability to listen to others. 2. I think about how others see things. 3. I am confident in my ability to ask questions.

Cross‐Cultural Skills 1. I am confident in my ability to learn about different cultures. 2. I enjoy learning about different cultures. 3. I take time to learn about different cultures.

21st Century Skills in the Classroom 1. My teachers use technology in our classrooms. 2. I use computers in my classes.

v.1

Mobile Learning Institute: Student Post-Reflection Survey

This survey is a part of the Mobile Learning Institute (MLI) residency. It will help us by identifying the kinds of skills you learned from participating in the MLI program, how you might use the skills you learned in the future, and your overall perception of the experience. Try to fill in every question and be as honest as possible. The questions we ask about your personal information will only be used to describe the kinds of young people who fill out the survey. The information will not be used to identify you.

What is the name of the MLI site you attend? ________________________________________________________

Core Subject & Relationship to 21st Century Skills. Please choose the one answer that best reflects how true the statement is for you.

1. During your MLI experience, you worked to develop a presentation/project that was focused on a central topic

relating to a subject you were studying such as writing, literature, history, science, math, sports, etc. To what extent did working on your presentation/project help you learn more about the main topic of you were studying? Select from 1 to 7, with 1 meaning “I didn’t learn anything about the topic,” to 7 meaning, “I learned a lot about the topic.”

Not

at a

ll tr

ue

A li

ttle

true

Som

ewha

t tr

ue

Pret

ty tr

ue

Rea

lly tr

ue

2. Being able to use mobile and digital technology helped me to be more involved in my project/presentation. O O O O O

3. Being able to use mobile and digital technology helped me to learn more about the main topic I studied. O O O O O


O O O O O

v.1

Learning and innovation Skills. Please choose the one answer that best reflects how true the statement is for you.

Because of my work on this project/presentation, I have improved my ability to…

Not

at a

ll tr

ue

A li

ttle

true

Som

ewha

t tr

ue

Pret

ty tr

ue

Rea

lly tr

ue

5. Express my creativity. O O O O O

6. Work on a team. O O O O O

7. Communicate with others. O O O O O

8. Use digital technology to share information with others. O O O O O

9. Work with others to solve problems when working on a project/presentation. O O O O O

Information, Media, and Technology Skills. Please choose the one answer that best reflects how true the statement is for you.

Because of my work on this project/presentation…

Not

at a

ll tr

ue

A li

ttle

true

Som

ewha

t tr

ue

Pret

ty tr

ue

Rea

lly tr

ue

10. I have improved my ability to use digital technology. O O O O O

11. I better understand how digital media are created. O O O O O

12. I am better able to teach others how to use mobile devices. O O O O O

13. I understand better how digital technology can be use to find information. O O O O O

14. I am more confident in my ability to use mobile technology to find information. O O O O O

v.1

Intentions & Transfer. Please choose the one answer that best reflects how true the statement is for you.

Because of my experience working on my project/presentation…

Not

at a

ll tr

ue

A li

ttle

true

Som

ewha

t tr

ue

Pret

ty tr

ue

Rea

lly tr

ue

15. I would like to use a mobile device for my learning in the future. O O O O O

16. I would like to use a mobile device in my regular classroom work. O O O O O

17. I would like to use a mobile device in my personal life. O O O O O

Overall Reaction and Youth Development Outcomes. Please choose the one answer that best reflects how true the statement is for you.

Not

at a

ll tr

ue

A li

ttle

true

Som

ewha

t tr

ue

Pret

ty tr

ue

Rea

lly tr

ue

18. The experience of working on my project/presentation was a good way for me to learn. O O O O O

19. My work on this project/presentation has increased my creativity skills. O O O O O

20. My work on this project/presentation has increased my confidence in my ability to accomplish a goal. O O O O O

21. My experience using mobile and digital technology has made me more interested in learning. O O O O O

22. My experience using mobile technology has increased my desire to be successful in school. O O O O O

23. Overall, my experience during the mobile learning institute program was… O Extremely Poor O Very Poor O Poor O So-So O Good O Very Good O Extremely Good

v.1

24. Because of my experience working on my project/presentation, I am more likely to create a digital presentation in the future. O Strongly Disagree O Disagree O Neutral O Agree O Strongly Agree

25. If you were to change ONE thing about the MLI residency experience, what would it be?

Demographics 1. Do you own a mobile phone?

O Yes O No

2. If you answered “yes” to Question 7, which of these activities do you do on your mobile phone? (Choose all that apply.)

O Send/receive text messages O Send/receive picture messagesO Send/receive email O Use the internet O Send/receive instant messagesO Listen to music O Watch videos O Play games

3. What is your gender? O Male O Female 4. What is your age? [pull down menu, 6-21 years old] 5. Please indicate your ethnicity/race.

O African-American O Caucasian O Asian American O Hispanic O Native American O Other

v. 1

Mobile Learning Institute: Teacher Post-Reflection Survey

This survey is a part of the Mobile Learning Institute (MLI) residency. It will help us by identifying the kinds of skills you learned from participating in the MLI program, how you might use the skills you learned in the future, and your overall perception of the experience. Try to fill in every question and be as honest as possible. The questions we ask about your personal information will only be used to describe the kinds of young people who fill out the survey. The information will not be used to identify you.

What is the name of the MLI site you attend? ________________________________________________________

Information, Media, & Technology. Please choose the one answer that best reflects how true the statement is for you.

Because of the MLI Teacher Residency, I have improved my ability to…

Not

at a

ll tr

ue

A li

ttle

true

Som

ewha

t tr

ue

Pret

ty tr

ue

Rea

lly tr

ue

1. Use mobile and digital technology in my classroom instruction. O O O O O

2. Use mobile and digital technology to find content for my classroom instruction. O O O O O

3. Use mobile and digital technology to find research on content and best practices. O O O O O

4. Use mobile and digital technology to monitor student progress.

5. Solve most problems that may occur when using mobile and digital technology.

6. Use mobile and digital technology, in general.

7. Teach students to use mobile and digital technology.

v. 1

Availability of Technology. Please choose the one answer that best reflects how true the statement is for you.

Not

at a

ll tr

ue

A li

ttle

true

Som

ewha

t tr

ue

Pret

ty tr

ue

Rea

lly tr

ue

8. My school building is not equipped with mobile and digital technology. O O O O O

9. My school building lacks technology support staff. O O O O O

10. The utilization of technology in instruction is a priority at my school building. O O O O O

11. Do you use mobile and/or digital technology in your classroom?

O Yes O No

12. Do you use mobile and/or digital technology to teach subject content? O Yes O No

Intentions. Please choose the one answer that best reflects how true the statement is for you.

Because I used mobile and digital technology in the MLI…

Not

at a

ll tr

ue

A li

ttle

true

Som

ewha

t tr

ue

Pret

ty tr

ue

Rea

lly tr

ue

13. I would like to use mobile and digital technology more in my classroom. O O O O O

14. I think that mobile and digital technology is a valuable resource for educators. O O O O O

15. I would like to use mobile and digital technology for my teaching in the future. O O O O O

16. I am more interested in using mobile and digital technology to teach subject content. O O O O O

17. I would recommend that more teachers integrate this technology into their classrooms. O O O O O

v. 1

General Residency Reactions. Please choose the one answer that best reflects how true the statement is for you.

Not

at a

ll tr

ue

A li

ttle

true

Som

ewha

t tr

ue

Pret

ty tr

ue

Rea

lly tr

ue

18. The MLI teacher residency was a good way for me to learn. O O O O O

19. I have learned new skills at the MLI teacher residency. O O O O O

20. I would recommend the MLI teacher residency to my coworkers. O O O O O

21. I would rate the overall MLI Teacher Residency as: O Extremely Poor O Very Poor O Poor O So-So O Good O Very Good O Extremely Good

Demographics 1. Do you own a mobile phone?

O Yes O No

3. What is your gender? O Male O Female 4. What is your age? [pull down menu, 6-21 years old]

2. If you answered “yes” to Question 7, which of these activities do you do on your mobile phone?

(Choose all that apply.)

O Send/receive text messages O Send/receive picture messages O Send/receive email O Use the internet O Send/receive instant messages O Listen to music O Watch videos O Play games

v. 1

5. Please indicate your ethnicity/race.

O African-American O Caucasian O Asian American O Hispanic O Native American O Other 6. How many years of working in schools have you had?

O 1-5 years O 6-10 years O 11-15 years O 16-20 years O 20+ years 7. What is your age in years? _______________ 8. In general, what teaching assignments do you have? (mark all that apply)

O General Education O Alternative Education O Special Education O Other (please specify) __________________________

Sample with Curriculum

MLI Session Log, to be completed by Instructor

MLI: Session Log

Activity Sequence Mobile Literary Devices Session Date: _______

Instructor Name ______________________________________________

1) Total length of instruction time: ___________ minutes

2) Objectives Objective Taught? Objective met?

1. Understand how literary devices are used in writing. Y N Y N 2. Apply the understanding of literary devices in a real‐

world context. Y N Y N

3. Apply existing knowledge to generate new ideas, products, or processes.

Y N Y N

4. Communicate information and ideas effectively to multiple audiences using a variety of media and formats.

Y N Y N

3) Please list any modifications you made to the original lesson outlined in the Curriculum. Modifications include the deletion of any elements, steps, or materials as well as the addition or expansion of any components or materials. Please describe the reason(s) for any modification(s) as well.

4) Please rate the participants (in general) on each of the following activities of today’s session.

Not engaged

at all Minimally engaged

Somewhat engaged

Very engaged

Extremely engaged

This component

was omitted.

1. Share basic procedures and expectations that they need to understand in order to succeed in the project.

O O O O O O

2. Explain that a series of Mobile Literary Devices entries are required for the assignment.

O O O O O O

3. Guide students through the creation of a Mobile Literary Devices entry.

O O O O O O

4. Students use mobile devices to capture and publish their photos

O O O O O O

5. Students review their entry using their Ovi album.

O O O O O O

6. Pair Share and Comment on each other’s entries.

O O O O O O

Sample with Curriculum

MLI Session Log, to be completed by Instructor

Not engaged

at all Minimally engaged

Somewhat engaged

Very engaged

Extremely engaged

This component

was omitted.

7. Explain that students will continue to follow the same procedures for the duration of the Mobile Literary Devices project.

O O O O O O

5) List the strengths of today’s session:

6) List the weaknesses of today’s session:

7) Please rate how appropriate today’s activities were for the participants. In other words, were they too hard or too easy?

Activities were

extremely easy for

participants.

Activities were

appropriate for

participants.

Activities were

extremely difficult for

participants.

How appropriate were today’s activities for the participants?

O O O O O

Please describe why you chose your selection above. (What was appropriate/easy/difficult etc.?)

MLI Student Site Observation Checklist Date:______________________ Site Name Observed: __________________ Observer:__________________ Instructor(s):_________________________ Start/Stop Time: ________________

1. Subject/Title of Session (Life Skills): _____________________________________

2. Content/Skill Connection: ________________________________ 3. Activity Sequence: _____________________________

3. Total number of participants: _____________

a. Participants engaged in activity when you arrived? _____ b. 10 min. later? _____ c. End of session? _____

4. Record time(s) leader started/stopped giving instructions/directions. ____________________________________________________

5. Split participants into smaller groups? Yes or No

a. Adequate adult supervision for each group? Yes or No

6. Number of participants who served as leaders within the activity? ___________

7. Consider the equipment. Was there enough equipment for the number of participants and the activity? Was it utilized?

__________________________________________________________________________________________________________________

__________________________________________________________________________________________________________________

8. Was there adequate time for participants to complete the session’s activity? If no, please explain. __________________________________________________________________________________________________________________

__________________________________________________________________________________________________________________

Other observations: ________________________________________________________________________________________________________________________ ________________________________________________________________________________________________________________________ ________________________________________________________________________________________________________________________

Program Features Instructors Instructor discussed learning objectives of the session with participants Yes No Instructor explicitly connected tasks with learning objectives Yes No Instructor worked one-on-one with participant Yes No Instructor used technology to enhance instruction Yes No Instructor demonstrated competence with technology (i.e. able to troubleshoot technical problems) Yes No Instructor acknowledged participant’s improvement in a certain skill Yes No Instructor allowed participants to express their own creativity Yes No Student Learning Opportunities Opportunities to ask questions Yes No Opportunities to understand how digital media are created Yes No Opportunities to use and become familiar with mobile and digital technology and digital media creation software

Yes No

Opportunities to participate in project-based activities Yes No Opportunities to locate content-focused information using digital media Yes No Opportunities to share understanding of mobile and digital technology with others Yes No Opportunities to use and express creativity Yes No Opportunities to participate in content-specific activities that integrate the use of mobile and digital technology Yes No Opportunities to participate in teams Yes No Opportunities for collaborative problem-solving Yes No Opportunities to create digital artifacts through the use of initiative-building activities Yes No

MLI Student Site Observation

MLI Instructor Debriefing Questionnaire Activity Sequence ____________ Session Date: _______ Instructor Name ______________________________________________ 1. What worked well today? 2. What didn’t work well? Do you know why? 3. What components/concepts did participants seem to understand well? 4. Were there any concepts that participants appeared to not understand? 5. What (if anything) would you do differently next time in presenting content?

MLI Observation Summary: 03‐15‐10

Frida Kahlo High School MLI, Los Angeles, CA

The OSU team and Pearson staff observed the first session of a new Mobile Learning Institute (MLI) located at Frida Kahlo High School in Los Angeles, CA. The session was facilitated by two MLI instructors with the assistance of two classroom teachers from Frida Kahlo. Twenty‐nine students were present for the first day. As indicated by a classroom teacher, the participating students were selected in small cohorts based on academic performance and engagement. In sum, these students included some with low engagement and low performance, some with low engagement but high potential for improved performance, and some with high engagement and high performance. Typically, these students spend the first period of every day in an independent study period. These 29 selected students will participate in the 12‐week MLI in place of their regular independent study.

After initial contact from Frida Kahlo’s principal, MLI and leadership within the school began planning for the MLI in November 2009. Together, the principal and lead MLI instructor developed a plan of implementation, including key goal areas. In addition, the MLI instructor identified several goals with the two classroom teachers. The primary goal was to increase student engagement in learning.

A Day in the Life: Day 1

The theme for the observed 12‐week MLI was “A Day in the Life.” This included a project‐based curriculum encouraging students to consider the integration of their communities with their identities, culminating in digital artifacts that allow students to share their own perspectives of their communities. This observation reflects the first day this MLI. The lead MLI instructor clearly outlined the session’s activities, which were: 1) to become familiar with the MLI social network (i.e. Remix), 2) to personalize student profiles, and 3) to begin personal blogs. More specifically, students were required to “surf” the Remix social network, comment one blog or video, upload pictures of themselves to their profiles, and begin answering questions in their first blog assignment. Once the activities for the session were explained, the two MLI instructors and two classroom teachers then walked around the classroom to assist students as needed.

Areas of Strength

Several components of the observed MLI session supported student learning, including the facilitators’ interactions with students, efficient use of available equipment, and the students’ interactions with each other. In regards to facilitators’ interactions with students, all four facilitators (from MLI and Frida Kahlo High School) utilized positive feedback and timely prompts for effective problem‐solving throughout the session. Additionally, the lead MLI instructor clarified directions repeatedly and provided specific , step‐by‐step instructions as needed. The efficient use of available equipment was especially important in this session as technological problems posed a potential barrier for student participation. While there were nearly enough laptops for each student to have his/her own, problems with the internet browser required many students to share laptops in pairs or trios. With the guidance of the facilitators, the students shared computers and mobile devices effectively and appropriately. In addition, the students

demonstrated prosocial skills as they helped each other navigate the Remix site and view other student videos posted in the forum.

Potential Areas for Improvement

While the instructors clearly related to the students and provided explicit directions, the roles of the school principal and staff were somewhat unclear. These roles may be dependent upon site‐specific context and needs. At this MLI, however, the classroom teachers initially offered support throughout the session. They did not lead in the instruction of the MLI. This may due, in part, to the nature of this introductory day in the 12‐week program. Throughout the program, ongoing support provided by the MLI instructors will be essential for creating long‐lasting change. The classroom teachers indicated that they had one informal review session with the lead MLI instructor. As the MLI instructors provide weekly sessions, the classroom teachers will monitor and assess student progress. In the end, the classroom teachers may be able to take on increasing responsibility in the MLI sessions. Further, Frida Kahlo’s principal is clearly committed to this partnership between the school and MLI. It will be important to consider the role of educational leaders at this school in future planning for MLI implementation.

As MLI continues to develop its program design, it will be important to consider how the program’s broader goals focused on 21st Century skills (e.g. teamwork, effective communication) are evident in the classroom. At this initial MLI session, the program and session’s 21st Century goals and objectives were not explicitly stated and explained to the students. Instead, the three main activities for the session were framed as goals for the day. Instructor feedback was limited in these areas as well. An improved emphasis on fostering 21st Century skills will most likely lead to improved outcomes in these areas. This could include more explicit emphasis on the acquisition of such skills, clear feedback on students’ strengths and weaknesses in these skills areas, and didactic instruction when necessary.

Mobile Learning Institute Program Profile Cultural Institution

Name: The Field Museum

Location: Chicago, Illinois

Focus: Museology Program

Climate Change Workshops

Background

Curriculum Features

Digital Moments

Mobile Learning Institute Program Profile Frida Kahlo High School, Big Picture Learning

Name: Frida Kahlo High School

Location: Los Angeles, California

Focus: Advisory Program

Background

Curriculum Features

Digital Moments

Mobile Learning Institute Program Profile High School, Envision Schools

Name: Metropolitan Arts & Technology High School

Location: San Francisco, California

Focus: Workplace Learning Experience Program (WLE)

Background

Curriculum Features

Digital Moments

Mobile Learning Institute Program Profile Community-Based Organization

Name: Education Without Borders International

Location: Oakland, California

Focus: Biodiesel Diaries

Background

Curriculum Features

Digital Moments

MLI: Engagement LogIntroduction

Mobile Learning Institute: Engagement LogThe questions in this log will address expected engagement outcomes, modification and activitiescovered during your MLI Program.

Name:

MLI Program:

MLI Program Date(s):

1. Total approximate length of instruction time (per session, in minutes):

Page 1 of 3

MLI: Engagement Log

Created using SurveyGizmoPage 1 of 3

Curriculum Objectives

2. Curriculum Objectives. For each curriculum objective, indicate if it was taught. If not applicable toyour MLI program, note that the curriculum objective is "not applicable."

Yes

No

Not

Appli

cable

Understand how media technology artifacts are created.Communicate information and ideas effectively to multiple audiences using a variety of media and formats.Use and become familiar with media technology.Develop collaboration skills.Create digital artifacts.Learn how to use media technology to pursue individual interests.Use media technology to express creativity and innovate.Acknowledge audience, receive critique, and act on it to improve products.Use digital media technology to find information, qualify it and act on it.

Yes

No

Not

Appli

cable

Understand how media technology artifacts are created.Communicate information and ideas effectively to multiple audiences using a variety of media and formats.Use and become familiar with media technology.Develop collaboration skills.Create digital artifacts.Learn how to use media technology to pursue individual interests.Use media technology to express creativity and innovate.Acknowledge audience, receive critique, and act on it to improve products.Use digital media technology to find information, qualify it and act on it.

3. Please list any modifications you made to the original lesson outlined in the Curriculum.Modifications include deletion of any elements, steps or materials, as well as the addition orexpansion of any components or materials. Please describe the reason(s) for any modification(s)as well.

Page 2 of 3

MLI: Engagement Log


Activities and Comments

4. Activities. Please rate how appropriate the following activities of this MLI program were (forstudents). In other words, were they too hard or too easy? If the activity was not included in theMLI program, indicate "not applicable."

Too

easy

Easy

Appr

opria

te

Diff

icul

t

Extre

mel

ydi

fficu

lt

Not

appl

icab

le to

this

pro

gram

Geo-locative Media Applications: mapping, tours, geotaggingParticipatory Media and Civic Engagement: calls to action, political and social change,mobilization, fan sitesSocial Networks and Collaborative Learning: wikis, study groups, formal critiquesGames and GamingCitizen Journalism and Writing-based Digital MediaDigital Media Production: photography, video, music, podcasting, digital storiesApplication Specific Program I: Augmented Reality Applications (Point and Find)Application Specific Program II: SCVNGR

Too

easy

Easy

Appr

opria

te

Diff

icul

t

Extre

mel

ydi

fficu

lt

Not

appl

icab

le to

this

pro

gram

Geo-locative Media Applications: mapping, tours, geotaggingParticipatory Media and Civic Engagement: calls to action, political and social change,mobilization, fan sitesSocial Networks and Collaborative Learning: wikis, study groups, formal critiquesGames and GamingCitizen Journalism and Writing-based Digital MediaDigital Media Production: photography, video, music, podcasting, digital storiesApplication Specific Program I: Augmented Reality Applications (Point and Find)Application Specific Program II: SCVNGR

5. List the strengths of this MLI Program:

6. List the weaknesses of this MLI Program:

7. Please describe why you chose your selection above. (What was appropriate/easy/difficult, etc.?)

Page 3 of 3

MLI: Engagement Log


v. 1

Mobile Learning Institute: Summit Evaluation

The following survey asks you to consider your experience during the Leadership Summit. The answers you provide are completely anonymous. Please be as honest as possible. In addition, the questions that ask you about your personal information will be used to describe those who completed the survey in general.

What was the location of the MLI Leadership Summit you attended? What date did you attend the MLI Leadership Summit?

Please choose the one answer that best reflects how true the statement is for you.

Not

at a

ll tr

ue

A li

ttle

true

Som

ewha

t tr

ue

Pret

ty tr

ue

Rea

lly tr

ue

1. I attended today’s session because I am looking for strategies to develop 21st century skills in support of student achievement. O O O O O

2. I attended today’s session because I am looking for strategies to apply the latest mobile technologies in support of student achievement.

O O O O O

3. Now that the Summit is over, I think the issues discussed and presented today have the potential to help me better meet my students’ needs.

O O O O O

4. I would have preferred a session in which my district’s leadership teams could also have taken part. O O O O O

5. I would have preferred more time to network with colleagues. O O O O O

6. I would have preferred more presentation of information and less discussion. O O O O O

v. 1

As a result of today’s Mobile Learning Institute Leadership Summit, I have a better understanding of…

Not

at a

ll tr

ue

A li

ttle

true

Som

ewha

t tr

ue

Pret

ty tr

ue

Rea

lly tr

ue

7. 21st Century skills and their role in student learning. O O O O O

8. The potential role mobile devices and related technologies can play in student learning. O O O O O

9. The ways in which the Mobile Learning Institute is working to support students and teachers. O O O O O

10. Concrete approaches I can take to bring the latest technologies to my teachers. O O O O O

11. Concrete approaches I can provide to help my teachers make better use of these technologies. O O O O O

Skill. Please consider how GOOD you think the following Leadership Summit components were and then choose the option that best represents your opinion.

Not

goo

d at

all

Not

goo

d

So-S

o

Goo

d

Very

goo

d

12. I found the opening introductory remarks O O O O O

13. I found the Summit segment about 21st Century skills O O O O O

14. I found the Summit segment related to mobile technology O O O O O

15. I found the Summit segment related to school district transformation O O O O O

16. I found the Summit segment related to teacher quality O O O O O

17. I found the day’s final conversations O O O O O

v. 1

General Summit Reactions. Please choose the one answer that best reflects how true the statement is for you.

As a result of today’s MLI Summit…

Not

at a

ll tr

ue

A li

ttle

true

Som

ewha

t tr

ue

Pret

ty tr

ue

Rea

lly tr

ue

18. I would recommend attending a future MLI Summit to my peers. O O O O O

19. I would be interested in attending another MLI Summit on another theme important to my district. O O O O O

20. I would like to explore a Mobile Learning Institute program for teachers in my district. O O O O O

21. I would like to explore a Mobile Learning Institute program for students in my district. O O O O O

22. I would welcome the chance to invite members from my district leadership teams to a similar summit intended just for them.

O O O O O

23. I better understand how to include project-based programs within my curriculum. O O O O O

24. I found the way my time was spent today to be: O Not good at all O Not good O So-So O Good O Very Good

25. I thought the way in which the Summit met my own needs to be: O Not good at all O Not good O So-So O Good O Very Good

26. Compared to my expectations, I thought today’s Summit was: O Not good at all O Not good O So-So O Good O Very Good

Nokia/Pearson Foundation Mobile Learning Institute ...

Documents

Transcript of Nokia/Pearson Foundation Mobile Learning Institute ...