ELI Online Seminar: Determining the Effectiveness of Your Faculty Development Program (236670470)

Educause ELI Online Seminar: The Effectiveness of Your Faculty Development Program

1

Hello everyone and thank you for joining today's ELI seminar, "Determining the Effectiveness of Your Faculty Development Program." I want to mention that those participating today are eligible to receive a digital badge for doing so, and you don't need to do anything special to request the badge. We're just going to be checking our registration list and issuing the badge automatically to those that were present and actively participated. We want to encourage you to accept and display the badge when you receive it on social network, and to do that, you just follow the instructions in the e-mail that comes notifying you that you go the badge. I also want to mention that today's online seminar is the first in a three-part series of faculty development programming that ELI will be doing in 2014. For those that participate in all three parts, we will be issuing a sort of super badge, certifying that you participated in several hours worth of faculty development, professional development programming. So I'll be introducing our speakers in just one moment but first let me give you a brief orientation to the learning environment. Our online learning space is subdivided into several windows, and the presenter slides are now showing in the main window, which is the largest. As we move through the presentation, the slides are going to advance through that very window. The tall window on the left is the chat window for all of us to use, and we encourage you to make comments or pose questions or resources, whatever you'd like. Everybody will be able to see that, and to do it you just type your message at the bottom of the chat window and press "return" or click on the quote balloon icon just to the right of the box. You can also direct a comment to just a single individual, and to do that you just find their name in the attendee list or participant list, whatever. Sometimes we change it. And then you just put your cursor over that person's name and then a menu of options will appear. Choose the option to initiate a chat with that person, and then the person's name will then appear as a separate tab beneath the chat space so you can direct additional message to them using that same tab if you like. If you have any problems with the audio, you can call in on that number that see on the lower right-hand corner there. And if you run into any additional issues, just feel free to send a message to EDUCAUSE Help Private Message. That would be appreciated. So now let's turn to today's seminar. We've got three speakers today. Tanya Joosten is the first, as she's the director at the University of Wisconsin Milwaukee's Learning Technology Center where almost 3,000 instructors are supported in their use of digital technologies to facilitate teaching and learning. UWM offers over 40 online and blended degree programs to it's approximate 30,000 students, and Tanya is the author of "Social Media for Educators" published by Jossey-Bass. Her work and expertise on social media, virtual worlds, and other emerging technologies have been highlighted in the Chronicle of Higher Ed, in Tech Magazine, E-Campus News, EDUCAUSE Review, EDUCAUSE Quarterly, and other news media publications. In leading the leading the UWM LTC she oversees the recognized faculty development team for online and blended learning, the UWM certificate in blended and online teaching, the online and blended teachers users group and the Sloan C. Blended Faculty Development Program. Dylan Barth is the interim associate director at the University of Wisconsin Milwaukee's Learning Technology Center with Tanya, where he coordinates the faculty development program for online and blended teaching at UWM and UWM certificate for online and blended teaching. He has presented at several educational technology conferences and has a forthcoming chapter on blended learning to be released -- I think actually this has already been released, "Blended Learning Research Perspectives Volume 2." Dylan has served on the program chair for the 2012 Sloan C. Blended Conference and the track chair for faculty development for the 2013 Sloan C. Blended Conference as well.


2

And also joining Tanya and Dylan is their colleague, Nicole Weber, who is an instructional technology consultant at the University of Wisconsin Milwaukee. Tanya, Dylan, and Nicole, welcome to today's seminar. Please begin. Great. Thank you for the introduction, Veronica. So what we are going to talk about here today -- and just one second as I pop up our slides. All right, there we go. So what you're all joining us for today is talking about determining the effectiveness of your faculty development program. And just to give you a little background of sort of where this came from, you know, obviously all universities are dealing with sort of this demand for blended and learning, and some of the opportunities that are afforded by these new modes of teaching and learning. And so we've had our faculty development program for some years, and we had certain ways at a very small or micro level that we evaluated that program, but since then, with some of the -- you know, the fact that blended and online is becoming more prominent not only at our institution but amongst lots of institutions. I think there's a little bit more competition there. And I know our institution, as well as lots of other institutions, are dealing with, you know, tough fiscal times here. And so we've been rethinking, in the last couple years, sort of how we evaluate the effectiveness of our faculty development program. Some of us are researchers. I'm a social scientist, and so we have been thinking about different ways that we could evaluate the effectiveness of our program and document that for different audiences. And so we actually had a learning circle at the ELI Conference in New Orleans on a very similar topic just to sort of have a discussion. By no means are us at UWM I think are anywhere, really, experts in this, and we're really looking to create a dialogue and sort of share in some of our practices. And so at the learning circle we really, firsthand, realized that there was a lot of diversity around where folks were when it came to determining whether their faculty development program was effective or not. We have some folks -- and I'm sure some of you joining us today -- that haven't designed to your faculty development program yet and are just thinking about what are your faculty needs. And I'm sure, like us, some of you have been, you know, providing professional developments and different instructional support mechanisms for your faculty to design and deliver blended and online classes for years, and just trying to think about maybe a more systematic approach to evaluation. And so that's sort of what brought us here today. So we're going to share a little bit about what we're doing. We're hoping to hear from all of your about what you're doing. We're definitely providing this in a framework, which I think will help guide you all, either in better understanding some of the processes that you have now, or making them more strategic. And so this should be really exciting today. So just to give you a little bit of an overview, I'm going to talk a little bit about UWM and just who we are so that you understand the lens that we come from when we're talking about determining the effectiveness of the faculty development program. And then also, since this is a workshop, we want folks to roll up their sleeves and get dirty, so to speak. So we want to put you all in some Adobe groups and have some breakout activities to really help with us and sort of, you know, having discussions around these questions that really are key considerations in planning, you know, whatever strategies for determining the faculty development effectiveness. Once we share out some of these summaries of activities from these breakouts, we're also going to share a little bit more detail about our model for blended and online courses, our pedagogical model, talk briefly about our faculty development, and then also delve into some more evaluation planning, as well as some specific strategies and tools. And then also, we'll share a little bit about some of the research we've been doing over the year, and some of the information that we have gathered from other institutions. And, again, we hope that all of you will contribute, and so we can hear great things from all of you as well. All right, so let's get started with a little bit of an introductions. I don't think there's much introducing that we need to do. I think Veronica gave a very thorough bio of me. But I'm Tanya Joosten. I'm the interim director. I have been here at the University of Wisconsin for ten years. I have taught online for 13 years,


3

and blended as well. And I teach in the Department of Communication, and so was part of the team that developed the faculty development program. I'm here with my colleagues Dylan and Nicole. Dylan? Thanks, Tanya. So I'm Dylan Barth. I think Veronica also did a great job introducing me. I think that I would just sort of add too, that I've been teaching online and blended also for several years. I teach in the Department of English, focusing mainly on a business writing course that's in the blended format. And Nicole, do you want to tell us a little bit about your different experiences and what you're doing here today -- no. Yeah, sure. My name is Nicole Weber. I am a learning technology consultant with the Learning Technology Center here at UWM. I teach fully online courses and have done so for the past, I don't know, seven, eight years with the Department of Educational Policy and Community Studies, where I teach both at the undergraduate and graduate level. At the LTC I co-lead the Online and Blended Teaching Program, which is a faculty development program for those looking to redesign their traditional courses for the online and blended environments. I also assist Tanya with our research efforts and evaluation efforts. Excellent. Thank you so much. Sorry, toggling back and forth here on mute so we don't get any feedback and so forth, so I appreciate everyone's patience. I'm very happy to have my colleagues here. As you heard, we all teach blended and online. We all participate in research, as Veronica mentioned. We just had a chapter out last fall, looking at the impact of faculty development on several things, but on retention and grades, and so we'll talk a little bit more about that moving forward here today. So I just wanted to emphasize some of the outcomes or things that we 'hoping that you're going to take aware away when you leave here today, and so those things are the fact that we want you to be able to identify the characteristics of a good blended and online course, including the pedagogical model, and we'll talk a little bit more about all of these, of course; determine what elements and formats should be considered in designing a faculty development program. Again, we'd like to hear all of what you're do, but we're going to share what we're doing and what we've heard from some other folks as well, and how all of this ties into really being able to have a full picture and a holistic view of the evaluation of your faculty development program. We also are looking not only for us to share strategies but for you all to share strategies for evaluating the effectiveness of your faculty development program at different levels. So we want to go from more of the micro level to the macro level, so individual courses, programs, as well as up to the institutional level. I know sometimes it's beyond that to a system level for some of you. And we also want to talk about the evaluation coming from multiple perspectives, so students evaluating, also your colleagues, evaluating, research administration, and so forth, and so we're talk about some of those things. And then finally, we want to talk about how these steps are going to fit into a larger model of evaluation for learning technologies and pedagogical evaluation. As you already can tell, specifically today, we're focusing on our blended and online faculty development and some efforts that we have taken to develop a strategic approach to evaluating its effectiveness, but also a lot of the strategies and the framework that we're sharing with you today can be used for other things as well. All right, so I wanted to just tell you a little bit about us so that you understand our framework and where we're coming from, coming from University of Wisconsin Milwaukee and coming from the Learning Technology Center. A lot of times when people here the terminology, Learning Technology Center, they tend to make an assumption that we are focused on student support. But, really, we are focused on supporting faculty and other instructors, and using technology in pedagogically effective ways. But at the end of the day, our overall mission is to improve student learning, which is why we, you know, call ourselves the "Learning Technology Center." So we provide, obviously, faculty development, which is a part of why we're here today. But we also provide pedagogical consultation to faculty, not only for blended and online but for an array of


4

technological interventions that might help improve their teaching and improve student learning. We don't do production, per se, so we don't help people actually build their core sites or build digital content for their core sites, but we, in addition to providing sort of this pedagogical consultation, we also do provide technology training and support. We sort of always had this idea around here of teaching faculty and instructors how to fish and then, therefore, this makes it so we can have scalable practices around here, and processes, so that we can sometimes to do other fun things like evaluation and research. And so rather than obviously fishing for them, by teaching faculty to use technology, use it in pedagogically effective ways, we hope to have an impact on different sorts of student outcomes, which, again, is back into determining our effectiveness of some of these things. Most of what we support is from tech-enhanced courses, all the way to fully online. What's very rare is we are in academic affairs, although we actually administer the Learning Management System, so it's not NIT, it's with our unit, and we're a group of instructors that administer the LMS. So it's nice to have teachers making decisions about the LMS and supporting others in LMS. And that's really where our tech-enhanced courses come. And then, as you can see, this goes all the way to online, where we used to have this traditional online, which I think it's funny that's even come out of my mouth, "traditional online," but now we have lots of different kinds of online here at UWM. So we now have the self-paced or the you-pace model. We also have flex classes, which is part of our competency-based, which really unbundles the instruction from that there, and then also, we have some other models as well, you know, MOOCs and all of that fun stuff. So online is really advancing, which really keeps us on our toes to how do we support faculty and make sure that we, you know, can manage our workload and still lead to effective outcomes. So a little bit about UWM, a lot of people may or may not be aware of us. So at the University of Wisconsin Milwaukee, we have about 30,000 students here. We're a doctoral granting institution. And, actually, one out of every three of our students takes an online class. What's very interesting in some of the research we did is, you know, we do have students that are far away and are substantiated with space. Butt it's very interesting, about 70% of our students are right in our own backyard here in the metro Milwaukee area, and they're looking to take online classes, in part, to push time. And so we have about over 1,500 blended and online classes that are delivered each year, and that's increasing, I think, every semester. We have about 40-plus online and blended degree programs. And if you want to know more about us, you can just go to online.uwm.edu and there is a full list. We actually have something called the "First 60 credits," so the first two years students can take completely online. And then also now there's an array of degrees, bachelor's degrees and graduate degrees. So there's lots of opportunities here for our students that was really driven based on the student demand and the student need, more than anything else. And so at the end of the day, it's good to let you all know that determining the quality of online and blended courses, at the end of the day, really is the responsibilities of the units, the departments, the schools, and colleges. It's not, per se, our responsibility. We don't necessarily want to be policing faculty and so forth. But we do want to facilitate a culture of quality, and we do want to better understand whether -- you know, it's not just about delivering the faculty development to prepare faculty, it's now after years and years, really understanding the greater impact that we're having on this institution with such a large diffusion of this mode of learning now across campus. So what is blended and online? Up in front of you is just a little bit of a chart of an institutional definition that we put together. I know some of you have different definitions on your campus for blended and online. So for online, for us, is that it needs to be 100% online. We have a predominantly asynchronous model. It's very rare that we see fully online classes, requiring the entire class to meet synchronously. Usually any synchronous meetings are more in small groups, and that comes from our research that students really want to push this time. We also have a definition for blended. It actually used to be called "hybrid" here. We have a faculty resource we'll throw up on our wiki, hybrid.uwm.edu, which provides more information about resources for


5

those of you who are supporting faculty and blended/online. We originally developed this resource over a decade ago. Since then, we have revamped it. But some of the information, you know, endures through time. And so we received a very large grant from the Sloan Foundation, and so now we called "blended." And so blended, for us, we have three classifications, but really, it's anything beyond 20% of seat time being replaced by online activities. So 21% to 99% at our institution is considered a blended class. I know that's a little different than some of the national surveys. I think with "Seemen" and so forth that we saw around 30% or so forth, but for us, if it's going to be online, we need to make sure for students' expectations, it's 100% online. And one of the keys for blended for us is that we're -- and this is will come out in evaluation, we really push an active learning student-centered model. We're really looking for increasing engagement in the online and the face to face with students and integrating those two environments so we're most effectively able to meet our pedagogical goals and improve student learning. Now I know that's little different than some of the other blended high-flex hybrid flipped models that are out there. You know, traditionally in the blended model we see more of the content delivery taken online, along with the reduction in seat time, and more of the experiential or the higher order active learning activities taking in the face to face. And we're going to talk a little bit about this to get us started, because I think it's really challenging for us to understand how to figure out, you know, how to evaluate our programs and whether they're effective or not when, you know, what really is being effective. So we have developed sort of what we're tagging a backwards design approach to evaluation. And so this is a framework that we use for our faculty development program. More of us in the last several years ran across this approach and are using it in designing our own classes. And it was very interesting, I thought it would be very appropriate here as we were discussing, you know, what would be the best way to present this information to you all to make it really consumable and helpful. All right. So this is just a little overview of what I was thinking about in evaluation planning from sort of a big picture view. You know, we have lots of inputs. We have faculty that come with different experience. We have instructional staff to tenured faculty, and so these are traditionally some ways that researchers and so forth will discuss different models here. Folks come from different disciplines. As all of us know, some disciplines move forward a little more quickly than others do. And then we've all been obsessed for years about the face to face versus online, you know, and so we'll talk a little bit more in detail about that. The process variables, which is what we're going to spend a good amount of our time on today, is just sort of thinking about what we're sort of putting into the soup or into the crock pot or whatever fun metaphor I can think of here. You know, what are we putting in it to know at the end of the day that we have a really good meal. And so we're going to talk a little about that, and then I'll wrap up later this afternoon, talking more about sort of the outcomes, you know, what are some specific things that you can do to help document these things. So just to put it in a little bit of a framework there. And so Kevin -- I'll just break real quick here -- Kevin had a question, from the University of Delaware, about who collates all the data, regional accreditation self study? That comes out of our institutional research units, and it's done in collaboration with the different department schools and colleges. And so we'll talk a little bit more about that later this afternoon here as well. All right, and so one of the things that we always see in research that I just sort of wanted to spend a little bit of time addressing is when we talk about inputs, a lot of people, for the past couple decades, have this tendency to always want to compare face to face and online. And I don't have a shot here, but I actually taught a large lecture at Arizona State with 400 students, and I will tell you that my teaching in that class and the student learning definitely was not the gold standard of which we should be comparing anything to. I think if you want to talk about distance education, that that large lecture hall with 400 students was definitely distance education. And it came with a little bit of a different philosophy. I think very little had to do with pedagogy and a lot had to do with the fact that it was the way that we weeded out those who weren't going to be communication majors.


6

And so I think that we want to move away from studies that are going to compare online to face to face and focus more around the pedagogy and the blended or online and what's unique about that that's going to impact different student outcomes. And so I think a lot of us who are experienced blended and online teachers or provide instructional support and faculty development for blended online can fully understand that. And I just think we've been comparing the face to face and online not only in teaching and learning but in lots of different disciplines for the last few decades, and I think we could spend a lot more time, you know, better understanding the process and what pedagogy affords us in regards to blended and online that's going to lead us to that really awesome meal at the end called student learning. I love it when I make these funny things up on the fly. All right. So this is a little bit focusing -- as I said, we wanted to focus some time here on the process, and we're going to break out into groups here just in a minute. But, you know, the first thing that you need to do to determine whether your faculty development program is effective or not is think about what, you know, what are you trying to produce in your faculty development program. So this is a little bit where we sort of took the backwards design approach. You know, first of all, you need to determine what's a good blend and online course for your campus. Now I talked very briefly about what good blended and online is for us, you know, it's very engaged learning, it's active learning, it's student centered, focuses more on asynchronous activities, these sorts of things. But, you know, what is that for your course? I know that's not necessarily the same for your courses. And so I think you need to know what's a good online and blended course. What are you hoping; that you're going to increase engagement; that you're going to increase student learning, and then what pedagogical model is going to make that happen? Maybe you are just looking to increase enrollments and you're less concerned with the learning community piece of it, the social learning piece, and so you're going to just focus on the numbers, you know, how many people can you push through this course like a large lecture course? Obviously we're a little partial to the active learning and engaged learning model. But then you've got to define what model is going to help you achieve those outcomes. And then from there you're going to design your faculty development program that's going to facilitate that pedagogical model and lead to courses that can be designed and delivered in that way. So to kick us off here a little bit today, we're going to get active right away. And so just to give you a little introduction, we're not doing there yet. We're not breaking into groups yet. I'm going to talk a little bit more after this about what we're doing in these groups. But what we're going to do is break into discussion groups using Adobe Connect. Victoria will place you randomly into three different groups. Each of us, Dylan, Nicole, and I, will be facilitating those groups. And in those groups, you're going to be able to use audio or text chat, whichever works for you. That's why Victoria has worked so hard to get you all connected to the phone line. As you can see on the bottom of your screen, to the right here, there is a call-inbox. And so you're going to get into those Adobe Connect rooms, and then we're actually going to get into some Google Docs where we can help document some of the discussions that we're having. Because I think one of the really important things about these events that we have is that we build these digital repositories and these archives, so we're sort of capturing this knowledge that takes place so that we can share it out with each other and beyond here. And so we have a wiki set up, elionlineseminar.wikispaces.com. Don't worry, I'll bring it up again when we're about to break into our groups. But this is where we'll be have links to the Google Docs that you can easily access and participate. The wiki is public, the Google Docs are public, so you shouldn't have any access issues. But then what we're going to do is, in your groups, we're going to discuss and we're going to share and we're going to brainstorm some responses to really -- it's sort of the three initial steps to planning your evaluation. And so we have these series of questions. We broke them into three different areas around the model I just shared with you. And so we're going to do that here in just a bit.


7

Before we do that, let's talk a little bit more about the series of questions. So step one, defining effective. One of the things that we need to do is we need to understand what is success. What is effective for your faculty development program? And so I think, you know, it was one of the interesting questions I had for myself, because I think we have a really broad group of people here; right? Some of us are individual faculty, some of us run faculty support units, and beyond. Some of us are in administration and so forth. So I think, first of all, we need to think about what are you trying to accomplish by having a faculty development program or an instructional improvement effort. You know, what is your desired result at the end? You know, you're obviously trying to do something, you know, increase engagement, increase student learning, you're trying to increase enrollments. Whatever it is, I think we need to first sort of unpack what is effective, what is success, before we can start, you know, planning anything else down the road. I thought this would be a great discussion because I know some of us are going to come from different lenses and perspectives and we're going to have a different idea about what is success for our institution or for us as a teacher. And so the first series of questions was around, you know what is a good blended or online course? You know, for me, as a social scientist, you know, I very much want a course where my students are engaged. I want to, you know, make sure that they learn something, and I also like my students to be happy with the course or satisfied with the course, you know, and so it's just thinking about those sorts of things. You can also think about this as a program or an institutional level, you know, why an institution does it. I know, in the beginning, you know, with blended there was this whole idea that we were going to recapture space, you know, so that was an idea. That didn't really ever happen -- I don't think for lots of institutions, but that might be what a good course was, was just recapturing some physical space. So just throwing some ideas out there. So when we break into groups, this will be the first thing, is determining what's effective. The second thing is sort of facilitating success is what I'm calling it. And so, you know, what's going to help you get to those outcomes for that blended and online course? And I think specifically what we're talking about here is, you know, what elements and formats should be considered in designing and developing a faculty development program. So now you've identified you know, what's a good online or blended course, you know, what does it do. And then you need to determine, you know, what's going to facilitate that success in that course, what sorts of things -- not only what things should be in a faculty development program but what is the format. We are just doing some research. We did a workshop, I think about six times in the last year, and asking folks about, you know, what sorts of things ensure quality and lead to success. And people had a wide range of ideas on the format of faculty development and support. The other thing is sort of what opportunities and experiences should be available to help instructors learn effective practices in design, delivery, and teaching in online and blended. So I think there's going to be different elements or topics in a format. But I think also, I know for us, is we want it to be experiential, we want it to be practice based, and so thinking about some of those things. The third activity or series of questions is around documenting success. This is the, really, meat of it, and this will carry us into later this afternoon as well. But we need to know a little bit more here about, you know, how are you going to know that the faculty are providing these quality online and blended courses that you've identified in step one? You know, what tools or services could be provided for evaluating the effectiveness of online and blended courses? And then how will quality be communicated to others? We are definitely going to springboard this final step three, documenting success, into a richer discussion in this afternoon, but I think it's a great way for us to use this backwards design approach to evaluation planning and talking about what is effective, what is success for you, secondly, talking about how do you facilitate this success through faculty development program and experiences, and then the third thing is how are you documenting the success that you've achieved, you know, what you wanted to achieve and


8

set out to achieve in the first place. And you know what's very interesting is a lot of us never did this too much in first place, which I think it makes us hard at this point to evaluate. We started sort of offering blended and online and some institutions, either maybe because it was a grounded approach, faculty were already teaching with it because they thought it was better, or it could have been a demand for the market, you know, people were doing online and we need to be doing online. And so I think this helps us take a step back and think a little bit more strategically about how these different things relate to each other. Okay. Now it is time to break into our breakout rooms. I'm just going to go over this really quickly, again, before Victoria puts us into our rooms. So you're going to break into discussion groups using Adobe Connect. You will either be entered into group, one, group, two, group three. I know we're really creative over here, group one, group, two, or group three randomly. Again you can use your audio or your text chat. So if you called in, as you can, down at the bottom of the screen, you'll be able to participate with your audio. Otherwise you can always text in the chat. We also are going to be using a Google Doc to sort of summarize some of our discussions so that we can share this out with others. You can always just type your comments and question ins the Google Doc as well. There's an actual comments function that you can comment on things that are typed, or you could just go ahead and type in your ideas. Each of the docs, what you're going to do is identify your group. Nicole, Dylan, and I will be in each of the group breakouts to help facilitate that to identify the group, as well as the members in that group. And then, you know, at the end of the day, what we're doing is we're going to discuss, share, and brainstorm responses to these three things; okay? Any questions before Victoria moves us to the break out rooms? All right, we'll have about a half hour to do the three sets of questions. We're going to do about ten questions each -- or I'm sorry, ten minutes each -- ten questions each. We're going to do about ten minutes for each question. Each question has its own separate Google Doc. Go ahead and go to the wiki spaces if you want to jump in those docs right now, and Victoria is going to move us over into our Adobe Connect groups. All right. I need to make those connections between what we want to understand and learn and do to make that connection back to their personal, professional, and academic lives so it makes more of an impact. Yeah, absolutely. I think sometimes, you know, having those activities that are authentic or experiential, I think some of the other folks that are in the document talked about that as well. I think we saw something about authentic activities and meaningful, so, yeah, that's a really great point. All right, it looks like we're pretty much here, done with the questions in step one, and I know some of you have sort of already moved into the question -- you're in Google Doc two. I'm going to post in Google Doc two right there in the chat, so if you don't have it, please go ahead and click on it and move over to step two; okay? Again, please feel free to use chat if you have questions and so forth, or you can please feel free to accidentally leave you mic on and we will call you out and make you talk -- no. All right, so step two it was a little bit more specific about, you know, what elements and formats should be considered in designing and development a faculty development program, what opportunities and experiences should be available to help instructors learn effective practices in design delivery and teaching online and blended courses . And as before, go ahead and insert your comments, type in your name under your group. You know, we sort of lost our groups here, so don't worry, you won't lose points if you did not report to your exact group. It's more important that we just, you know, document the folks' ideas and so forth. While we're talking about this, I just wanted to share a little bit of some resent research that we were doing, Nicole and I -- whoops, I'm typing in the wrong document, job hazard. And so it was very interesting. We did this workshop, and one of the things that we asked people what were thing that is needed an institution needed to have to motivate faculty. And it was interesting how many people actually felt that it was how a faculty development program was delivered, the time period, whether it was


9

online or face to face, whether it was in manageable chunks. How many of those sort of things about faculty development that folks felt were, you know, really important to that motivation and to faculty. So some really believed that there should be longer sessions. I know some sessions that we saw in some of our document were 80 hours. Some folks felt very much it needed to be experiential, so delivered in the same way in which they were going to be teaching. Some felt that they needed shorter sessions, you know, very small chunks for pedagogies small chunks for tech tools, even lunch hour trainings or really small clinics. Others really like the fact of this fellows idea or peer-to-peer training, or, you know, just having this just in time or one-on one, some real individual attention. Some folks even talked about, as I mentioned the fellows program, but also this idea of faculty development coaching. So we just found it really intriguing that these sorts of things, you know, could tie into this when we're talking about facilitating success. All right, you guys are typing away here in the second Google Doc. If you're not in the second Google Doc, the step two, I did post it in the chat. Again, if you have questions or comments or would like to share, please feel free. We are monitoring the chat as well here. All right. All right, so it's really great to hear from Victoria that all of the phones are off mute. So if you have something in particular that you would like to share that your institution is doing, or a question that you have, please go ahead and put it in the Doc or go ahead and turn off your mute button and share it with us. All right, somebody is talking. David, did you turn your mic off? The leader has unmuted your line. I hear laughs and lots of typing. This is the University of Notre Dame. Did you just unmute us? Yes. We're hoping that you'll talk to us. Did you have your hand raised? No, we didn't. [Inaudible]. Oh, see, this is the way we're secretly pulling you all into discussions today, so you have to be active learners, as we're unmuting your mics -- I'm just kidding -- and making you talk. Anyways what were some of your ideas their at Notre Dame. I know that you have, what, three or four of you there, or even more? So what were some of your, oh, go ahead. We're currently four. I'm going to let Kevin answer you question. Okay. Well, a big part for us is making sure that it fits into the time constraints for the faculty. So you had mentioned longer sessions, that doesn't work here. Shorter sessions tend to be more effective. Okay, excellent. I know we still have somebody else unmuted because I hear them typing away. All right. Does anybody else like to share what they're doing to facilitate success? Hey, Tanya, this is David. Hey, David, I thought you were on there typing away. Well at Southeast we have a five-day crash course. We call it a "crash course," literally, for new faculty. And it used to be one day, and then it just kept on expanding, and we found that most of the day for a new faculty is not so much pedagogy or anything like that, but it's focused on the technology. Thanks for sharing, David. Does anybody else want to share a little bit about what they're doing, what opportunities and experiences they're providing their faculty, or what sort of different things are included in their faculty development program.


10

This is Sarah, can you hear me? Hi, Sarah, how you doing? Yeah, I can hear you. [Inaudible] and healing for our faculty is they work with me in one-on-one sessions for technology, and then they work with Louise in some pedagogical and other educational items that have to do more with the ideas and concepts. And when they go online, we meet with them in a series of one-on-one meetings, and then also let them know about university-wide opportunities, such as Moodle courses or things through our Academic Learning Center, that are things that they can schedule individually on their own time. And then we also do larger group sessions occasionally, and that has been based on both technological tools and also theory in pedagogy. That's what we do. How many instructors do you have at your institution? We have a lot who actually have never taught, as well as people who are faculty -- about 29 or 30 faculty in total. But, yeah, as I said, a lot of practitioners out in the field, they don't have much experience teaching either. Oh, okay. Either in person or online. Well it was funny, I can't remember who had said that. I think it was Susan Metros, when we were at the ELI conference in New Orleans, she's like the dirty secret is that, as faculty, we don't know how to teach, you know, because it was very rare when you were working on your PhD that you took a class on teaching or on pedagogy, I think that's one thing. I think, too, you know, you bring up a really good point about scalability. So we have about 3,000 instructors here that we support, and there are seven of us. And so a lot of times it's very interesting in developing or settling up faculty development and then thinking about how do we scale that and so forth, so. All right, I'm going to jump back here to the Google Doc, before we move to three, and just sort of see what folks here have. It looks like Nicole has inserted some comments. And, please, I encourage you all to insert comments about summaries about what folks have posted. You know, what's very interesting is how many people come and want to know about evaluating a faculty development program, and a lot of folks don't have a faculty development program or haven't started it yet, and that's okay too. I think that's part of the backwards design thing, is sort of figuring out in what ways you're going to evaluate yourself and then building a program so that you achieve that success. Yeah, it looks like we have some really great stuff. Some folks are really putting in some details. Please, if you have links to a more thorough explanation of your faculty development -- and I think this is a good time, too, to be thinking about is your faculty development program producing that evidence that you identified in step one, okay. All right, so as we wrap up question -- or step two, I should say, we're going to go ahead and move to step three. All right, so this is sort of the meat of it here, is documenting success. What I'm going to do is go ahead and post step three's Google Doc in there. So now, like I said, we're going to be sharing out some things with you all, but we're very interested in knowing what you guys are doing as well. So go ahead and click on that step-three doc that I just put into the Adobe Connect chat, or you can access it through the wiki, elionlineseminar.wikispaces.com, and we're on step three now; okay? So if you want to move you want to move your way over there, and we'll try to keep up with the chat as well. So step three, wow, look at you guys go. Step three is about documenting success; right. So we identified what success is in step one. In step two we talked about how our faculty development program was going to facilitate that success, and now here, in step three of our backwards design model for evaluating effectiveness of faculty development, we're going to talk about how are we going to document


11

that success; okay? So how are you goes to know faculty are providing that quality online and blended course that you identified in step one, you know, it's tools and services -- again, please include links -- you know, could be provided for evaluating the effectiveness of those courses, and then how will that quality be communicated to others on your exam campus. And we're going to talk in a bit more detail about that here. So it looks like we have quality matters, that was brought up already. That's always a big one we hear about. And so we have lots of other folks in here. Emily of Trinity is sharing about course evaluations, informal conversations, review of course materials. We have Rebecca from Alabama is typing away, is talking about their are multi-tiered review process. I don't know who this is, but I love this "increased student satisfaction faculty confidence increase, faculty sharing and leading development of other faculty." The minute I see "increased student satisfaction" -- oh, that's Julie. I think of that person must be a social scientist. No, I'm just kidding. All right. This is good, Pete's talking about this lifecycle review; that courses could be reviewed before launching midterm and after. We call that at UWM the "lifecycle of course review." I hear somebody who wants to chime in. Who do we have here? Oh. Okay, who just coughed? If somebody just coughed it's at Minnesota. Here we are, what institution are you guys from? I guess we don't -- All right, Minnesota, you're on the line. Why don't you tell us a little bit about what you guys are doing there to document success. [Inaudible] as a kind of a quick chat, but our other technique is that we have a template for online courses that we ask our instructors to copy. And that has a lot of built-in practices right away, so, you know, that has a lot of instructions to students. It's got a consistent interface. So we're already building in what we've tested and think is effective. So it's sort of a way of evaluating. But it's a way of also -- yeah, we've done the evaluation prior in terms of student usability, and now we're just requiring it. Yeah. So you're focusing more on the course level; right? So that each course has good design, you know, and good online effective practices. Are you guys do anything at the program level or institutional level? Yeah, we do get the students satisfaction. Like looking at satisfaction in online courses or looking at retention. So retention is a good idea. I don't know that we looked at that. Uh-huh. Uh-huh, yeah. Yeah, all right, good deal. It looks like -- all right, thanks, ladies. If there's anything you want to add, let me know, otherwise go ahead and mute your phone, or we're going to pull you in again for more discussions.


12

All right, Kevin at University of Delaware is talking about he was digging into the literature on evaluating faculty development, and he's talking about the maturity model for faculty development. Kevin, do you want to go ahead and share with us a little bit about what you're doing. Can you hear me. Yeah, we can hear you. Go ahead, thanks. Yeah. I'm sorry, I'm just starting to dig into the literature now. This came up in our center. It looks like we're still trying to tackle it. So we're just now getting into the literature. There's quite a bit for you to look out. Pod has a really good list of resources in their wiki, which is a really good place to start. I'm also very happy to find reviews we've done recently by Nancy and some other folks -- and Nancy's last name escapes me. She just retired from Indiana University. There's some really good stuff out there that I'm just starting to dig into. And the difficulty that we have is direct assessments of the impact of project development work. I think there's not just difficulty evaluating it but also some disappointment. Whenever we do evaluate it, it doesn't seem to made much of an impact. So I think that's why some folks have started to leave that behind and to look more at maturity models of development in teaching centers, so they can evaluate what is, essentially, the complexity or robustness of the service that they offer. What methodology are they using when they're looking at the maturity model? That's a great question. I'm not there yet. You know, sorry about that. Yeah I think it's great. That might we something this group, since we have all of these folks in this event that are obviously interested in evaluating faculty development and its effectiveness, it might be great if you have a link or have any links, Kevin, if you could put them on the wiki. You have full permission to edit. It would be really great if we could put together a collective, you know, list of the literature references around literature for those of us who aren't involved in Pod or in some of the more, you know, formal organizations that have access to the literature. And I know for some of us its just hard to find time to pull literature and do a lot of the reading with the other 500 things we have to do. So I think if we can use each other as a community to develop this body of literature or this knowledge base, as well as instruments and tools and help each other out as a community, that would be wonderful. Yeah, Sierra College and Trinity has been in there. Sorry, I wasn't paying to the chat [inaudible]. So I think to push a little bit further, it's not so much about us developing the literature, I think it's more important that we understand the literature that exist. There's a lot of things out there from other folks and other disciplines and related fields that have already done, so we don't have to reinvent the wheel, we just need to find the people who have already reinvented it. Yeah. Yeah. Well, I think it's good to understand the existing literature. I also, though, believe that there's a lot of room to further develop the literature and the research body that's out there, specifically when we're talking about faculty development for technology enhanced processes, because I think sometimes that can be a little limited. And what's really great is this offers the opportunity for more multidisciplinary approach here. It's just one of those areas that there is so much going on across the disciplines, and really fantastic. All right, let me go ahead and pop back over into the doc here and see how folks are doing. We're just about to wrap up our step three. It look like we have some fantastic information here that folks have shared. And, again, so it's really, here we're trying to sort of build this connection; right. That's why we call it backwards design. You know, what is success, how do you facilitate that, and how do you know it happened or not? And so as you go through these series of questions,, these are some things that you,


13

in your own minds, and for your institution, should ensure there is an alignment between these three things. I think a lot of times, if there's not this alignment, we lack a coherent strategy, and then sometimes, you know, we can have faculty development programs, which aren't doing what we are hoping they're going to be doing, or, like us, there's pending budget cuts, and we have a problem, you know, showing evidence that we are valuable as a unit, and, I mean, of course, we're all valuable, but sometimes it really helps these days, if you can -- and I hate using the term, show a little bit of return on investment. You know, one of the things that we'll talk about is sort of these institutional considerations for having success in faculty development. And a lot of times in the beginning people thought blended and online was going to be like this institutional moneymaker. But I think it's sort of like investing if your kids' education, you know, you make this investment in blended/online, you know, whether you necessarily see a return from it, I think it is something that many institutions are still asking. All right, so what we're going to do now is we are going to take a 15-minute break. I know that folks need to get up, stretch their legs, shake their body, get the wiggles out. Can you tell I have a six-year-old? And so we'll go ahead and we'll meet back here, I guess at, like, 1:38 -- no, I'm just kidding. We'll meet back here at 1:40. In the meantime, if you haven't had a chance to add your information to the three Google Docs for step one, what is success; step two, what facilitates success; and step three, for documenting success, please go ahead and do that. We'll jump back in here in a little bit and sort of move onto part two of today's event. And I appreciate everyone's patience and being such a great group and community, and overcoming our limitations here with free software and audio issues. So thanks, everyone, and we'll see you back here in a little bit. -- success and do the same thing and then I'm going to take step three, and then I'm going to wrap up the session today, talking a little bit about a series of questions that should go into your evaluation planning, which really focuses around step three. So I want to remind you, as Victoria has mentioned here in the chat, that everybody is off mute, so we can hear you. And please feel free to ask questions, either type them in the chat or go ahead and interrupt us, or if you have something to share, that would be wonderful. Right now what we want to do is we want to change our screen back to the presentation screen, and we are going to move on with our presentation for today to help guide us, if everyone wants to hold on for just one second. There it is. Whoops, I think a few of us are trying to move this at the same time. Let me push that down out of the way. All right. So time's up. Well I guess we already got that when my mic was turned off. All right, guys. All right, so step one, defining effective. You know, we talked about what is success and sort of working our way back from there. So, Nicole, do you want to summarize a little bit of what you saw in the discussion, and, Dylan, if you have anything to add in, feel free. All right, so in step one, we ask all you guys and gals out there to define what effective was. So what is good and blended or online course, and what pedagogical model facilitates a good blended or online course? There was a number of great comments out there, and I'll just try to highlight a few of them. One individual talked about the importance of comparability between online and face to face, so they should have maybe the same kinds of content, have the same kinds of things be important, so like interactivity and things like that. A number of you talked about the importance of learning objectives, so, that they should be clearly stated and that they should align with activities and assessments. A lot more of you talked about how important it is to have things like student engagement, participation and meaningful exchanges between students and instructors, and students and themselves, and, like, group projects and things like that.


14

And one of the final things I wanted to mention was the importance of -- that I saw in the Google Doc -- was the importance of making technology transparent for students and having a very organized course, and making sure that students can easily navigate the things that you're putting up there. I would also add that I saw quite a bit related to active learning as sort of the main focus that showed up in a few of those, and I think that's definitely something that we advocate for in our program that Nicole is going to talk a little bit about. Yeah. And I think building off of that, I think that's very important. You know, there are some folks, believe it or not, that will still argue that that they don't support the active learning model. And we do have some folks in certain disciplines, and just in general, that still attempt to drive a more teacher-tenured model online, which we know can be problematic, could lead to high attrition and those sorts of things. I also noted that a lot of you are sort of identifying, you know, what is a good blended/online course, you know, one that there's learning taking place, that there is engagement, that there is student satisfaction. I think these are all key variables that can help us in our evaluation planning if we know that these are the goals; that we want students to be engaged and we want them to be active and have interactivity and communication and social presence, and we want them to, you know, learn, and we want them to be satisfied. So I think those were some really great things, seeing in the doc, that are going help you plan that, you know. And then also that pedagogical model, we saw more of this authentic experiential learning, active learning, things being documented in portfolios that were more true to professional life, and those sorts of things. So I think that this was a really great first place for us to be to understanding what the definition of success is. Nicole is going to talk a little bit more here about our pedagogical model. But before we get there, I just wanted to sum up, you know, these are some things at UWM that we sort of think about with a good course. So we want a course with higher retention when they go through the faculty development program. We know from our some of our research that some folks try to teach that teacher-centered. We also want to have, you know, I'm going to call it "better grades," because on assessment some people call that "learning." Some people call it "performance." It depends on what you look at in the literature and in the research and the different research designs. I'm just going to call it what it is, it's grades. I think there is this underlying assumption of validity in assessments, but I think it's that, it's an assumption. I think also satisfied students, a lot of times we don't realize that student satisfaction is so important. The other things I already mentioned, learning engagement. And one I mentioned, too, which is, as a university, that's looking at decrease in enrollments, you know, one of the things that we're looking at is how can we offer new forms of online learning that will increase enrollments, and so although this is something that we're not necessarily systematically documenting how new course -- new online course leads to this, or how our faculty development program or working with faculty produces courses that leads to this, it's definitely something on our mind. I'm going to go ahead and turn it over to Nicole who will talk about our pedagogical model, which we've already talked about a bit, but she'll bring it home. All right, thanks, Tanya. So for our faculty development program and the pedagogical consultations that we do here at the Learning Technology Center, we really focus on the following pedagogical model that focuses on content, so things like text, images, audio and video interactive, which includes discussions, group activities, and feedback, and assessments, which, you know, we know that there's traditional written and oral examinations. There's cursive assessments and portfolios and things like that. And through all these components, we really try to focus in on getting instructors to really think about, you know, active learning, students at the center, instead of, you know, instructors being kind of the sage on


15

the stage and turning into more of a facilitator, so like a guide on the side, through really engaged learning, active kind of activities and doing so through research-driven effective practices that they can figure out through the literature, which we try to tell them, and also through their own course evaluations and the lifecycle of evaluation, which Dylan will talk about a little bit later. And I'll talk about each of them a little bit more right now. So for content, from the literature, from our experiences, and from student data that we've collected, we know content is really important for students; right. So one of the most common questions I think that we get when we run faculty development programs or consult with faculty are kind of questions around content. They're so concerned about content, how to get student it is content that they need to get them. And, you know, probably the most common questions that we see are, you know, I'm teaching online next semester, I'm teaching face to face and I want to video record my lectures and give students my 40-to-60-minute lectures because, you know, that's the way we think about things. Or they just come to us saying, "Oh, I want to do a voice-over PowerPoint because they still need to hear my voice in all these things." We talk to them a lot about what they really want to solve with what they want to do. But, in general, what students are kind of looking for is they don't want to sit through a 40-to-60-minute lecture that's not made for them. So we really try to tell them to really think about focusing content down on what does that student particularly need to know. Instead of 40 to 60 minutes, let try to focus around chunking around 10 minutes or so and get them the things from the instructor that they really need to know that way. We also focus on trying to advocate for, you know, using a variety of media; right. So, as you can see here on the slide, we have a video clip from Donna Mertens on research methods. This is a clip that I use in my class. She's a leader in the mixed methods field and just makes abstract concepts really, really clear. And what we found out from some student data that we just collected around this pilot that we're currently -- that we just finished up, was around a product called "Gingko Tree," where, as you can see here, it's the Holocaust Encyclopedia. That course utilized Gingko Tree, which was kind of like a course wrapper where you can put copyrighted material and open educational resources all together in one easy-to-access area for students. But from that study what we learned was that students really, really liked the use of videos, because it helped them to learn. One student even commented that it was something different and it made the class so much more exciting. So we advocate for a use of a mix of content, along with these current materials. So, you know, no longer do the students have to read about concept materials that were in a textbook ten years ago, but you can provide them with updated content materials. Yeah, it was very interesting. We were recently looking at the Gingko Tree study, and we found out there was a difference between the classes that use the copyrighted material or digital text, and then the classes that were using more the web links or the videos and those sorts of things. And come to find out that the students had a higher satisfaction and felt more engaged and those sorts of things when we were using this rich current content. And so, you know, it's very interesting that we can use this research, then, to drive our recommendations and pedagogical models. So one thing we really try to advocate for when we have the pedagogical consultations in our development program in our pedagogical model is this idea of interactivity, and a lot of you brought that up in the Google Doc as well, just how important interactivity between individuals in the course, so interactions with the instructor and with other students in the course is just so important, because it gives kind of this rich kind of learning experience. So this can be done multiple ways, and we tell instructors it can be done multiple ways. It can be done through, as you see there, introductory discussion areas. Can be done through a course overview like you see in the up upper right-hand corner. That's Tanya's pretty face there, talking about her course. It can also be done through other kinds of discussion areas like support areas, where, you know, people can post course questions and help each other out by answering them and sharing other kinds of scholarship, other things that they might be working on.


16

Social media, of course, is something instructors can use as well, like Twitter and Facebook, to keep up course pages and Twitter feeds. And we also focus in on the importance of giving students feedback. And this can be done on multiple levels; right. So like narrative feedback, you did great on this, this is something you should think about. Also, you know, of course the things that students look for is the numeric feedback; right, so they got five out of five or something like that. We also try to say, so, you know, you should have instructor interactivity but you should also have collaboration and peer interactions. So having small group projects or larger group projects that span the course of a semester that are scaffolded, those are all kinds of things that we kind of advocate for as well. Because I'm so -- I just was looking at the Gingko Tree data from our students, they gave us a lot of feedback on how they use that particular tool, and so many students commented how they really liked being able to read classmate examples, and it helped them to better understand their work in the topic areas, so really working with others to build a shared knowledge is just so important for students, especially in this day and age in the 21st Century learning. We also advocate for kind of a low stakes frequent assessment plan, so getting students engaged early and often during the course is really important. So it's low stakes, so if students miss one little thing, it's not going to totally break their whole semester. But it also is enough to encourage them to login at multiple times during the week or module to get them engaging with their fellow classmates, which has been shown to kind of aid their retention in online and blended courses. And we always try to get instructors, especially if we look at their assessment plans, to kind of avoid this catastrophic assessment, which you could see there with the big "just say no," I guess, through it where, you know, midterm and final examines account for a hundred percent of the grade. So, instead, we try to advocate for instructors to focus in on assessing their students in the different kinds of things like cognitive, effective and psychomotor and behavioral skills, and so also do things through both individual and in group assessments, so individual assessments like, you know, quizzes and individual writing and projects and group projects through, you know, discussions and, you know, longer term or short term for group projects. And, of course, it's important to always keep in mind if you're doing any kind of proof work to have both components of an individual assessment and assessing the group as a whole as well. Great. Thanks. Before we move on, really quick, I just wanted to mention -- oh, I thought I had pasted it in there. There is a link in here too. You know, Nicole went over the three different areas that exemplify our pedagogical model, in the content, the interactivity, and the assessment, you know, which we had developed very much separately. I recently had saw that Calvin had shared -- Calvin Thompson had shared University of Central Florida's pedagogical practice, which has their model and their practices divided into the same three categories, which is interesting. So I shared that link here, and then we'll also make sure to put that on the wiki for you as well. But if you're looking to really sort of identify the keys to your pedagogical model in these three areas to help you better, you know, design and develop your faculty development program, as well as evaluation planning for the effectiveness of your faculty development, this is a really great resource there. Okay, so step two, facilitating success, so this is how we can design and develop faculty development programs to help produce that pedagogical model and those outcomes from blended and online courses that you guys discussed. So I'm going to turn it over to Dylan so he can summarize some of the discussions there, and as well as -- and I think Kevin had already brought this up -- share with you all sort of what some of our faculty development program sort of characteristics are. Dylan? Thanks, Tanya. So the two questions that were in the second question was what elements and formats should be considered in designing and developing an faculty development program, and what opportunities and experiences should be available to help instructors learn effective practices and design delivery in teaching online and blended courses.


17

So looking through the Google Doc that you all participated in, I noticed that -- one thing that emerged was really about peer-to-peer learning within a faculty development program, so having instructors learning from one another and sharing back best practices throughout the program, I think, is vital, at least as it appeared on the Google Doc, and this is something that we do quite a bit as well. Interestingly, I was sort of looking at the format and time element, and there was one group that said, you know, you can't have workshops longer than an hour-and-a- half, ranging all the way to sort of full-day workshops, also, varying in terms of length, you know, some may be a single day or a single workshop leading up, or some are as long as 16 weeks. And one thing I should say about that is a lot of it has to do with sort of institutional expectations, you know, and how you can engage a faculty members or instructors, you know, in terms of what kinds of incentives they have in place, are these incentives where they're required to go through a certain a program in order to teach online, or is it a volunteer kind of thing? So those are definitely factors. Another focus was definitely active learning instructors who are sort of learning to teach online and blended need to be doing things. We need to be modeling active learning in those spaces. I thought one way that came up that was great was talking about flipping the faculty development experience, having more of the content pushed online so that if there are face-to-face discussions or activities, those are much more active, and, again, that's a model that we've sort of moved to. Another kind of group was talking about multiple methods and formats, so having, even within one institution in online program, a face-to-face program, a synchronous program, asynchronous programs. And I think that, you know, this is a great way to sort of target different demographics on campus. Of course that's going to depend on what kind of resources you have and what those needs are, and identifying those needs on campus, which would be, I think, an important process to undergo. So that's kind of my overall summary of what I saw in that Google Doc. I just wanted to thank everyone for also sharing resources in those Google Docs, and I encourage you all to sort of go back in there and read through what everyone has written, and maybe click on a couple of those links to find, you know, some more resources that would help you. Tanya, Nicole, anything you want to add? Oh, sorry, I had to find out if my mute button was on or off, but it's off. Sorry. That's like the skill of the day, finding your mute button. No, I think there were some really key themes there, just as you had mentioned. So I think the idea of delivering a faculty development program, providing a more experiential -- you know, experiential experience. That's funny. You know, this more experiential professional development opportunity for folks in delivering at that mode, and a lot about sort of that idea of peer-to-peer or community or those sorts of things, and supporting folks. You know, when we talk about this today, in my mind, I was talking about, specifically, our faculty development program, which Dylan is going to talk a little bit about. But this faculty development program that he's going to talk about is actually just one piece of a giant puzzle in which we support faculty for blended and online. So this is our official program, but we also do a series of one on ones, and a series of ad hoc workshops and certification program, and so, you know, I guess we need to look more at how we can better understand how all of those things impact faculty. But we'll talk a little bit about those implications and documenting and measuring those sorts of things a little bit later. Dylan, you want to talk a little bit about our program? Nicole, did you have anything to add, by the way? Just on the left side, Jennifer just brought up a really good point about, you know, the length of workshops in faculty development programs and sessions that, you know, some one-hour sessions are painfully long and some three-hour sessions can go so quickly. I think that has a lot to do with the kinds of activities that you have, you know, individuals do in those particular settings. If you just are constantly giving presentations and they're sitting there not active, then it could seem like a really, really long time, only if it's just an hour. But if you have them actively engaging with one another, you know, even a longer period


18

of time, like three hours, can just go by in a flash, which I don't want to steal Dylan's thunder at all, because I think he's going to talk about that a little bit. And I think we saw that in some of our data; right, and our research. Some people wanted, like, lots of one-hour meetings and some of them wanted more formal program. And we've even seen that in our attendees. Some folks rather just come to a dozen ad hoc workshops before they actually come to the program and so forth. Exactly. Thanks, Tanya and Nicole. So I'm going to talk about little bit about our faculty development program, like Tanya said. This has sort of been in, more or less, this form since 2007. We have about 70 to 100 instructors who come through every year. These are faculty members, teaching academic staff, teaching assistants. Sometimes even folks from nearby institutions as well. And so I'm just going to give you a bet of an overview of what we do so you can get some context for our discussion. What you can see here is kind of the overall model, the kinds of topics and concerns that we have in our faculty development program. Centered in the middle there, in orange, is transitioning. We're getting folks to move from a face to face to online or blended. The red box focuses on course redesign, which is kind of where we start to give them sort of conceptually what some of the major concerns are going to be as they continue to work throughout the program, defining learning modules, focusing on backward design, talking about learning objectives, and those kinds of things. And then we move to the right, along three of those boxes that Nicole went quite in depth into, related to content interactivity, and assessment. So these are sort of the major pieces that we work with in terms of that program. Also, the three that instructors are really most interested in in talking about and thinking about, like Nicole said, content, for example, is something that they may have some misconceptions about how to do that online. They may have quite a bit of anxiety about how to do that, and so we spend quite a bit of time focusing on that element. And then we also have built in quite a bit about the course administrative issues, so, for example, helping your students in an online or blended course, providing them with the expectations, what they can expect in terms of, you know, what an online course is, how it benefits them, how they can manage their time more effectively in these kinds of courses, which, of course, leads to retention, and then technology support, where they can go for help with that sort of thing. Same with sort of overall course management, where it's a little bit more focused on the instructor, so how they can stay organized and manage workload in order to avoid course-and-a-half, which I think is really a key, because this is something that, you know, we face quite a bit. I think all of us, in terms of, you know, feeling overloaded, and when folks come to us right at first it seems so overwhelming, so trying to provide those kinds of strategies, I think, is really a key. We also have course evaluation elements, so how will instructors in their courses be evaluating the effectiveness of their own course. So we talk quite a bit about progressive and summative student evaluative feedback, you know, self evaluation by the instructor, peer evaluation from either us in the LTC, or by other, you know, folks in their department who is have some experience in this area. I think another element that's really important to this, there's some familiar faces here, a couple people on the left there, Amy Mangrich and Jerry Brookston, they're our colleagues here at the LTC. These are everyone who is involved in our faculty development program, and I think the importance is that we all teach online and blended courses, and so we can come from both the experience of being instructors, we can bring the research that we've done in the area, and we can bring the years of faculty development experience we have and seeing what other folks have done to really help facilitate a conversation around online and blended teaching. We also represent all the different disciplines, really. We have social sciences, natural sciences, humanities, professions, arts, so we can address, I think, the different kinds of issues that folks will have in different kinds of disciplines. So it's really about -- you know, I think our teaching experience allows us


19

to both experiment in our classes, but then also it gives us little bit more different in terms of, you know, what we're presenting to them. So this is kind of an overall look at what the program is. We have, as you can see from this chart, we have sort of three face-to-face sessions these last three hours, and they're two weeks apart. So you can see that it runs roughly month one in terms of our program. And, of course, this is a blended model, so we have instructors doing online work as well, and the idea there is that they get the experience they need, as students, in seeing what that format is like, what kinds of activities they can utilize, in that environment. We do this program between four and six times a year, depending on the need. And, again, these are usually during the semester, you know, a couple over the summer, and they've been pretty successful in engaging folks. So we have these three face-to-face sessions where we do all sorts of different kinds of activities. Sometimes there are group discussions. We've moved really far away from doing presentations. If we do anything that's presenting, it's usually limited to just a few minutes, and then we'll do some sort of activity or discussion. And the online work that we ask them to do is really focused on being as practical as possible. So one of our outcomes is that they walk away from this experience having developed part of their course. You can also see that we have a showcase that we do at the end of the year, and that's, I think, a great way to encourage the peer-to-peer learning that was mentioned in the Google Doc, where, you know, people can see what great things, you know, others have done over the course, you know, having gone through the program, maybe taught online or blended, and to really get some good ideas there. And it was great for us, you know, the last time we ran this because we really got to see, in our minds, how effective our program was in helping these folks. I mean they did the work and did some wonderful things, and it was great to see that come back through. We also have a post program, so that, you know, we follow up with instructors. We have a deliverable that's due a month after the last session, and it's way to kind of touch base with them and make sure that, you know, they're continuing along their course redesign path. I should also mention that we don't have any firm financial or other incentives for our instructors to come here. We don't have any money to give them. They sometimes get them through seed funding or from their departments. But most people come through of their own free will and interest, and so that's why we've really targeted this number of sessions, keeping it a cohort model, but then also, you know, not trying to stray too long, otherwise we tend to -- we would probably lose folks. I think as we talked about earlier, backward design, we get some [inaudible] here. We advocate for this strongly in our program, and it's sort of the starting point by which we are engaging with faculty and instructors. But it's also informed our own process for redesigning. I guess this is a year-and-a-half ago, Tanya, Nicole and I sat down and sort of did a redesign process for our faculty development program, and I think it's been very successful. And we did this by looking back at what we wanted, what are our objectives were for this program, and then working back in terms of activities and content and things. So it works not just in a course, it also has worked for us in our redesign. Active learning as well, this is something that came up, and here is a picture of our new Innovation Hub and Active Learning lab, and it's where we held our faculty development program just this -- it just ended up last week. And, really, we were astounded by the difference that this space made over what we could usually find, which was more like a lecture hall or maybe a classroom with some desks. And, you know, we don't really have the data or evidence to suggest that it was completely this space, but I think we -- I know that we had way far greater attention. Almost everyone participated in all of the online activities and did all of the work that we asked them to do, which can sometimes be difficult to do. And it also allowed to us to really model active learning in the classroom as well. So we do this in the online assignments, but then we can also do this for people teaching blended or face-to-face classes, we can model what we think are effective active learning practices.


20

So, again, this is practice-based, so as I mentioned before, we really want them to walk away with something that is usable; right, so that they have part of their course design and they can continue along the way. So we have them create -- by the end of our program, they all will create a learning module, complete learning module, in a desire-to-learn sandbox course that they can copy over into their real course when it runs. And, you know, this practical approach, I think, has been one of the reasons we've been very successful, so we want them to really get started in redesigning that course, developing that course material, and then get the teaching skills that they're going to need to effectively implement this -- implement their course when it eventually runs. And then finally, our program evaluation, we have several ways in which we try to determine the effectiveness of our faculty development program, and we're going to talk a little bit more about this. But this grid sort of shows we have students -- or we have our participants doing self reflection during every face-to-face session. If you look on the wiki, I added a link to our document that has all of these. It's called "UWM LTC Faculty Development Evaluations." And these are sort of classroom assessment techniques, where we're asking them to reflect on what they've learned, what they might be confuse about. So it can help guide us where they may be still having some trouble and gives us good feedback about that. We also do -- the purple box there is facilitator self reflection, so we also do a more formal process of writing up how we, you know, determine how the sessions went, what kinds of improvements we may make in the future, if we need to, and then have a future discussion around that. The green box, progressive participant evaluation, is kind of built into those self reflections, and then the blue box there, summative participant evaluation, we have kind of a lengthier, more comprehensive evaluative instrument that we use in order to get their feedback on the course. We also kind of focus on, you know, the five main program outcomes that, as we're thinking about how -- the extent to which our faculty development program is meeting the needs of these participants, we really, again, these five, you know, focus on starting a redesign course, have the they acquired new teaching skills and knowledge? Do they know what to expect in an online or blended course? They often have quite a bit of questions that we want them to get answered. And then in blended particularly, the extent to which they are rethinking both the face-to-face and online components and not just sort of, you know, thinking they can sort of do what they have done in the past, that even in a blended course, a redesign process is really necessary. So we use these kinds of survey instruments, we use the materials that they've collected, and I think Tanya is going to talk a little bit more about all of that. But I just want to give you an overview of what we do for that and our focus on the main program outcomes. I think we're on to step three. All right, thanks everyone. I was getting so engaged in our conversation in the chat area. There's really great stuff going on. There's obviously a lot of interest in the specifics of how we do our faculty development program and those sorts of things. Please know that there's more information on our website, ltc.uwm.edu, which is on the wiki, and which is on the PowerPoint slides, which are also on the wiki. You can click on online and blended and find out more about our faculty development program, or please feel free to reach out to any of us. I think for those of you who have faculty development programs -- and I should mention, we actually call ours now -- we changed it to call it the "Online and Blended Teaching Program," so we did not want to alienate any of the instructional staff that may not be tenured track faculty or tenured faculty on our campus. So our Online and Blended Teaching Program you could call and talk to us. We'll get you information about it. Otherwise, if you have a program on your campus, please feel free to throw it in that step two Google Doc or throw in a link to it or throw in a description of it, because I think a lot of us are just still trying to develop our programs. We're trying to improve our programs.


21

You know, first, we had the blended faculty development program here 13 years ago. When I came onboard, we developed the online faculty development program, and it's a constant improvement cycle, because the needs are changing of our faculty and we're growing as an institution, and so I think it's very important for us to share in our faculty development practices, which, you know, hopefully lead us to step three, here, which was documenting success. So the question that we're asking is how will you know when faculty are providing quality in online blended courses? What tools or services could be provided for evaluating the effectiveness of online and blended courses, and how will quality be communicated to others? You guys can jump in the Google Doc and see what folks have posted. But just to overall summarize, and what I'm going to talk about here moving forward, is a lot of you were talking more about documenting at the course level, and so, you know, does this individual course -- is this a good course, you know, is it engaging, and these sorts of things? You know, some of you talked about using quality matters rubric, which is very popular, although, because of academic freedom, having such prominence at our institutions, quality matters just didn't seem like a realistic process, so we have created something else, which I'll talk about just in a minute. But, you know, a lot of times we're just looking at the course level, like this one course is a good course. This one course is a good course. But sometimes we're a little challenged to have a more systematic look. And, you know, at the end of the day, we just want to make sure that the learning objectives are being accomplished; right? And some of you talked about some new and innovative ways of documenting that. You know, I like the idea of using electronic portfolios, and electronic portfolios can obviously reflect whether learning has taken place, and the learning objectives were met. What I think is interesting, though, is looking at, you know, sort of this control variable. So when people don't go through faculty development, are the students still not learning then? And so it's one of those questions, like, how do we really know whether faculty development is effective, you know. And do we have a group of folks that aren't going through it that we could sort of look at or look at, you know, those different sorts of things. So do we know the faculty development program is what made it a good online or blended course, or maybe it's just the teacher or the discipline, or if we have all these other things to control for. So anyways, I don't have any answers. Well I might have a few. But I just want to throw out, these are all of the million thing that is are going through my mind, which was why I was really interested to do this session with you all today. All right, so at the end of the day, what we want to know is, did it work? Okay, do you get the star, you know, or the badge that it worked, your faculty development program worked and you have produced success or effectiveness, whatever that is or whatever you have defined that as. And as I just mentioned, that could be done at the course level or the program and institutional level. Now we've been doing the course level for a while, so I'll share in some of our tools and strategies with that. But I also want to talk about taking a more program- and institutional-level approach to evaluating faculty development and/or producing documentation or evidence at that level that it's effective. So first of all, and it was great to see this, you know, it's lovely when I see in Google Docs when people are sharing some of the same strategies or philosophies that we have. So one of the things that we have -- and this is from, wow, back in 2007, we were probably using something earlier then. So this is several years old, and we called it sort of the lifecycle of evaluation. And so -- and I can't remember who, but somebody in the Google Doc had talked about evaluating the course before it's even delivered, during delivery, and after delivery, and so I feel like we're talking about birth here, the birthing of online courses. So, anyways, it's really great that there's different ways you can do evaluation, even before the course is launched. We use something called a "course evaluation checklist," which is adapted from the UC Chico rubric, which I'll talk little bit more in detail. But there's also ways that you can do evaluation during the teaching, and so, you know, obviously you can have your colleagues, or some people ask us to evaluate their courses for them in the different three different like lifecycles. But, also, it's great to get feedback from your students on the course. We do a lot of midterm assessments. And then after the course.


22

You know, traditionally, we don't do a lot of evaluation until the course is done, you know, and then we have these few measures on this student evaluation, and we're like, oh, whoops the course was a bad course, darn. Sorry for all of those students. So the great thing, by using innovation in our teaching and learning, it's just made us think a little bit differently about the idea of trying something new, changing and making adjustments as the course is underway, and really focusing more on student learning and an array of other variables. All right, so this is the evaluation checklist. You can access this on the wiki. This is just an example. I think there are six or seven axioms, different areas, and so this is something that the instructor can use before, during, or after the course. These are some effective practices, and, you know, I think this could very well be integrated into some of the University of Central Florida's new pedagogical practice document they put out. But at the end of the day, we actually use this as part of our faculty development program, not for them to just use once the course is done. We actually use this as a course redesign tool. So, again, going backwards, if this is the way they're going to evaluate whether they have a good course or not, why don't they start using this in the beginning. Now they might not necessarily have everything on this entire checklist, but the great thing is that this is really practical and it helps faculty identify what we call a "redesign plan," so what are the things that they find really important that they have to do, you know, the first semester. Now we have something called the "three semester rule." Redesigning and delivering an online course, or blended course, is very challenging, and we, you know, really encourage faculty to be forgiving of themselves. Realize these are new skills, new experiences. It's a work in progress, and I call it the "three semester rule." It takes three semesters teaching a course, online or blended, until you really get it where you want it to be. So using the evaluation checklist as a redesign plan will help them understand what the high priority things are for the first semester and then the semesters beyond. This is also an instrument, then, that can be used to evaluate the course before it's delivered, during, or after, either by the instructor themselves. I'll tell you I go through this checklist a week before my class starts every semester, and there's always something that I'm missing or forgot to put, so it's a really great handy thing, and it's just really practical. We found that faculty don't want to be lectured at. They don't want to hear about theory. Although you can tell them that a practice was informed by theory. They want things that they can walk away with without having to, you know, earn a PhD in learning theories. So this works out very well for us. So this is one tool that you could use. As I just mentioned, as a course is going on, we strongly encourage these midterm evaluations, so asking students what's one thing they like and what's one thing they don't. You know, we talk a lot about instructors experiencing what we call "course-and-a-half syndrome." I think about 12 years ago we coined that here at UWM, and all of us, although we're aware of it, still experience it. We published a chapter back in 2007, in the "Blended Research" book, and it was funny, we went back to faculty. We've done our faculty development program all over the country, and we went back to different institutions and interviewed their faculty, and all of them said that they still ran to course and a half, even though we told them about it. So it's a great way -- you know, for me it's always the three nights I didn't go to sleep because I was still upgrading that I realized it might be course and a half, and I just thought my student were whining. But this is a great way to really streamline your processes by using these things. These are all, by the way, really practical tools that faculty can use at the course level. We'll get into some other things here shortly. Now what we did is we developed what's called the "Handbook for Peer Evaluation of Online and Blended," again, a link to this is on the wiki, and maybe we can pull some of those links in there in the chat area for you guys who are interested in them as well. So what we did is we took the UC Chico rubric, which is in part, the checklist, we took quality matters, we took all of those sorts of things and we developed this handbook for evaluation of online and blended courses, and let me talk just a little bit about that.


23

Quality matters, I think, in the initial reports I heard from Purdue years ago, I think maybe it was one of the initial schools, it just felt like it didn't have enough flexibility to work for our faculty at our institution. You know, we believe that we hire really great faculty that are really wonderful, and, therefore, you know, to feel that we need to police them or require something or these sorts of things, we don't really take that position. We're really there to provide a faculty support when need be in order to gain, you know, skills and expertise and their teaching just overall. So our approach in this was sort of doing an environmental scan, seeing what's out there, and pulling in the different pieces of the different instruments and processes that were out there and developing this. I think it's -- I can't remember how many pages this is, but maybe it's about 20 pages. But this is definitely sort of for us to train faculty who are going to do evaluations. We also use this in our own evaluations. So, for example, as part of our certification program, those that receive a certificate in blended and online -- as you see Merriam Merris [ph] in the upper right-hand corner -- they then can go through training and become an evaluator and Evaluate other folks' courses in their area using this Handbook of Peer Evaluation." We also have some departments or some program that actually require it. So, for example, our translation program requires that all of the courses go through this process. And that's at a unit level. That's up to programs to make decisions about how they're going to do things, and so, therefore, either we will evaluate those courses for them, we've trained evaluators in their program, or there's others with similarly disciplinary backgrounds that can provide that evaluation. And so, again, this is another course-level tool that folks can use. So traditionally when you teach a course, and I should mention, you know, I talked a little bit about the face to face being a gold standard, you know, in some units they don't evaluate the face-to-face courses. So it's very interesting that we want to evaluate the online courses, and the face-to-face courses sometimes have a different instrument than the online courses should be. And so I think institutional, especially one as sort of decentralized as ours is, you need to think a little bit differently about we actually don't even have a standard evaluation tool, I should mention. Every program has its own student evaluation for courses. So I think sometimes we just need to be careful. And in my department, communication, when I taught my -- when you teach any course for the first time in the department as tenured track faculty or whatever, another colleague will review your course and provide you feedback. That's sort of the model we took here. So I think it very much represented some of the culture that already existed. Now this is how we're going to wrap up the half hour today. So we talked about some different course-level tools that you could use to evaluate course, either as an individual faculty member or things that you could do to evaluate the courses of the faculty and/or instructors that went through your program. So this is sort of a little bit different, so I wanted to switch gears a little though, and talk about more developing an institutional evaluation plan for faculty developments here in the last half hour. And I know that there's very few of you that I could see in the doc that were maybe sort of looking at the overall impact of faculty development on student satisfaction, on grades, on retention, and those sorts of things. So this is moving up a few levels here, possibly. Now everything we're going to talk about could be done at the course level, but it's coming more from a post positivistic viewpoint here. So what I have done is I have some questions up here to help you with your evaluation planning. These are questions that, boy, I've probably been using for about eight years, and researching, not just blended online but researching learning technologies in general and helping other institutions do these same things. So very practical advice here. Now if you go back to the wiki here, we actually -- let me just share that quick. This would be the fourth document there. Let me fix that in. Oh, that fits so nice. All right, so let's go here. There we go. All right, so if you go back here to the wiki and scroll to bottom, I'm going to walk through these different questions that should guide you in more formal and strategic evaluation planning. But as I was doing that, I thought it would be great if you could each answer these questions for yourself, you know, ever so


24

briefly, and so folks could review those. So if you go down -- wow, I went way down. Woo, reign it in, Tanya. Reign it in. So if you look under activities on that first page of the wiki, and go ahead and click on the developing and evaluation plan, and I'll paste this link right here into the chat as well. There you go. And everyone can grab a table in the doc, and we'll see whether we can crash it or not, again. Ha, ha, ha, just kidding. So and, too, if you want to scroll down and copy and paste it at the end so you're not running into everyone. What I'd like each of you to do is type in your name, your institution, and then walk through these questions. If you want to copy it and throw it at the end so nobody is over you, but I think it would be great to see what other people are answering in regards to these questions to help inform your decisions. You know, sometimes I'm like, "Oh, I have the answers to these," but then you see somebody else's, and you're like, "wow, I never thought about that. Maybe I should be doing that." So Barbara is getting typed right here in the beginning, from University of Pittsburg. Go ahead, everyone, and type in a name and your institution. And I'm going to walk through just some considerations. I'm going to switch back now to the PowerPoint. But as I'm going through discussing these, please feel free to, you know, post in your answers there. All right. Yeah, let's hide that. Excellent. All right, so here we are, back to the evaluation plan. And maybe this is the communication person in me, but, you know, I think whenever we're doing anything that we really need to think about who our audience is. So for example, you know, maybe it's just for me, and maybe I want to know if I have a good course or not, or maybe there is a bigger institutional demand, and I think you need to think about the different audiences and who is going to see this data, because it can have some implications, positive and negative. So, you know, are you going to share this data just internally? We do lots of data collection and analysis here that never leaves our doors, and it's just for us so that we can improve our faculty development program and know whether we're having an impact. But sometimes you need to think, you know, where is this going beyond this? Are we going to share it with the program leads? Are we going to share it with the chairs and deans? Are we going to share it with the provost and chancellor? Are we going to share it with students? We always forget them as an interested audience. And then for me, as I'm doing right now, you know, today in this event, as we're all here today, you know, are you going to share it all out beyond your institution? So are you going to share it on a national stage? Obviously you guys know I'm all about sharing, you know, documenting and sharing and documenting and sharing. So, you know, and you know, where can you share these sorts of things out, because the audience is greatly going to impact your evaluation planning process, and so you need to maybe identify your primary audience or your secondary audience, and then we're going to, you know, sort of go through some other questions and you can think about, okay, again, you're preparing a message. You're going to prepare some documentation of some evidence for this audience. So if you're in the Google Doc, go ahead and type in who your primary audience is, maybe who your secondary audience is. I love it when somebody on my team will come and say, "Tanya, who is this for?" And I'll be like, "Well, you know, first and foremost, it's fur us and our faculty," and then I list about 15 other audiences. So, anyways, just something to think about. The second question has to do what are the variables? I know variables is, you know, a very funny word. From a social scientist, I get really excited about this. And this is isn't all inclusive by any means, but I think that it can get you thinking about some of the different things that are out there. I use a little bit more of a systems approach with the input process and output, because this is just how my sick mind thinks all the time about things. But I think that this is very important when you look back at what was success for you in identifying, you know, what variables, because your variables are very much, again, going to lead you to make other decisions. And if you don't know what your variables are, you're going to have a problem, you know, collecting data and analyzing data and those sorts of things. So, for example, some of you said that success for you was engagement. All right, then engagement is one of your variables. Some of you talked about satisfaction, student satisfaction, you know, then that's going to be one of your variables there. So it's good to identify what sorts of things that you need to, you


25

know, be concerned with. I get excited about all of them, which is why I probably never get anything done. All right. The next piece of this, as we just said, is if you know your variables, how are you going to collect data? So, you know, for example, if you're going to do engagement, you know, how are you going to collect whether this class was engaging? This is a hard one, you know. So we can look at it and you could identify that this class may be based on what's in the LMS, was engaging or interactive or those sorts of things. You can have that either qualitative or anecdotal, observation of classes that are underway, or those sorts of things. But otherwise, if you're going to do more quantitative measures, it really takes either a student survey on engagement or a faculty survey on engagement. I sometimes like to get both the student and the faculty's perspective. But with collecting data, there can be some challenges there, you know. Are you an expert in survey development? What surveys are out there that you could use? This is a little one behind there, you can't see. But, you know, I have taken items from, like, the Nace and different engagement surveys out there from Q and from Pace and pulled them into -- because majority of our technology solutions here at the LTC really focused on engaged learning. And, of course, because I'm a social scientist, I think you have to be social in order to learn and interact and all of that good stuff, so I'm a little bias. But I think most of you would agree with me. So, you know, where can you get this type of survey? Who can you collaborate with on getting these survey instruments so you can collect data? If you're go doing it in a more qualitative, a systematic qualitative way and you're going to go into courses and look at some of the texts -- by texts I mean anything, you know, that you can interpret that's in a course site, you know, then that's another area, but can you get access to the course sites? We're very lucky that our unit administers the LMS, so we have access to all course sites in order to go get that data. Of course there's going to be IRB implications, depending on how you're going to use it. But, anyways, just something to keep in mind. So when it comes down to identifying who is your audience, then you need to think about what are your variables, remember, like what's success, and then thinking about how are you going to collect data. Now another way, which was popular at the ELI session in New Orleans, focused on this idea of grade and retention data or attrition data. I know retention we think of retaining, learning. Sometimes I view the retention as staying in a class, which can be called "attrition" as well. So, you know, those are two really important institutional measures. You know, are the students getting decent grades? At University at Central Florida they call it "success," a grade of C or better. I like to call it "grades" or "performance," but maybe that's with my background and managerial comp. And then the other thing is retention, so are the students staying in class? You know, it's very interesting, there's very, very, very little published research out on retention because I don't think a lot of people, although we all know it anecdotally, want to report that online classes sometimes can have higher attrition, you know, because of missed expectations, technology problems, and so forth. And so, however, you know, this is something that we should be, at this point, taking into account in our faculty development programs to create a pedagogical model that keeps students engaged; therefore, there is going to be higher retention, less attrition and that they're going to be doing better in these classes, because as my students have told me and we're hearing institutionally with some studies that we've done, is that students take online classes not only to push time, but because they think the learning is better than it is in the face-to-face classes, and we believe that, because in the online classes, you know, for years and years now, we've more pushed the student-centered active learning, this engaged learning model, which we didn't always see in the face to face. Not to say there weren't any good face-to-face courses. I'm just saying we're seeing a push more for that now. So go ahead in your document, again, write your audience, write your variables, and write how you're going to collect the data. We need to get the grade and retention data from the data warehouse we're very lucky to have worked out with our institution that we can go ahead and pull that data, which actually led to our chapter in the book that Veronica had talked in the introduction, so we have a chapter in "Blended Learning Research Perspectives," which is edited by Tony Picciano, Chuck Dziuban, and


26

Charles Graham, in which we talk about looking at faculty who went through our faculty development program versus those who didn't, and then looking at the impact on their students' grades and on their retention. And we actually found statistical significant in grades; meaning, you know, the students of those who went through our faculty development program did better. And we only had a .058, so, you know, by a very small fraction there, we didn't find statistical significance with attrition, but it was very close, and we're doing a bigger study. So we're hoping we'll find some good stuff. The other thing is how you're going to analyze the data. Even if you're not a genius at survey instruments, you can usually find one out there. Even if you don't know how to analyze, you can usually make the connections and contacts you need with people at your institutional research or the IT people that hold your data hostage -- no, I'm just kidding, guys -- and get grade and retention data for those courses. But the bigger thing is how you're going to analyze data. And so we're very lucky that not only are all of us at our unit teachers and we support faculty, but a lot of us are researchers in different ways. And so we know quantitative and qualitative analysis and some other analyses and all that good stuff. So we can usually run some univariate, multivariate statistics, you know, looking at the relationships that we want. Sometimes it's just really easy when you do surveys, though, and this goes back to who your audience is, is just doing some basic descriptives, like you see in this bar graph up here. So, in the beginning when I got here to the LTC, there were no social scientists or quantitative researchers, it was some ethnographers and those sorts of things. But for me, I think numbers can be convincing. So, you know, using easy tools like just summing them up, we have Qualtrics as our survey tool. Qualtrics gives really great reports. You could put the survey results in very consumable formats in PowerPoint, which is a really easy-to-use tool. So even if you don't have the skills and resources to analyze, using some of the univariate or multivariate statistics, or doing thematic analysis using qualitative methodologies, you still have the opportunity to present data in a descriptive way. And I'll tell you, you know, one of my first sessions I went to, leaving the field of communication and coming over here, and I remember giving a presentation and talk about my multivariate analysis, and this multiple regression I run. It was very interesting, because the audience, I don't think, had any clue what I was talking about. So that's, again, how thinking about who your audience is really helps you answer some of these questions. So if I am going to the ELI or the EDUCAUSE conference, I will usually use descriptive data like this bar graph. I may follow up with an article that we're submitting for publishing if people are interested in hearing about some more of the advanced analysis. Now we're going to talk a little bit here about human resources, but if you don't have the skills to analyze the data, there are some resources on your campus that you can tap, like PhD students, to help you with that. And I'll talk a little bit more about this here. The other thing is thinking about, as I was saying, like, where are you going to present this, or where are you going to disseminate it? So I was just talking about, you know, if you're going to share your results at the ELI conference next year, at EDUCAUSE this fall, you know, you have to go back to who your audience is and these sorts of things, and what sorts of things are they going to understand. I think also, too, you know, we have things locally. You know, what groups can you share your results with on campus? We have something called the "Online Program Council," so many times we share that out. It's program leads, folks who have online programs that are coming to learning more about stuff. The other thing is, you know, maybe you want to publish. We definitely need more scholarly research that's published in these new and innovative areas. Not only about your results but about your methodology. I don't think there's a lot of sharing about the different methodological approaches, even also mixed method approaches that help us better understand the phenomenon, and so I don't think we should devalue that. I think we should definitely prioritize that, especially when we see the new and evolving interest in research like MOOCs and other forms of mediated learning. So thinking about that is always something, so please note in your research -- your evaluation plan different areas where you think you might disseminate. Woo, woo, we're moving there.


27

And as I mentioned, I didn't want to jump ahead here, but think about who your human resources are. So do you have PhD students? Can you contact PhD advisors in the different programs, like communication or psychology or sociology, and have them -- you know, have folks that can intern for you, or that, actually, you can pay a very -- you know, a TA or a graduate student does not require a large salary to do really magnificent work on your analysis that you can use. So definitely -- and the better thing is is you are giving you graduate students at your institution practical real-life experience, which can be very hard for them to find. So other human resources is reaching out to your IT folks that are manning your data or that might have the survey tool, reaching out to folks in education and structural communication or other areas that might have survey instruments for you. Obviously we're a human resource, all of us here on this event today. And I really look forward to continuing this conversation, because I hope we can come together as a community and share our methodologies and share our plans and share our instruments and our tools and so forth. And as we are running out of time, this is an important one. You know, I usually have, like, 20 research questions, things that I want to know, you know, did our faculty development program increase retention. But I tend to have so many questions that I forget to prioritize, and then we just run out of time and run out of time, and we don't allocate the right resources. So I think it's very important that you develop a timeline for your efforts, you identify when you want to be done by, when you're going to deliver or disseminate to which audience. Sometimes it really helps, like today, like signing yourself up for a presentation where you'll have to share out that information. But also, you know, move backward from that time line, schedule time. Again, time is the most valuable commodity. I mean we never have enough time to do all of the evaluation that we want to around here. Schedule time, engage the right human resources and stakeholders in that process, you know, and realize that time is exactly that, there's not a lot of it, and you need to be a little flexible and make sure to prioritize. All right, I just wanted to touch quickly in the last few minutes here, before we end today, on some of the challenges that you might run into. And so one of the things I just want to take a little step back is challenges and faculty development altogether, you know, one of the things that's come out of our ensuring quality sessions is the idea of incentivizing and motivating faculty. It's very hard to have an effective faculty development program if the faculty aren't really engaged or it's a top-down, you must do this process. So there's some institutional or infrastructure challenges that you might have to do deal with in determining this. And so you might want to think about those, even before you're thinking about -- or how that's going to impact your evaluation. I think another challenge, like I just mentioned, is time and focus. You really need to make it a priority to evaluate your faculty development program and to harness the resources to make that happen. I know we talked about human resources. I should have mentioned engaging with faculty and co-authoring. One of the limitations to that, though, is sometimes, unlike in the field of communication, we actually have a whole track for instructional communication. But, you know, in biology, while there is an instructional biology, in some of the disciplines they might not have, say, engineering, such an interest, and it would be harder to collaborate with faculty in certain disciplines on this sort of research. IRB, for us, has not been a challenge, but I've heard it's been a challenge on other campuses. Most of what we do is research that's exempt. It's already taking place in common educational practices. But I know some other campuses have a bit of a challenge with that. And just some other challenges, you know, access to your data sometimes can be an issue, whether that's in a data warehouse somewhere or through a lack of expertise around survey tools. So think about when you're doing this backwards design process and doing your evaluation planning, always be thinking about how you're going to get that data. I remember Chuck Dziuban and Patsy Moskal from University of Central Florida, I think it was, like, eight years ago, maybe longer, we were doing a session on evaluation of blended, and, you know, they're like, the only bad data is the data you didn't collect. So start collecting data now any way that you can,


28

whether it's art facts that could be, you know, looked at qualitatively or some sort of rubric to assess them, or it's actual survey data, you know, collect it. We have lots of survey data I don't even know if we'll ever analyze, but at least we have it. And that's the final thing. Data analysis, I know, can be really hard, because we, as faculty development people, are expected to, you know, evaluate our programs and do this rich, you know, multifaceted evaluations, but a lot of us don't come with those skills. And so I think in that way, you know, find those folks on campus that are really great at this, that you can connect with and look to partner with them, as well as us, as a larger community. So those are just a quick things there that I wanted to address at the end. So I will mention Victoria has pasted in the chat the survey. Please, please complete the survey. And I just wanted to thank you all for coming. I'll try to go through if there's any questions and answer those. But, again, thanks you all. Have a great day and look forward to connecting with you in the future. Great. Thank you. Thank you, Tanya. And on behalf of all of our participants, I want to thank all of our speakers that participated today for sharing their perspectives and their insights. This was very valuable information, so many thanks for that. We very much want you feedback on today's seminar and showing now on the screen is the URL that's going to take you to a one-minute survey on the session, and we encourage you to complete the survey now while the details of the seminar are still fresh in your mind, and we thank you for doing that. The session has been recorded and will be available for later viewing by you or anyone from your institution, and you won't need any special login, just look for the link in an e-mail that will be coming later today. And we look forward to having you at our next ELI event, and I wanted to remind you all, yes, of the focus session, and also, earlier, I mentioned all the other faculty development events that will be taking place this year. And you can find all of those events on our event calendar. On behalf of my associate, Malcolm Brown, this is Veronica Diaz. Thank you again very much for joining us today.

ELI Online Seminar: Determining the Effectiveness of Your Faculty Development Program (236670470)

Documents

Transcript of ELI Online Seminar: Determining the Effectiveness of Your Faculty Development Program (236670470)