PROGRAM EVALUATION PROJECT 2013 PRELIMINARY FINDINGS.

PROGRAM EVALUATION PROJECT 2013PRELIMINARY FINDINGS

INTRODUCTION

• This presentation aims to share some of the preliminary findings that were reached as a result of the data collected and analyzed for Program Evaluation Project – 2013.

• The slides mainly have tables and graphs, and following each slide, we added some explanations to help interpret them.

• The rest of the findings were presented in the slides used on 18 September, 2014.

CONTEXTUAL VARIABLES

This slide shows the components that are subject to evaluation at the School of Foreign Languages. The main components are the two main departments, and the EPE is viewed as a component in between the two.

INITIAL PHASE OF PROGRAM EVALUATION 2013

Needs AssessmentThe existing curriculum and syllabusThe materials in useThe assessment toolsThe learnersThe instructors (DBE, DML, Faculty)The resources

DBE MLD Faculty

Ss Q Ins Focus Group Int Interviews

Ins Q Ss Q Materials Analysis

Ss Focus Group Int Classroom Obs

Ins Interview

Classroom Obs Document Analysis

Materials Analysis Ss Product Analysis Freshman Int

Ss Product Analysis Ss Interviews (?) Freshman Product

Analysis

DATA SOURCES AND DATA COLLECTION INSTRUMENTS

• This slide shows the departments from which data were collected, the sources, and the data collection instruments used.

• The darker color used in the boxes in the lower half of the slide indicates that the data collection procedure is still in progress.

COMPLETED JOBS

Tools DBE MLD Faculty

Questionnaire

2,612 (ss)

95 (Ts)

~800

Interviews 20 29

Focus Group Interviews 45 11

Classroom Observations 15 5

PENDING JOBS

Qualitative Data Analysis

Freshman student samples

MLD Student samples

DBE Student samples

Interviews

Observations

PENDING JOBS

Quantitative Data Analysis

MLD Student Qs

PRELIMINARY RESULTS FROM DBE

Levels Ins # Participation %

Ss # Participation %

Beginner 24 48 % 682 83 %

Elementary 35 67 % 1,043 88 %

Intermediate 21 84 % 429 76 %

Upper-Intermediate 6 100 % 118 89 %

Repeat 9 50 % 328 76 %

Total 95 63 % 2,612 82 %

PRELIMINARY RESULTS FROM DBE

• This table displays the number of students and instructors who filled in the questionnaires and the degree of participation in the two groups.

Listening Reading Writing Speaking Pronunciation Vocabulary Grammar3

4

5

6

7

Importance vs Adequacy of Program Coverage of Skills & Components (INS)

Importance Adequacy

IMPORTANCE VS ADEQUACY OF PROGRAM COVERAGE OF SKİLLS & COMPONENTS (INS)

• Both the instructors and the students were asked to rate the skills in terms of their importance and the adequacy with which they are covered in the program.

• This graph compares the instructors’ responses to “importance” and “adequacy of coverage” of individual skills.

• Please remember that 1 indicates “strongly disagree”, 4 indicates “undecided”, and 7 indicates “strongly agree”.

• In this line graph, the only area where the degree of importance and the amount of coverage are seen as close to equal is “grammar”. Instructors perceive grammar as important and adequately covered in the program.

CONT.

• Instructors’ views indicate that they rate “reading” and “vocabulary” as the most important areas, followed by listening, writing and speaking, and grammar (in descending order). Pronunciation is rated the lowest, with a score above 5.5, indicating that it is considered “important”.

• The coverage of the skills and subskills (except for grammar) is seen as below the importance ratings.

• There is a small difference between the importance (at around 6.5) and adequacy of coverage of “listening”, “reading” and “writing” (at around 5.5) skills.

• A larger difference emerge in “speaking” (importance=6.25 but adequacy of coverage=3.88). We may say that instructors believe that speaking is an important skill to master but the instructional coverage is not quite adequate.

• Similarly, differences exist between importance and coverage in “pronunciation”, “vocabulary” as well.

• The following five slides have two aims: • to show and compare students’ and instructors’ views of the

degree of importance and the degree of adequacy of coverage of skills and sub-skills in their group’s program. • The top lines indicate “importance” and the bottom lines

indicate “adequacy of coverage”. • To be able to interpret the lines, it would be a good idea to have

a look at lines that cross each other or move away from each other in one graph. Then, have a look at those in the other graph. Compare and contrast. Please keep in mind to check the scores (from 1 to 7).

Speaking Writing Listening Reading Grammar Vocab Pronun3.0000

3.5000

4.0000

4.5000

5.0000

5.5000

6.0000

6.5000

7.0000


3.5000

4.0000

4.5000

5.0000

5.5000

6.0000

6.5000

7.0000

BG Ss BG Ins

EXAMPLE: BEGINNER GROUP

Students’s views

• Speaking: most important BUT least adequately covered

• Listening and vocab: very important BUT not so adequately covered

• Grammar: important BUT covered too much

Instructors’ views

Speaking: important BUT not adequately covered

Vocab: important BUT not so adequately covered

Grammar: quite important BUT covered a little bit too much


4.0000

4.5000

5.0000

5.5000

6.0000

6.5000

7.0000


4.0000

4.5000

5.0000

5.5000

6.0000

6.5000

7.0000

EL Ss EL Ins


3.5000

4.0000

4.5000

5.0000

5.5000

6.0000

6.5000

INT Ss INT Ins


3.5000

4.0000

4.5000

5.0000

5.5000

6.0000

6.5000

7.0000

7.5000

UIN Ss UIN Ins


4.5000

5.0000

5.5000

6.0000

6.5000

7.0000


4.5000

5.0000

5.5000

6.0000

6.5000

7.0000

REP Ss REP Ins


3.5000

4.0000

4.5000

5.0000

5.5000

6.0000

6.5000

7.0000


3.5000

4.0000

4.5000

5.0000

5.5000

6.0000

6.5000

7.0000

Click icon to add picture

Level Taught BEG EL INT UIN REP

Mean Mean Mean Mean Mean

I have my students do pair or group work activities. 5.68 5.63 5.76 5.75 5.29

I have my students work individually. 5.23 5.54 5.33 4.50 5.29I assign activities that involve group-work outside class. 2.09 2.54 2.33 1.75 2.57

I assign activities that involve individual work outside class. 6.09 5.60 4.90 5.25 5.29

I make use of posters, flash cards, realia, etc. 4.91 4.17 3.20 2.25 3.71

I make use of games, contests and puzzles. 5.00 4.34 4.19 4.75 4.00I lead a teacher-centered classroom. 3.55 4.00 4.00 4.00 4.86I involve my students in role play/scenario activities and discussions. 5.27 5.14 4.57 5.25 4.57

I involve my students in whole-class debates. 4.50 4.91 4.86 5.50 4.14I encourage peer teaching in my classroom. 4.77 4.46 3.86 3.50 4.00I have my students discover grammar rules from texts. 5.50 5.12 4.75 4.50 5.14

I explain the grammar points myself. 5.00 4.74 4.62 4.75 5.67I use online resources for my students to practice English outside class. 3.77 3.71 3.76 4.75 3.86

I make use of technology in class. 5.82 4.94 4.81 4.75 3.71I have my students use their own technological devices 4.82 4.00 4.76 4.50 3.71

I regularly allocate time to revise input. 5.27 4.80 4.90 4.50 5.00I make use of weekly office hours. 5.09 4.89 4.05 2.25 4.29I have my students provide feedback on each other’s work. 4.95 4.15 4.14 3.50 3.86

I provide the feedback on student output myself. 6.00 5.94 5.81 5.75 6.29

• This table shows the DBE instructors’ preferred styles in teaching. The list includes activities and tasks that they can have students do during the lesson.

• Each activity is rated from 1 to 7. 1 indicates “never”, 4 indicates “half of the time”, and 7 indicates “always”.

• Each colon on the right shows the average score of a group.

• It would be a good idea to examine groups one by one and to locate the highest and lowest scores in each.

• Then, findings commonalities and differences among groups may indicate trends in the department.

• Pairs of activities that seem related gives some idea about the preferred styles in teaching (e.g.. The first two activities in the list).

EXAMPLE

• The most frequently preferred activities among beginner group instructors seem to be “I assign activities that involve individual work outside class” and “I provide the feedback student output myself”.

• The least frequently preferred activities among beginner group instructors “I assign activities that involve group work outside class”, “I lead a teacher-centred classroom”, “I use online resources for my students to practice English outside class”.

• The common points among all groups are: • the high score in the last item: “I provide the feedback on student output

myself”. • the low score in items 3 and 13.

BEG EL INT UIN REP0% 0% 0% 0% 0%

13%0% 5% 0% 0%

4%20% 10% 17% 13%

13% 11% 24% 17% 25%

9%

40%19% 33%

13%

48%

26%38%

33%

38%

13%3% 5% 0%

13%

I regularly allocate time to revise input.

“I REGULARLY ALLOCATE TIME…”

• This graph shows us the responses of instructors to the item “I regularly allocate time to revise input” according to groups.

• Since the question was related to frequency, the scale was in that nature.

1= never, 2= rarely, 3=occasionally, 4= half the time, 5= often, 6= frequently, 7= always

• In each colon, the color on the top (maroon) indicates 7 = always. The other colors indicate frequency in descending order. Thus, lilac indicates 4= half of the time. No one indicated 1= never, therefore, there is no color to show it.

• Example: In Beginner group, more that half of the teachers allocate time to revise input at 6 & 7 (frequently and always).

BEG EL INT UIN REP0% 3% 0% 0% 0%0%

11%10%

0%

25%

9%

11%14%

17%

25%

13%

9% 14% 33%

13%

17%

23%24%

17%

25%

17%

14%

24%0%

0%

43%

29%14%

33%

13%

I make use of technology in class.

MOST IMPORTANT SKILLS

S P W R L G V

BG S/T T T S/T S/T

EL S T S/T S/T

INT S T S/T T

UIN S/T T T

REP T S T

MOST IMPORTANT SKILLS

• This table aims to display differences and commonalities between the views of the students and instructors regarding the importance of skills.

• One thing to remember is that this does not mean to say the groups do not find the other skills as unimportant. ALL emerged as important in the graphs; however, we wanted to see which one/s came out as the “highest”.

• BLUE: Only students; RED: Only teachers; MAUVE: both groups

• S: speaking; P: pronunciation; W: writing; R: reading; L: listening; G:grammar; V: vocabulary

• Example: Reading is seen as the most important skill in all groups by instructors but this skill is not seen as the most important by students.

LEAST IMPORTANT SKILLS

S P W R L G V

BG T S/T

EL T S

INT T S/T

UIN T S S/T

REP T T S

2.58 2.58 4.01 4.16

2.54 2.58

4.13 4.58

6.82 6.3 6.22 6.21 6.03 5.954.94 4.82

Importance of factors affecting language im-provement

Ss Ins

IMPORTANCE OF FACTORS AFFECTING LANGUAGE IMPROVEMENT

• This bar chart displays the students’ and instructors’ rating of the importance of factors in language improvement.

• The bars on the right (light colored bars) indicate students’ ratings and the bars on the left (dark colored bars) indicate instructors’ ratings.

• In the rating scale, 1 indicates “totally unimportant”, 4 indicates “undecided”, and 7 indicates “very important”.

• Example: Students do not regard “reading texts in English” as an important factor but the instructors do.

• Example: Students’ highest score was “studying together”. This may indicate that they actually prefer and benefit from studying with someone else, perhaps with a “buddy” or “a big sister or brother”.

DBE SS - FGISKILL/AREA

WRITING 2 drafts, more often, more variety in topics

READING MTR good, more dep. related texts / extensive rd

SPEAKING extra time in program

LISTENING more practice / more challenge

GRAMMAR less detail & terminology

VOCABULARY more practice – in class

Main book: redundant / tasks too simple /DBE own book ? Lessons: less book following / more student activityHomework: more HW / more guidance for self-study

DBE SS - FGI

• This table shows the preliminary findings obtained from the focus group interviews held with the students at DBE.

• The ideas that emerged in all the groups (from Beginners to UIN) were summarized here.

• Only the suggestions and ideas have been displayed here.

FACULTY REPRESENTATION

Architecture 4

Arts and Sciences 12

Economic and Ad. Sciences 3

Education 2

Faculty of Engineering 8

Total 29

• This table shows the number of instructors that have been interviewed from different faculties.

• The tables in slides with the title “faculty needs” provide a list of the tasks and activities that the students are required to perform in the first year first semester in their departments.

FACULTY NEEDS

WRITINGAssignments Response paper (1-3 pages)

Report (3-10 pages)Assessment(MT & Final) Short answer (1/2 page to 1 page)

Essay (2-6 pages)Multiple ChoiceTrue FalseFill in the Blanks

Recurrent Theme: Paraphrasing, Comparing, Analyzing, Synthesizing

FACULTY NEEDS

READINGAssignments Book chapters

ArticlesRecurrent Theme: Need to compare, analyze, synthesize, read long texts

FACULTY NEEDS

LISTENINGOut of Class Documentaries (BBC, CNN, etc.)

Films

In Class Lectures

Seminars

Recurrent Theme: Problems in understanding different accents (international instructors and native speakers), speed

FACULTY NEEDS

SPEAKING

In Class Conversational skills

Presentations

Recurrent Theme: Lack of confidence, too grammar focused

FACULTY NEEDS

GRAMMAR

Expectations Simple present & past tense

Adj clause, noun clauseActive and passive structures

Recurrent Theme: Short simple sentences

FACULTY NEEDS

VOCABULARY

Expectations Knowledge of an acceptable range of vocab

Knowledge of collocations

Recurrent Theme: Discipline-specific vocab should be taught by content course instructors

MODERN LANGUAGES DEPTSKILL/AREA ?WRITING Prg org & concepts Content (minors) READING Low level skills Higher level skillsSPEAKING A bit Not muchLISTENING A bit No note-taking (?)GRAMMAR A bit Simple mistakesVOCABULARY A bit Simple level

Points: It would be a good idea to know how much is covered at DBE so we can make adjustments in our tasks/expectations.

• This slide displays the findings reached from the interviews conducted with two groups of instructors at the Department of Modern Languages.

• It lists the points that are seen as “done well” and those that are “not done so well” by students in the first year first semester courses.

CONCLUSION

Complete the analyses

Factual information from the faculties

How do we share instructional goals w/ the MLD?

Exchange information with all parties involved

Make use of research conducted at DBE (previous studies)

WHAT’S NEXT

2014-2015 Fall & Spring Term

SFL Mission

SFL Curriculum

DBE Goals & Objectives

DBE Syllabus

Materials Development

Assessment Tools Development

As always

Thank you.

PROGRAM EVALUATION PROJECT 2013 PRELIMINARY FINDINGS.

Documents

Transcript of PROGRAM EVALUATION PROJECT 2013 PRELIMINARY FINDINGS.