THE EVALUATION OF THE SEVENTH-GRADE ENGLISH …I should also thank to my first English teacher in an...

THE EVALUATION OF THE SEVENTH-GRADE ENGLISH TEXTBOOK

WHEN ENGLISH RINGS THE BELL: A SURVEY RESEARCH

A THESIS

A Thesis Presented to

The Graduate Program in English Language Studies

in Partial Fulfillment of the Requirements

for the Degree of Magister Humaniora (M.Hum.)

in English Language Studies

by

Teguh Ariebowo

116332027

THE GRADUATE PROGRAM OF ENGLISH LANGUAGE STUDIES

SANATA DHARMA UNIVERSITY

YOGYAKARTA

February, 2017

i

PLAGIAT MERUPAKAN TINDAKAN TIDAK TERPUJI

Member

------

---

ATHESIS - '

-----~ --- --

Presented hy, -

-~~

_; =1 -=- -::: l

~- - =-1 -- -j

=~~ - :_:

-

" ·- -----

- ~

THE EVALUATION OF THE SEVENTH-GRADE ENGLISH TEXTBOOK

WHEN ENGLISH RINGS THE BELL: A SURVEY RESEARCH


v


ACKNOWLEDGEMENTS

In the name of Allah The Most Gracious and The Most Merciful. Praise and

thanks are due to Allah, who has given me strength to complete this thesis. I also

wish to express my gratitude to the following for their contribution throughout the

completion of this study.

I owe an enormous debt of gratitude to F.X. Mukarto, Ph.D., my supervisor,

for all his patience, continuous supervision, invaluable guidance, encouragement,

help, and feedback all the way through the processes of writing this thesis. My

gratitude also goes to Dr. J. Bismoko and Dr. B.B. Dwijatmoko for the inspiration,

feedback, and suggestion so that I am able to revise this work and make it better. I do

not forget to say thanks to all the lecturers and staffs in the Graduate Program of

English Studies, Sanata Dharma University, Yogyakarta for the knowledge and skills

that I have got throughout the undertaking of my graduate study.

I am also sincerely grateful for having wonderful experience with all English

teachers who enthusiastically participated as respondents of this research. Without

their contribution and participation, this research would have not be completed. Those

supportive and cooperative teachers are from SMP N 1 Sleman, SMP N 1 Galur, and

SMP N 1 Gunung Kidul.

I should also thank to my first English teacher in an English course, Ms. Erni

Dhukun and Ms. Pipin for introducing English to me in an interesting and fun way.

The gratitude is also addressed to Ibu Henny Herawati who gave support and

involved me in fruitful discussion about teaching which was able to enlighten me in

finishing this work.

Finally my special thanks are due to my beloved parents, my lovely wife and

my siblings. Thank you for giving me unstoppable support to accomplish this

mission. Last but not least, great thanks also go to people who I impossibly mention

those people one by one. Thank you for invaluable source of support and inspiration

for my over years studying here.

vi


TABLE OF CONTENT

STATEMENT OF THE ORIGINALITY .................................................................... iv

LEMBAR PERNYATAAN PERSETUJUAN PUBLIKASI KARYA ILMIAH

UNTUK KEPENTINGAN AKADEMIS ..................................................................... v

ACKNOWLEDGEMENTS ......................................................................................... vi

TABLE OF CONTENT .............................................................................................. vii

LIST OF FIGURES ..................................................................................................... ix

LIST OF TABLES ........................................................................................................ x

LIST OF ABBREVIATIONS ...................................................................................... xi

LIST OF APPENDICES ............................................................................................. xii

ABSTRACT ...............................................................................................................xiii

ABSTRAK ................................................................................................................. xiv

CHAPTER I INTRODUCTION ................................................................................... 1

A.

B.

C.

D.

E.

Background ........................................................................................................ 1

Problem Identification and Limitation ............................................................... 4

Problem Formulation ......................................................................................... 5

Purpose of Study ................................................................................................ 5

Benefits of the Study .......................................................................................... 5

CHAPTER II REVIEW OF LITERATURE................................................................. 7

A. Theoretical Review ............................................................................................ 7

1.

2.

3.

4.

Textbooks ....................................................................................................... 7

Evaluation..................................................................................................... 16

Review on Indonesia National Curriculum 2013 ......................................... 23

Review of Previous Studies.......................................................................... 26

B. Theoretical Framework .................................................................................... 28

CHAPTER III RESEARCH METHOD ..................................................................... 30

A. Research Method ................................................................................................ 30

B. Research Respondents ........................................................................................ 31

C. Research Instrument ........................................................................................... 32

vii


D. Data Gathering Techniques ................................................................................ 32

E. Data Analysis ...................................................................................................... 33

CHAPTER IV RESEARCH FINDINGS AND DICUSSION.................................... 35

A. General Findings ................................................................................................ 35

B. Detail Findings.................................................................................................... 37

1.

2.

3.

4.

5.

6.

The Aim of the Textbook ............................................................................. 37

The Approach of the Textbook .................................................................... 39

Design and Organization of the Textbook ................................................... 46

Content of the Textbook ............................................................................... 51

Language Skill Presented in the Textbook ................................................... 54

Topic Presented in the textbook ................................................................... 56

C. Discussion ........................................................................................................... 59

1.

2.

3.

4.

5.

6.

The Aim of the Textbook ............................................................................. 59

The Approach of the Textbook .................................................................... 61

The Design and Organization of the Textbook ............................................ 62

Language Content of the Textbook .............................................................. 66

Language Skill Presented in the Textbook ................................................... 66

The Topic Presented in the Textbook........................................................... 67

CHAPTER V CONCLUSION .................................................................................... 70

A. CONCLUSION ................................................................................................ 70

B. Suggestions ...................................................................................................... 73

Bibliography................................................................................................................ 75

Appendices .................................................................................................................. 78

viii


LIST OF FIGURES

Figure 2.1 The Theoretical Framework ..................................................................... 29

Figure 4.1 Sample of activity found in textbook: Activity 5 ..................................... 41



Figure 4.4 Aim of Chapter 1 ...................................................................................... 48

Figure 4.5 Sample of activity found in textbook ....................................................... 53

Figure 4.6 Sample of a simple lyric and a very simple dialog ................................... 56

Figure 4.7 The new version of Bloom's taxonomy .................................................... 63

ix


LIST OF TABLES

Table 2.1 Criteria of an Effective Textbook .............................................................. 15

Table 2.2 Checklist Suggested by Rea-Dickins and Germain ................................... 22

Table 4.1 Overall Survey Result ................................................................................ 36

Table 4.2 Survey Result on the Aims of the Textbook ............................................. 38

Table 4.3 Survey Result on the Approach of the Textbook ....................................... 40

Table 4.4 Survey Result on the Design and Organization of the Textbook............... 48

Table 4.5 Complexity Level of the Activity .............................................................. 51

Table 4.6 Complexity Level and Number of Activity ............................................... 52

Table 4.7 Survey Result on the Content of the Textbook .......................................... 53

Table 4.8 Survey Result on the Skills Presented in the Textbook ............................. 56

Table 4.9 Survey Result on the Topic Presented in the Textbook ............................. 58

x


LIST OF ABBREVIATIONS

C13

TEFL

EFL

WERB

Curriculum 2013

Teaching English as Foreign Language

English as a Foreign Language

When English Rings the Bell

xi


LIST OF APPENDICES

Appendix 1: Questionnaire Draft 1


Appendix 3: The Blueprint of the Questionnaire

Appendix 4: The Respondents’ on Statement

Appendix 5: The Respondents’ Answer on Open Question Questionnaire

Appendix 6: Translated transcript of the interview

79

90

98

106

110

115

xii


ABSTRACT

Teguh Ariebowo. 2016. The Evaluation of Seven Grade English Textbook “When

English Rings the Bell”: A Survey Research. Yogyakarta: English Language

Studies. Graduate Program Sanata Dharma University.

As the process of curriculum development, Indonesia developed a new national

curriculum in 2013 which is called Curriculum 2013 (C13). This new curriculum raised a

great discussion and debate among people of Indonesia. The textbooks of any subject which

were made based on the C13 then became problematic since its basic competence has

become tremendous concern of textbooks especially the English textbook. Therefore, the

researcher focuses on the grade textbook of secondary school to be evaluated. In this

evaluation study, the researcher try to find out to what extend textbook meets the criteria.

This study belongs to a descriptive quantitative research. This research took the data

from the survey among the teachers of English using this textbook in their classroom. To be

able to conduct the survey, the researcher developed a questionnaire distributed to 17

teachers as the respondents in the Special Province of Yogyakarta. When this research was

conducted, there were only few schools appointed by the government to implement the C13.

The questionnaire administered the Likert using scale 1 – 4. The statements were developed

from some criteria of effective textbook based on the experts for instance Brian Tomlinson

(2008), David Nunan and Cunningsworth (1995). The criteria include a) the aim, b) the

approach, c) the design and organization, d) the language content, e) the skill and f) topic

presented in the textbook.

The findings showed that some criteria were fulfilled in the textbook but some other

criteria were not. According to the survey result, the aims of textbook met perfectly and

suited with the aim of the C13. The approach implemented in the textbook met the criteria

quite well although some respondents had no idea of what approach they should adopt in this

curriculum. A different finding happened in the design and organization of the textbook. In

some aspect of this criterion, the respondents saw that textbook did not meet the criteria very

well. Different attitude found in responding language content of the textbook. All

respondents did not appreciate the language content. Then respondents saw that the topic

presented in the textbook met the criteria so much well.

In conclusion, the researcher sees that the textbook has some strengths and

weaknesses. The presented topic, for instance, is one of the strengths since it is suitable with

the learners’ level and social condition in Indonesia. The illustration found in the textbook

represents the gender, ethnic, and religion diversity in Indonesia. However, the textbook is

like other books which always have pitfalls. One of the most obvious pitfalls in textbook is

the language content since this book does not have grammar reference and its level of

difficulty way too simple for grade students. Finally, the researcher suggests material

developers to revise this textbook and to English teachers to always conduct evaluation

toward books or new materials before utilizing them in classroom. Hence teachers are able to

choose the most suitable textbook for their learners.

Keywords: Effective textbook, Textbook evaluation, Material Evaluation.

7th

7th

xiii


ABSTRAK

Teguh Ariebowo. 2016. The Evaluation of Seven Grade English Textbook “When

English Rings the Bell”: A Survey Research. Yogyakarta: English Language

Studies. Graduate Program Sanata Dharma University.

Sebagai bentuk proses perkembangan kurikulum, Indonesia mengembangkan

sebuah kurikulum nasional yang baru pada tahun 2013 yang disebut Kurikulum 2013

(K13). Kurikulum baru ini memunculkan diskusi dan perdebatan besar di kalangan

masyarakat Indonesia. Buku-buku teks pelajaran yang disusun berdasarkan K13

kemudian menjadi masalah disebabkan kompetensi dasar yang menjadi kekhawatiran

besar dari buku-buku teks pelajaran khususnya Bahasa Inggris. Dengan demikian,

peneliti berfokus pada buku teks pelajaran Bahasa Inggris untuk siswa kelas 7 SMP

untuk dievaluasi. Dalam kajian evaluasi ini, peneliti mencoba untuk mencari tahu

seberapa jauh buku teks bahasa Inggris tersebut memenuhi kriteria.

Kajian ini digolongkan dalam sebuah penelitian kuantitatif diskriptif.

Penelitian ini mengambil data dari survei yang dilakukan di kalangan guru-guru

Bahasa Inggris yang menggunakan buku teks pelajaran ini di kelas. Untuk

melaksanakan survei ini, peneliti mengembangkan sebuah kuisioner atau angket yang

dibagikan kepada 17 guru di DIY sebagai respondennya. Ketika penelitian ini

dilaksanakan, hanya terdapat beberapa sekolah yang ditunjuk oleh pemerintah untuk

mengimplementasikan K13. Angket tersebut menggunakan skala Likert dari 1 – 4.

Pernyataan-pernyataan dalam angket dikembangkan dari kriteria buku teks yang

efektif yang berdasarkan dari beberapa ahli seperti Brian Tomlinson (2008), David

Nunan (1989) and Cunningsworth (1995). Kriteria tersebut meliputi a) tujuan, b)

pendekatan, c) desain dan organisasi, d) bahasa isi, e) keterampilan dan f) topik yang

disajikan di dalam buku teks.

Hasil dari survei menunjukkan bahwa buku teks tersebut memenuhi beberapa

kritera dan juga beberapa hal lain tidak memenuhi kriteria. Berdasarkan hasil survei,

tujuan buku teks berkesinambungan dengan tujuan K13. Pendekatan yang

diimplementasikan di dalam buku teks cukup memenuhi kriteria walaupun beberapa

guru yang mengisi angket tidak mengerti dan memahami pendekatan apa yang

seharusnya mereka gunakan di dalam kurikulum ini. Berbeda dengan hasil jajak

pendapat mengenai desain dan organisasi buku teks bahwa di beberapa kriteria buku

ini tidak dapat memenuhinya dengan baik. Sangat jauh berbeda dengan hasil jajak

pendapat mengenai bahasa isi dari buku teks tersebut di mana semua guru yang

menjadi responden bahwa mereka bersikap kurang setuju dengan bahasa isi dengan

xiv


buku teks tersebut. Kemudian, di bagian topik, para guru berpendapat bahwa topik

yang disajikan sudah memenuhi kriteria buku dengan baik.

Pada kesimpulannya, peneliti melihat bahwa buku teks terebut memunyai

kelebihan dan kekurangan. Topik yang disajikan, misalnya, merupakan kekuatan dari

buku teks ini mengingat bahwa topik yang disajikan cocok dengan tingkatan

pembelajar dan kondisi sosial masyarakat Indonesia. Ilustrasi yang ditampilkan dalam

buku teks ini memperihatkan keanekaragaman gender, etnis dan agama di Indonesia.

Namun demikian, buku teks ini tidak luput dari kekurangan seperti halnya pada buku

teks lainnya. Salah satu kekurangan buku teks ini yang sangat terlihat jelas adalah

tidak adanya referensi tata bahasa yang bisa dijadikan rujukan oleh pemelajar dan

tingkat kesulitan bahasa yang digunakan dalam buku ini terlalu mudah untuk

pemelajar tingkat 7 SMP. Pada akhirnya, peneliti menyarankan kepada pengembang

materi untuk merevisi buku teks ini. Selain itu paling penting adalah menyarankan

kepada setiap guru Bahasa Inggris untuk melakukan evaluasi terhadap buku-buku

setiap sebelum menggunakan buku-buku tersebut di dalam kelas. Dengan demikian,

para guru dapat memilih buku teks yang cocok bagi para pemelajar di kelas mereka.

Kata kunci: Evaluasi materi pembelajaran, evaluasi bahan ajar, evaluasi.

xv


CHAPTER I

INTRODUCTION

This chapter comprises the elaboration of the background of the study, the

problem identification and limitation, the problem formulation, the purposes of the

research, and the research benefits.

A. Background

It has been declared by experts that material evaluation is significant in

education field. First, Rod Ellis (1998) underscored that it is mandatory to evaluate

every single book to teach a language. Second, Ellis (1998) illuminated that the

evaluation of a textbook is important because it aims to see how far it has worked and

is a partial procedure of a macro evaluation of curriculum development. Third,

Sheldon (1988) said that a textbook evaluation needs to be administered since a

textbook which is imposed on parties, both learner and teacher, by a higher authority

(in this case the Indonesia Ministry of Education) and no room for change and

modification will result a discontent. From what experts say, it can be concluded that

an evaluation process of a textbook is fundamental because it affects effectiveness of

teaching and learning process. If it happens that textbook is discontent for students

and teachers and it is kept to be used, there will be a hole in the teaching and learning

process. So, textbook evaluation is needed to be a patch on the hole through inputs

and feedback resulted from the process for its revision.

Another reason why this research is about material evaluation is because

textbook is an important resource for (1) presenting language components, (2)

1


2

activities, (3) stimulating ideas, (4) self-directed learning, and in some contexts, (5)

used as a syllabus and a support for less experienced teachers (Cunningsworth, 1995).

Agreeing to Cunningsworth (1995), Tomlinson (2008) stated that textbook evaluation

is a crucial process that can have a direct effect on the teaching procedure. Textbook

evaluation can be carried out to help a publisher or an institution to make decisions,

to help ourselves in developing and selecting textbook, to find out if the book being

used meets the needs of the students and to try carry out a research project (2003).

Without having good materials to teach, the program will not run very well.

Therefore, the writer sees evaluating material is absolutely important and useful for

teachers, stakeholders and the program.

There are two other reasons why teachers, experts and material developers

should do a material evaluation besides determining whether the material “works” or

not. Empirical evaluation of teaching materials can be found in accounts of trialing of

new materials (Bernard and Randall, 1995). In this case, the evaluation of teaching

materials is taken when there is a new material used in the teaching and learning

process. Hence, the main purpose of the teaching materials evaluation is to determine

whether the materials are suitable for the teaching and learning process. Another case

is when the teaching materials are evaluated to find out whether it is worthwhile

using the materials again. In such case, the evaluation is taken when teachers, experts

and practitioners plan to evaluate the existing materials. Since it has been explained

above that in Indonesia there is a new teaching material for the C13, this evaluation

research is conducted to see the teachers’ perspective toward textbook. Last but not


3

least, according to the need analysis survey in TEFL/TESL teachers and researchers

considered textbook evaluation to be one of the most important components of a

teacher training course (Hutchinson & Waters, 1991).

The Indonesia Minister of Education decided to load the new in 2013. Hence

in 2013, Indonesia government imposed the new curriculum called Curriculum 2013

or C13. The C13 was implemented for some rationales mentioned by the Minister of

Education (2012). Those are current global challenges, required competencies, and

current negative phenomena especially among young people and discouraging

perceptions among Indonesians regarding education (Hamied, 2014). However,

Karnodi (2014) in Hamied (2014) stated that the first implementation of C13 in 2013

seemed to raise different reactions among teachers in Indonesia. Some with defiance,

as many of them still argue that C13 is a type of curriculum developed in a top-down

and centralistic fashion. Therefore, the researcher points that a textbook evaluation of

C13 is in the urge to be conducted.

In conclusion, this present study is evaluating the new English textbook used

in seventh grade of junior high school in Indonesia mainly concerning that material

evaluation is very important today to determine to what extent a material has worked

and broader is to facilitate the whole process of curriculum development (Ellis,

1998). When we talk about evaluating teaching material, or textbook, it means that

we talk about textbook as a whole package. However, this present study limits the

dimension of evaluation. The limitation of the evaluation will be explained in the next

subtopic.


4

B. Problem Identification and Limitation

As mentioned earlier that the C13 raised different reactions among teachers

toward its administrative matters to pedagogical matters. Administrative matters

include dissemination of the curriculum, budget, distribution of textbooks and teacher

training (Weir & Roberts, 1994). Some questions in the evaluation may be raised

such as on how the dissemination of the curriculum will be arranged, how much

money that the authority should be put in the budget for the whole program, and

many other critical questions towards the program. Classroom teacher behaviors,

teaching method or the learning process, classroom learner activities and the teaching

materials take part in the discussion of pedagogical matters. This explains what Ellis

(1998) suggests in his journal that program evaluation covers administrative matters

and curricular matters where materials evaluation, teacher evaluation and learner

evaluation employ in it. These evaluations later will be called as macro-evaluation.

More specific than macro-evaluation is the micro-evaluation. This will involve more

specific topic for each aspect mentioned above. For instance, in material evaluation,

task evaluation can be done in micro-evaluation.

Too many aspects must be evaluated in macro-evaluation. As a matter of fact,

there are only small numbers of a complete holistic evaluation that involves various

kinds of information relating to one or both administrative matters (i.e. the logistical

and financial underpinnings of the program) and curriculum matters which later can

be broken down into materials, teachers, and learners. Hence, this study evaluates a

more specific subject. A micro-evaluation will be undergone to examine the English

textbook of junior high school in Indonesia. In this case, this research will focus only


5

on textbook published by the Indonesia Ministry of Education entitled When English

Ring the Bell (WERB) for seven grade level. Then, to evaluate the textbook, the

researcher determined the criteria that a textbook should meet. The textbook criteria

will be elaborated in detailed in chapter 2.

C. Problem Formulation

To evaluate the tasks in the English textbook of 7th grade of junior high school

using 2013 curriculum WERB research questions are raised as follows:

1. To what extent does textbook meet the criteria?

2. What are the strengths and weaknesses of textbook according to the survey?

D. Purpose of Study

The purposes of this study are, first, to find out to what extent the textbook

meets the criteria and to find the strengths of textbook and the weaknesses of

textbook according the users. Second, the researcher then can make any justification

based on the empirical data and the theory whether the textbook is suitable to be used

as the main material, needs additional material, is suitable as the additional material

or needs to be revised. Then the researcher then can make suggestion to teachers who

use this book, to material developers and to future researchers related to textbook

evaluation.

E. Benefits of the Study

7th This evaluation research on English textbook of grade of junior high

school is absolutely beneficial for material developers (in this case the authority),

teachers, learners, and stakeholders.


6

For materials developers, this research evaluation is beneficial as reference in

developing textbook to design better and more effective textbook. The material

developer in this case the authority may refer to what has been discussed in this

evaluation research.

For teachers, this research evaluation will give detailed explanation on how

the textbook meets the criteria. Consequently, teachers may respond towards the

findings. Then teachers can take any anticipated action to take a positive response to

do in relation to the findings.

This research will also be beneficial for stakeholders since they can use result

as a reference to develop the whole English program in junior high school. Besides,

this particular evaluation research will be valuable for other evaluators especially

program evaluators to help them evaluate English program for junior high school.


CHAPTER II

REVIEW OF LITERATURE

This chapter is mainly presenting the literature reviews related to the topic of

the research. First, theoretical review is elaborated briefly to build the foundation of

this research. Second, related studies are presented and explained in this chapter to

point out what have been discussed so far by other researchers and evaluators. Last, a

framework is presented in the end of this chapter to give a clear construct of this

research.

A. Theoretical Review

This section reviews relevant constructs and critically analyzes their relations

in building up theoretical framework of the research. Based on the research problems,

it is recognized into three major constructs, namely English textbook, Evaluation in

English Language Teaching, and Indonesian curriculum of 2013.

1. Textbooks

Commonly, textbooks are a book about a particular subject that is used in the

study of that subject especially in a school. Those are commonly used by the learners

as well as the teachers in a school that provides structured syllabus of the course

program (Ur, 1991). Usually, the books are accompanied by teacher’s book as the

manual in teaching the subject and workbook to support learners in practicing the

subject.

7


8

a. Definition of Textbooks

Before going further talking about course books evaluation, this section

initially talks about the subject of the study which is a textbook. As a matter of fact

textbooks are taken for granted in some places while in some other places textbooks

are not used at all. In such case, textbooks are only used as the supplementary

material. However, some teachers select textbooks and use them in their classes (Ur,

1991). In other words, in some situation, textbook is used selectively and it is used to

as supplementary materials in the English course program. Therefore, it is important

to clarify the definition on what it is called as a textbook.

It seems that Ur’s (1991) definition about textbook is the easiest to

understand. She gives a clear position of the term textbook when the discussion

comes to the necessity of using a textbook. It is explained that textbook is book of

which the teacher and each student has a copy, and which later on in the learning

process it is followed systematically as the basis of a language course.

Textbooks now become a must-have element in EFL/ESL teaching. Those are

widely accepted to be the heart of any English language situation (Sheldon, 1988;

Hutchinson and Torres, 1994). It is also argued that textbooks have become the

common element in the teaching of English as the second language or foreign

language (Tsiplakides, 2011). This happens because it seems that it is too difficult for

teachers to create their own teaching materials (Sheldon, 1988). Another reason by

Sheldon (1988) to justify the argument above is that textbooks lessen the preparation

time by providing ready-made teaching texts and learning tasks.


9

b. Roles of Textbooks

In our discussion about textbooks, a role play of a textbook is inseparable

from the discussion. Ur (1991) says that a textbook is a framework for the teachers

and learners making them know where they are going and what is coming next.

Therefore, there is a sense of structure and progress. A textbook can be also a

syllabus of an EFL/ESL course since a textbook should provide a syllabus in it.

Besides providing a syllabus, a textbook even provides texts and learning tasks.

Agreeing to Ur’s opinion (1991), a textbook is a source for learner practice and

communicative interaction (Cunningsworth, 1995). A textbook is a convenience in

teaching and learning EFL/ESL since in a textbook the components stick together in

order and structured (Ur, 1991). According to Ur (1991), a textbook also roles as a

guidance for inexperience teacher or for teachers who occasionally unsure of their

knowledge of the language. In other meaning, it can become a support for less

experienced teachers (Cunningsworth, 1995). Finally, a textbook may become a

reference and source where learners can use textbook to learn new material, review

and monitor progress with some degree of autonomy (Ur, 1991; Cunningsworth,

1995).

In conclusion, textbook plays a very important role in teaching and learning

process since from the above explanation of the experts; textbook is inseparable from

the teacher and student. Three important roles of a textbook in English course

program dominate in teaching and learning English in EFL context. They are as a

guide book for teachers and learners, a reference book for teachers and learners and a

ready-made syllabus for teachers.


10

c. Advantages and Disadvantages of Textbooks

In regard idea or concept, the discussion definitely dragged to search the

advantages and disadvantages of it. In the same time, advantages and disadvantages

of textbook will be described based on some experts of language materials

developers.

Richard (2001) states in his book entitled Curriculum Development in

Language Teaching that the advantages and disadvantages of the use of textbook

depend on how it is used and the context for its use. He concludes that textbooks

provide structure and a syllabus for a program, helps standardize instruction,

maintains quality, provides a variety of earning resources, and is efficient. Besides,

textbooks also provide effective language models and input, can train teachers and are

visually appealing.

However, textbooks may still potentially have negative effects (Richards,

2001) such as textbook may contain inauthentic language, distort content, not reflect

student’s, and can deskill teachers. Lastly, textbook is usually expensive.

Furthermore, textbooks fail to help the learners to make full use of the language

experience available to them outside the classroom (Tomlinson, 2008).

In conclusion, textbooks have important roles in teaching and learning

EFL/ESL. The relationship between teacher and textbooks is an important

consideration and is at best when it becomes partnerships when both share common

goals to which each party contributes to the teaching and learning process

(Cunningsworth, 1995).


11

d. Textbook Criteria

It is such an uneasy task to judge whether a textbook or teaching material, in

this case is ELT material, is good or bad (Tomlinson, 2008) since every textbook or

ELT material has its own strengths and weaknesses and there is no perfect textbook

exists. Instead of using the word good and bad, Cunningsworth (1995) used the word

effective and ineffective. Consequently, many experts and researchers have been

doing evaluation on textbook as part of the curriculum development process to find

the best possible one that will fit and be appropriate to a particular learner group.

Hence in this section of the study, the researcher is explaining what aspects of

textbook that are usually evaluated by the experts and researchers as well as teachers

to seek the best and most appropriate textbook for a particular group.

Cunningsworth (1995) suggested that it is significant to ensure that careful

selection is made and that materials selected closely reflect the need of the learners,

the aim, methods and values of the teaching program. To carefully select the

materials or textbooks, a predictive evaluation can be administered (Ellis, 1997) and

some criteria of a textbook should be considered (Allwright, 1981, Cunningsoworth,

1995, Richard, 2001 and Sheldon 1988).

Goal of a textbook seems to be an important criterion to be evaluated. In many

evaluation checklists from the experts such as Allwright (1981), Sheldon (1988),

Cunningsworth 1988 and Richards (2001) evaluative question regarding the goal of a

textbook is always put at the beginning of the checklist. As also suggested by Nunan

(1989), in developing teaching materials the objective, aim or goal of a textbook

should be generated in the first hand. In addition, the aims of a textbook should


12

correspond with the aim of the syllabus and curriculum (Cunningsworth, 1995;

Nunan, 1989 and McDonough and Shaw 1993:64). However, conflict of goals cannot

be avoided since teachers, learners, materials developers and sponsors involve in

teaching and learning management (Allwright, 1981). There is always a conflict of

teachers, leaners, material developers and sponsor goals found in textbook. Shortly,

Allwright (1981) argued that a textbook may contribute to some ways but it cannot

determine goals.

The approach of a textbook is also considered to be evaluated in a textbook

evaluation process. Cunningsworth (1995) and (Brown, 2001) suggested some

questions to evaluate this criterion whether a textbook is flexible to any teaching and

learning situation and style. In a slight different opinion to this approach issue by

Allwright (1981) was that in materials evaluation assessors or teachers should be able

to measure to what extent the management of the activities are maximally profitable.

However, he pinned important clause that textbook can and do contribute to the

teaching and learning process, including the teaching approach, but cannot possibly

cope with many important decision made by the teachers regarding their various

situation (Allwrigth, 1981). In conclusion to the discussion about the approach, the

researcher agrees to Allwright’s opinion that detailed local decision is clearly beyond

the scope of publication (1981). The materials or textbook then can only impose

suggestion to the teachers.

Human being loves everything which is beautiful. It is also not too much to

say that aesthetically positive experience through the use of attractive illustration and

its design can facilitate learners in language acquisition and development (Tomlinson,


13

2008) since it can arouse the learners’ interest (Rowntree 1997,92). A good material

must provide a pleasurable learning experience which can be achieved by practicing

learning items in an interesting and novel way (Richards, 2001). This can be done by

using attractive illustration and design (Tomlinson, 2008). Besides the design and

illustration, a good textbook should help the learners feel at ease (Rowntree 1997).

Therefore the sequence and the grading of textbook should be able to make the

learners at ease and suitable for the learners (Cunningsworth, 1995). In the same time,

Cunningsworth (1995) also believes that a textbook should allow the teachers to

undergo various teaching styles. In a simple way, a textbook should be flexible for

the teachers.

The next criterion of a textbook to be reviewed is the language content. In this

aspect of textbook, three main elements are described. Those are grammar,

vocabulary and phonology. Cunningsworth (1995) sees that those three elements are

important in term of learning the language. Therefore, in his textbook checklist, he

tried to find out whether a textbook provides adequate grammar item for the students

to learn autonomously and teachers as the source of grammar knowledge. Besides the

grammar item that should be presented in textbook, vocabulary in terms of quantity

and range must be adequately provided in a textbook. Another important aspect that

should be presented in textbook related to content is the pronunciation. A textbook

has to give teachers and students the pronunciation work in terms of individual

sounds, word stress and intonation.

Related to the content of textbook, language skills presented in textbook

should be carefully paid attention of since referring to what Tomlinson (2008)


14

suggests in his critical review book that an effective textbook must provide the

learners with the experience of extensive listening and/ or extensive reading. His

opinion definitely supports the input Hypothesis by Krashen (1981).

The topic presented in a textbook ideally should have enough variety and

sufficiently fulfill the learners’ interest. The topic presented in a textbook also should

be content-wise enough for the learners’ language level. The social cultural context

presented in a textbook is also being considered to the use of English in the real life.

A textbook must also consider the social structure such as gender group, ethnic

group, occupational group and other group such as disability (Cunningsworth, 1995;

Brown, 2001; and Tomlinson, 2008).

Many experts in textbook evaluation field share ideas regarding the criteria of

a textbook. They seem explaining and putting the name for each criterion in different

way. However, whatever the term they are using for each criterion, actually they are

referring to the same ideas. Therefore, in this study the researcher respectively

adapted Cunningsworth’s (1995) evaluation checklist which has covered all criteria

used in this study. The six aspects of textbook will be evaluated are its aims,

approach, design and organization, language content, skills and topic. In this study

the researcher came with six aspects a textbook should have. Each aspect should at

least meet with one criterion. The criteria used in this study are as follow:


15

Table 2.1: Criteria of an Effective Textbook

(Tomlinson, 2008)

source of grammar knowledge (Cunningsworth, 1995)

skills needed

Aim

The aims of a textbook should correspond with the aim

of the syllabus and curriculum (Cunningsworth, 1995; Nunan,

1989; and McDonough and Shaw 1993:64).

Approach

a. A textbook should be flexible to any teaching and

learning situation and style (Cunningsworth, 1995 and

Brown, 2001).

b. Material should take into account that learners differ in

learning styles (Tomlinson, 2008)

Design and

organization

a. attractive illustration and its design can facilitate

learners in language acquisition and development

b. Therefore the sequence and the grading of textbook

should be able to make the learners at ease and suitable

for the learners (Cunningsworth, 1995).

c. a good textbook should help the learners feel at ease

(Rowntree,1997 and Tomlinson, 2008).

Language

content

a. a textbook should provide adequate grammar item for

the students to learn autonomously and teachers as the

b. vocabulary in terms of quantity and range must be

adequately provided in a textbook (Cunningsworth,

1995)

c. A textbook has to give teacher and student the

pronunciation work in terms of individual sounds, word

stress and intonation (Cunningsworth, 1995)

Language

Skills

a. Integrated language materials should provide language

b. Materials should facilitate learners with adequate and

rich reading text (Tomlinson, 2008)

Topic

a. To achieve impact, a textbook should present appealing

topics of interest to the target learners. (Tomlinson,

2008)

b. A textbook should present topics which offer the

possibility of learning something new. (Tomlinson,

2008)


16

2. Evaluation

In the second language education, the main purpose of evaluation is ultimately

for decision-making based on the informed judgment. That is why enough data

gathering is highly important in the evaluation process (Richards, 1996). The

information can be gained from the programmer, administrators, teachers, learners or

the stakeholders as part of the implementation of a program and can be used as the

program development.

a. Definition of Evaluation

As it is mentioned above, the ultimate purpose of evaluation is to collect

information systematically in order to indicate the worth or merit of a program or a

project then to make a decision. Therefore, Brown (1989: 223) suggests a good

working definition to be the systematic collection and analysis of all relevant

information necessary to promote the improvement of the curriculum and assess its

effectiveness and efficiency, as well as the participants’ attitudes within a context of

particular institutions involved. Similar definition proposed as follow, evaluation is a

natural activity as intrinsic part of teaching and learning that enables evaluators to

make judgments towards the program or curriculum that is done systematically and in

principled way (Rea-Dickinds & Germaine, 1992). Obviously, both definitions agree

that evaluation is a systematic information gathering to make a judgment. They also

mention a program and curriculum in the context of educational evaluation. In the

case of educational evaluation, it can be done before the program commences or

before. It of course can also be done during the program is being commenced

(Brown, 1994). This is supported by Hewings (1996). The scope of evaluation has


17

moved from a concern with test result to the need to collect information and make

judgements about all aspects of the curriculum, from planning to implementation.

In relation with language program, in this context is English program,

evaluation may focus on many different aspects. First, it may focus on curriculum

design which is to provide insights about the quality of program planning and its

organization. Second, the concern can be on syllabus and program content in which it

will deal with the relevancy and engagement of the syllabus and program content and

the successfulness of the assessment and test procedures. Third, that is classroom

process which aims to provide insights about the extent to which a program is being

implemented appropriately. Fourth, material of instructions where evaluators may

focus on materials of instruction in which provides insights about specific materials

are aiding student learning. Fifth, that is the teachers which need to consider main

questions such as who conducted the teaching, what was the teachers’ perception of

the program, and what did the teachers teach. Sixth, it can be about teacher training

which aims to assess whether training teachers have received is adequate. Seventh,

the point is about the students in which the research focuses on investigating what the

students learn from the program, what their perception toward the program is, and

how the students participate in the program. Eighth, the concern is on monitoring of

the students’ progress.

One of evaluation process which is categorized based on time frame it is

conducted is formative evaluation or in-progress evaluation of student learning.

Based on Sanders (1992) and Weir and Roberts (1994), the focus aims of conducting

this type of evaluation are on five matters. First, that is learner motivation where


18

evaluators try to seek insights about the effectiveness of teachers in aiding students to

achieve goals and objectives of the school. Second, that is learner motivation where

the evaluators try to seek what drives learners to learn. Third, it is about the

institution where the evaluators try to what administrative support was provided, what

resources were used, what communication networks were employed. Fourth, learning

environment is to provide insights about the extent to which students are provided

with a responsive environment in terms of their educational needs. Fifth, staff

development aims at providing insights about the extent to which the school system

provides the staff opportunities to increase their effectiveness. Last but not least, that

is about decision making which is to provide insights about how well the school staff-

principals, teachers, and others – make decisions that result in learner benefits.

According to the above explanation, it can be concluded that evaluation is a

long process conducted to provide information. The information is to improve

curriculum, teacher development, students, the effectiveness of teaching and learning

process and materials used. All things mentioned are to achieve educational and

instructional goals.

b. Type of Evaluation

In general, evaluation in ELT mainly occupies two kinds of evaluation namely

macro-evaluation and micro-evaluation. The distinction between two evaluations is

that macro-evaluation deals with the administrative and curriculum matters that

undergo evaluation for the whole programs. Besides, in micro-evaluation, it only


19

deals with the break-down of the curricular matters such as teachers, materials and

learners (Ellis, 1998).

Weir and Roberts (1994) distinguish the types of evaluation for language

program based on two major purposes, program accountability and program

development. Accountability refers to the quality of the work of those involved in a

program (Richards, 2001). He also makes a point that accountability evaluation

usually examines the effect of a program and its significance towards educational

system is usually conducted by an independent institution as the external audience of

the education system or by the decision maker. In contrast, development-oriented

evaluation aims to improve the educational quality of a program or project (Weir and

Roberts, 1994). This kind of evaluation is normally done while the program is in

progress.

Richards (2001) gives different opinions about purposes of evaluation to what

Weir and Roberts suggested. He distinguishes it into three different kinds of

evaluation. The first type is formative evaluation. Formative evaluation refers to an

evaluation that is undergone as part of the process of the program to seek what is

working and what is not and what problems need to be addressed. Since this kind

evaluation focuses on ongoing development and improvement of the program, it is

usually conducted during the program. An evaluative question related to textbook that

typically raised in formative evaluation is how well textbook is being received.

Another type of evaluation is Iluminative. In this kind of evaluation,

evaluators try to find out the different aspects in the program and their

implementation. This kind of evaluation focuses on the teaching and learning process


20

that occurs in the program. Therefore, it is rarely to ask about the materials in this

kind of evaluation.

The last type suggested by Richard (2001) is summative evaluation. This third

type of evaluation is the type with which most teachers and program administrators

are familiar. Summative evaluation concerns on whether a program is effective and

efficient and to some extent acceptable. Richard points that this kind of research is

conducted after the program has been implemented. This kind evaluation is also

asking whether the materials work well. Brown (1994), related to the material

evaluation, makes an important point that any evaluation can be carried out before the

program commences or after and or during the program.

By relying on the above explanation, this present study agrees on what Brown

pointed that any evaluation can be carried out before, after or during the program. In

this study, the evaluation is conducted during the program. Another point to

reconfirm is that this present study undergoes a material evaluation in form of

textbook evaluation.

c. Check List of Textbook Evaluation

Related to textbook evaluation, McDonough, Shaw & Masuhara (2013) give

two different stages instead of the type of textbook evaluation. The first stage is a

brief external evaluation. This includes criteria which gives an overview of the

organizational foundation of the book. Cunningsworth (1995) give similar ideas with

different term, impressionistic evaluation. In this stage an assessor only does a brief

look to textbook to give a brief description about textbook. Following this is an in-


21

depth internal investigation of textbook, ‘to see how far the materials in question

match up to what the author claims as well as to the aims and objectives of a given

teaching program’ (McDonough, Shaw, & Masuhara, 2013). In other words, in-depth

internal investigation must be done to analyze to which extent result of external

investigation match up with internal investigation.

Another criterion is illuminated by McDonough, Shaw & Masuhara (2013).

They comprise nine criteria before conducting in-depth internal evaluation. These

criteria are usually found at the cover or in the introduction of the book. The criteria

or checklist are in a form of questions that will be answered by evaluators.

In parallel with Shaw (2013) and Rea-Dickins (1992) suggest a checklist in a

form of preliminary questions. Then, evaluators answer the questions before

analyzing which teaching and learning materials are suitable. The questions are as

follows.

Rea-Dickins (1992) provide these preliminary questions above. Though in the

end of the procedure, they point important consideration that since teaching and

learning materials are sensitive to the language process, evaluation criteria should

relate not only to the aims and contents of language learning but also to the

procedures of working with texts and performing tasks in the classroom. That is why

they give credits to Cunningsworth’s work since as it can be seen in appendix, the

criteria cover Germain have pointed. To compare to the criteria suggested by

McDonough, Shaw & Masuhara (2013), what Cunningsworth marked gives more


22

detailed and comprehensive questions related to the methodology in which Rea-

Dickins (1992) suggested.

Table 2.2 Checklist Suggested by Rea-Dickins & Germaine

A more recent study held by J. Mukundan, (2011) in developing English

language textbook evaluation checklist was done in a focus group discussion. Six

participants joined the developers to improve the items of checklist generated by

Cunningsworth’s evaluation checklist (1995) in reference to their clarity and

inclusiveness. In this study, researchers successfully developed an evaluation

checklist for ELT textbook. They added, omitted, and revised the Cunningsworth’s

evaluation checklist.

In relation to Indonesian context and considering the above explanation, this

present study sees that Cunningsworth’s evaluation check list will work well to

Criteria 1: What do materials mean for you?

1. Do you refer exclusively to textbooks, or do you include teachers’

guides, teaching manuals, supplementary units, readers, audio and

visual materials, etc?

2. Do you make a distinction between materials designed specifically for

first and second language teaching, and also between those targeted

specifically for use in school and materials that are non-pedagogic but

authentic? 3. Do you include materials produced by the teachers and the learners?

Criteria 2: What are the roles of material within your teaching and learning

context?

1. What role(s) are they expected to play?

2. What role(s) are they expected to achieve?

Criteria 3: How are the materials to be used?

1. Are they to be used as the sole source and resource for teaching?

2. Are they one of several available resources?


23

evaluate the examined book. Therefore, this present study will be spelt out the

evaluation criteria by Cunningsworth in form of questionnaire to be given to the

teachers to evaluate the English textbook of seventh grade junior high school in

Indonesia.

3. Review on Indonesia National Curriculum 2013

A new curriculum has been implemented in Indonesia since June 2013. This

new curriculum is called curriculum 2013 (C13) which is based on the Regulation of

Ministry of Education of Indonesia number 22, 2006. C13 has been implemented in

all around Indonesia for all levels of education from Elementary School (grade 1-6),

Junior High School (grade 7-9), and High School (grade 10-12). In the first step of

the implementation, not all grades were using the new curriculum. Only grade 1, 4, 7

and 9 run the C13. This new curriculum was also not implemented in all school in

Indonesia. For the first stage of the implementation, Ministry of Education only

pointed some schools to become the pioneering schools for this curriculum

implementation.

To give an illustration on the C13, the following explanation is a brief

description on the core competence and base competence. There are four core

competences for English subject in K 13. All of them are related to religious and

moral conduct and conceptualized in the base competence.

To make the explanation crystal clear, samples of a base competence stated in

C13 (Core and Basic Competence of Curriculum 2013) in the following six points.

First, it is composing oral text to express and respond greeting, leave taking,


24

thanking, and apologizing correct language features in context. Second, it is

composing oral and written text to express and respond introducing oneself in short

and simple way by paying attention to social function, text structure, and language

features correctly and contextually. Third, it is composing oral and written text to

mention names of days, month, and time in days, time in numeric form, dates, and

year by using the language features correctly and contextually. Fourth, it is

composing oral and written text to mention names of animals, things, and public

places found in the real life by using the language features correctly and contextually.

Fifth, it is composing oral and written text to make names in label and list down

things by using language features correctly and contextually. Sixth, it is composing

oral and written text to describe personal characteristics, animal and things by using

the language features correctly and contextually.

As stated above, it is very obvious that current curriculum deploy three main

aspects reflected in the core competence. Those are attitude, skills and knowledge. In

addition to the explained aspects, C13 performs the thematic and integrated concept

of syllabus which can be seen in the official website of education and cultures

ministry.

Taken from the preface of the teacher’s book of English textbook, it is said

that C13 gives a significant change in terms of teaching and learning process.

Previous curriculum enables teacher to deliver the subject or knowledge. In contrary

to the prior curriculum indeed, C13 enables the students to share knowledge among

them by the guidance of the teacher. In pedagogical term, the teaching and learning is

a two-way teaching and learning process. Though, in the real implementation some


25

teachers in Indonesia have implemented a two-way teaching process instead of one

way. However, such approach, had not been realized until the C13 was issued. As a

consequence, by applying this curriculum, students are intended to be more active in

doing observation, thinking, questioning, and communicating as well as presenting

what they have earned from the knowledge receiving process

(http://kemdikbud.go.id/kemdikbud/uji-publik-kurikulum-2013-1).

Four standards that happen to be changed in curriculum 2013 from curriculum

2010 covers graduate competence, process content, and standard of learning

evaluation. Related to learner evaluation/assessment, there are four criteria of learner

assessment: 1) level of achievement of the use of social function text; 2) level of

completion and arrangement of text structure; 3) accuracy of language features such

as grammar, vocabulary, pronunciation, word stressing, intonation, and spelling; and

4) level of appropriateness of writing and delivery format. Four methods of learner

assessment are provided in this curriculum to meet with the core and base

competence. The four methods of learner assessment are assessing student

performance, student observation, student portfolio, reflection and pair assessment.

In conclusion to the explanation above, it is highly required to evaluate

textbook of new curriculum 2013. It is since many new constructs applied in the

curriculum that become valuable source to adjust the materials used. In addition,

significant shifted paradigm such as the teaching methodology suggested in the

curriculum.


http://kemdikbud.go.id/kemdikbud/uji-publik-kurikulum-2013-1

26

4. Review of Previous Studies

In this section of the study, two previous studies will be reviewed briefly to

give depiction and broader knowledge in doing the research. The review of previous

studies focus on the methodology of each research to compare one to another in order

to get depiction of what and how the researcher conduct an evaluation especially

textbook evaluation.

A thesis by Alamri (2008) entitled An Evaluation of Sixth Grade English

Language Textbook for Saudi Boy’s School was conducted to evaluate a new

textbook. Conducted in King Saud University, Riyadh, this research was a survey

research which used a set of questionnaire to elicit the perspective of 93 English

language teachers and 11 supervisors in Riyadh Educational Zone about textbook in

question. The questionnaire comprised 64 groups under 12 main categories. For the

questionnaire design, he adopted four different evaluation checklists from four

different experts and researchers. As the result of the adoption of those four

checklists, (a) general appearance, (b) design and illustration, (c) accompanying

materials, (d) objectives, (e) topic contents, (f) language contents, (g) social and

culture contexts, (h) language skills, (i) teachability, (j) flexibility, (k) teaching

methods, and (l) practice and testing are used as the categories in the questionnaire.

After getting the data, the researcher analyzed them through descriptive statistic. The

findings showed out of 64 items in the questionnaire, only 13 items had means less

than 2.50. The category that had the highest mean was the one on learning

components. Meanwhile, the category that had the lowest mean was the one on the

teaching methods. Based on the findings, it was also found that there were no


27

significant differences between the means of two populations of the study, teachers

and supervisors, except on the flexibility of textbook. The different natures of their

jobs could be a possibly reason for this difference.

Another example of impressionistic evaluation using an empirical qualitative

approach is the one done by Nemati (2009) in India. This research involved 26 Pre-

University teachers. Because this research also focused on vocabulary factors, besides

a questionnaire, the data were also taken from vocabulary response. The

questionnaire that was distributed to the respondents then followed by an interview

for each respondent to triangulate the data, it was arranged based on textbook

evaluation checklist. She adopted several checklists from the authors such as

Cunningsworth, (1995) and Ur (1996). After studying the checklists from the authors,

she decided to select features which are in common. The criteria used in the

questionnaire are as follow (1) logistical factors, two questions about the availability

and price of the book, (2) pedagogical factors, or methodology, with two subparts of

skills and contents, (3) human factors, including teacher friendly and learner friendly

factors, (4) vocabulary factors, entail questions regarding vocabulary learning

strategies. From the research result, teachers’ belief toward textbook was revealed

based on the criteria. In addition, suggestions to improve textbook were also

obtained.

The other example of textbook evaluation research discussed in this section is

conducted by Zamanian (2015). Aiming at evaluating ‘Learning to Read English for

Pre-University Students’ used in Iranian high schools, the researchers distributed a

questionnaire to 75 high school English as a Foreign Language teachers. In designing


28

the questionnaire, Litz’s checklist (2000) was adopted. After the data were collected,

those are analyzed based on a Likert scale. Based on the research results, it was

known that textbook was quite far from teachers’ standard. It happened because

textbook ignored listening and writing skills. Moreover, it did not follow a

communicative approach in teaching different skills and language components.

Similar to those three research, this study is conducted empirically. After

being used to teach junior high school students in Indonesia, textbook entitled ‘When

English Rings the Bells’ is evaluated. Data are going to be collected by distributing

questionnaire to evaluate whether the material is in line with what teachers expect.

Similarly, this research also captured textbook evaluation process done by Indonesian

teachers after they used textbook in a classroom context. Framework used for this

research was Cunningsworth’s (1995). Using the framework, the research focuses on

the aim, the approach applied, the language skills presented in textbook, the design

and organization, the content of the language, and the topic.

B. Theoretical Framework

From the literature reviewed above, the theoretical framework can be drawn

as follows. In conducting this evaluation research, the criteria of a book evaluation

are the most important and significant aspects to be paid attention of in resulting a

reliable research. The criteria established in this research are selectively adapted from

the Tomlinson’s, Bworn’s, Cunningsworth’s and Nunan’s book evaluation criteria.

Furtherly, the criteria are conceptualized into a research instrument in form of

questionnaire. The instrument is used to measure textbook by the practitioners in this


29

case are English secondary school teachers. Each item of questionnaire is also

considering what is currently formulated in C13. Therefore, the concept of textbook

evaluation by Tomlinson (2008), Brown (2001), Cunningsworth (1995) and Nunan

(1989) contributes largely in drawing the concept of the criteria applied by the

researcher in this evaluation study.

The framework is developed from some underlying theories about evaluation

in English language teaching (ELT) and textbook. Those encompass the framework

on textbook evaluation and textbook development. From the literature reviewed

above, the framework can be theoretically drawn as follows.

Figure 2.1 The Theoretical Framework

Cunningsworth (1995)

Nunan (1989)

Textbook

Evaluation

Curriculum

2013 (C13)

Checklist

Aim Approac Language Design and

Organization

Language

Content

Topic

Brown (2001)

Tomlinson (2008)


CHAPTER III

RESEARCH METHOD

This chapter discusses the research methodology which was employed in an

attempt to answer the research question empirically. In the relation to the path of data

collection, this chapter discuss research method applied. After that, research setting

and participants, research instrument, data gathering and analysis technique are

scrutinized in this chapter.

A. Research Method

This research was under an umbrella of applied research that enabled the

researcher to evaluate a program, in this case, it was to evaluate a textbook. It was

adopted in this research since it embraces a set of process in evaluating the quality of

textbook. Evaluation research study was designed to find out to what extent a new

program, policy, textbook, a new way of doing something and so forth, has met the

criteria mentioned in the second chapter of this report.

Evaluation research can be undergone quantitatively or qualitatively. In favor

to conduct quantitative research, researcher may use a survey research. To conduct

evaluation research using qualitative data, the researcher must use the field research

(Neuman 2006). In survey research Neuman suggests to researchers to collect data

from a sample or could be smaller group of selected people. He suggests around 150

respondents to generalize result to a larger group of 5,000 people.

30


31

B. Research Respondents

This research targeted junior high school teachers who implemented

curriculum 2013 as the respondents. As it has been described in the previous chapter,

the implementation of curriculum 2013 did not cover all junior high schools

throughout Indonesia. Each region and province in Indonesia has pointed selected

school to implement and use textbook of curriculum 2013 in odd semester. In odd

semester 2014, more schools implemented C 13.

Since it was survey research, it was conducted in 10 junior high schools in

Special Province of Yogyakarta. These schools were pointed to implement the C13 in

Yogyakarta. Special Province of Yogyakarta has five districts including the city of

Yogyakarta. The researcher picked up two schools for each district as sampling. From

each school, the researcher took two teachers to be participated in the survey

research. The teachers vary in terms of the length of teaching experience. The very

least year of experience was two years. While the most experienced teacher had 35

years of teaching experience.

To sum up, this research was supposed to involve twenty teachers teaching in

different junior high schools in the province of Yogyakarta. However, due to some

circumstances, there were two teachers who refused to join the survey and one

teacher filled out the questionnaire with the same answers for every item. As result,

this respondent is considered failed. Therefore, by the end of the data gathering, there

were only 13 respondents whose data was considered as valid.


32

C. Research Instrument

Two research instruments were utilized in order to attain feedback from this

study. First, it was an evaluative questionnaire adopted from Cunningsworth’s

checklist. Second, to triangulate the data taken, an open ended questionnaire was

distributed then.

For the evaluative questionnaire, the items were established by adopting

Cunningsworth’s checklist which focused on the aim, approach, language skills,

design or organization, language content, and topic used. There were seventy-one

items prescribed in the questionnaire which these items were measured using Likert

scale. In order to reduce the number of invalidity of the survey result, the researcher

only used 4 scales instead of 5 scales namely scale 1 for strongly disagree, 2 for

disagree, 3 for agree, and 4 for strongly agree to the item.

Beside closed item which was prescribed in form of statement, the

questionnaire also encompassed open questions for the respondents to answer. This

was aimed to obtain the triangulation data. Besides using open questions, the

researcher also used textbook close reading to get the factual evidence from textbook

to crosscheck whether the survey results were confirmed.

D. Data Gathering Techniques

In gathering the data, at first, the researcher made an appointment with twenty

teachers. However, like what it has been stated above that due to some circumstances,

two teachers refused to join the survey and one teacher filled out the questionnaire

with the same answers for every item. As result, only thirteen teachers who kindly

agreed to the appointment.


33

The data gathering was conducted by the end of the even semester 2012/2013

Indonesian academic year in June 2013. The researcher started to meet respondents

from school to school. So, it took a month to take the data. From thirteen sets of

questionnaire were distributed, amazingly all of them returned. However, there was

one questionnaire was returned remain unfilled by a respondent.

E. Data Analysis

The very first step of data gathering was distributing questionnaire to the

respondents. The respondents were given a week to fill out the questionnaire

completely. The researcher came over to each respondent to explain the procedure in

filling out the questionnaire although the researcher had written a clear instruction in

the questionnaire. This aims to make sure that the respondents were able to fill out the

questionnaire correctly. Then, after a week the questionnaires were collected. After

the questionnaire were returned to researcher, the results then were put into the tally

system as seen in the table below.

After putting all the result in tables, the researcher calculated the data to get

the MEAN for each item of the questionnaire. In addition, the standard deviation was

also gained to find out whether each item was valid. In the table presented in this

survey research report, it comprises the percentage of all fourth scales (1 = strongly

disagree, 2= disagree, 3 = agree, and 4= strongly agree), the Mean, and the standard

deviation.

To read the table, it can be inferred from the mean of the item. If the mean

more than 2.5, it means that respondents or teachers agree to the statement. In


34

contrary, if the mean of an item is showing less than 2.5, it means the item is not

agreed by the respondents. Therefore, the researcher presented the data for each item

in two digits of decimal. It aims to see the difference one item to another more

precisely.

After completing the tally sheet, the researcher then created tables based on the

result of the survey. The table helps the researcher to describe the result of the survey easily.

Hence, later on the researcher will be able to analyze then discuss the result based on the

diagram and the answers of the open questions.

After obtaining the result survey, the data inform of score are then presented

into table in order to make the interpretation easier to elaborate. The researcher later

describes the survey result elaborately in chapter IV section one. And then, in the

second section of chapter IV the researcher is discussing the survey results and

relating them with the theory in learning and material development. This discussion is

aimed to get a comprehensible conclusion and decision toward textbook since the

main goal of evaluation is to make decision based on the data gathering.

Although the researcher had time constrain, the researcher successfully interviewed

respondents to triangulate the data. In addition, the researcher also happened to give them

several open questions on the questionnaire. They had to write the open questions on the

questionnaire sheet. While the researcher collecting the questionnaire, the researcher tried to

confirm all the answers written on the sheet by. By doing this, the researcher was able to

understand the answers from the respondents comprehendingly. Then, all the answers of the

open questions were written in a table and categorized. Hence, these data are ready to use in

the analysis to support the survey result.


CHAPTER IV

RESEARCH FINDINGS AND DICUSSION

In this chapter, it is described the survey result begin with the general

findings. The following elaborative description is explaining the survey result of each

aspect. Six aspects of textbook being evaluated are its aim, approach, design and

organization, language content, skills and topic.

A. General Findings

In general, the evaluated textbook remained credible as a textbook used in

English language classroom for secondary school. Despite the debate regarding the

C13 and its textbook, the survey result proved everything regarding the textbook. The

result for each aspect of the evaluated textbook shows that teachers believe this

textbook is still credible and reliable to be used in the English language classroom for

secondary school students.

Later in this section, a table showing the overall result of the survey is

displayed. The Mean of each aspect is telling the average of the respondents’

response toward the aspects. Each aspect was evaluated by several statements

formulated in the questionnaire. Therefore, the mean represents how each aspect

meets the criteria based on the teachers’ or respondents’ perspective. Since the scale

employed in the questionnaire is 1 to 4, the Mean which is more than 2.5 means that

the aspect meets the criteria. To see to what extent the aspect meet the criteria, it can

be seen how far the Mean overly reach the limit (2.5).

35


36

Teachers believed that the aims of the textbook were definitely in line with the

curriculum and the program objectives. As it is seen from the table, the mean for this

aspect shows 2.97 and is ranked as the highest mean among other five aspects to be

evaluated. The second highest mean of the aspect is 2.76 for the topic of the textbook.

This unavoidably shows that teachers consider the topics of the textbook are

appropriate for Indonesia context as well as representing different group and

ethnicity. As though aims and topic of the textbook, the approach used in the

textbook happens to be one of the aspect proving that this textbook remains reliable.

This is shown in the table which 2.61 happens to be the mean for approach of the

textbook. The other three aspects namely the design and organization, content, and

language skill have almost the same result for each.

Table 4.1: Overall Survey Result

Aspect Mean No

2.61 2. Approach

3. Design and organization

2.52 4. Content

5. Language skill

2.76 6. Topic

However, teachers and material developer are obliged to see further and

deeper on each aspect. Therefore, addition to the material as well as revision to the

existing material may be necessary to be done in favor to the improvement of the

textbook. Hence teacher may find the additional material focus on the specific

2.32

2.55

1. Aims 2.97


37

component of the textbook and the material developer may revise the textbook on that

specific component as well. Accordingly, the next subchapter will elaborate deeper

on the survey result of each item of the aspects.

B. Detail Findings

In this part, all criteria will be elaborated in detail by giving the real survey

data and related theory as well as data taken from textbook to enrich findings. The

researcher will also add small part of discussion related to certain item of the criteria.

In the next section, the researcher will give general discussion about the evaluated

textbook. In this sub-section, the detailed findings will start from the very first

criterion.

1. The Aims

The first analysis goes to the first aspect, the aims of textbook. As it is seen in the

table, there are four items or statements to measure the agreement of the aims of

textbook. The first item is rated at 3.11 by the respondents. This mean shows that it is

higher than 2 and even higher than 3. This means that teachers are mostly agree on

the statement in item 1. Teachers saw that the aims of this textbook closely

correspond with the aim of the curriculum. The respondents rated the second item the

aims of textbook correspond closely with the needs of learner at 3 for the mean. As

though the first item, the second item was agreed by the respondents. In conclusion,

almost all respondents saw that the aim of this textbook has no issue and think that

the aim of textbook had been in line with the curriculum and learner needs. See the

table for the detail.


38

Table 4.2: Survey Result on the Aims of the Textbook

Item Statement Strongly

Disagree

(%)

Disagree Agree Strongly

Agree

(%)

Mean SD

(%) (%)

2. The aims of

the textbook

correspond the

core of the

competence.

0 16.7 66.7 16.7 3 0.594

4. The aims of

the textbook

correspond

closely with

the needs of

learners.

0 27.8 66.7 5.6 2.76 0.545

However, teachers and material developers must carefully see that the mean

of item number 4 namely the aims of the textbook correspond closely with the needs

of leaners is not that high. It is only 2.76 a little higher than 2. In other words, 27.8 %

of the respondents disagreed to the statement. Therefore, to determine whether each

statement in this criterion is factually confirmed, the researcher needs to look at the

results of the next criteria and the actual evidence found by reading closely to

textbook and contrasting them with what the survey results are. Therefore, the

researcher will elaborate each statement at the discussion section of this chapter.

3. The aims of 0 16.7 66.7 16.7 3 0.594

the textbook correspond closely with the aim of the teaching program.

1. The aims of 0 11.1 66.7 22.2 3.11 0.582

the textbook correspond closely with the aim of curriculum.


39

2. The Approach

The second aspect to measure is the approach of textbook. There are seven

statements to measure this criterion. See table 4.3. It can be clearly said that

respondents agreed that textbook is suited to the learning situation. However as a

researcher, it is important to analyze it deeply. Based on the survey, the score of this

statement is 2.7. It means the textbook suited to the learning situation. Respondents

see that textbook is effectively suited to the learning situation although the score is

low. It is a survey research, therefore teachers’ statement towards this topic cannot be

proven empirically since the researcher did not conduct observation to see how

effective and judge whether or not the aim of textbook is suited to the learning

situation.

The second statement is seen as effective one by the respondents. It means

that respondents agree that textbook is suited to the teaching situation. However, the

number of agreement to this statement is absolute which means some other

respondents see that textbook is not suited to the teaching situation.

Remarkably, most respondents disagree to the third statement. The Mean is

2.23 See table 4.3. Respondents did not agree that textbook covers most of all student

needs since it does not reach 2.5. Another negative result happens at the fourth

statement. Respondents did not agree that textbook is a good learning resource for

student. But the value of the disagreement is low. It is 2.47.

Most teachers agreed that textbook does not cover all students’ need

(Diagram). For sure it is not easy to facilitate all students need. However, at least a


40

textbook must be able to give the most important need for the students for example

the need of language input. This study attempts to confirm the empirical data taken

from the survey and textbook itself with the theory. As Krashen (1982) proposed the

input hypothesis, comprehensible input is very important for second or foreign

language learners. Comprehensible input means the learners know what people say

and what the message is, instead how people say. Let us refer to textbook –When

English Rings the Bells. Here are two picture captured from the book on page 7.

Table 4.3 Survey Result on the Approach of the Textbook

Item Statement Strongly Disagree

(%)

Disagree Agree Strongly Agree

(%)

Mean SD

(%) (%)

6. The textbook is suited to the teaching situation.

5.6 22.2 66.7 5.6 2.70 0.665

8. The textbook is good resource for students.

5.6 44.4 44.4 5.6 2.47 0.696

10. The textbook is flexible so it allows various teaching styles.

0 27.8 61.1 11.1 2.82 0.616

Another important aspect in language learning is grammar (Krashen, 1982).

Grammar inevitably cannot be separated from the language learning since grammar

11. The textbook is 0 27.8 66.7 5.6 2.76 0.545

flexible so it allows learning styles.

9. The textbook is 0 44.4 50 5.6 2.58 0.599 good resource for teachers.

7. The textbook 5.6 61.1 33.3 0 2.23 0.545

covers most of all student needs.

5. The textbook 5.6 22.2 66.7 5.6 2.70 0.665

suited to the learning situation.


41

has a great deal in language learning as Monitor (Krashen, 1982). Obviously, a good

textbook is when textbook can be used by the students for individual learning

(Tomlinson, 2008). Then, the question is ‘what is the relation between grammar

reference and individual learning?’

As it is mentioned above according to Krashen (1982) that grammar plays

important role as Monitor while in individual learning, students have no supervision

from the source of knowledge except textbook itself. Therefore, grammar reference is

surely needed in a textbook to fulfill students’ need in individual learning process. To

support individual learning in ELL, a textbook must be equipped with features or

section which can be used without any supervision or assistance from teachers or

tutors. Hence, the grammar reference can be the Monitor in their language learning

process.

Figure 4.1 Sample of activity found in textbook: Activity 5

On page seven in activity 5 as you can see at figure 1 it can be seen from the

instruction that students are asked to practice the dialogs given with people around


42

them. In the dialog it is found words or phrases like feeling great and fine to express

positive response in greetings as reading input applied in activity. On this page at

chapter 1 is the only presentation of positive response to greetings.

In activity 6, different type of input is applied. See figure 4.2. The teacher has

to read aloud the dialog then students have to repeat after him/her. As seen in the

dialogs, new expressions are introduced such as have a headache and got flu. Inputs

are very important for students since they will be able to produce the language.

Learners will not be able to produce adequate outputs if learners receive very few

inputs.

Figure 4. 2 Sample of activity found in textbook: Activity 6

A case where learners cannot produce adequate language outputs in terms of

writing and speaking are very possible to happen in the next activity, activity 8 on

page nine. As you can see, textbook does not give adequate inputs for learners to

produce the language effectively. Only small number of words and expressions are


43

given in activity 5 and 6. In fact, in productive activity on this chapter, textbook

requires students to ask 10 people about their feelings.

Figure 4.3 Sample of activity found in textbook: Activity 8

Textbook does not anticipate any possibility might happen in this activity

such as students ask more vocabularies to express feelings. Textbook doesn’t provide

a rich vocabulary range related to expressing feelings. In contrary, textbook provides

picture expressing feeling without its English word.

Interestingly, the respondents who are teachers see that this textbook is a good

learning resource for teachers. Yet the value of agreement for this statement is low. It

is 2.47.

Related to the flexibility of various teaching and learning styles, respondents

see that this textbook is flexible for both teaching and learning style. However, in fact

teachers must follow the teaching method used in the curriculum. To find further

factual evidence stating about the teaching method used in C13 curriculum, the


44

researcher went thoroughly to the teacher manual. It was found that in C13, teachers

must implement the learning process suggested in the curriculum.

The first learning process is observation. At this beginning process, students

are aimed to read/watch/listen to the examples of the text being learnt directly from

the sources and/ or from the recorded sources by paying attention to the social

function, text structure, linguistics elements and the writing format. Then, teachers

must scaffold the students to imitate/ copy/ use the examples.

Second, students are aimed to questioning what they have observed at the

observation stage. In this stage, teachers must assist the students to question the text

being learnt regarding social function, utterance and text structure, linguistics

elements as well as the writing format. After questioning the text, students are

supposed to learn the social function, utterance and text structure, linguistics elements

and writing format directly from the teacher or other sources.

The third learning process is exploration/experiment. At this point, according

to the brief explanation written in the teacher manual, students are aimed to read/

listen/ watch similar kinds of the text being learnt from other different kind of sources

including text book and guide book by paying attention to the social function of the

text, structure of the text, linguistics elements and the format of the text. However,

the researcher failed to understand what ‘guide book’ means.

The fourth learning process suggested in the manual book is associating/

analyzing. The students are put in groups then asked to study a set of text with their


45

group. Subsequently, they have to indicate the social function, utterance and text

structure, linguistics elements and text format of the texts they are discussed. After

that, teachers must give feed back towards their discussion.

The last stage is communicating. Students are aimed to

read/listen/present/demonstrate/ publicize/speak/recite/watch the texts being learnt.

Then teacher has to give feedback to what they have done on the texts and message

they can get from the text. Eventually, students are asked to share what difficulty and

ease they face in learning the text. They are also asked to share their strategies to deal

with their difficulties.

Back to the diagram, the score of both statements number 6 and 7 is almost

the same, 2.70 and 2.23. The researcher cannot give any further comments on this

finding since the researcher did not conduct any observation which actually can be

one the data to evaluate the teacher’s method and approach in teaching activities.

Therefore, researcher in this study could not give any justification towards textbook

flexibility in terms of teaching and learning styles. However, the researcher will

explain and give a very gentle comment towards this particular topic in the discussion

section of this chapter.

Generally, respondents see the approach of this textbook has met the criteria

rated at 2.71. It means that respondents agree that the approach of this textbook has

met the criteria since the mean of this aspect is more than 2.5.


46

3. Design and Organization of textbook

Various results happen in this criterion that some statements are disagreed and

some others are agreed. The mean of each statement is also varied. Respondents seem

do not agree with that this textbook support media such as CD, workbook, and

teacher’s manual. The mean is 2.47. It means half of respondents agree to the

statement and another half do not. It can happen since textbook support by teacher’s

manual but it does not support CD and workbook.

However, the result shows that respondents still consider that textbook is not

effective as it does not contain workbook and CD(s). It is definitely true that this

textbook is not accompanied with workbook and CD(s) for multimedia support. In

other words, the survey does reflect the reality. Unfortunately, it is an inconvenient

truth that textbook does not fulfill what Tomlinson (2008) suggest that ELT materials

must make use of multimedia to provide reach language experience.

A finding based on the survey raised an intriguing question whether the result

of the survey does represent the reality. The result shows that teachers saw textbook

has a well sequencing in term of complexity. To be surer about the result, the

researcher tried to look closely at the content of textbook. The focus on the

investigation was the spread of textbook and the sequencing of textbook. To make the

explanation become clearer, Chapter 1in textbook is used as an example. See figure

4.4. As it can be seen, students will learn ‘how to greet’, ‘how to thank’, ‘how to take

leave’ and ‘how to apologize’.


47

Table 4.4 Survey Result on the Design and Organization of the Textbook


Disagree

(%)


Agree

(%)

Mean SD

(%) (%)

13. The organization of the

textbook is right for the

teacher.

0 50 38.9 11.1 2.58 0.691

15. The content is well

sequenced on the basis of

complexity.

0 33.3 66.7 0 2.64 0.477

17. The content is well


usefulness.

0 27.8 72.2 0 2.70 0.455

19. There is adequate review. 5.6 44.4 44.4 5.6 2.47 0.696

21. The materials promote

individual study.

0 50 33.3 16.7 2.47 0.775

23. It is easy for teacher to find

his/her way around the

textbook.

0 38.9 55.6 5.6 2.64 0.588

24. The textbook has good 0 33.3 44.4 22.2 2.88 0.757

layout.

22. It is easy for learners to 0 41.2 52.9 5.9 2.64 0.588

find their way around the

textbook.

20. There is reference section 16.7 61.1 22.2 0 2 0.594

of grammar in the book.

18. There is adequate 11.1 33.3 55.6 0 2.41 0.691

recycling.

16. The content is well 0 27.8 66.7 5.6 2.76 0.545


learnability.

14. The organization of the 0 44.4 50 5.6 2.41 0.844

textbook is right for the

learner.

12. The textbook is supported 11.1 38.9 38.9 11.1 2.47 0.874

with media such as CD,

workbook, and teacher’s

manual.


48

Talking about sequence on basis of complexity, this book has fulfilled by

placing the learning materials in sequence since ‘how to greet’ and ‘how to thank’ are

easier than ‘how to take leave’ and ‘to apologize’. The investigation was not finished

to this point. A deeper analysis was done and it was found that the book consists of

18 activities (activity 1 to activity 18). There are five activities teaching and

practicing ‘how to greet’. They are activity 1, 2, 3, 4 and 11.

Figure 4.4 Aim of Chapter 1

Then, there are six activities presenting and practicing ‘how to thank’. They

are activity 5, 6, 7, 8, 14 and 16. ‘How to take leave’ is presented and practiced in 2

activities only, namely activity 9 and 10. Finally, ‘how to apologize’ is also presented

and practiced only in 2 activities, activity 17 and 18.

In the prior paragraph, it seems that there is no problem about sequencing in

terms of content. In contrary, the sequencing of the activity seems to be problematic

since there is an activity which is misplaced. Activity 11 practicing ‘how to greet’,


49

but it is placed in activity number 11. It should be placed in activity 5. Activity 14

and 16 should be placed in activity 9 and 10.

It is proven that actually this book, even the first chapter of this book does not

seem well sequenced in term of complexity. Hence the result of the survey for this

statement does not really represent the fact. As Richards (2001) suggested that in

developing ELT materials, developers must consider the sequencing of the content.

They must determine which content need to be placed prior to other content.

Table 4.5 Complexity Level of the Activity

Another related topic to be discussed is the scope of the content. As Richards

(2001) purposed that material developer must consider the scope of content. It means

how many meetings students need to learn a certain topic. Developers must be able to

determine such detail. They have to know whether or not the time allocation is

Chapter

Activity

Topic

Complexity level

(1= lowest level, 5= highest level)

1

1 How to greet

1 2

3

4

5 How to thank

2 6

7

8

9 How to take leave

3 10

11 How to greet 1

12

How to greet and take

leaver

1 and 3

13 How to thank

2 14

15 How to ask for help / favor 5

16 How to thank 2

17 How to apologize

4 18


50

enough, not enough or too long. In this textbook, it seems that the material developers

didn’t really pay attention to the scope of the content of each chapter. In chapter 1 as

explained above, there are 5 activities to learn and practice ‘how to greet’. The fact,

as it is concluded from the teacher’s manual, 2 activities are taught in one meeting. It

means to learn and practice a simple thing called greeting, students must spend a

week. Surprisingly, chapter 1 is created for 4 up to 5 weeks learning. Take a look at

the table. It is obviously seen that the highest level of complexity indeed has only 2

activities and is taught in 1 meeting only.

Table 4.6 Complexity Level and Number of Activity

This section of textbook teaches how to greet and respond the greeting. The

respondents also agree that this textbook does not have adequate recycling in it and

respondents give 2.41 to measure the adequateness of recycling in this textbook. It

shows half of the respondents agree and another half disagree. Textbook does not

have adequate review either according to the result as it is 2.47. The next item should

be paid attention more by the teachers and textbook developer is that the result shows

Complexity

level

Content in

Chapter 1

number of

activity

number of

meeting

1

greet

6

2 ½

2

thank

7

3 ½

3

take leave

2

1

4

apologize

2

1

5

ask for help

1

1


51

there is no reference section of grammar in this book. Almost all respondents agree

that this textbook is very poor of reference section of grammar. The result was 2. The

last for this category, respondents do not see that this textbook promote individual

study.

4. Content of the textbook

Next result to be elaborated is the result of content of textbook. The result

apparently is pretty shocking that mean for all statements in this criterion were rated

at score 2 meaning that the respondents tended to disagree to each statement. As it is

seen in the table 4.7 on page 52, most teachers responded almost disagree to each

statement. According to the teachers, this textbook does not cover main grammar

item. While Brown (2001) explained in his book that teaching grammar is always

needed since students need to know the correct form and structure of English

sentences. Then he proposed judicious question what kind of techniques would be

best for teaching grammar. Then Brown (2001) suggested referring Fotos (1994) and

Doughty & Williams (1998) who talked about appropriate techniques in teaching

grammar.

To be thoughtful responding to the result of the survey, it is better to take a look

at the textbook itself. What is in it? Does this book really not covering the grammar

item? If it is not so, then does this book use the techniques proposed by Fotos (1994)

and Doughty & Williams (1998)?

This textbook seems to be on track with what both experts suggested about

appropriate teaching grammar techniques. At list the first technique they proposed


52

that it must be embedded in meaningful and communicative context (Fotos 1994,

Doughty & Williams 1998). To be surely informed that this discussion is not

discussing the teaching method. Instead, it is still teaching the material (textbook)

itself, since it seems in this textbook teachers must follow the instruction step by step

told in textbook. Therefore, it is clearly seen in textbook that textbook reflects the

teaching techniques should be used in the class.

Table 4.7 Survey Result on the Content of the Textbook Item Statement Strongly

Disagree

(%)

Disagree

(%)

Agree

(%)

Strongly

Agree

(%)

Mean SD

26. Material for

vocabulary teaching

is adequate in terms

of quantity.

5.6 55.6 33.3 5.6 2.35 0.680

28. The textbook

includes the material

for pronunciation

work.

22.2 50 22.2 5.6 2.05 0.802

30. The material for

pronunciation work

covers word stress.

16.7 61.1 11.1 11.1 2.11 0.831

31. The material for 16.7 61.1 11.1 11.1 2.11 0.831

pronunciation work

covers intonation.

29. The material for 55.6 55.6 16.7 0 2 0.766

pronunciation

covers individual

sounds.

27. Material for 5.6 50 38.9 5.6 2.41 0.691

vocabulary teaching

is adequate in terms

of range of

vocabulary.

25. The textbook covers 16.7 61.1 22.2 0 2.05 0.638

the main grammar

item.


53

As seen in figure 4.5 on page 40, the grammar item is actually taught embedded

in the meaningful and communicative context. Textbook teaches the students how to

construct simple presentence using to be (is, am, are) and using verb for present tense.

This textbook also shows the concept of subject-verb agreement in simple present

sentence.

Another appropriate technique to teach grammar used in this textbook is that

this textbook does not use linguistic terminology.

Figure 4.5 Sample of activity found in textbook

Related to the reading skill, teachers from 10 different schools in Special District

of Yogyakarta said that this textbook does not provide the students adequate

vocabulary in terms of quantity. Therefore, most of the teachers had to find

supplementary materials out of this book to fulfill the need of vocabulary for students

although a textbook supposedly roles to fulfill inadequacy of materials (Ur, 1996).

Teachers also saw that the vocabulary in terms of range is not adequate either.


54

5. Language Skill

Based on Tomlinson (2008), a good ELT material must provide a rich

experience of different genre and text types. While text types can be mainly drawn as

two different types. They are spoken text and written text. Years before, Brown

(2001) strongly stated that in highly literate society like today, hundreds of different

types of written text can be used as reading inputs. He mentioned more than 20

different text types for example reports, editorial, article, novels, short stories, letters,

greeting card, etc. He even believed that the written texts are much more than spoken

ones. Unfortunately, the result of the survey shows that most of teachers who

evaluated When English Rings the Bells agree that the book even does not provide

them to improve students’ reading skill.

To assure that the book does not provide reading inputs, a close reading to

textbook should be done. It is found that four out of eleven chapters in this book

completely do not provide any kind of written text. They are chapter 5, chapter 7,

chapter 9 and chapter 10. Chapter 11 is the only chapter which provides longer dialog

and longer descriptive text comparing to other chapters. Except chapters with no text

at all, they always have lyric text. Some of them have short monolog and very short

dialog. Surprisingly, in chapter 6 the book provides a long lyric with difficult

vocabularies comparing to other chapters with simple lyrics. On page 43 figure 4.6 is

the picture showing kind of texts available in the book.

In conclusion, this book does not really provide enough reading text to make the

students experience different genre of text as mentioned by Tomlinson (2008). Even


55

more this textbook does not provide supplementary materials that make the students

experience the extensive reading as Tomlinson (2008) also stated in his book. It is the

fact that this book does not make the book itself become the source to present

adequate materials for learners (Cunningsworth, 1995).

Table 4.8 Survey Result on the Skills Presented in the Textbook


Disagree


Agree

Mean SD

(%) (%)

(%) (%)

35. The textbook

covers writing

skills.

0 27.8 66.7 5.6 2.64 0.836

44. The textbook

covers speaking

skills.

0 17.6 82.4 0 2.81 0.390

A similar result shows at the item of listening skill. The questionnaire tried to find

out how teachers see textbook in relation with listening skill. Surprisingly, almost all

teachers see that this textbook does not cover any listening skill for learners. The

survey result reached 1.93. Since the Mean did not reach 2.5, it means teachers or

respondents did not agree to the statement stating that the textbook covers listening

skills. Contrary to the result of the survey, the researcher of this study sees that in fact

textbook does have listening skill for students since any text in the book can used as

38. The textbook 16.7 66.7 16.7 0 1.93 0.658

covers listening

skills.

32. The textbook 18.8 56.3 25 0 1.88 0.757

covers reading

skills.


56

the input. As it is explained by Krashen (1981) in his input hypothesis that students

can acquire a language when they are given lots of input. The input refers to reading

text or listening text. As in further studies, some resulted in response to the input

hypothesis that there are some characteristics of input which later can be used by ESL

teachers in their teaching. Teachers may modify the input by using caretaker talk

when they read aloud the dialog in textbook. Any dialog in textbook in fact can be

used as the listening input for the learners. Therefore, the researcher in this study

doesn’t see that this textbook does not cover listening skill.

Figure 4.6 Sample of a simple lyric and a very simple dialog

6. Topic Presented in textbook

The last criterion is the topic of textbook. C13 curriculum is well known as a

curriculum with rich of topics in each subject. Other subjects of C13 textbooks are

constructed based on the topic implemented in the base competence. This evaluation


57

is done to find out how far this textbook conceptualizes the topics in the curriculum.

Therefore, to measure the topic of textbook, the researcher gave four statements

related to this criterion. looking closely to the teacher manual. In the manual book, it

is described that textbook covers topics which related to one self and social and

natural environment around the house and school.

Table 4.9 Survey Result on the Topic Presented in the Textbook


Disagree

(%)


Agree

(%)

Mean SD

(%) (%)

48. There is enough

variety of topics.

5.6 38.9 50 5.6 2.58 0.691

50. The students are able to relate the social culture context presented in the textbook to the use of

English in real life.

0 22.2 66.7 11.1 2.87 0.599

52. The characters in the

textbook represent

people from different

region in Indonesia.

0 11.1 77.8 11.1 3 0.5

54. Different groups of occupation are well represented.

0 16.7 72.2 11.1 2.93 0.555

The first is statement is that material of genuine interest to learners is sufficient.

In this statement, the researcher attempts to know whether textbook covers sufficient

55. Other group such as 0 44.4 50 5.6 2.62 0.599

disability is also

represented in the

textbook.

53. Different ethnic 0 22.2 66.7 11.1 2.87 0.599

groups are well

represented.

51. Female is portrayed 0 33.3 61.1 5.6 2.68 0.582 equally with male.

49. The topic presented is 5.6 38.9 50 5.6 2.58 0.691

content-wise enough

for learner’s language

level.

47. There is sufficient 0 38.9 55.6 5.6 2.70 0.570 material of genuine interest to learners.


58

materials that accommodate students’ genuine interest for example textbook gives

topic which is recently relevant to young students. As shown in the table, it can be

seen that teachers see that textbook has already provided genuine topic to learners.

The teachers agreed on the first statement. However, a double check must be done to

confirm whether this textbook does provide sufficient topic of genuine learner

interest. The researcher needs to confirm it by interviewing two respondents. One the

respondent said:

“colorful, many pictures and … look at this picture [pointing to a page of

textbook] this represents Indonesia. So Indonesia, in my opinion. This is

good. The topic is varied.”

However, the researcher thinks that it is not enough to give the topic around the

school and home. In the information technology era like today, where a social media

does really affect the entire way of living most students today, this textbook has not

inserted yet any such topic in textbook.

Generally, teachers see that this textbook has no issue regarding the topic

presented in textbook. The content of textbook is even content-wise enough for

learner’s language level. In contrary to the survey result, the open question results

indicate that this textbook generally is too easy for the students. Teachers also see that

students are able to relate the social context presented in textbook to the use of

English in the real life.


59

C. Discussion

To make this research more valuable and useful, a fruitful discussion regarding to

textbook and its evaluation using the survey given to 20 teachers in Yogyakarta is

explained in this section. Although the curriculum used in the evaluated textbook is

not currently administered by the Minister of Education any longer, some schools in

Indonesia especially the schools which were selected becoming the pilot school to

administer the C13 are still using it until today. Hence, the researcher believes that

this evaluation will be very useful for teachers and material developers in the future.

In order to make the data more valid, this research also uses triangulation data

obtained from interview done to two randomly selected teachers out of 20

respondents. It aims to confirm whether the survey results are confirmed by teachers

directly.

1. The Aim of the Textbook

First discussion goes to the aim of this textbook. This study tries to see whether

the aim of this textbook corresponds with the core competence, basic competence,

teaching program of the curriculum and the needs of the students. In fact, the result of

the survey shows that generally the aim of textbook doesn’t have any issue to this

point. All respondents view that the aim of textbook agrees with the aim of the

curriculum. To see whether it is true, the researcher tries to check what textbook

really aims for. In the preface of textbook, it is written:

“The English lesson for grade 7 of junior high school students presented in this book was

developed to increase the language competency.” (page iii)


60

This means that the English lesson for 7 grade of junior high school presented in

this book was developed to improve English language competency. Meanwhile, a

high English competence student masters the fourth English skills – listening,

reading, speaking and writing skill. In contrary, the fact obtained from the survey

result and close textbook reading show that this textbook does not supply the students

with enough reading texts and its comprehension tasks as well as listening tasks. As it

has been discussed in the result section, most teachers agree that this textbook does

not provide enough reading text. In the interview, which is used as the triangulation

data, both respondents said that this book is lacking of reading texts and listening

activities.

To clarify that this textbook does not provide enough reading texts and listening

activities, two respondents who were interviewed after filling out the questionnaire

elaborated that:

“Definitely, speaking and writing skills are well covered in this book. But the sample of the

conversation for speaking task is too short and simple. It is not challenging. This book doesn’t

cover listening and reading skill pretty well.” (respondent 1)

“I am a little bit confused to answer this. Well, this book doesn’t have listening section. It is

not good because teachers then must find additional material for listening. It bothers us as

teacher [laughing]. And what else? [laughing].” (respondent 2)

Hence, from the evidence above the researcher strongly believe that this textbook

does not celebrate the aim of the syllabus and more over the aim of the curriculum

since this textbook does not give learners room to improve their reading and listening

skills through this textbook.

However, the researcher believes that the first reason underlies the result on the

aim of textbook since no one can tell whether the aim of a textbook correspond to


61

curriculum until the lesson material presented in each chapter is evaluated. If the

lesson material presented in each chapter is not in line with the aim of textbook, this

means that textbook doesn’t correspond the curriculum. Therefore, the researcher

believes that the adapted questionnaire used as the instrument in this research has no

issues regarding the item on the aim of textbook.

2. The Approach of the Textbook

A surprising finding from the survey happened during some interviews conducted

by the researcher. In some interviews, many of the teachers apparently either did not

understand what approach is or they were drowned in a confusion while being asked

about the difference between approach and method. This situation is also reflected in

the way they answer the open questions in the questionnaire about the approach. In

fact, more than half of the teachers have more than 10 years of teaching experience.

From the situation above, the researcher is confident enough to make a

preliminary conclusion that the quality of the English teacher in Indonesia needs

more attention from the government. This can be assured that the quality of English

teacher in remote area throughout Indonesia is even worse.

Officially the C13 applied a new approach called Scientific Approach (SA) which

is derived from the science education framework (Helena, 2014). As the

consequence, from the teachers’ point of view (taken from the interview), most of

ELT teachers find difficult to follow the curriculum since the ELT teacher must build

the skill to communicate while in other hand the government imposed SA, which is

basically aimed to gain or develop new knowledge.


62

Obviously and many would agree that the curriculum seemed to have much

bigger shockwaves to the teachers than ever before (Helena, 2014). Although C13

was only implemented in some appointed school when this study was conducted,

other schools seemed to have the same tense since it seemed that teachers were not

ready for the new curriculum with its new approach. Specifically in ELT context, C13

adopts a Genre-Based Approach (GBA) which aims to develop the ability to

communicate in both spoken and written language.

Both contradictory results which are explained above may happen for reasons.

One of them is that the respondents might not evaluate the book deeper by looking

thoroughly what is presented in textbook and realize that what are presented in

textbook should be established based on the aim of textbook as suggested by Nunan.

Another reason for this was that the questionnaire itself did not give a clear

instruction and explanation.

3. The Design and Organization of the Textbook

The same case happens to the content of textbook. As it is prescribed in the

preface of textbook as well as the teacher manual book, this book is aimed to help the

students to be able to speak English spontaneously or in guided situation with correct

pronunciation, intonation and word stress in the transactional communication.

Surprisingly, there is no single material or activity assisting the students to be able to

speak with correct pronunciation, intonation and stress found in this textbook. The

teachers have the same opinion as the researcher which is shown by the result of the

survey.


63

Still related to the aim of textbook, in the new version of Bloom’s Taxonomy of

learning, there are six steps of difficulty in learning. The very first step of learning is

remembering. Then it is followed by understanding applying, analyzing, evaluating

and creating.

Figure 4.7 The new version of Bloom's taxonomy

Source:(http://sites.duke.edu/eelandscape/2015/03/31/blooms-taxonomy/)

The developer of this textbook definitely does not consider this taxonomy as the

basis to establish the aim of this textbook. After reading closely to the basic

competence written in the teacher manual book, the researcher found that there are

two major problems occur in the basic competence of C13. First, that is the basic

competence seems does not consider Bloom’s Taxonomy as the basis in formulating

the basic competence. As the evidence, there are only two steps from the Bloom’s

taxonomy are applied in the basic competence of C13. The first is understanding,

then jump to the final step which is creating. So in term of difficulty order, this this

textbook doesn’t really follow the order as it is seen from the survey result as well.

The second issue is about the base competence of textbook. The researcher could

not see that the first four base competences mentioned in the teacher manual have any


http://sites.duke.edu/eelandscape/2015/03/31/blooms-taxonomy/

64

relation to the English language teaching and learning. To make this clearer, the

translation of the first four base competences are provided as follow:

1. To be grateful of having opportunity to learn English as the means of International

communication.

2.1 To appreciate polite and care attitude in teacher-student communication and among the

students’ communication.

2.2 To appreciate the attitude of being honest, disciplined, self-confident, and responsible in

transactional communication to the teacher and other students.

2.3 To appreciate the attitude of being responsible, caring, helpful in group working, peaceful

in doing functional communication.

To evaluate student base competence mentioned above, in the manual of textbook

it is described the criteria to evaluate the first four base competences. Those four

basic competences are categorized into the aspect of attitude encompassing respect,

honesty, care, bravery, confidence, communication, social awareness, and curiosity.

In each criterion, teachers must evaluate the students using score from 1 (the lowest)

to 5 (the highest). The students can be evaluated using four different kinds of

evaluation. The first is performance assessment which is meant for the learning result

(outcome) and the learning process. It means that teachers must assess student

performance through their learning process and outcome. In the manual, however, is

not explained the specific form of the learning outcome should be applied, neither the

learning process assessment. It means that this curriculum give teachers freedom to

decide the form of assessment they are desiring to administer. The second is In the

manual of textbook, it is also explained. As the consequence, many teachers complain

about the assessment systems since the variables in base competence of number one

and two are difficult to assess. Both variables do not related to the English teaching


65

and learning. However, teachers must evaluate the students using them during the

program.

These three cases indicate that the developer of this textbook does not pay

attention to the basic principles of materials development and to important theories in

developing teaching materials. Lesson units presented in textbook are supposed to be

based on what the aim of textbook has established. Interestingly, most teachers as the

evaluators of this textbook also have the same notion regarding material

development. Because, logically if the teachers saw that the lesson units presented in

textbook had the problem such as no enough reading and listening text or no

pronunciation materials, they then would not say that the aim of textbook did not

have any problem at all. However, fortunately there are still two teachers out of 17

who see that the aim of textbook does have problem. Surely, these two teachers are

highly able to evaluate textbook comprehensively.

Another evidence shows that this textbook is not a good resource for the students

when the teachers answer the open question in the questionnaire. Many of the

respondents say that this textbook has very limited materials. In addition to it,

textbook does not challenge the students. It is considered too easy. This definitely is

against what Piaget has proposed about cognitive development which is divided into

4 stages according to the age.

The typography of the textbook is intangible and the lively colorful pictures seem

to well suit the needs of adolescents for some respondents. However, the number of

the pictures presented in the textbook seemed to be not well suited and tended to be


66

distracter since in the age of junior high school students, usually called as adolescents

(between 11 – 15 years old), the students can learn something more abstract and

containing logical ways. Therefore, for late language learners such as adolescents and

adults, the most cases occur in Indonesia, new vocabulary is easier to learn than new

sounds or new grammar. In contrary, as the result survey shows, this textbook hardly

provides adequate vocabulary in term of quantity. This condition was substantiated

by the respondent as shown in the interview transcript below:

“Oh God, this textbook is definitely under standard. I am really sure that this book was made

in a rush. A very noticeable part is that many typos in this book. And it is too easy for my

students, too many pictures but lack of tasks and texts. The government really didn’t pay

attention to it, I think.” (respondent 2)

“This book is good. It is colorful. Many pictures support the activity. But sometimes I feel it

is too many. There is no listening section I guess and the text is very limited though I add

with my own texts.” (respondent 2)

4. Language Content of the Textbook

The survey result also tells that this textbook does not provide pronunciation

exercise as well as word stress and intonation. According to the survey result, this

textbook do not promote either the vocabulary building or grammar exercise. In

which, again, this proves what Tomlinson (2008) explained that textbooks fail to help

the learners to make full use of the language experience available to them outside the

classroom. The researcher seems to doubt that this textbook was created and

developed based on the theories underlying the material development.

5. Language Skill Presented in the Textbook

Those two survey results are confirmed by other survey result. The teachers

disagree that this textbook provides adequate reading and listening texts for the

students. In result, it is not surprising that teachers do not promote this textbook as a


67

good resource book for the students. In the survey result, the teachers do not agree

either this textbook covers the students need or a good textbook. In fact, Nation

theorizes that material developers must agonize over the four strands to create and

develop a balanced textbook. They have to assure that the learners later can learn

from textbook through the reading and listening materials presented in textbook as

the meaning-focused input. In other word, in Krashen’s (1981) term, this textbook

does not provide enough language input text namely listening and reading text. In

short, this textbook obviously is not considering the language learning and teaching

principles as well as the material development’s principles.

6. The Topic Presented in the Textbook

The research shows that this textbook entitled When English Rings the Bell

presents some positive qualities. This textbook represents both gender groups, various

ethnic group in Indonesia and different religion groups. This textbook has

accommodated the cultural and group diversity in Indonesia. This textbook is also

easy to get because students and teachers can get this book by downloading from the

Indonesia Education Ministry ‘s website. Regarding the topic presented in textbook,

this book also has accommodated the relevance topic to students although in terms of

the content, teachers found that this textbook was too easy for the students at that

time. The writing and speaking skill are adequately presented in textbook according

the survey result.

In spite of the positive quality of textbook, the research also presents the pitfalls

of textbook. It is worth mentioning that the criteria which teachers least satisfied is


68

the skill presented in the textbook and the language content that comprises grammar

reference, recycling and review, and vocabulary. With respect to the big number of

the writing and speaking exercise of this textbook, teachers felt shortcomings of this

textbook in terms of providing listening and reading exercises are real. It seemed that

the developer of this textbook managed to take the emphasize on the productive skills

namely writing and speaking skill. As the matter of fact language learning is

definitely needing the receptive and productive skills in balanced as suggested by

Krashen (1981). On different sub criterion, it was found that this textbook seemed to

not giving enough room for students to use this book independently since it is lacked

of grammar reference. This textbook also shows that there is inconsistency between

the curriculum and the lesson unit presented in textbook. The researcher found that

this textbook was not developed with a strong foundation of basic principles in

developing textbook. Shortly this textbook in some aspects is positively responded

and in some more other aspects negatively responded by the teachers.

Commenting on this research itself, the researcher believes that this research

stands close to other similar research conducted by experts and other scholars in

terms of the method by conducting a survey. It is worth mentioning such as what Litz

(2002) researched in South Korea on a textbook evaluation and ELT management.

Another scholar to mention is Sadehvandi (2012) who also conducted textbook

evaluation by administering a survey in Iranian EFL context.

However, this research unavoidably has shortcomings that need to be improved in

the future. The main shortcoming that is obviously seen and conceptually may raise


69

questions is the validation process. A single validation process, namely expert

judgment, was only undergone for this research which undoubtfully it is still under

the standard of validation process. This research should undergo the validation

process to test the items of the questionnaire. Therefore, in the future the researcher

is validating the items of the questionnaire to administer other same kind of

evaluation research.


CHAPTER V

CONCLUSION

In this very last chapter, the conclusion chapter consists of two parts namely, the

conclusion and suggestions. The conclusion is briefly restating the answer of the

research question which are 1) is the textbook entitled When English Rings the Bells

effective? And 2) What are the strengths and weaknesses of the textbook according to

the survey? In answering the research questions, the researcher found some

drawbacks of the study. Therefore, besides presenting the conclusion, this very final

chapter is also presenting suggestion for future researchers.

A. CONCLUSION

By the end of the day, there are scarcely textbook or materials which are

developed close to the state of perfect. Students, teachers and readers must be able to

locate the pitfalls or the drawbacks of any material, not least this textbook. This

textbook according to the survey result, in some aspects is very effective and

ineffective in some other aspects. To start the concluding chapter, the researcher is

giving the strength of this textbook prior to the drawbacks of the textbook.

The most prominent strength of this textbook would be the attractiveness. This

textbook was designed in a colorful way and illustrated with attractive comic-like

characters. This definitely will impress the students in a good first impression.

However, the attractiveness of a textbook is not the only aspect to determine whether

a textbook is good and effective. It is true that it is quite effective to attract the

students to learn using this textbook. But the teachers must be very critical to the

70


71

students’ response since however, by the end of the day students are autonomously be

able determine whether a book is worth to study of not considering the content and

the usefulness of the book itself. For the Indonesia context, this textbook effectively

accommodates the plurality in terms of culture, ethnic group and religion as well as

gender since it is easily pointed that there are many pictures symbolizing certain

ethnic group, culture, religion and gender. In regard to the discussion of the language

skill presented in the textbook, it seems that writing skill exercises are quite well

developed although the some of the instruction of the tasks need to be revised. The

speaking skill exercises and tasks pretty much show their quality though some

revisions need to conduct.

In some other aspect, the textbook is see as ineffective to the respondents. The

evaluation result on the aim of this textbook shows that the developing of this

textbook did not consider the basic foundation which should be generated in the very

beginning process of its development. Although in some ways around in this textbook

the developers state clearly the basic competence and foundation, the lesson units in

this textbook do not confirm what have been set in the basic competence.

Regarding to the approach, this textbook was written according to the newest

curriculum in Indonesia called C13. The teachers looked over textbook then decided

that this textbook has been in line with what the C13 suggested as approach to be

employed namely scientific approach (SA). However, the results of the evaluation

show that many English teachers in Indonesia especially in junior high schools have

very limited knowledge regarding the approach of teaching. Besides, according to


72

some experts in English language teaching in Indonesia, the approach implemented in

the curriculum is hardly suitable for English teaching and learning process. As the

consequence, this textbook raised confusion among the teachers as it is shown in the

result of the open question questionnaire. The ultimate conclusion regarding approach

of this textbook is that material developers fail to determine and describe rigidly to

the EFL teachers in junior high school in Indonesia.

The respondents of this textbook evaluation appreciate the organization of this

textbook in various results. The respondents rate that this textbook does not hold

enough recycling and review. Even though this textbook is aimed to improve

communicative competence, this textbook was assessed by the teachers as textbook

which does not have grammar reference where students can refer to in their individual

learning. Therefore, teachers do not promote this textbook for independent learning.

In terms of the content of textbook, teachers positively agree that this materials

ware very poorly developed. Besides the teachers couldn’t find any grammar

reference in the textbook, apparently, the grammar items are hardly found in the

textbook. The teachers even got difficulty to conduct their classes using this textbook

since this book does not provide either adequate vocabulary learning in terms of

quantity and range or pronunciation works in terms of individual sounds, word stress

and intonation. As the result, this textbook alone is far from being enough to

becoming the main textbook in the class. As the evidence, all teachers who

participated in this survey gave their students more materials outside the textbook as

the supplementary materials.


73

This textbook is not either completely effective or ineffective. In some aspects of

the textbook, When English Rings the Bells is really effective. While in some more

other aspects of the textbook, it does not support effective teaching and learning

process. Shortly, the researcher sees that there are more aspects of the textbook which

do not support EFL teaching and learning process. Therefore, it is concluded that this

textbook tends to be ineffective one.

B. Suggestions

All the drawbacks resulted in the evaluation of this textbook are believed because

the textbook was not designed based on a strong fundamental grounded theory of

developing materials. Additionally the curriculum itself was prematurely tested and

validated prior to the field implementation. Therefore, this study highly recommends

the curriculum and materials developers to carefully develop them by considering the

most fundamental principles in developing curriculum and materials. In addition, they

also need to consider theories on English language teaching and learning to support

their decision in order to create a reliable and effective textbook.

Considering the trend of Indonesia’s government in keep changing the curriculum

every time a new government is elected, teachers in Indonesia must be able to

conduct autonomous textbook evaluation to make decision in choosing the teaching

and learning materials in their class. They can use the available evaluative questions

such as developed by Cunnisngworth, Nunan, etc. Thus, teachers in Indonesia are not

depending on the unreliable and effective materials. However, to do so, teachers must

improve their knowledge in EFL learning and teaching principles as well as materials


74

development by having focused group discussion among the teachers in certain

region or attending national and international conference and training in teaching

EFL.

The researcher of this textbook suggests to the next researcher to limit and narrow

down the scope of the evaluation. In other words, future researcher must be more

specific to the certain elements of the textbook. If the research will be conducted in a

survey research like this, the future researcher must be able to give clearer instruction

in the questionnaire by giving guideline to fill out the questionnaire. However, the

researcher suggests that future researchers must do a more empirical evaluation

respectively like what Ellis (1997) pinned in her article entitled The Empirical

Evaluation of Language Teachinng Materials using pre-test and post-test to see a

more precise evaluation on the materials although it is believed that there is no

definite yardstick or evaluation system to analyze a book.


75

Bibliography

Alamri, A. A. (2008). An Evaluation of Sixth Grade English Language Textbook for

Saudi Boy's School. Kingdom of Saudi Arabia: King Saud University.

Alderson, J. C. (1992). Guidelines for the Evaluation of Language Education. In A.

Baretta, & J. C. Alderson (Ed.), evaluating second language education.

Cambridge: Cambridge University Press.

Allwright, R. L. (1981). What do we want teaching materials for? ELT Journal, 1, 5-

18.

Brown, H. D. (2001). Teaching by Principles: An Interactive Approach to Language

Pedagogy. New York: Addison Wesley Longman.

Cunningsworth, A. (1995). Choosing Your Coursebook. Oxford: Heinemann.

Doughty, C., & Williams, J. (1998). Focus on Form in Classroom Second Language

Acquisition. New York: Cambridge University Press.

Ellis, R. (1997). The Empirical Evaluation of Language Teaching Materilas. ELT, 36-

42.

Ellis, R. (1998). Evaluation of Communicative Task. Cambridge: Cambridge

University Press.

Fotos, S. S. (1994). Integrating Grammar Instruction and Communicative Language

Use through Grammar Consciousness-raising Tasks. TESOL Quarterly 28, pp.

51-323.

Graves, K. (2000). Designing Language Course: A Guide for Teachers. Boston:

Heinle & Heinle Publishers.

Hamied, F. A. (2014). Curriculum Change: What Does It Mean to Indonesia TEFL?

In H. P. Widodo, & N. T. Zacharias, Recent Issues in English Language

Education: Challenges and Directions (pp. 13-37). Surakarta: UNS Press.

Helena, I. R. (2014, October 7). The English Curriculum: The Paradigm,

Interpretation, and Impementation . The Recent Issues in English Language

Education: Challenges and Directions, pp. 39-64.


76

Hewings, M., & Dudely-Evans, T. (Eds.). (1996). Evaluatiion and Course Design in

EAP. Hertfordshire: Prentice Hall Macmillan.

Hutchinson, T., & Waters, A. (1991). English for Specific Purposes. New York:

Cambridge University Press.

J. Mukundan, V. N. (2011). Developing an English Language TextbookEvaluation

Checklist: A Focus Group Study. International Journal of Humanities and

Social Science, 1(12), 100-106.

Jones, S. (2009). A Retrospective Evaluation of ELT for Korean University

Conversation Course. Seoul.

Karnodi, O. R. (2014). Implementing the Authentic Assessment Embodied in the 2013

English Curriculum. School of Postgraduate Studies. Bandung: Indonesia

University of Education.

Krashen, S. (1981). Second Language Acquisotion and Second Language Learning.

Pergamon Press Inc.

Krashen, S. (1982). Principles and Practice in Second Language Acquisition.

Pergamon Press Inc.

Lawrence, W. P. (2011). Textbook Evaluation: A Framework for Evaluating the

Fitness of the Hong Kong New Secondary School (NSS) Curriculum. Hong

Kong City: University of Hong Kong.

Litz, D. R. (2002). Textbook Evaluation and ELT Management: A South Korean

Case Study. Asian EFL Journal, 48, 1-53.

McDonough, J., Shaw, C., & Masuhara, H. (2013). Materials and Methods in ELT.

Oxford: Balckwell.

Nemati, Z. (2009). Evaluation of an ESL English Coursebook: A Step towards

Systematic Vocabulary Evaluation. Karnataka: Univeristy of Mysore.

Neuman, W. L. (2006). Basic of Social Research: Qualitative and Quantitative

Approaches (second edition). Boston: Pearson Education Inc.

Nunan, D. (1989). Designing Tasks for the Communicative Classroom. Cambridge:

the Press Syndicate of the University of Cambridge.

Rea-Dickinds, P., & Germaine, K. (1992). Evaluation. Oxford: Oxford University

Press.


77

Richards, J. C. (2001). Curriculum Development in Language Teaching. Singapore:

Cambridge University Press.

Rowntree, D. (1997). Making Materials-based Learning Work. London: Kagan Page.

Sadehvandi, M. A. (2012). Textbook Evaluation: EFL Teachers' Perspective on

"Pacesetter Series". ELT Journal, 5, 64-74.

Sadehvandi, M. A. (2012). Textbook Evaluation: EFL Teachers' Perspective on

"Precestter Serries". ELT Journal, 5, 64 - 74.

Sanders. (1992). Evaluating School Programs: An Educator's Guide. Newburry Park:

Corwin Press.

Sheldon, L. E. (1988). Evaluating ELT Textbooks and Materials. ELT Journal, 237-

246.

Textbook Evaluation: EFL Teachers' Perspective on "Pacesetter Series". (2012). ELT

Journal, 5, 64-74.

Tomlinson, B. (2008). English Language Learning Materials: A Critical Review.

New York: Continum International Publishing Group.

Tsiplakides, I. (2011). Selecting an English Coursebook: Theory and Practice. Theory

and Practice in English Studies Volume 1 No 7, pp. 758-764.

Ur, P. (1991). A Course in Language Teaching. Cambridge: Cambridge University

Press.

Weir, C., & Roberts, J. (1994). Evaluation in ELT. Oxford: Blackwell.

Zamanian, M. R. (2015). Textbook Evaluation ELT Teachers' Perspective on

"Learning to Read English for Pre-University Students". International Journal

of Research Studies in Language Learning, 4, 105-118.


78

Appendices


Categories

Questions

YES

NO

A Aims and

approaches

1

Do the aims of textbook correspond closely with

the aims of the teaching program and with the

needs of the learners?

Apakah tujuan buku ‘When English Rings the Bell’

sejalan dengan tujuan program pengajaran dan

sejalan dengan kebutuhan pembelajar?

2

Is textbook suited to the learning/teaching

situation?

Apakah buku ‘When English Rings the Bell’cocok

untuk situasi belajar mengajar di kelas Anda?

3

Does it cover most of all what is needed?

Apakah buku ‘When English Rings the Bell’

mencakup semua apa yang dibutuhkan?

4

Is it good resource for students and teachers?


merupakan sumber yang bagus bagi murid dan

guru?

5

Is textbook flexible? Does it allow teaching and

learning styles?


fleksibel? Dalam hal ini, apakah buku ini memberi

keleluasaan dalam model 78u8 mpengajaran dan

pembelajaran?

B Design and

organization

1

Is there any other supporting media such as CD,

workbook, teacher’s book?

Apakah ada media pendukung lainnya untuk buku

‘When English Rings the Bell’ contohnya seperti


79

CD, buku latihan, atau buku guru?

2

Is the organization right for teachers? (e.g.

According to structures, functions, topics,

skills, etc)

Apakah pengorganisasian buku ‘When English

Rings the Bell’ sudah baik untuk guru? (missal

berdasarkan struktur bahasa (grammar), fungsi,

topik, ketrampilan dll)

Is the organization right for the learners? (e.g.

According to structures, functions, topics,

skills, etc)

Apakah pengorganisasian buku ‘When English

Rings the Bell’ sudah baik untuk pembelajar?

(missal berdasarkan struktur bahasa (grammar),

fungsi, topik, ketrampilan dll)

Is the content sequenced on the basis of

complexity?

Apakah isi buku ‘When English Rings the Bell’

disusun berdasarkan tingkat kerumitan?


‘learnaibility’?


disusun berdasarkan tingkat keterbelajaran?


usefulness?


disusun berdasarkan kegunaan?

6

Is the grading and the progression suitable for

the learners?

Apakah system penilaian dan aktifitas di

dalam buku ‘When English Rings the Bell’ sesuai


80

dengan pembelajar?

Does it allow them to complete the work

needed to meet any external syllabus

requirements?


memberi kesempatan pembelajar untuk

mengerjakan semua tugas-tugas di dalamnya

dalam rangka memenuhi ketentuan-ketentuan

dalam silabus?

5

Is there adequate recycling and revision?

Apakah ada daur ulang materi dan revisi

materi?

6

Are there reference sections for grammar, etc?

Apakah ada bagian referensi untuk grammar

dll?

Is some of the material suitable for individual

study?

Apakah beberapa materi yang ada di dalam

buku ‘When English Rings the Bell’ cocok/sesuai

untuk pembelajaran secara individual?

7

Is it easy to find your way around textbook?

Apakah mudah menggunakan buku ini?

Is the layout clear?

Apakah layout buku ini jelas?

C Language

content

1

Does textbook cover the main grammar items

appropriate to each level, taking learners’ need

into account?

Apakah buku ini mencakup item grammar

yang sesuai dengan level, yang


81

mempertimbangkan kebutuhan murid?

2

Is material for vocabulary teaching adequate

in terms of quantity and range of vocabulary,

emphasis placed on vocabulary development,

strategies for individual learning?

Apakah materi untuk pengajaran kosakata

cukup memadai dalam hal kuntitas dan range

kosakata, penekanan pada pengembangan

kosakata, strategi untuk pembelajaran

mandiri?

3

Does textbook include material for

pronunciation work?

Apakah buku ini memasukkan materi untuk

pengnucapan?

If yes, does the pronunciation work cover

individual sounds?

Jika iya, apakah materi tersebut mencakup

bunyi yang berdiri sendiri?

Does the pronunciation work cover word

stress?

Apakah materi tersebut mencakup penekanan

pada kata?

Does the pronunciation work cover sentence

stress?

Apakah materi tersebut mencakup penekanan

pada kalimat?

Does the pronunciation work cover

intonation?

Apakah materi tersebut mencakup intonasi?


82

4

Does textbook deal with the structuring and

conventions of language use above sentence

level, eg. How to take part in conversations,

how to structure a piepce of extended writing,

how to identify the main points in a reading

passage? (More relevant at intermediate and

advanced levels.)

5

Are style and appropriacy dealt with? If so, is

language style matched to social situation?

Categories

Questions

YES

NO

D Skills

1 Are all four skills adequately covered,

bearing in mind your course aims and

syllabus requirements?

Apakah keempat skill berbahasa

secara proporsional tecakup dalam

buku ini dengan mempertimbangkan

tujuan pembelajaran di kelas Anda dan

persyaratan silabus?

2 Is there material for integrated skill

work?

Adakah materi di dalam buku ini

dengan skil terpadu (integrated)?

3 Are reading passages and associated

activities suitable for your students’

levels, interests, etc?

Apakah teks bacaan dan latihan-

latihannya sesuai dengan :

a. level pembelajar di kelas Anda?

b. minat pembelajar di kelas Anda?

Is there sufficient reading material?


83

Adakah meteri bacaan yang

mencukupi?

4 Is listening material well recorded, as

authentic as possible, accompanied by

background information, questions and

activities which help comprehension?

Apakah materi mendengarkan

(listening) :

a. direkam dengan baik?

b. otentik?

c. disertai infomasi pendahuluan?

d. disertai pertanyaan dan aktifitas

yang membantu pemahaman?

5 Is material for spoken English

(dialogues, roleplays, etc) well

designed to equip learners for real-life

interactions?

Apakah materi untuk percakapan

bahasa Inggris ( seperti dialog dan

role play) dirancang dengan baik

untuk membekali pembelajar di

kehidupan nyata?

6 Are writing activities suitable in terms

of amount of guidance/control, degree

of accuracy, organization of longer

pieces of writing (eg paragraphing) and

use of appropriate styles?

Apakaha aktifitas menulis sudah sesuai

dalam hal

a. jumlah panduan/control

b. tingkat keakuratan


84

c. pengorganisasian panjang

pendeknya tulisan

d. penggunaan gaya penulisan?

Categories

Questions

Yes

No

E Topics

1 Is there sufficient material of genuine

interest to learners?

Adakah materi yang cukup berkenaan

dengan minat asli untuk pembelajar?

2 Is there enough variety and range of

topic?

Adakah variasi dan rentangan topic

yang cukup ada di buku ini?

3 Will the topics help expand students’

awareness and enrich their experience?

Akankah topic yang disajikan

membantu memperluas kesadaran

pembelajar dan memperkaya

pengalaman mereka?

4 Are the topics sophisticated enough in

content, yet within the learners’

language level?

Apakah topic yang disajikan cukup

canggih/rumit/sulit dalam hal:

a. isi?

b. tingkat penguasaan bahasa para

pembelajar?

5 Will your students be able to relate the

social cultural contexts presented in the

course book?


85

Akankah pembelajar Anda mampu

menghubungkan konteks sosial budaya

yang disajikan dalam buku ini?

6 Are women portrayed and represented

equally with men?

Apakah figure perempuan

digambarkan dan ditampilkan

seimbang dan sepadan dengan figur

laki-laki?

7 Are other groups represented, with

reference to ethnic, origin, occupation,

disability, etc?

Apakah kelompok minoritas dari:

a. suku

b. ras

c. pekerjaan

d. orang-orag cacat

e. agama

juga direpresentasikan?

F Methodology

1 What approach/approaches to language

learning are taken by textbook? Is this

appropriate to the learning/teaching

situation?

Pendekatan pemebelajaran bahasa apa

saja yang digunakan di dalam buku

ini?

Apakah pendekatan ini sesuai untuk

situasi pembelajaran dan pengajaran

di kelas Anda?


86

2 What level of active learner

involvement can be expected? Does

this match your students’ learning

styles and expectations?

Level pembelajar apa yang bisa secara

aktif terlibat dalam pembelajaran

menggunakan buku ini?

3 What techniques are used for

presenting/practicing new language

items? Are they suitable for your

learners?

Teknik apa yang digunakan untuk

mempresentasikan/ mempraktekkan

item bahasa yang baru?

4 How are the different skills taught?

Bagaimana setiap skill berbahasa

diajarkan?

Membaca:

Menulis:

Mendengarkan:

Berbicara:

5 How are communicative abilities

developed?

Bagaimana kemampuan berkomunikasi

dikembangkan di dalam buku ini?

6 Does material include any advice/help

to students on study skills and learning

strategies?

Apakah buku ini juga memasukkan

tips/saran kepada pembelajar dalam


87

mempelajari skill dan strategi belajar?

7 Are students expected to take a degree

of responsibility for their own learning

(eg by setting their own individual

learning targets)?

Apakah ada di antara

pembelajar/murid yang mempunyai

target pribadi mereka dalam hal

belajar bahasa Inggris? (missal

setelah lulus harus sudah bisa lancer

berbahasa Inggris)

G Teacher’s

books

1 Is there adequate guidance for the

teachers who will be using textbook

and its supporting materials?

Apakah ada panduan yang menuckupi

untuk para guru yang akan

menggunakan buku ini?

Dan adakah materi-materi

pendukungnya?

2 Are the teachers’ books comprehensive

and supportive?

Apakah buku guru tersebut lengkap

dan membantu?

3 Do they adequately cover teaching

techniques, language items such as

grammar rules and culture-specific

information?

Apakah buku guru dan materi

pendukung mencakup:

a. teknik pengajaran?

b. item kebahasaan seperti grammar?


88

c. serta catatan informasi budaya

secara spesifik?

4 Do the writers set out and justify the

basic premises and principles

underlying the material?

Apakah penulis penulis buku guru

tersebut memeberi pernyataan dan

penilaian tentang aturan dasar dan

prinsip dasar tentang buku murid ini?

5 Are keys to exercise given?

Apakah kunci jawaban latihan

diberikan di buku guru?

H Practical

considerations

1 What does the whole package cost?

Does this represent good value for

money?

Berapa harga buku ini?

Apakah gratis?

2 Are the books strong and long lasting?

Are they attractive in appearance?

Apakah buku ini akan tahan lama?

Apakah buku ini cukup menarik secara

penampilan?

3 Are they easy to obtain?

Apakah buku ini mudah didapatkan?

4 Do any parts of the package require

particular equipment, such as a

language laboratory, listening center or

video player? If so, do you have the

equipment available for use and is it


89

reliable?

Apakah di antara bagian di dalam

buku ini membutuhkan peralatan

khusus lainnya seperti laboratorium

bahasa, listening center atau videa

player?

Jika iya, apakah sekolah Anda

mempunyainya?


90


A Questionnaire of

English Textbook Evaluation of Curriculum 2013

A. Statements

Please, tick accordingly.

1=strongly disagree, 2=disagree , 3= agree, 4=strongly agree

Statements

1

2

3

4

1. The aims of textbook correspond with the aim of the curriculum.

2. The aim of textbook correspond the core competence (KI)

3. The aims of textbook correspond with the aim of the teaching

program.

4. The aims of textbook correspond with the needs of learners.

5. Textbook is suited to the learning situation.

6. Textbook is suited to the teaching situation.

7. Textbook covers most of all student needs.

8. The courebook is a good learning resource for students.

9. Textbook is a good learning resource for teachers.

10. Textbook is flexible so it allows various teaching styles.

11. Textbook is flexible so it allows the learning styles.

12. The book support media such as CD, workbook, and teacher’s

manual.

13. The organization of the book is right for the teacher. (for

example the organization is based on the structure order, topic

based order, or difficulty based order)


91

14. The organization of the book is right for the learner. (for

example the organization is based on the structure order, topic

based order, or difficulty based order)

15. The content is well sequenced on the basis of complexity.

16. The content is well sequenced on the basis of learnability.

17. The content is well sequenced on the basis of usefulness.

18. There is adequate recycling. (Example of recycling is an

expression such as refusal “I am sorry, I am afraid I can’t” is

presented in unit 1 and it is repeated in unit 3, 6 and 8.

19. There is adequate review. (For example there is always review

after finishing 3 units)

20. There is reference section of grammar in the book.

21. The materials promote individual study.

22. It is easy for learners to find their way around textbook.

23. It is easy for teacher to find his/her way around.

24. The book has good layout.

25. Textbook covers the main grammar item.

26. Material for vocabulary teaching is adequate in terms of

quantity.

27. Material for vocabulary teaching is adequate in terms of range of

vocabulary.

28. Textbook includes the material for pronunciation work.

29. The material for pronunciation work covers individual sounds.

30. The material for pronunciation work covers word stress.

31. The material for pronunciation work covers intonation.


92

Continued

1=strongly disagree, 2=disagree, 3= agree, 4=strongly agree

Statements

1

2

3

4

32. Textbook covers the reading skill.

If your respond for this statement is Agree (3) or Strongly Agree (4), and then please

answer number 33 and 34.

33. The level of difficulty of the reading passages is appropriate to

the students.

34. The level of associated activities for reading passages is

appropriate to the students.

35. Textbook covers the writing skill.

If your respond for this statement is Agree (3) or Strongly Agree (4) and then please


36. The level of writing activities is appropriate to the students.

37. The writing activities are suitable in terms of amount of

guidance/control.

38. Textbook covers the listening skill.


answer number 39 up to 43.

39. The level of listening texts is appropriate to the students.

40. The listening material is well recorded.

41. The listening material is authentic.

42. The listening material is accompanied by background

information which helps comprehension.

43. The listening material is accompanied by question and activities

which help comprehension.


93

Continued

1=strongly disagree, 2=disagree, 3= agree, 4=strongly agree

Statements

1

2

3

4

56. The teacher’s manual is comprehensive.

57. The teacher’s manual is supportive.

58. The teacher’s manual adequately covers teaching technique.

44. Textbook covers the speaking skill.



45. The level of speaking activities is appropriate to the students.

46. The material for speaking activities such as dialogues and

roleplays is well designed to equip students for real-life

interactions?

47. There is sufficient material of genuine interest to learners.

48. There is enough variety of topics.

49. The topic presented is content-wise enough for the learner’s

language level.

50. The students are able to relate the social cultural context

presented in textbook to the use of English in the real life.

51. Female is portrayed equally with male.

52. The characters in the book represent people from different region

in Indonesia

53. Different ethnic groups are well represented.


55. Other group such as disability is also represented in textbook.


94

B.

1.

Open Questions

What approach/approaches to language learning are taken by textbook?

2. Is the approach to language learning taken by textbook appropriate for your class

situation? Why? (Please give a brief explanation of the reason.)

59. The teaching technique in the teacher’s manual book in line with

what the curriculum proposes.

60. The teacher’s manual adequately covers language items such as

grammar and pronunciation.

61. The teacher’s manual book adequately covers culture-specific

information.

62. The keys to exercises are given in the teacher’s manual book.

63. The book is affordable for students.

64. The book is affordable for the haves-not students.

65. The book is strong and long lasting.

66. The book is easy to obtain.

67. Some parts of textbook require a language laboratory.

68. Some parts of textbook require listening center.

69. Some parts of textbook require video player.

70. Your school has the equipment available for the proper use of

the book.

71. The equipment your school has is reliable.


95

3. What approach/approaches to language teaching do you use?

4. What level of active learner involvement can be expected from textbook? Does this

match your students’ learning styles and expectations?

5. What techniques are used for presenting/practicing new language items? Are they

suitable for your learners?

6. How are the different skills taught? Explain briefly.

a. Reading

b. Writing

c. Listening


96

d. Speaking

7. How are communicative abilities developed?

8. In your opinion, what is the major problem of this textbook? (Please give a

explanation)

brief

9. Do you always make lesson plan for every single lesson for you class? If not, why?

10. Do you think that lesson plan is important? Why or why not?

11. If the book is accompanied by a set of lesson plan for each unit to help the teacher,

do you think that you will use it or adapt it? Why or why not?


97


98

Appendix 3: The Blueprint of the Questionnaire

A Questionnaire for Teachers to Evaluate

A Textbook Entitled When English Rings the Bell

Dear participants,

This questionnaire is a part of a research for attaining a Master degree in English

Language Studies, Sanata Dharma University. The research is to find out whether

textbook proposed by the Indonesia Education Ministry entitled When English Rings

the Bells is effective according to the result survey to teachers who are applying the

C13 curriculum and using this book in their class. The aim of this questionnaire is for

the participants to check give evaluation to each statement according to the

experience and knowledge as English teacher of Junior High School in Indonesia.

There are two parts of the questionnaire should be fill out. Part A is the statement in

which the participant must tick to each statement accordingly. Part B is the open

questions. In this part, participant must answer the questions briefly and clearly by

writing the answer at the provided blank space.

Name

School

Experience

(in year)

:

:

:

C. Statements

Please, tick accordingly.

1=strongly disagree, 2=disagree , 3= agree, 4=strongly agree

Statements

1

2

3

4

72. The aims of textbook correspond with the aim of the

curriculum.

73. The aim of textbook correspond the core competence (KI)

74. The aims of textbook correspond with the aim of the

teaching program.

BLUEPRINT


99

75. The aims of textbook correspond with the needs of

learners.

76. Textbook is suited to the learning situation.

77. Textbook is suited to the teaching situation.

78. Textbook covers most of all student needs.

79. The courebook is a good learning resource for students.

80. Textbook is a good learning resource for teachers.

81. Textbook is flexible so it allows various teaching styles.

82. Textbook is flexible so it allows the learning styles.

83. The book support media such as CD, workbook, and

teacher’s manual.

84. The organization of the book is right for teacher (for

example the organization is based on the structure order,

topic based order, or difficulty based order).

85. The organization of the book is right for the learner (for

example the organization is based on the structure order,

topic based order, or difficulty based order).

86. The content is well sequenced on the basis of complexity.

87. The content is well sequenced on the basis of learnability.

88. The content is well sequenced on the basis of usefulness.

89. There is adequate recycling. (Example of recycling is an

expression such as refusal “I am sorry, I am afraid I can’t”

is presented in unit 1 and it is repeated in unit 3, 6 and 8.

90. There is adequate review. (For example there is always

review after finishing 3 units)


100

91. There is reference section of grammar in the book.

92. The materials promote individual study.

93. It is easy for learners to find their way around textbook.

94. It is easy for teacher to find his/her way around.

95. The book has good layout.

96. Textbook covers the main grammar item.


quantity.


range of vocabulary.

99. Textbook includes the material for pronunciation work.

100. The material for pronunciation work covers

individual sounds.

101. The material for pronunciation work covers word

stress.

102. The material for pronunciation work covers

intonation.

103. Textbook covers the reading skill.

If your response for this statement is Agree (3) or Strongly Agree

(4), and then please answer number 33 and 34.

104. The level of difficulty of the reading passages is

appropriate to the students.

105. The level of associated activities for reading

passages is appropriate to the students.

106. Textbook covers the writing skill.


101

If your response for this statement is Agree (3) or Strongly Agree

(4) and then please answer number 36 and 37.

107. The level of writing activities is appropriate to the

students.

108. The writing activities are suitable in terms of

amount of guidance/control.

109. Textbook covers the listening skill.

If your response for this statement is Agree (3) or Strongly

Agree (4) and then please answer number 39 up to 43.

110. The level of listening texts is appropriate to the

students.

111. The listening material is well recorded.

112. The listening material is authentic.

113. The listening material is accompanied by

background information which helps comprehension

114. The listening skill material is accompanied by

question and activities which helps the comprehension.

115. Textbook covers speaking skill

If your response for this statement is Agree (3) or Strongly

Agree (4) and then please answer number 45 and 46.

116. The level of speaking activities is appropriate to the

students.

117. The material for speaking activities such as

dialogues and roleplays is well designed to equip students

for real-life interactions?

118. There is sufficient material of genuine interest to

learners.

119. There is enough variety of topics.


102

120. The topic presented is content-wise enough for the

learner’s language level.

121. The students are able to relate the social cultural

context presented in textbook to the use of English in the

real life.

122. Female is portrayed equally with male.

123. The characters in the book represent people from

different region in Indonesia

124. Different ethnic groups are well represented.


126. Other group such as disability is also represented in

textbook.

127. The teacher’s manual is comprehensive.

128. The teacher’s manual is supportive.

129. The teacher’s manual adequately covers teaching

technique.

130. The teaching technique in the teacher’s manual

book in line with what the curriculum proposes.

131. The teacher’s manual adequately covers language

items such as grammar and pronunciation.

132. The teacher’s manual book adequately covers

culture-specific information.

133. The keys to exercises are given in the teacher’s

manual book.

134. The book is affordable for students.


103

D. Open Questions

12. What approach/approaches to language learning are taken by textbook?

13. Is the approach to language learning taken by textbook appropriate for your

class situation? Why? (Please give a brief explanation of the reason.)

14. What approach/approaches to language teaching do you use?

15. What level of active learner involvement can be expected from textbook?

Does this match your students’ learning styles and expectations?

16. What techniques are used for presenting/practicing new language items? Are

they suitable for your learners?

135. The book is affordable for the haves-not students.

136. The book is strong and long lasting.

137. The book is easy to obtain.

138. Some parts of textbook require a language

laboratory.

139. Some parts of textbook require listening center.

140. Some parts of textbook require video player.

141. Your school has the equipment available for the

proper use of the book.

142. The equipment your school has is reliable.


104

17. How are the different skills taught? Explain briefly.

e. Reading

f. Writing

g. Listening

h. Speaking

18. How are communicative abilities developed?

19. In your opinion, what is the major problem of this textbook? (Please give a

brief explanation)

20. Do you always make lesson plan for every single lesson for you class? If not,

why?

21. Do you think that lesson plan is important? Why or why not?


105

22. If the book is accompanied by a set of lesson plan for each unit to help the

teacher, do you think that you will use it or adapt it? Why or why not?


Appendix 4: The Respondents’ on Statement

106

respondent

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

Experience

(years)

16

-

19

15

-

20

7

16

10

10

10

20

26

14

17

35

2

Statement

1.

3

3

3

3

3

3

2

3

2

4

3

4

4

4

3

3

3

2.

3

3

3

3

3

3

2

3

2

3

3

4

4

4

3

2

3

3.

3

3

2

3

3

3

2

3

2

3

3

4

4

4

3

3

3

4.

3

3

2

3

3

3

2

2

2

3

2

4

3

3

3

3

3

5.

3

3

3

3

3

2

1

3

2

3

4

2

3

3

3

2

3

6.

3

3

3

3

3

3

1

2

2

3

4

2

3

3

3

2

3

7.

3

3

2

2

2

2

1

2

2

3

2

2

3

2

2

2

3

8.

3

3

1

3

3

2

2

2

2

3

4

2

3

2

2

2

3

9.

3

3

3

3

3

2

2

2

2

2

4

2

3

2

3

2

3

10.

3

3

2

3

3

2

2

3

2

3

4

2

3

3

3

3

4

11.

3

3

3

3

3

2

2

2

2

3

4

2

3

3

3

3

3

12.

3

3

3

2

3

2

1

2

2

3

4

3

4

1

2

2

2

13.

3

3

2

2

3

2

2

2

2

3

4

3

4

3

2

2

2

14.

3

3

2

3

3

2

2

2

2

3

4

3

3

2

2

2

15.

3

3

2

3

3

2

2

3

2

3

3

3

3

3

3

2

2

16.

3

3

3

3

3

2

2

3

2

3

3

3

4

3

3

2

2

17.

3

3

3

3

3

2

2

3

2

3

3

3

3

3

3

2

2

18.

3

2

3

3

3

2

3

3

2

3

1

2

3

1

3

2

2

19.

3

2

3

2

2

2

3

3

2

3

3

2

4

1

3

2

2

20.

3

2

2

2

2

2

1

3

2

2

2

2

3

1

2

1

2


107

21.

2

3

2

3

2

2

2

3

2

3

2

2

4

1

3

2

4

22.

3

2

2

3

2

2

2

3

3

3

3

2

4

3

3

2

3

23.

3

3

2

3

2

2

2

3

2

3

3

2

4

3

3

2

3

24.

3

3

2

4

4

2

4

3

2

3

3

2

4

3

2

2

3

25.

2

2

3

3

2

2

1

3

2

2

2

2

3

1

2

1

2

26.

3

2

2

2

2

2

1

2

3

3

4

2

3

2

3

2

2

27.

3

2

2

3

2

2

1

2

3

3

4

2

3

2

3

2

2

28.

3

2

1

2

2

2

1

2

2

2

3

2

4

2

3

1

1

29.

3

2

1

2

2

2

1

2

2

2

2

2

4

2

3

1

1

30.

3

2

2

2

2

2

1

2

2

2

4

2

4

2

2

1

1

31.

3

2

2

2

2

2

1

2

2

2

4

2

4

2

2

1

1

32.

3

2

2

2

2

2

1

3

0

2

2

2

3

1

2

1

2

33.

3

0

0

0

0

0

0

3

0

0

0

0

4

0

0

0

0

34.

3

0

0

0

0

0

0

3

0

0

0

0

4

0

0

0

0

35.

3

3

3

3

3

2

2

3

0

3

3

2

4

3

3

2

3

36.

3

3

4

3

3

0

0

3

0

3

4

0

4

3

3

0

3

37.

3

3

3

3

2

0

0

3

0

3

3

0

3

3

3

0

3

38.

2

2

3

3

2

2

2

2

0

2

1

2

2

1

3

1

1

39.

0

0

3

3

0

0

0

0

0

0

0

0

0

0

3

0

0

40.

0

0

2

2

0

0

0

0

0

0

0

0

0

0

3

0

0

41.

0

0

3

2

0

0

0

0

0

0

0

0

0

0

2

0

0

42.

0

0

2

2

0

0

0

0

0

0

0

0

0

0

2

0

0

43.

0

0

3

2

0

0

0

0

0

0

0

0

0

0

2

0

0

44.

3

3

3

3

3

2

2

3

0

3

3

3

3

3

3

2

3

45.

3

3

3

3

3

0

0

3

0

3

4

3

3

3

3

0

3


108

46.

3

2

2

2

2

0

0

3

0

3

4

2

3

3

3

0

3

47.

3

2

2

3

2

3

2

3

3

3

4

2

3

3

3

2

3

48.

3

2

3

3

3

3

2

2

2

3

4

2

3

2

3

1

3

49.

3

3

2

3

3

3

2

3

2

3

4

2

3

2

3

1

2

50.

3

3

3

3

3

2

2

3

3

0

4

2

4

3

3

2

3

51.

3

3

2

3

3

2

2

3

2

0

3

3

4

3

3

2

2

52.

3

3

2

3

3

2

4

3

3

0

3

3

4

3

3

3

3

53.

3

3

2

3

3

2

4

3

3

0

2

3

4

3

2

3

3

54.

3

3

3

3

3

2

4

3

2

0

2

3

4

3

3

3

3

55.

3

3

3

2

2

2

4

3

3

0

2

2

3

2

2

3

3

56.

3

3

3

3

3

2

2

3

3

0

4

3

3

1

3

3

3

57.

3

3

3

3

3

2

2

3

3

0

4

3

3

1

3

3

3

58.

3

3

3

3

3

2

2

3

3

0

4

3

4

1

3

2

2

59.

3

3

3

3

3

2

2

3

3

0

4

3

3

1

3

2

2

60.

3

2

2

2

2

2

2

3

3

0

3

2

4

1

2

3

3

61.

3

2

2

3

2

0

1

3

3

0

4

2

4

1

3

2

3

62.

2

3

4

2

2

2

1

1

3

0

4

2

3

1

2

2

2

63.

2

3

4

3

3

3

3

2

2

0

4

3

3

3

3

3

3

64.

2

3

3

3

3

3

3

2

2

0

4

3

3

3

3

3

3

65.

2

3

3

3

3

2

1

2

2

0

4

3

3

1

2

3

3

66.

2

3

1

3

3

3

4

2

2

0

4

3

3

3

3

3

3

67.

2

2

3

3

1

3

3

2

2

0

3

2

4

2

3

2

2

68.

2

2

1

3

1

2

1

2

2

0

4

2

3

2

3

2

2

69.

2

3

1

4

2

2

2

2

2

0

3

2

3

2

3

2

2

70.

2

3

2

3

3

2

4

2

2

0

4

3

3

4

3

3

3


Note:

1.

2.

3.

4.

5.

Zero (0) means the participant didn’t respond to the statement.

One (1) means the participant strongly disagree with the statement.

Two (2) means the participant disagree with the statement.

Three (3) means the participant agrees to the statement.

Four (4) means the participant strongly agrees to the statement.

109

71.

2

3

1

3

3

3

4

2

2

0

3

3

3

4

3

3

3


Appendix 5: The Respondents’ Answer on Open Question Questionnaire

110

Resp

on

den

t

Ex

perien

ce

What

approach/appr

oaches to

language

learning is

taken by the

coursebook?

What

approach/appro

aches to

language

teaching do you

use?

Is the approach to

language learning

taken by the

coursebook

appropriate for you

class situation? Why?

In your opinion

what is the

major problem

of this

coursebook?

1

-

Scientific

approach

Scientific

approach

The steps are okay but

the forms of task in

some ways are not

suitable. As teachers, I

have to find other tasks

which are suitable for

my students. Most of

tasks are not

challenging for my

students.

It is not enough

if we as teachers,

solely use the

book as a source.

Teachers should

make/create

other learning

experiences for

their students in

the form of task

and practice.

2

20

Scientific

approach

Scientific

approach

Yes, because the

students’ English

competence is above

the average.

The content of

the book is too

easy for the

students. As the

teachers, we

need much

sources for the

teaching

materials.

3

7

Scientific

approach

Scientific

approach

Yes, because the book

is too easy for the

students of SMP 1

Sleman. So the teacher

must make another

work sheet to

influence their

understanding.

The coursebook

is too easy for

our student so

we need another

sources to teach

our student. We

make worksheet

to influences the

students’

knowledges.

4

26

-

The students are

asking actively

Because it contains the

picture which suitable

with the situations of

Students can

only do exercise

if they are

explained by


book is not so

give a challenge

111

our area.

their teacher.

5

14

Based on the

statement in the

coursebook, the

approach is

text-based

learning.

Various

approaches

And

communicative

approach

Yes. Because a text-

based approach

provides a very wide

space for the teacher

and students to vary

their class with any

creative method.

This coursebook

is too simple. In

other words, the

deep and doesn’t

to the students.

6

35

Scientific

approach

Scientific

approach

No, too simple. There

are no task for

exposure.

No text. So the

students have no

chance to

observe.

7

2

Scientific

approach

Scientific

approach

-

It’s used some

text so student

can practice their

reading

(pronunciation).

But using some

pictures can

make student

explore more

information from

theirself.

8

20

The curriculum

suggests that all

subjects are

taught based in

scientific

approach i.e.

observing,

questioning,

exploring,

associating, and

communication.

This time being,

I am still trying

to find the link

between this

approach and

language

learning

approach I’ve

So far, based on

KTsP I use

genre-based

approach to

language

teaching. This

approach is used

to teach year 8

and 9. For year 7

I am suing the

Scientific

approach as

recommended by

the curriculum.

Quite appropriate as

far as the teachers are

creative enough to

create students tasks

by themselves,

because the

coursebook doesn’t

display any clear steps

or tasks for students.

There are no

clear tasks

designed to

develop which

skills. This book

presents mostly

pictures and

simple

expressions or

sentences with

no clear

purposes until

the teachers refer

themselves to the

teacher’s book.


the coursebook

of the

not good.

112

known.

9

11

Scientific

learning

Both Scientific

learning and

three phase

technique

Not really. Sometimes

my students still need

to be guided in

understanding the

material. For example

teacher must write the

explanation on the

board, etc.

It is lack of texts.

10

17

The book takes

a text-based

learning

approach.

Communicative

and text-based

approach.

Yes it is. The text-

based approach

provides the teaches to

use various methods.

The materials is

no so deep and

doesn’t give

more challenge.

11

10

The coursebook

use Speech

approach

I think, we will

be better if we

use easy

approach (like

pictures, CD and

video)

No, it is not. Because many students

don’t know the

knowledge of English

especially the student

that have low

understanding in

English when they are

in SD.

The coursebook

doesn’t complete The materials of

are limited.

The arrangement

coursebook is

12

10

Discovery

learning, project

based learning,

communicative

learning (CLT),

role play, CTL

teaching.

Discovery

learning for

reading and

writing.

Communicative

learning for

speaking.

No, because the

approach to language

learning in general

appropriate of the

material.

I think, the

coursebook of

2013 curriculum

is more colorful

picture so it is

suitable for

speaking.

But in teaching

reading and

writing is not

suitable. So my

problems are in

teaching reading

and writing if

used this

coursebook of

the ring bells. So

I used the others

book to teach


example of

enough.

teacher has to

resources to

material.

experimenting, and

networking) the

113

reading and

writing.

13

16

Scientific

approach

Scientific

approach

Yes, because it is not

only transfer

knowledge from

teacher to student but

student can see , ask

and analize what

teacher present.

Limited

materials

There are not

conversation

14

19

Scientific

approach

Scientific

approach

Yes it is but not

always. It depends on

the materials.

There is no

grammar focus.

There is no

listening and

speaking.

15

15

Scientific

approach

Communicative

approach

I use active

learning

approach. I try to

make the

students active

in learning, and

the most

important thing

that they can

apply it in their

real life.

Sometimes I also

use co-operative

and collaborative

learning.

I think it is appropriate

but sometimes it

depends on the topic of

learning.

Using 5 steps in

learning (observing,

questioning,

associating,

communicating/

students can learn

more active and enjoy

the learning.

The book covers

minimal

material, so the

find other

develop the

16

16

Scientific

approach

Scientific

approach

Yes, sesuai. Membuat

anak lebih creative.

Masih ada

beberapa

kekurangan/

kelemahan

diantara:

Skill listening

tidak ada

Skill reading

kurang


114

17

10

Learning based

on text. Speech

or written by an

English

language as a

communication

language

Contextual

language

teaching

Yes, it is. Because

students can

comprehend easily if

in language learning

use learning by texts

It is very simple

because our

students have

had the materials

from primary

school.


Appendix 6: Translated transcript of the interview

Interview 1

Researcher: What is your comment towards English textbook for grade 7

entitled When English Rings the Bell?

Respondent 1: Oh my God, this textbook is definitely under standard. I am really

sure that this book was made in a rush. A very noticeable part is that many typos

in this book. And it is too easy for my students, too many pictures but lack of

tasks and texts. The government really didn’t pay attention to it, I think. Really…

I can’t believe it. At the end, as usual… teachers must work harder to find more

materials which are suitable for the students. There are too many problems in this

book AND the curriculum itself.”

Researcher: What is wrong with the curriculum?

Respondent 1: Well, the major problem is the approach. In the curriculum it is

clearly stated that teachers must apply the scientific approach. However, I never

heard of such approach in language teaching and learning. Neither have other

teachers. We are in the big confusion because we are mandated to apply what the

curriculum says but in other hand we ARE aware that such approach doesn’t exist

in language teaching and learning.

Secondly, one of the scoring aspects for final examination is way too much. How

can we measure the attitude through English language learning process? Yes we

are equipped with teacher’s guide. However, the implementation is way more

difficult than what it is prescribed in the teacher’s manual. Too many aspects that

are not correlated to language learning need to be measured. Teachers are

overwhelmed.

Researcher: What does this book lack?

Respondent 1: Hmm,, obviously, this book lacks reading texts and listening tasks.

Do you agree, mas? Right? If you take a look at it closely, this book is lacked of

reading texts. And there is no listening section. So teachers must prepare them all.

Researcher: So do you think that this book doesn’t cover all skills?

Respondent 1: Definitely, speaking and writing are well covered in this book. But

the sample of the conversation for speaking task is too short and too simple. It is

not challenging. This book doesn’t cover listening and reading skill pretty well.

Researcher: Do you think this book has covered all the student needs?

Respondent 1: Absolutely no. You know, this book is too easy and students need

more texts, need more vocabulary. Look, even there is no grammar in it.

Researcher: Sorry to interrupt, so you think that grammar is important to be

taught to the students?

Respondent 1: Well, not directly. But at least students can refer to the book for

grammar if they find any difficulties when they study at home by themselves.

Researcher: Sorry, so are you saying that this book does not promote auto

learning or independent learning? Respondent 1: Hmm… yes, sort of. Students cannot learn by themselves. They need teachers or instructors when they study using this book. Researcher: So what do you do responding this situation?

Respondent: I do anything possible to do for my students such as finding more

relevant materials. I mostly find more texts to support my teaching. And to make

more challenging I change a little bit the tasks in textbook and sometimes I

115


replace it with other different tasks because, you know, the exercises in the book

are too easy for my students.

Interview 2

Researcher: What is your comment towards English textbook for grade 7

entitled When English Rings the Bell?

Respondent 2: [release breath] This book is good. It is colorful. Many pictures

support the activity. But sometimes I feel it is too many. Hmm … [opening

textbook then look at it briefly] oh ya… there is no listening section I guess and

the text is very limited though I add with my own texts. I use previous textbook

and other textbook to complete When English Rings the Bell.

Researcher: So what is the strength of this textbook?

Respondent 2: colorful, many pictures and … look at this picture [pointing to a

page of textbook] this represents Indonesia. So Indonesia, in my opinion. This is

good. The topic is varied.

Researcher: And what do you of the lack of this textbook?

Respondent 2: I am a little bit confused to answer this. Well, this book doesn’t

have listening section. It is not good because teachers then must find additional

material for listening. It is bother us as teacher [laughing]. And what else mas?

[laughing]

Researcher: What about the approach, Sir?

Respondent 2: Ah… that one. It is good. Sometimes the Scientific Approach (SA)

can be applied in my class but sometimes cannot. And I think I never heard of

scientific approach in English teaching, is it? It’s been years since I graduated

from my university. I forgot many things about theory of teaching English

language. [laughing]

Researcher: So do you think the SA is suitable for your class?

Respondent 2: [sigh] sometimes [smiling]. I think there isn’t such approach in

English teaching. But since the government told us to apply it in our class, so I

must apply it.

Researcher: So do you apply this approach completely?

Respondent 2: Sometimes. So I modify the approach sometimes.

Researcher: So do you use textbook every time you teach?

Respondent 2: Not all the time. But most of the time, I use textbook as the guide.

So when I feel I need to add materials, then I will find it because this textbook is

too easy for many of my students.

Researcher: Do you have anything else to say about this textbook?

Respondent 2: Yes, this book still needs to be revised. [smile]

116


THE EVALUATION OF THE SEVENTH-GRADE ENGLISH …I should also thank to my first English teacher in an...

Documents

Transcript of THE EVALUATION OF THE SEVENTH-GRADE ENGLISH …I should also thank to my first English teacher in an...