YDS AS A BENCHMARK IN TURKEY: THE DIFFICULTY LEVELS OF ...

95

AbstractThe ability to speak a language covers all skills, namely reading, writing, speaking,

and listening, together with a reasonable degree of performance and competence knowledge. Specializing over a single skill by overlooking the others may hinder speaker proficiency. In other words, all skills equally weigh; hence, the tests that aim to measure language proficiency should be able to gauge all skills in a good balance. In spite of that reality, YDS (Foreign Language Exam) in Turkey scores test-takers only through reading questions, and accordingly, this test may not be a good benchmark for Turkey because reading questions do not seem to have a stable difficulty level. The exam is composed of different kinds of questions which are vocabulary, grammar, comprehension, dialogue, and reading. The reading questions take up the biggest portion in the exam -20 out of 80 questions-; therefore, they highly determine the test-takers’ scores. This study investigated eleven YDS tests held in the last 5 years. In total eleven YDS tests (220 reading questions) were analysed. Coleman-Liau Index was used to measure how difficult each test is. This study aims to reveal whether reading questions of YDS have a similar difficulty level by calculating the Coleman-Liau Index. This study is of great importance because YDS is a crucial benchmark test in Turkey that determines the English proficiency level of those who take it, and any discrepancy in difficulty levels may negatively affect the reliability of the test. The results showed that the reading questions are in a large range of difficulty, and do not follow a stable track of difficulty.

Keywords: YDS, Benchmark, Difficulty, Reading, Coleman.

*) Dr.Öğr.Üyesi,YabancıDillerYüksekokuluMütercimTercümanlık(İngilizce) (e-posta:[email protected])ORCIDID:https://orcid.org/0000-0003-2588-372X

YDS AS A BENCHMARK IN TURKEY: THE DIFFICULTY LEVELS OF READING COMPREHENSION QUESTIONS

Cüneyt DEMİR (*)

EKEV AKADEMİ DERGİSİ • Yıl: 23 Sayı: 80 (Güz 2019)Makalenin geliş tarihi: 4.11.20191. Hakem rapor tarihi: 12.12.20192. Hakem rapor tarihi: 14.12.20193. Hakem rapor tarihi: 15.12.2019Kabul tarihi: 16.12.2019

96 / Dr. Cüneyt DEMİR EKEV AKADEMİ DERGİSİ

Türkiye’de Yabancı Dil Becerisini Ölçme Sınavı Olarak YDS: Okuduğunu Anlama Sorularının Zorluk Düzeylerinin İncelenmesi

ÖzBir dili kullanabilme, onun okuma, yazma, konuşma ve dinleme gibi bütün

becerileriyle kullanabilmeyi gerektirmektedir. Bir beceri üzerinde gelişirken bir diğerinin zayıf bırakılması dili öğrenen kişinin tam akıcı olması önünde bir engeldir. Diğer bir ifadeyle, bir dili ölçerken bütün becerileriyle ölçmek gereklidir. Bu gerçeğe rağmen Türkiye’de uygulanan YDS sadece okuma becerisi üzerinden puanlama yapmaktadır. Fakat ne yazık ki bu test standart bir zorluk seviyesine sahip olmadığı için iyi bir ölçme sınavı olmayabilir. Bu sınav kelime bilgisi, dil bilgisi, okuduğunu anlama, diyalog ve okuma sorularından oluşmaktadır. Seksen sorunun yirmisini oluşturan okuma soruları sınavda en fazla ağırlığı olan soru türüdür ve bundan dolayı puanlamada esas teşkil eder. Bu çalışma YDS’de son 5 yıl içerisinde sorulan toplam on bir sınavın okuduğunu anlama sorularını analiz etti. Toplamda 220 okuduğunu anlama sorusu analiz edildi. Her testin hangi zorluk düzeyinde olduğunu bulmak için Coleman-Liau analiz tekniğini kullandı. Bu çalışma Coleman-Liau tekniğiyle YDS sınavında okuduğunu anlama sorularının aynı zorluk düzeyinde olup olmadıklarını araştırmayı amaçlamaktadır. YDS Türkiye’de yabancı dil becerisini ölçen en önemli sınav olduğu için, bu çalışma testin güvenirliliği konusunda bir fikir edinmede büyük bir önem arz etmektedir ve soruların zorluk düzeylerinde ki büyük değişiklikler sınav güvenirliliğini etkileyecektir. Sonuçlar okuma parçalarının zorluk düzeyleri arasında farklılıkların olduğunu ve standart bir zorluk düzeyine sahip olmadıklarını göstermiştir.

Anahtar Kelimeler: YDS, Ölçüt, Zorluk, Okuma, Coleman.

1. IntroductionThepurposeofanexamis tomeasureknowledge.Differentfromclassicalexams,

language examsmeasure not only the knowledge but also the proficiency, andwhiledoingthis,thequestions,eitherwrittenorspoken,arepreparedmeticulouslysothatthereliabilityoftheexamswouldnotbenegativelyaffected.Languagetestinghasbeengainingpopularityallovertheworldandcountriesmaketheirexamssothattheycandeterminethelanguageproficienciesoftheircitizens.Forexample,CanadahasitslanguagetestingofCELPIP(TheCanadianEnglishLanguageProficiencyIndexProgram);AmericahasTOEFL;UKhasIELTS;TurkeyhasYDSandsoon.OfallinternationalEnglishlanguagetests,TOEFLand IELTSare themost prevalent ones; valid nearly in all countries.Acommon feature of these exams is that theymeasure not a single skill but a generalproficiencyofthespeakersuchaswriting,reading,listeningandspeaking.Comparedtolanguagetestsgauginggenerallanguageskills,YDSdoesnottestthegenerallanguageproficiencyofaspeakerbutonlyvocabulary,grammarandreading,andallmeasuringrequireyoutosolvemultiple-choicequestions.

97YDS AS A BENCHMARK IN TURKEY: THE DIFFICULTY LEVELS OF READING COMPREHENSION QUESTIONS

Therearemanymotivesfortakinglanguageproficiencytestssuchasqualificationforapositionandacceptanceforauniversity.Whateverthereasonis,manypeoplestrugglewithteststohaveagoodmark,andtakethetestrepeatedlyparticularlywhentheygetfail,orarenotabletogettherequiredscoretheyneed.Thereareoccasionswhenatesttakerwasnotabletopassinthefirstexamwhiles/hedidinthesubsequentone,whichshouldbebecausethetest-takerstudiedmore;however,thecaseisnotalwayslikethis,andthetesttakerpassjustbecausethedifficultylevelofthetestwasreduced.Thisisaninfelicityintermsoftestreliability,andmustbedealtwith.Languageproficiencytestingisparticularlysensibletotestreliabilityasanymeaninglessfluctuationsindifficultymayresultinunfairscores,henceinaccurateassessmentoftest-takers.

Assessmentiscrucialtodeterminethelanguagelevelsofspeakersandhasasignificantrole in the process of teaching and learning (Lam, 2015). Traditional human-basedassessmentisaffectedbydifferentfactorssuchascontext,psychologicalstateofthetesttaker,assessorandmotivation;therefore,anythingthatmayaffectoneofthesefactorsmay promptwrong-assessment, and accordingly scoring error.However, the standardqualityofabenchmarkisthemostimportantfactorparticularlyifitisimplementedonlargescales.Anyfluctuationindifficultymayresultininsurescoring,i.e.thelevelsoftest-takerscannotbemeasuredfairly.Inlinewiththis,thisstudyaimstoanalysereadingcomprehensionquestionsinYDS(ForeignLanguageexam)between2013and2018totrackthedifficultylevelsoftheexams.ThisstudyproposesthatthestabilityofreadabilityofcomprehensionquestionsinYDSshouldbemaintainedtoacquirereliabletestresults.Onlyreadingcomprehensionquestionswereevaluatedbecausereadingcomprehensionisacomplexandmultidimensionalprocess(Carlson,Seipel,&McMaster,2014);thereforeitcanreadilymeasureEnglishproficiency

1.1. What is YDS?YDSisatestheldindifferentlanguagestomeasurelanguageproficiencyinTurkey.

TheacronymofYDS,calledKPDSbefore2013,standsforYabancı Dil TestiinTurkish,the English of which would be Foreign Language Test. It was being. YDS is heldbiannuallyandhas80questions.ItisnotatestmeasuringgeneralEnglishskillsbutjustreadingskill.Thequestionsarecomposedofvocabulary(6),grammar(10),clozetests(10),sentencecompletion(10),translation(6),readingcomprehension(20),dialogue(5),closestinmeaning(4),paragraphcompletion(4),andirrelevantsentence(5).Youneedtotakethistesttocontinueyourhighereducation,tobeanacademicianorforotherspecificpurposes.Atest-takershouldhaveascoreofatleast50tobesuccessful.Thescoringisdoneonascaleof100.TheequivalentqualificationsofYDSareprovidedintable1.


Table 1.EquivalentQualificationsandTests

YDS TOEFL iBT PTE Academic CPE CAE100 120 90 A90 108 84 B80 96 78 C A70 84 71 B60 72 55 C50 60 45

1.2. Reading Comprehension Questions and ScoringCompared to other types of questions, reading comprehension questions requires

multiplelanguageproficienciessuchasnotonlyknowledgeofwordrecognitionskillsbutalsootherlinguisticandextralinguisticskills.Thismultidimensionaldomainembodiesanumberoflanguagecomponentsthatnecessitatereaderstohaveacompetentaptitudeforlanguageproficiency.Therefore,apartfromfluencyinwordrecognition(Perfetti,1999),the reader is to be capable of analysing morphological awareness (Deacon, Kieffer,& Laroche, 2014), metacognitive knowledge (Perfetti, Landi, & Oakhill, 2005), andsyntacticawareness(Nagy,2007).Accordingly,onlyreadingcomprehensionquestionswereevaluatedinthisstudymainlyfortworeasons:thefirst,theyconstitutethemajorityof the questions inYDS; therefore, they are themost influential part of scoring.Thesecond,readingcomprehensionisaunitaryskill,i.e.readingcomprehensionisacomplexandmultidimensionalprocess(Carlson,Seipel,&McMaster,2014);therefore,inordertodecodethetexts,atest-takerneedstohaveacompetenceingrammar,ahighlexicalreservoir,andpragmaticknowledgeofthecontext.Concisely,properlypreparedreadingcomprehension questions aremore qualified to gauge theEnglish proficiency of test-takersthananyotherquestionstypes,whichiswhyreadingcomprehensionquestionsisofparamountimportanceforanytestaimingatmeasuringlanguageskill.

1.3. Research Aim and Research QuestionsThe present study aims to measure the difficulty level of reading comprehension

questionsofYDSimplementedbetween2013and2018byusingColeman-Liauindex,andtoshowthemostdifficultandtheeasiestreadingpassages(researchquestion1).AcommonviewwithcomprehensionquestionsinTurkeyis that theynecessitatealargelexical reservoir, hence are difficultwhen compared to other types of questions.Thisstudy aims to investigatewhether there is a positive or negative correlation betweenCLIandmeanexamsuccessesof test-takers (researchquestion2).Finally, eachYDShas five reading passages and the calibration of these passages is important; it is not


reliableforanexamtohavereadingpassagesatdifferentdifficulties.Inlinewiththat,thisstudyaimstoinvestigatewhichYDShasthemostvolatilityinthedifficultyofreadingcomprehensionpassages(researchquestion3).Specifically,thisstudyaimstoanswertheresearchquestionsbelow.

1. What term of YDS has the highest and the lowest CLI regarding readingcomprehensionpassages?

2. IsthereanycorrelationbetweenCLIandmeanexamsuccess?3. What term of YDS has the most and least volatility in difficulty of reading

comprehensionpassages?

2. Methodology2.1. DataThedataarecomposedofelevenYDSexamsheldbetweenonly2013-2018years

because theexamshadbeenheldunderadifferentnamebefore2013;namelyKPDS.Furthermore,thenumberofquestionshasbeendecreasedto80from100inthewakeofKPDS,andalsothenumberofreadingcomprehensionquestions.Therefore,theexamsbeforeYDS,whichwereKPDS,wereexcludedfromthedata.EachYDSexamincludesfive reading passages and four questions for each passage; in total twenty readingcomprehension questions. Table 2 provides detailed information about the questionsanalysed.

Table 2.TheDetailedContentoftheData

Exam Number of Passages Number of questions Number of words2018SpringTerm 5 20 10122017 Fall Term 5 20 10582017SpringTerm 5 20 11052016 Fall Term 5 20 10602016SpringTerm 5 20 10292015 Fall Term 5 20 9822015SpringTerm 5 20 11162014 Fall Term 5 20 9702014SpringTerm 5 20 10202013 Fall Term 5 20 10242013SpringTerm 5 20 1005TOTAL 55 220 11381 (1035 x)


2.2. Missing DataTwoexamsareheldeachyear;spring termandfall term.Inspiteof this, thedata

includedonlyasingleexamin2018(springterm)becausetheinstitutionholdingYDSdidnotsharetheexamalthoughitwascontactedtothem.Additionally,thisstudydidnotanalyseallquestion types inYDSbutonly thoseof readingcomprehensionquestionsbecausereadabilitytestssuchasCLIneedacertainnumberofwordstobeabletocalculatethedifficultylevel.Theotherquestionssuchasvocabulary,grammar,andsoforthdonothaveenoughwordstoreachareliableramification.Forthisreason,thepresentstudyonlyanalysedreadingcomprehensionquestions;inaddition,theyconstitutealargepartofthewholeexamwhencomparedtootherquestiontypes.

2.3. Coleman-Liau index (CLI)ThisstudyusedColeman-Liauindex(CLI)toinvestigatethedifficultylevelofeach

exam.CLI is a readability test that investigateshowdifficult a text is, thusallowsustounderstand thegrade level. It iswidelyused in theUSAandsomeothercountries.Unlikemostof theothergrade-levelpredictors,CLIdependsoncharacters insteadofsyllables perword. In otherwords, insteadof referring to syllable/word and sentencelength indices,“MeriColemanandT.L.Liaubelievedthatcomputerizedassessmentsunderstandcharactersmoreeasilyandaccurately thancountingsyllablesandsentencelength” (Readable Blog, 2017); therefore, this index is superior to other readabilityindicesrelyingonsyllable-countingtechniques.TheformulathatCLIusedisshowninfigure1.Thesummaryoftheformulaisthis:(5.89*characters/words)−(0.3*sentences/(100*words))−15.8.

Figure 1. Theformulaofcoleman-liauindex.

ThehighnumberofCLIshouldbe interpretedasadifficultreadingpassagewhilelowerCLIfiguresshowtheeasinessofthepassages.Thenumberyouobtainisthegradingequivalenceoftheresults;forexample,CLIof12referstograde12(collegelevel)or4referstograde4(primaryschoollevel).

2.4. Procedure The exams are legally available on the web page of the institution implementing

theYDSexams(OSYM,2016).Havingcompiledtheexams,theresearcherpickedupreadingcomprehensionquestions.Atotalfifty-fivereadingpassageswerecollectedandeachreadingpassagewasanalysedthroughtheformulainfigure1.Asecondassessor,a

widely used in the USA and some other countries. Unlike most of the

other grade-level predictors, CLI depends on characters instead of

syllables per word. In other words, instead of referring to syllable/word

and sentence length indices, “Meri Coleman and T. L. Liau believed

that computerized assessments understand characters more easily and

accurately than counting syllables and sentence length” (Readable

Blog, 2017); therefore, this index is superior to other readability indices

relying on syllable-counting techniques. The formula that CLI used is

shown in figure 1. The summary of the formula is this: (5.89 *

characters/words) − (0.3 * sentences / (100 * words)) − 15.8.

Figure 1. The formula of coleman-liau index.

The high number of CLI should be interpreted as a difficult

reading passage while lower CLI figures show the easiness of the

passages. The number you obtain is the grading equivalence of the

results; for example, CLI of 12 refers to grade 12 (college level) or 4

refers to grade 4 (primary school level).

2.4. Procedure

The exams are legally available on the web page of the

institution implementing the YDS exams (OSYM, 2016). Having

compiled the exams, the researcher picked up reading comprehension

questions. A total fifty-five reading passages were collected and each

reading passage was analysed through the formula in figure 1. A second

assessor, a specialist on statistics, called for calculating CLI of

randomly selected six passages (11% of whole data) in case the

researcher might make miscalculation, and found the same results as

me. The CLI of each passage and each exam was delivered through

tables in the result section. Each table presents the number of characters,

number of words, number of sentences, the average number of

characters per word, the average number of syllables per word, and the


specialistonstatistics,calledforcalculatingCLIofrandomlyselectedsixpassages(11%ofwholedata) incase the researchermightmakemiscalculation, and found the sameresultsasme.TheCLIofeachpassageandeachexamwasdeliveredthroughtablesintheresultsection.Eachtablepresentsthenumberofcharacters,numberofwords,numberofsentences,theaveragenumberofcharactersperword,theaveragenumberofsyllablesperword,andtheaveragenumberofwordspersentencebecauseallthesedataarenecessarytomakeaccuratecalculationstheyareimportantincalculatingtheCLIaccurately.

3. Results Resultswereprovidedthroughtables,eachreferringtoadifferentexam.Theanalyses

started from2013 fall termYDSexam to2018 spring termexam.Table3 shows theresultsregardingYDS2013fallterm.

Table 3.CLIAnalysisResultsofYDS2013FallTermExam.

YDS 2013 Fall Term Passage1

Passage 2

Passage 3

Passage 4

Passage 5 Average

Number of characters 1058 1058 1089 1033 1023 1052

Numberofwords 202 216 229 192 203 208

Number of sentences 10 9 9 8 13 10

Averagenumberofcharactersperword 5.24 4.90 4.76 5.38 5.04 5

Average number of syllables per word 1.80 1.68 1.58 1.86 1.76 1.7

Averagenumberofwordspersentence 20.20 24 25.44 24 15.62 21.3

CLI 13.56 11.80 11.03 14.64 11.96 12.53

TheCLI differences between passages are notmarked and relatively close to oneanother,whichrepresentsasmoothtransitionbetweenpassages.The4thpassagehasthehighestCLIwhilethe3rdhasthelowest,andthemeanCLIforthisexamis12.53.Table4showsresultsregarding2013YDSspringterm.


Table 4.CLIAnalysisResultsofYDS2013SpringTermExam.

YDS 2013 Spring Term Passage1

Passage2

Passage 3

Passage 4

Passage 5 Average


Numberofwords 187 194 219 216 202 204

Number of sentences 8 8 11 13 8 9.6

Averagenumberofcharactersperword 5.61 5.1 5.3 4.6 5 5.1


Averagenumberofwordspersentence 23.4 24.3 20 16.7 25.3 21.2

CLI 15.96 13 13.78 9.64 12.49 12.90

Different fromthe formerexam, thisexamhasahighCLIvolatility.Furthermore,meanCLIisoneofthehighestcomparedtootherexams;12.90.TherangebetweenthepassagewiththehighestCLIandthelowestis6.32,respectivelythefirstandthefourthpassages.Table5showstheresultsconcerning2014YDSfallterm.



Passage2

Passage 3

Passage 4

Passage 5 Average


Numberofwords 221 192 204 169 200 197


Averagenumberofcharactersperword 4.9 5.7 4.9 5 5.3 5.17


Averagenumberofwordspersentence 24.6 21.3 15.7 16.9 18.2 19

CLI 11.92 16.42 11.16 12.08 13.50 12.99


Similarly,thisexamhashighvolatility;itisnotashighastheformer,though.Themostdifficultpassageinthisexamisthesecondonewhereastheeasiestisthefirstone.Ontheotherhand,theexamhasoneofthehighestmeanCLI;12.99.Resultsof2014springtermwereprovidedintable6.

Table6.CLIAnalysisResultsofYDS2014SpringTermExam.


Passage2

Passage3

Passage4

Passage5 Average

Number of characters 1123 1114 1071 985 1010 1060Numberofwords 220 190 214 186 226 207

Number of sentences 11 9 9 7 11 9.4Averagenumberofcharactersperword 5.1 5.9 5 5.3 4.5 5.1

Average number of syllables per word 1.7 2 1.7 1.8 1.6 1.8

Averagenumberofwordspersentence 20 21.1 23.8 26.6 21 22

CLI 12.77 17.31 12.42 14.26 9.06 12.99

PassagetwohasthehighestCLIwhilepassagefivehasthelowest.TheaverageCLIofthisexamis12.99.Similartothefalltermexamofthesameyear,thisstudyalsohashighvolatilityinthedifficultyofreadingpassages.Table7providesthedetailsof2015YDSfallterm.

Table 7. CLIAnalysisResultsofYDS2015FallTermExam.


Passage2

Passage3

Passage4

Passage5 Average


Number of sentences 7 8 10 9 9 8.6Averagenumberofcharactersperword 4.9 5 5.3 4.7 4.9 45


Averagenumberofwordspersentence 29 24.8 19 24.3 21 23.1

CLI 12.09 12.68 13.96 10.64 11.84 12.20


It is understood that the third and fifth passages are the easiest and hardest ones,respectively.ThemeanCLIis12.20forthisexam.Thisstudyhassmoothvolatilityindifficult.Table8providestheresultsof2015YDSspringterm.


YDS 2015 Spring Term

Passage 1

Passage2

Passage3

Passage4

Passage5 Average


Numberofwords 254 233 195 222 229 227


Averagenumberofcharactersperword 4.8 4.8 5 5.1 4.8 4.9


Averagenumberofwordspersentence 18.1 21.2 24.4 22.2 20.8 21

CLI 10.51 10.92 12.45 12.80 11.18 11.51

ThepassagewiththelowestCLIisthefirstonewhilethefourthonehadthehighestCLI.TheaverageCLIforthisexamis11.51.Smoothvolatilityindifficultyexists.Table9showsthedetailsof2016YDSfallterm.



Passage2

Passage 3

Passage 4

Passage 5 Average


Numberofwords 238 200 192 202 239 214


Averagenumberofcharactersperword 4.5 5.6 5 4.7 4.9 4.9


Averagenumberofwordspersentence 21.7 20 17.5 16.7 18.4 18.8

CLI 9.12 15.39 11.96 10.26 11.60 11.57


Theresultsshowvolatilityinthedifficultofreadingpassagesof2016YDSfallterm.ThelowestCLIis9.12(passage1)whilethehighestis15.39(secondpassage).ThemeanCLIis9.12.Table10informsabouttheresultsof2016YDSspringterm.



Passage2

Passage 3

Passage 4

Passage 5 Average


Numberofwords 193 234 220 196 191 207

Number of sentences 8 15 10 12 9 10.8Averagenumberofcharactersperword 5 4.8 5.1 4.3 4.4 4.7


Averagenumberofwordspersentence 24.1 15.6 22 16.3 21.2 19.2

CLI 12.41 10.29 12.85 7.7 8.60 10.43

Totheresults,theaverageCLIis10.43.PassagesfourandthreehaverespectivelythelowestandhighestCLI,7.7,12.85.10.43isthemeanCLI.Table11presentstheresultsof2017YDSfallterm.

Table 11. CLIAnalysisResultsofYDS2017FallTermExam.


Passage2

Passage 3

Passage 4

Passage 5 Average


Numberofwords 208 231 221 197 215 214

Number of sentences 16 16 10 9 10 12.2Averagenumberofcharactersperword 4.7 4.6 5 5 4.9 4.8


Averagenumberofwordspersentence 13 14.4 22.1 21.9 21.5 17.6

CLI 9.73 8.89 12.43 12.61 11.49 10.99


ThereissmoothCLIdifferenceamongpassagesinthisexam.ThelowestCLIbelongstopassagetwowhilethehighestbelongstopassagefour.ThemeanCLIis10.99.Thenexttableshowsthedetailsof2017YDSspringterm.



Passage2

Passage 3

Passage 4

Passage 5 Average


Number of sentences 12 11 14 10 10 11.4Averagenumberofcharactersperword 4.9 4.9 4.9 4.7 4.9 4.9


Averagenumberofwordspersentence 17.5 19.6 16.9 24.9 20.3 19.6

CLI 11.26 11.66 11.32 10.86 11.88 11.37

ThereisaperfectharmonyofCLIwiththisexam.Almostnovolatilitywasdetected.PassageswiththehighestandthelowestCLIisrespectively,11.88and10.86.TheaverageCLIis11.37.Table13providesthedetailsofthelastexam:2018YDSspringterm.



Passage2

Passage 3

Passage 4

Passage 5 Average


Number of sentences 8 11 12 8 10 9.8Averagenumberofcharactersperword 4.7 4.4 4.9 4.7 4.8 4.7


Averagenumberofwordspersentence 19.3 20 22.5 27.4 16.6 21

CLI 10.37 8.70 11.73 10.78 10.81 10.53

Thereisevenvolatilitywiththisexam.11.73(passage3)isthehighestCLIand8.70(passagetwo)isthelowestCLI.ThemeanCLIis10.53.


Table 14.MeanCLIandMeanSuccessRateofallExamsintheData.

Exam Number of words

Number of sentences CLI Mean success

rate*2013Fall 1042 49 12.53 36.762013Spring 1018 48 12.90 30.472014Fall 986 52 12.99 41.372014Spring 1036 47 12.99 31.622015Fall 997 43 12.20 37.72015Spring 1133 54 11.51 34.12016Fall 1071 57 11.57 38.652016Spring 1034 54 10.43 322017Fall 1072 61 10.99 39.462017Spring 1114 57 11.37 362018Spring 1029 49 10.53 37.75

*Themeansuccessratescoresaredeliveredoutof100.

To the table, theexamwith thehighestmeansuccessrate is2014fall termwithascoreof41.37,andthen2017YDSfallterm(39.46)and2016fallterm(38.65)follow.Althoughtheyhaveanaverageofthehighestmeansuccessrates,theydonothavethelowestCLI.TheexamswiththehighestCLIare2013springterm,2014fallterm,and2014springterm.Table15presentsthesummaryofCLIdifferencesbetweenpassages.

Table 15.SummaryofCLIofEachExam.

Exam Highest CLI Lowest CLI Difference2013Fall 14.64 11.03 3.612013Spring 15.96 9.64 6.322014Fall 16.42 11.16 5.262014Spring 17.31 9.06 8.252015Fall 13.96 10.64 3.322015Spring 12.80 10.51 2.292016Fall 15.39 9.12 6.272016Spring 12.85 7.7 5.152017Fall 12.61 8.89 3.722017Spring 11.88 10.86 1.022018Spring 11.73 8.70 3.03


Astable15shows,thedifferenceofCLIbetweenthepassagesislowestin2017YDSspringterm-1.02-whileitishighestin2014YDSspringterm,8.25.

4. DiscussionThepresent studyaimed tomeasure thedifficulty levelof reading comprehension

questionsofYDSimplementedbetween2013and2018byusingColeman-Liauindex;toshowthemostdifficultandtheeasiestreadingpassages;toinvestigatewhetherthereisapositiveornegativecorrelationbetweenCLIandmeanexamsuccessesoftest-takers,andtoinvestigatewhichYDShasthemostvolatilityindifficultyofreadingcomprehensionpassages.Astothesepurposes,thisstudyansweredtheresearchquestionsasfollows.

1. What term of YDS has the highest and the lowest CLI regarding reading comprehension passages?

Theresultsshowedthatboth2014springandfalltermsYDShavethehighestCLI.CLIreferstothedifficultyofreadingpassagesandbecausetheseexamshavethehighestreadingpassages,itcanbeconcludedthat2014YDSspringandfalltermsaretheexamswiththemostdifficultpassages.Ontheotherhand,2016YDSspringtermcameforwardastheexamwiththelowestCLI,whichmeansthatitistheexamwiththeeasiestreadingpassages.

Ontheotherhand,thisdifferenceofCLIisnotappropriateforaqualifiedtestbecausethe testshouldfollowastraight line indifficultyso that the test-takerswouldnotfeelanxiety about the difficulty level of reading questions that take up the big portion inscoring.Thequestionscanbeordered fromeasy todifficult(Zimmaro,2016)but thishasnothingwithorderingpassagesinthesameway.Ensuringaccuratedifficultylevelofquestionsisessentialtohelptest-takerslearnmoreefficientlyandeffectively(Pérez,Santos,Pérez,deCastroFernández,&Martín,2012)fortheprospectiveexams.

2. Is there any correlation between CLI and mean exam success? Thisisacrucialbutdifficultquestiontoanswerbecauseyouneedtomakethorough

analysestoreachastableconclusion.However,thequestioncanbeansweredpartlyintermsofinvestigatingtherelationshipbetweenthemeanscoresandmeanCLI.Accordingto theresults, there isnotadirectpositiverelationbetweenCLIandsuccessrate.Forexample,2014YDSspringandfalltermsexamshavethemostdifficultreadingpassagesofallotherexamswhile themeanscoresarenot the lowest. Inotherwords,althoughthe CLI is 12.99 inYDS 2014 fall term (the highest CLI), themean success rate isthehighestofallexams(seetable14).Again,althoughtheCLIis12.99inYDS2014spring term, themeansuccess rate is thesecond-worst.This shows that there isnotadirectrelationshipbetweenCLIandsuccessrate,but this isnot tosaythat thereisan


asymmetricalrelationshipbecauseresultsshowthatthereareexamsinwhichbothCLIandmeansuccessrateislow(table14)

3. What term of YDS has the most and least volatility in difficulty of reading comprehension passages?

Thedegreeofdifficultymaychangefromoneexamtoanother;however,thechangeofdifficultymayoccurinsideasingleexam.Thisstudyfoundthateachexamdoesnothaveanevendegreeofdifficultyinreadingpassages(seetable15).Forexample,theCLIdifferenceamongthepassagesis8.25in2014YDSfalltermexam,whichmeansthatonepassageisveryeasywhiletheotherisverydifficult.Althoughitwasobservedthatthisdifferenceexistsinthemajorityoftheexams,someexams,2015and2017YDSspringterms,havesmoothvolatilityamongpassages.

Comprehendingatextisasophisticatedprocessdependingonthemodelsofreadingcomprehension(Trapman,Gelderen,Schooten,&Hulstijn,2016)becausereaderswouldneed to apply their linguistic knowledge aswell as appropriatemental representationofa text (Perfetti,Landi,&Oakhill,Theacquisitionof readingcomprehensionskills,2005).Therefore,providingpassagesatsimilardifficultyisnecessarytoavoidpossibleconfusionwiththetest-takers.Areadingpassagecomposedofsimpleordifficultwordswoulddeterminethecomprehensionlevelofthetest-taker(Susanti,Nishikawa,Tokunaga,&Hiroyuki,2016).However,asmoothtransitionindegreeofdifficultyisanobligationfor test reliability; therefore,“cautionshouldbeexercisedwhenselecting tests for theassessmentofreadingdifficulties(Nation,1997,p.359)”particularlyforthetestssuchasYDS that is composedofmore sophisticatedandcompound sentences (Aşkaroğlu,2013).

Individualdifferencesoftest-takersinreadingcomprehensionquestionsareattributedto the knowledge of linguistic knowledge particularly vocabulary and grammar (VanGelderen,etal.,2004);therefore,decodingfluencyisofparamountimportancetodefinethe level of reading comprehension, and accordingly irregular fluctuations betweenparagraphsmaynegatively affect theflair for decodingfluency i.e. speedof sentenceprocessingwhich is oneof the crucial factors that determines reading comprehensionability (VanGelderen, Schoonen, Stoel,DeGlopper,&Hulstijn, 2007).On the otherhand,carefullyplannedreadingcomprehensionparagraphsmayhaveapositivewashbackeffecton the readingskillsof test-takers,aswasevidencedbyAkpınarandÇakıldere(2013)whoalsoinvestigatedYDS.

Anothercaveatabouttests is thatwhethertheyarefor level-gradingor tomeasureproficiency.Level-gradedquestionsmaybeincludedinatesttodeterminethelevelsofthetest-takers,andaccordinglytorankthemaslow-achievingorhigh-achievinglanguagespeakers.However,high-stakes tests suchasYDS-considered tobeproficiency tests-addresstoagroupwithhighlanguageaptitudeandarecarriedouttobeusedinareassuchasacademiathatnecessitatehighleveloflanguageproficiency,whichiswhytestsas


YDSistohaveastablelevelofdifficulty.Giventheevidencethisstudyconcluded,sometests(table15)hadCLIovernormalisedlevelsintermsofparagraphdifficultywhereassomeothershadonlyminordifferences.Besides,majorCLIfluctuationsexist amongteststhatweresetatdifferentterms,whichmaybeduetotheirparagraphstyle.Kıray(2015) stated that there is not a standardizedparagraph style inYDSexams; inotherwords,someparagraphsseemtobedeficientbecausetheyarenotfullbutasmallfractionof awhole text.Therefore, simplypicking a paragraphout of a full textmay lead todifficultyatvariouslevels,ifnotambiguityintermsofreadingcomprehension.Finally,theresultsareinlinewiththeassumptionsofGur(Gür,2013)whoindicatedthatKPDS(formernameofYDS)didnotrepresentforstableparagraphorders,i.e.someparagraphsaremuchmoredifficultthantheothers.

5. ConclusionThefunctionofperforminganexamistoenableinstructorstomakejudgmentsabout

the quality of learning (Piontek, 2008) and to assess the level of preparation reachedby the test-takers (Frosini, Lazzerini, & Marcelloni, 1998). Accordingly, readingcomprehension questions out of all other question types are the most important onetomeasure theEnglishproficiencyofstudents,andfor thesuccess ineveryacademicdomain(Wigfield,Gladstone,&Turci,2016).Therefore,accuratequestionpreparingforreadingcomprehensionisobligatoryfortesting-institutions.

Giventheimportanceofassessment,testinginstitutionsshoulddevelopexamsthatincludequestionsbalancedindifficulty.Ifcreatinghigh-qualityeducationalassessmentsisdesired,strikingabalanceonthedifficultylevelofquestionswillincreasethereliabilityofexamresults.Anyfluctuationindifficultythroughouttheexamsmaypreventassessorsfromreachingastableassessmentontheproficiencyofthetest-takersbecauseunbalancedexamswith questions at different difficulty levelsmay prompt test-takers to do low-performingandhigh-performingalthough theyhavesimilar languageproficiencies. Inbrief,difficultycanaffectthetestsnegatively,whichiswhyitshouldbeconsideredontheprocessofpreparingquestions(Tan&Othman,2013).Accordingly,thisstudyemphasizesthat the calibration of reading passages is of importance for both reliable results andaccuratemeasuring.Testinginstitutionsmayusedifferentmethods-insteadofCLI-suchasFleschKincaid,SMOG,ARItocalculatedifficultylevelofreadingpassages.

This studymeasured the difficulty levels of only reading passages asked inYDSbetween 2013 and 2018. Further studies may measure the whole questions askedin the questions.Also, tests include vocabulary questions.They are also important indetermining the proficiency levels; however, some testing institutions ignore creatingequilibriumonthedifficultylevelsofthevocabularies.SomewordsmaybeatA2levelwhilesomeothersatC1level.Researchersarekindlyinvitedtoinvestigatethisunstablesituationbecauseitmayhaveaprofoundimpactontestreliability.Thoughtheyarestilldeveloping, there are somePC-basedprograms suchasCollins providingadegreeofdifficultyofvocabularies.


ReferencesAkpinar,K.D.,&Cakildere,B.(2013).Washbackeffectsofhigh-stakeslanguagetests

ofTurkey (KPDS andÜDS) on productive and receptive skills of academicpersonnel.Journal of Language and Linguistic Studies,9(2),81-94.

Aşkaroğlu,V.(2013).2013YDSile2012KPDSveÜDSSınavlarınınKarşılaştırılması.Karadeniz Uluslararası Dergi,162-173.

Carlson, S. E., Seipel, B., & McMaster, K. (2014). Development of a new readingcomprehension assessment: Identifying comprehension differences amongreaders.Learning and Individual Differences,31,40-53.

Deacon,S.H.,Kieffer,M.J.,&Laroche,A.(2014).Therelationbetweenmorphologicalawarenessandreadingcomprehension:Evidencefrommediationandlongitudinalmodels.Scientific Studies of Reading,18,432–451,DOI:10.1080/10888438.2014.926907.

Frosini, G., Lazzerini, B., & Marcelloni, F. (1998). Performing automatic exams.Computers & Education,281-300.

Gür,Ö.(2013).Ölçme,değerlendirmevekamupersonelidilsinavı(KPDS)–Businavneyiölçüyor?Sakarya University Journal of Education,23-32.

Kıray, G. (2015). Macro-structure Analysis of Reading Comprehension ParagraphsofKPDS andYDSExamswithinYears 2003-2013.Hasan Ali Yücel Eğitim Fakültesi Dergisi,219-233.

Lam,R.(2015).LanguageassessmenttraininginHongKong:Implicationsforlanguageassessmentliteracy.Language Testing,32(2),169–197.

Nagy, W. E. (2007). Metalinguistic awareness and the vocabulary-comprehensionconnection.InR.K.Wagner,A.E.Muse,&K.R.(Eds.),Vocabulary acquisition: Implications for reading comprehension (pp. 52-77). New York: GuilfordPress.

Nation, K. (1997). Assessing reading difficulties: the validity and utility of currentmeasures of reading skill.British Journal of Educational Psychology, 67(3):359-370doi.org/10.1111/j.2044-8279.1997.tb01250.x.

OSYM.(2016).www.osym.gov.tr.Retrieved23,2019,fromYDSÇıkmışSorular:http://www.osym.gov.tr/TR,15073/yds-cikmis-sorular.html

Pérez,E.V.,Santos,L.M.,Pérez,M. J.,deCastroFernández, J.P.,&Martín,R.G.(2012).AutomaticClassificationofQuestionDifficultyAutomaticClassificationofQuestionDifficulty.Frontiers in Education Conference(pp.1-5).IEEE.

Perfetti,C.A. (1999).Comprehendingwritten language:Ablueprint of the reader. InC.M.Brown,&P.(.Hagoort,The neurocognition of language(pp.167-208).Oxford:OxfordUniversityPress.


Perfetti,C.A.,Landi,N.,&Oakhill,J.(2005).Theacquisitionofreadingcomprehensionskills.InC.H.M.J.Snowling,The science of reading: A handbook(pp.227-247).London:Blackwell.

Perfetti,C.A.,Landi,N.,&Oakhill,J.(2005).Theacquisitionofreadingcomprehensionskills.InM.J.Snowling,&C.H.(Eds.),The science of reading: A handbook (pp.227-247).London:Blackwell.

Piontek,M.E.(2008).Bestpracticesfordesigningandgradingexams.Occasional Paper,24,1-12.

Readable Blog.(2017,522).Retrieved12,2019,fromReadabilityandtheColeman-LiauIndex:https://readable.com/blog/the-coleman-liau-index/

Susanti,Y.,Nishikawa,H.,Tokunaga,T.,&Hiroyuki,O.(2016).Itemdifficultyanalysisofenglishvocabularyquestions.The 8th International Conference on Computer Supported Education,pp.267-274.

Tan,Y.T.,&Othman,A.R.(2013).Therelationshipbetweencomplexity(taxonomy)anddifficulty.The 20th National Symposium on Mathematical Sciences (pp.596-603).AIPPublishingLLC.

Trapman,M.,Gelderen,A.,Schooten,E.,&Hulstijn,J.(2016).ReadingComprehensionLevel and Development in Native and Language MinorityAdolescent LowAchievers: Roles of Linguistic and Metacognitive Knowledge and Fluency.Reading & Writing Quarterly,239-257,DOI:10.1080/10573569.2016.1183541.

VanGelderen,A.,Schoonen,R.,DeGlopper,K.,Hulstijn, J.,Snellings,P.,Simis,A.,et al. (2004). Linguistic knowledge, processing speed and metacognitiveknowledgeinfirstandsecondlanguagereadingcomprehension:Acomponentialanalysis. Journal of Educational Psychology, 96, 19–30. DOI:10.1037/0022-0663.96.1.19.

VanGelderen,A., Schoonen,R., Stoel,R.D.,DeGlopper,K.,&Hulstijn, J. (2007).Developmentofadolescentreadingcomprehensioninlanguage1andlanguage2:A longitudinal analysis of constituent components.Journal of Educational Psychology,99,477–491.DOI:10.1037/0022-0663.99.3.477.

Wigfield,A.,Gladstone,J.R.,&Turci,L.(2016).Beyondcognition:Readingmotivationandreadingcomprehension.Child development perspectives,10(3),190-195.

Zimmaro,D.M. (2016,111).Writing Good Multiple-Choice Exams.Retrieved210,2019, from https://facultyinnovate.utexas.edu/sites/default/files/writing-good-multiple-choice-exams-fic-120116.pdf

YDS AS A BENCHMARK IN TURKEY: THE DIFFICULTY LEVELS OF ...

Documents

Transcript of YDS AS A BENCHMARK IN TURKEY: THE DIFFICULTY LEVELS OF ...