ThestudyonthecontentvalidityofthereadingcomprehensionsectionofthefirstmockNMETin2012,Yangzhou

ThestudyonthecontentvalidityofthereadingcomprehensionsectionofthefirstmockNMETin2012,Yangzhou

王晓莉

王晓莉英国布里斯托大学教育研究院

一、Introduction

1.Contextofthestudy

InChina,therehavebeenhundredsofstudiesworkingontheCollegeEntranceExam,ortheNationalMatriculationEnglishTest(NMET),butonlyarelativelysmallnumberofarticlesofstudieswereonthemockNMET,whichIthinkisofgreatimportanceaswelltotheseniorhighschoolteachers.

ThemockNationalMatriculationEnglishTest(mockNMET),whichhasbeenoftencarriedoutforyearsinChina,isanimportantexamduringtherevisionperiodinSenior3ofhighschools.ThesemockexamstrytototallyimitatetheNMET,includingthestructure,taskspecifications,testingrequirements,etc.TheNMETisheldonceayearinJune,whilethemockoneisusuallyheldtwiceorthreetimesayear.ThefirstmockexaminationisnormallyinlateMarch,theaimofwhichisnotonlytogivestudentsasenseofwhattheNMETis,butalsotoexaminewhetherthestudentshavemettherequirementsofthetestingsyllabusforNMET.Bothteachersandstudentscanhaveanopportunitytofindwhattheylackinordertomakepossiblechangesontherevisionplansaccordingtotheexamresults.Inthiscase,themockexamsneedtopresentthecharacteristicsoftherealNMET,andtheresultsofthemshouldbeabletorepresentstudents’truelevels.Inotherwords,onlywhenthemockpapersreachahighdegreeofvaliditycantheybeofgreathelpinpreparingforNMET.InthefirstmockNMETin2012,Yangzhou,Iamgoingtoanalyzeonlythereadingcomprehensionsectionwhichaccountsforonequarterofthetotalscoresandcanhavevitalinfluenceonstudents’finalscores.Iwillmainlydealwiththevalidity,especiallythecontentvalidity,ofthispartbecauseitmayreflectstudents’trueachievement.

2.Theoreticalconcerns

Validity,asdefinedbyHenning,“ingeneralreferstotheappropriatenessofagiventestoranyofitscomponentpartsasameasureofwhatitispurportedtomeasure.Atestissaidtobevalidtotheextentthatitmeasureswhatitissupposedtomeasure.”AccordingtothelatestversionoftheStandardsforEducationalandPsychologicalTesting,therearefivebroadtypesofvalidationevidence:testcontent,responseprocesses,internalstructureofthetest,correlationwithothervariablesandtheconsequencesoftestuse.Contentvaliditycorrespondingtothefirsttype,“istherepresentativenessorsamplingadequacyofthecontent-thesubstance,thematter,thetopics-ofameasuringinstrument”.Hughesstatesthat“thegreateratest’scontentvalidityis,themorelikelyitistoaccuratelymeasurewhatitissupposedtomeasure”.Acommonwayofcontentvalidationistoanalyzethecontentofatestandtocompareitwithastatementofwhatthecontentoughttobe.Suchacontentstatementmaybethetest’sspecifications,aformalteachingsyllabusorcurriculum,oradomainspecification.Inthisstudy,thecurriculumstandard,testingsyllabusorspecificationandtherecentNMETpapersareusedascontentstatements.

ThispaperisgoingtoanalyzethereadingcomprehensiontestofthefirstmockNMETthisyearinmycity-YangzhouinJiangsuProvinceinChina,andisconcernedwithtwomainpointsthrougharesearchdesign.Thefirstiswhetherthereadingcomprehensionsectionofthispaperreachesahighdegreeofcontentvalidity,i.e.whetherthetestcanmeasurewhatthecurriculumstandardandtestingspecificationrequireofstudents.TheotheriswhetherthisreadingtestaccuratelymockstheNMET.Indoingso,Ihopetofindoutwhethertheresultsofthistestcouldprovideusefulsuggestionstotheteachingandlearninginthepreparationforthe2012NMETinthefollowingtwomonths.

二、Standardsforjudgingthereadingcomprehensionsectioninthechosentestpaper

1.Thedemandsofcurriculumstandards

InStandardsofEnglishcurriculuminfull-timecompulsoryeducationandseniorhighschools,studentswhowanttograduatefromhighschoolsandenteruniversitiesoughttohaveacertainEnglishproficiencylevel.Asforthepartofreadingproficiency,students(oflevel8)arerequiredtodistinguishcharacteristicsofdifferenttextgenres,understandlongandcomplexsentencesbyanalyzingtheirsentencestructures,telltheviewpointsandattitudeofatextandappreciateEnglishliteraryworkswritteninplainwords.

2.Thestandardsofthetestingspecification

Chinahasauniquetestingsystemforcollegeentranceexaminations.Ratherthanusingthesametestpaperthroughoutthewholecountry,someprovinceshavetheirowntestpapersandmyprovinceJiangsuisamongthem.Inthiscase,boththenationalandtheprovincialtestingsyllabus—EnglishtestingsyllabusforcollegesdraftedbythePRCMinistryofEducationandtheIntroductionoftheEnglishExaminationdraftedbyJiangsuProvincialExaminationsAuthority—aretakenintoconsideration.

Thedemandsinthetwodocumentsincludeaspectsofreadingskillswhichneedtobetested:understandingthepurposeandmainidea,understandingspecificinformation,guessingunfamiliarwordsandphrasesaccordingtothecontext,makingsimplejudgmentandinference,understandingthebasicstructureofpassages,understandingtheauthor’sopinions,intentionsandattitude.

3.ThecharacteristicsofNEMTtestpapersintherecentyears

Althoughthetestingsyllabusprovidesgeneralideasofatestpaper,nodetailedstandardsaregiven.Forexample,fromthesyllabuswedon’tknowwhatkindofgenres,topicsshouldbeadoptedinthereadingtest.Wealsohavenoideaoftheexactrateofeachsub-skillstobetested.Inthiscase,itisnecessarytoanalyzetheNMETtestpapersaswell,whichareconsideredtobestrepresentthetestingsyllabusandcurriculumstandard,asadditionalstatements.What’smore,thecomparisonbetweentheNMETandthemockNMETcanalsoprovideevidenceofhowsuccessfullythemockpaperimitatestherealones.TheNMETpapersofyears2009,2010and2011inJiangsuProvincewillbetakenassamples.Thecharacteristicsofreadingcomprehensionpartinthesetestpaperswillbesummarized,andacomparisonbetweenthesecharacteristicsandthoseofthemockpaperwillbemade.

三、Researchdesign

1.Researchquestions

Thispaperseekstoinvestigatetwomainareas:

(1)DoesthereadingtestinthefirstmockNMETin2012,Yangzhoureachahighdegreeofvalidityaccordingtothestandardsmentionedabove?

(2)DoesthereadingtestinthefirstmockNMETin2012,YangzhousucceedinmockingtheNMETpapers?

2.Researchobjects

TheresearchobjectsaremainlythereadingtestpaperofthefirstmockNMETin2012,Yangzhou.Besides,interviewsfromsixstudentswhotookthetest,andtwoteachersofthemarealsoinvolvedintheresearch.

3.Datacollection

Thisstudycollectsbothquantitativedataandqualitativedata.Thedatahavebeencollectedfromtherecentyears’NMETpapersandthefirstmockNMETpaperthisyear.Theseincludeobjectivedatasuchaslength,readingspeed,andsubjectivedatasuchasgenres,topics,questiontypes.Computershavebeenadoptedincollectingtheobjectivedatawhicharerelatedtothereadingmaterials,thequestionsandstudents’scores.Besides,inquiriesintotheattitudeofsixrandomlypickedstudentsandtheirteacherswerealsoconductedsoonafterthetestwiththehelpoftwoofmyformercolleagues.AccordingtoMcNamara,test-takerfeedbackshouldbegatheredfromthetrialsubjects,oftenbyasimplequestionnaire.Thiswillincludequestionsonperceptionsofthelevelofdifficultyofparticularquestions,theclarityoftherubrics,andgeneralattitudetothematerialsandtasks.Subjectscanquicklyspotthingsthatareproblematicaboutmaterialswhichtestdevelopersstruggletosee.

四、Dataanalysisandfindings

1.Theanalysisanddiscussionsofthereadingpassages

Whendesigningareadingtest,choosingappropriatereadingtextsarevitallyimportant.Textswithcertainreadabilityshouldbeapplied.Thiswillallowstudentstoresponsetothequestionsupontheirunderstanding,andwillresultinahighdegreeofcontentvalidityofthetest.Otherwise,studentsarelikelytoguessbyallmeanstheanswers,andtheirtestresultsareprobablyinvalid.Aldersonstatesthatmanyaspectsoftextmightfacilitateormakedifficultthereadingprocess.Sometextvariableswillbediscussedinthispart,includingtextlength,readingspeed,genres,andtopics.

（1）Lengthandreadingspeed

Thereisnotaspecificstandardofthetextlengthintestingsyllabus.However,byanalyzingtherecentNMETpapers,wecangetsomeideasofthisaspect.Asisshowninthetable,thetotallengthofallthereadingpassagesandallthetextquestionsinaNMETpaperrangesfrom2275-2358,theaveragebeing2332.Whileinthemockpaper,totalnumberofwordsinthereadingtestis2296,whichisjustintherangeofNMETandisveryclosetotheaveragenumber.

Thesuggestedtiming(bythetestingsyllabus)forreadingcomprehensionsectionis30minutes.Ifthetotalinputamountispidedbytesttime,wewillgettheresultofthereadingspeed.Inthisway,readingspeedofthemockexamis77wordsperminute,whichismoreorlessthesamewiththoseofNMET.Meanwhile,italsoseemstomeetthestandarddemandedbythenationalteachingsyllabusof70-80wordsperminute.However,YangandWeirindicatethatthedesigneroftestpapersneedtotakeintoaccountofthefactthatittakessometimetoreadthequestionsitemsandthinkabouttheanswerstothequestions.Inotherwords,the30-minutetesttimecannottotallybeusedtoreadthepassagesandquestions.Inthiscase,students’takingthismockNMETprobablyneedtoreadmorethan80wordsperminute,whichisbeyondtherequirementsofteachingsyllabusandwillmakethereadingtestmoredifficult.“Thedifficultyandperceptionofspeedednessmayleadtoguessing,orcompensatorytesttakingstrategies...oreventhecompletelackofresponsetothetesttask”,jeopardizingthetestvalidity.

(2)Genresoftexts

KobayashihasfoundthattexttypeandresponseformataffectreadingcomprehensionperformanceofL2learnersofdifferentproficiencylevel.Theimportanceoftexttypeisemphasizedandattentionneedstobepaidtoit.AccordingtoCoulthard,agenreisonetypeofstylisticstructurefororganizingsentencesandutterancesintolargerunits.Genresmaydifferintheirformalcharacteristicssuchasparagraphstructure,thematicsequence,thestereotypicalorderingofeventsintime,andthedistributionofsentencetypesandpronominalforms.InStandardsofEnglishcurriculuminfull-timecompulsoryeducationandseniorhighschools,studentsarerequiredthecompetencetoidentifydifferentgenresoftextsaccordingtothesecharacteristics.SoitisnecessaryforthemockNMETdesignerstoinvolvedifferentgenresoftextsaccordingtothestandardsofthecontentstatements.In2012mockNMETthatIamstudyingon,itsreadingtestpaperisvariedintexttypes,consistingoftwoexpositions,onenarration,andonepracticalwriting.Besides,accordingtothestatisticspresentedintheabovetable,2012mockNMETisquitesimilartothepreviousNMETintermsoftexttypes.Inotherwords,itdidagoodjobinmeetingthestandardsandmockingtheNMET.

However,althoughthe2009-2011NMETdonotcontainanyliteraryworks,curriculumsyllabusencouragestudentstolearntoappreciateliteraryworkswritteninplainwords.Thistypeoftextsiscompletelyneglectedinthemockpaperaccordingtothecollecteddata.

(3)Topicsoftexts

Bachmanexplainsthattopicofinputhasasubstantialeffectontesttakers’performance,whichcouldbereferredtoas“passageeffect”intestsofreadingcomprehension.Sotopicsthatareinterestingandrelevanttotesttakersarethemostappropriate,meanwhile,possiblebiasandeitherextremesshouldbeavoided.Asaresult,testdesignersselectedalotofgeneralorinnocuoustopicsforthereadingcomprehensionofNMETinthepast.Topicfamiliarityisexpectedto“haveafacilitatingeffect”onthetestresult.AldersonandUrquhartwereabletoshowthatreadingtestonthetextsthatarefamiliartothestudentsweresometimeseasiertoprocessthanthosewhichwerenot.Oneofthereasonsisthatinreadinganunfamiliartextforcomprehension,studentsmustbeabletorecognizeallthewordsandstructuresincontext,whichrequireshigherlanguageproficiencyofthestudents.ThenationalEnglishcurriculumstandardshasgivenoutarangeoftexttopics,consistingof121topicsof24categories.Readingcompetencecanbereflectedinreadingallkindsofmaterials;thereforedifferenttexttopicsanddiffernttextgenresshouldbeinvolvedinthereadingtestsasmanyaspossible.Inrecentyears’NMET,readingcomprehensionofteninvolvesvarioustopicsincludingdailylife,commonsense,news,advertisements,culture,economics,history,geography,etc.Thiscanbeembodiedbythedataintheabovetable.Mostofthetopicsareclosetothedailylifeandarefamiliartothestudents.In2012mockNMET,mostofthetexttopicsareverymuchclosetostudents’life,exceptPassage3.Passage3isaboutthesecurityoffood,focusingonproblemsexistingonfoodlabels.Althoughfoodsecurityhassomethingtodowithstudents’dailylife,thepoliciesoffoodlabelsarenot.Thisunfamiliarityresultsinstudents’incrediblypoorperformance.Forinstance,students’scoringaverageinmycityofthesecondquestionandthefourthquestionofthispassageisonly23.5%and24.5%respectively.Thisinappropriatenessintexttopicchoicecreatesflawsincontentvalidityofthetest.

2.Theanalysisanddiscussionofthetestquestions

Aldersonstatesthattherearevariablesaffectingtheassessmentofreadinginafirstandinasecondlanguage,andthequestionssetandthepassagesonwhichcomprehensionisassessedisequallyimportant.Sainsburyfurtherpointsoutthatinthecaseofmanytypesofreadingtests,theevaluationofcontentmustbeappliedbothtopassagesoftextandtothequestionsasked.Accordingtothe2011IntroductionofChinese,MathsandEnglishExaminationsforCollegeEntranceExaminationdraftedbyJiangsuProvincialExaminationsAuthority,sixtypesofreadingskillsneedtobetested.Usuallythesixskillsaresimplifiedasthefollowingfourskills:understandingthepurposeandgist,understandingexplicitlystatedinformation,makingjudgmentandinference,deducingthemeaningofunfamiliarwords.AccordingtoZou,agoodreadingtesttestsreadingskillsthosearerepresentative;testdesignersshouldavoidemphasizingoneorseveraltypesofreadingskills,suchasfocusingtomuchondeducingthemeaninganduseofunfamiliarlexicalitems.

Dataintheabovetableillustratethat2012mockpaperwellmeetsthestandardsandwellmockstheNMETintermsofthequestiontypes.Itcoversthreeofthefoursub-skillsofreadingcomprehension,withanappropriaterate.Questionsrequiringskillsofdeducingthemeaningofunfamiliarwordsarenotmentionedinthispaper.Yet,sincethecircumstancesoccurredinthe2011NMETaswellandthereisnorequirementinsyllabusoftestingallthesub-skillsinasingletest,itcouldberegardedasacceptable.Meanwhile,numbersoftheotherthreetypesofquestionsareverymuchclosetotheaveragedataof2009-2011NMET.Moreover,allthequestionsarebasedonthematerials,nodeviationfromthepassages.Thesefactsofquestiontypesaddtothevalidationevidenceofreadingcomprehensionpartofthismockpaper.

3.Theanalysisanddiscussionofstudents’performanceandscores

Inordertogathertheinformationofstudents’feelingsandperformances,aquestionnairetothestudentsandinterviewswithtwooftheirteacherswereconductedbyoneofmycolleague.Whenfinishingthequestionnaire,thestudentsrecalledthecoursetoobtainsolution.Allthesixsubjectsthoughtthatthereadingpassages,especiallythethirdone,areabitmoredifficultthanusual,inwhichtheycameacrossmanynewwordsandsomelongandcomplexsentences.Fourofthemspentover35minuteswhilethesuggestedtimingis30.Thetwoteachersbelievedthatthereadingsectionisonthewholeawell-designedone.Theythoughthighlyofthechoiceofmaterials,bothintopicsandingenres.Buttheyheldthesameviewonthedifficultylevelastheirstudents.Bythistoken,themajorprobleminthissectionisthedifficultylevel.

五、Conclusion

1.Summaryofthestudy

Throughthethoroughanalysisofmuchrelateddata,IthinkthatthereadingtestinthefirstmockNMETin2012,Yangzhou,succeedstosomeextentinthefollowingtwoaspects:First,itwellanswersthenationalEnglishcurriculumstandardandtheprovincialtestingsyllabus.Inotherwordsitreachesahighdegreeofvalidity.Second,itsuccessfullymockstheNMETpapers.Soitcanembodystudents’truelevelinreadingcomprehensionbyandlarge.Andwiththecorrectfeedbackfromthetest,bothteachersandstudentsmayappropriatelyadjusttheirstrategiesintherevisionofreading.

However,someflawsdoexistinthetestdesignthatcouldjeopardizeitsvalidity.Themostsignificantoneisthatitisalittledifficultfortheintendedtesttakersbasedonitshighreadingspeed,itsrelativelylowscoringaverageandstudents’feelings.Itisquitecommoninmycontextthatreadingtestsofwhichtherequirementsareabovestudents’proficiencylevelarepreferredbyteachers.Forinstance,theyoftenadoptthereadingtestsofCET-4(CollegeEnglishTestBand4)level,thinkingitcouldbebeneficialtopromotestudents’learning.YetIbelievethesetestscouldnothelptoincreasecomprehensibleinputandisnothelpfulforstudents’improvementofreadingproficiency.

2.Limitationsofthestudy

Dataofthetestpapersandquestionnairesarecollectedandanalyzedcarefullyinthisstudy.However,sincedifferentpeoplemayhavedifferentcognitionanddifferentdatacollectingprocedures,thedata,especiallythesubjectiveones,mightbeslightlydifferentfromthefacts.Besides,duetolackofofficialdataofstudents’NMETscoresinrecentyearsandnoaccesstothetestdesignersofthismockexam,thisstudydoesnotanalyzethoroughlyanddetailedlyaspectslikedifficultydegreeanditemdiscrimination.Also,thewashbackeffectsofmockNMET,whichisnotreferredtointhispaper,couldmeritfurtherenquiry.