王晓莉英国布里斯托大学教育研究院
一、Introduction
1.Contextofthestudy
InChina,therehavebeenhundredsofstudiesworkingontheCollegeEntranceExam,ortheNationalMatriculationEnglishTest(NMET),butonlyarelativelysmallnumberofarticlesofstudieswereonthemockNMET,whichIthinkisofgreatimportanceaswelltotheseniorhighschoolteachers.
ThemockNationalMatriculationEnglishTest(mockNMET),whichhasbeenoftencarriedoutforyearsinChina,isanimportantexamduringtherevisionperiodinSenior3ofhighschools.ThesemockexamstrytototallyimitatetheNMET,includingthestructure,taskspecifications,testingrequirements,etc.TheNMETisheldonceayearinJune,whilethemockoneisusuallyheldtwiceorthreetimesayear.ThefirstmockexaminationisnormallyinlateMarch,theaimofwhichisnotonlytogivestudentsasenseofwhattheNMETis,butalsotoexaminewhetherthestudentshavemettherequirementsofthetestingsyllabusforNMET.Bothteachersandstudentscanhaveanopportunitytofindwhattheylackinordertomakepossiblechangesontherevisionplansaccordingtotheexamresults.Inthiscase,themockexamsneedtopresentthecharacteristicsoftherealNMET,andtheresultsofthemshouldbeabletorepresentstudents’truelevels.Inotherwords,onlywhenthemockpapersreachahighdegreeofvaliditycantheybeofgreathelpinpreparingforNMET.InthefirstmockNMETin2012,Yangzhou,Iamgoingtoanalyzeonlythereadingcomprehensionsectionwhichaccountsforonequarterofthetotalscoresandcanhavevitalinfluenceonstudents’finalscores.Iwillmainlydealwiththevalidity,especiallythecontentvalidity,ofthispartbecauseitmayreflectstudents’trueachievement.
2.Theoreticalconcerns
Validity,asdefinedbyHenning,“ingeneralreferstotheappropriatenessofagiventestoranyofitscomponentpartsasameasureofwhatitispurportedtomeasure.Atestissaidtobevalidtotheextentthatitmeasureswhatitissupposedtomeasure.”AccordingtothelatestversionoftheStandardsforEducationalandPsychologicalTesting,therearefivebroadtypesofvalidationevidence:testcontent,responseprocesses,internalstructureofthetest,correlationwithothervariablesandtheconsequencesoftestuse.Contentvaliditycorrespondingtothefirsttype,“istherepresentativenessorsamplingadequacyofthecontent-thesubstance,thematter,thetopics-ofameasuringinstrument”.Hughesstatesthat“thegreateratest’scontentvalidityis,themorelikelyitistoaccuratelymeasurewhatitissupposedtomeasure”.Acommonwayofcontentvalidationistoanalyzethecontentofatestandtocompareitwithastatementofwhatthecontentoughttobe.Suchacontentstatementmaybethetest’sspecifications,aformalteachingsyllabusorcurriculum,oradomainspecification.Inthisstudy,thecurriculumstandard,testingsyllabusorspecificationandtherecentNMETpapersareusedascontentstatements.
ThispaperisgoingtoanalyzethereadingcomprehensiontestofthefirstmockNMETthisyearinmycity-YangzhouinJiangsuProvinceinChina,andisconcernedwithtwomainpointsthrougharesearchdesign.Thefirstiswhetherthereadingcomprehensionsectionofthispaperreachesahighdegreeofcontentvalidity,i.e.whetherthetestcanmeasurewhatthecurriculumstandardandtestingspecificationrequireofstudents.TheotheriswhetherthisreadingtestaccuratelymockstheNMET.Indoingso,Ihopetofindoutwhethertheresultsofthistestcouldprovideusefulsuggestionstotheteachingandlearninginthepreparationforthe2012NMETinthefollowingtwomonths.
二、Standardsforjudgingthereadingcomprehensionsectioninthechosentestpaper
1.Thedemandsofcurriculumstandards
InStandardsofEnglishcurriculuminfull-timecompulsoryeducationandseniorhighschools,studentswhowanttograduatefromhighschoolsandenteruniversitiesoughttohaveacertainEnglishproficiencylevel.Asforthepartofreadingproficiency,students(oflevel8)arerequiredtodistinguishcharacteristicsofdifferenttextgenres,understandlongandcomplexsentencesbyanalyzingtheirsentencestructures,telltheviewpointsandattitudeofatextandappreciateEnglishliteraryworkswritteninplainwords.
2.Thestandardsofthetestingspecification
Chinahasauniquetestingsystemforcollegeentranceexaminations.Ratherthanusingthesametestpaperthroughoutthewholecountry,someprovinceshavetheirowntestpapersandmyprovinceJiangsuisamongthem.Inthiscase,boththenationalandtheprovincialtestingsyllabus—EnglishtestingsyllabusforcollegesdraftedbythePRCMinistryofEducationandtheIntroductionoftheEnglishExaminationdraftedbyJiangsuProvincialExaminationsAuthority—aretakenintoconsideration.
Thedemandsinthetwodocumentsincludeaspectsofreadingskillswhichneedtobetested:understandingthepurposeandmainidea,understandingspecificinformation,guessingunfamiliarwordsandphrasesaccordingtothecontext,makingsimplejudgmentandinference,understandingthebasicstructureofpassages,understandingtheauthor’sopinions,intentionsandattitude.
3.ThecharacteristicsofNEMTtestpapersintherecentyears
Althoughthetestingsyllabusprovidesgeneralideasofatestpaper,nodetailedstandardsaregiven.Forexample,fromthesyllabuswedon’tknowwhatkindofgenres,topicsshouldbeadoptedinthereadingtest.Wealsohavenoideaoftheexactrateofeachsub-skillstobetested.Inthiscase,itisnecessarytoanalyzetheNMETtestpapersaswell,whichareconsideredtobestrepresentthetestingsyllabusandcurriculumstandard,asadditionalstatements.What’smore,thecomparisonbetweentheNMETandthemockNMETcanalsoprovideevidenceofhowsuccessfullythemockpaperimitatestherealones.TheNMETpapersofyears2009,2010and2011inJiangsuProvincewillbetakenassamples.Thecharacteristicsofreadingcomprehensionpartinthesetestpaperswillbesummarized,andacomparisonbetweenthesecharacteristicsandthoseofthemockpaperwillbemade.
三、Researchdesign
1.Researchquestions
Thispaperseekstoinvestigatetwomainareas:
(1)DoesthereadingtestinthefirstmockNMETin2012,Yangzhoureachahighdegreeofvalidityaccordingtothestandardsmentionedabove?
(2)DoesthereadingtestinthefirstmockNMETin2012,YangzhousucceedinmockingtheNMETpapers?
2.Researchobjects
TheresearchobjectsaremainlythereadingtestpaperofthefirstmockNMETin2012,Yangzhou.Besides,interviewsfromsixstudentswhotookthetest,andtwoteachersofthemarealsoinvolvedintheresearch.
3.Datacollection
Thisstudycollectsbothquantitativedataandqualitativedata.Thedatahavebeencollectedfromtherecentyears’NMETpapersandthefirstmockNMETpaperthisyear.Theseincludeobjectivedatasuchaslength,readingspeed,andsubjectivedatasuchasgenres,topics,questiontypes.Computershavebeenadoptedincollectingtheobjectivedatawhicharerelatedtothereadingmaterials,thequestionsandstudents’scores.Besides,inquiriesintotheattitudeofsixrandomlypickedstudentsandtheirteacherswerealsoconductedsoonafterthetestwiththehelpoftwoofmyformercolleagues.AccordingtoMcNamara,test-takerfeedbackshouldbegatheredfromthetrialsubjects,oftenbyasimplequestionnaire.Thiswillincludequestionsonperceptionsofthelevelofdifficultyofparticularquestions,theclarityoftherubrics,andgeneralattitudetothematerialsandtasks.Subjectscanquicklyspotthingsthatareproblematicaboutmaterialswhichtestdevelopersstruggletosee.
四、Dataanalysisandfindings
1.Theanalysisanddiscussionsofthereadingpassages
Whendesigningareadingtest,choosingappropriatereadingtextsarevitallyimportant.Textswithcertainreadabilityshouldbeapplied.Thiswillallowstudentstoresponsetothequestionsupontheirunderstanding,andwillresultinahighdegreeofcontentvalidityofthetest.Otherwise,studentsarelikelytoguessbyallmeanstheanswers,andtheirtestresultsareprobablyinvalid.Aldersonstatesthatmanyaspectsoftextmightfacilitateormakedifficultthereadingprocess.Sometextvariableswillbediscussedinthispart,includingtextlength,readingspeed,genres,andtopics.
(1)Lengthandreadingspeed
Thereisnotaspecificstandardofthetextlengthintestingsyllabus.However,byanalyzingtherecentNMETpapers,wecangetsomeideasofthisaspect.Asisshowninthetable,thetotallengthofallthereadingpassagesandallthetextquestionsinaNMETpaperrangesfrom2275-2358,theaveragebeing2332.Whileinthemockpaper,totalnumberofwordsinthereadingtestis2296,whichisjustintherangeofNMETandisveryclosetotheaveragenumber.
Thesuggestedtiming(bythetestingsyllabus)forreadingcomprehensionsectionis30minutes.Ifthetotalinputamountispidedbytesttime,wewillgettheresultofthereadingspeed.Inthisway,readingspeedofthemockexamis77wordsperminute,whichismoreorlessthesamewiththoseofNMET.Meanwhile,italsoseemstomeetthestandarddemandedbythenationalteachingsyllabusof70-80wordsperminute.However,YangandWeirindicatethatthedesigneroftestpapersneedtotakeintoaccountofthefactthatittakessometimetoreadthequestionsitemsandthinkabouttheanswerstothequestions.Inotherwords,the30-minutetesttimecannottotallybeusedtoreadthepassagesandquestions.Inthiscase,students’takingthismockNMETprobablyneedtoreadmorethan80wordsperminute,whichisbeyondtherequirementsofteachingsyllabusandwillmakethereadingtestmoredifficult.“Thedifficultyandperceptionofspeedednessmayleadtoguessing,orcompensatorytesttakingstrategies...oreventhecompletelackofresponsetothetesttask”,jeopardizingthetestvalidity.
(2)Genresoftexts
KobayashihasfoundthattexttypeandresponseformataffectreadingcomprehensionperformanceofL2learnersofdifferentproficiencylevel.Theimportanceoftexttypeisemphasizedandattentionneedstobepaidtoit.AccordingtoCoulthard,agenreisonetypeofstylisticstructurefororganizingsentencesandutterancesintolargerunits.Genresmaydifferintheirformalcharacteristicssuchasparagraphstructure,thematicsequence,thestereotypicalorderingofeventsintime,andthedistributionofsentencetypesandpronominalforms.InStandardsofEnglishcurriculuminfull-timecompulsoryeducationandseniorhighschools,studentsarerequiredthecompetencetoidentifydifferentgenresoftextsaccordingtothesecharacteristics.SoitisnecessaryforthemockNMETdesignerstoinvolvedifferentgenresoftextsaccordingtothestandardsofthecontentstatements.In2012mockNMETthatIamstudyingon,itsreadingtestpaperisvariedintexttypes,consistingoftwoexpositions,onenarration,andonepracticalwriting.Besides,accordingtothestatisticspresentedintheabovetable,2012mockNMETisquitesimilartothepreviousNMETintermsoftexttypes.Inotherwords,itdidagoodjobinmeetingthestandardsandmockingtheNMET.
However,althoughthe2009-2011NMETdonotcontainanyliteraryworks,curriculumsyllabusencouragestudentstolearntoappreciateliteraryworkswritteninplainwords.Thistypeoftextsiscompletelyneglectedinthemockpaperaccordingtothecollecteddata.
(3)Topicsoftexts
Bachmanexplainsthattopicofinputhasasubstantialeffectontesttakers’performance,whichcouldbereferredtoas“passageeffect”intestsofreadingcomprehension.Sotopicsthatareinterestingandrelevanttotesttakersarethemostappropriate,meanwhile,possiblebiasandeitherextremesshouldbeavoided.Asaresult,testdesignersselectedalotofgeneralorinnocuoustopicsforthereadingcomprehensionofNMETinthepast.Topicfamiliarityisexpectedto“haveafacilitatingeffect”onthetestresult.AldersonandUrquhartwereabletoshowthatreadingtestonthetextsthatarefamiliartothestudentsweresometimeseasiertoprocessthanthosewhichwerenot.Oneofthereasonsisthatinreadinganunfamiliartextforcomprehension,studentsmustbeabletorecognizeallthewordsandstructuresincontext,whichrequireshigherlanguageproficiencyofthestudents.ThenationalEnglishcurriculumstandardshasgivenoutarangeoftexttopics,consistingof121topicsof24categories.Readingcompetencecanbereflectedinreadingallkindsofmaterials;thereforedifferenttexttopicsanddiffernttextgenresshouldbeinvolvedinthereadingtestsasmanyaspossible.Inrecentyears’NMET,readingcomprehensionofteninvolvesvarioustopicsincludingdailylife,commonsense,news,advertisements,culture,economics,history,geography,etc.Thiscanbeembodiedbythedataintheabovetable.Mostofthetopicsareclosetothedailylifeandarefamiliartothestudents.In2012mockNMET,mostofthetexttopicsareverymuchclosetostudents’life,exceptPassage3.Passage3isaboutthesecurityoffood,focusingonproblemsexistingonfoodlabels.Althoughfoodsecurityhassomethingtodowithstudents’dailylife,thepoliciesoffoodlabelsarenot.Thisunfamiliarityresultsinstudents’incrediblypoorperformance.Forinstance,students’scoringaverageinmycityofthesecondquestionandthefourthquestionofthispassageisonly23.5%and24.5%respectively.Thisinappropriatenessintexttopicchoicecreatesflawsincontentvalidityofthetest.
2.Theanalysisanddiscussionofthetestquestions
Aldersonstatesthattherearevariablesaffectingtheassessmentofreadinginafirstandinasecondlanguage,andthequestionssetandthepassagesonwhichcomprehensionisassessedisequallyimportant.Sainsburyfurtherpointsoutthatinthecaseofmanytypesofreadingtests,theevaluationofcontentmustbeappliedbothtopassagesoftextandtothequestionsasked.Accordingtothe2011IntroductionofChinese,MathsandEnglishExaminationsforCollegeEntranceExaminationdraftedbyJiangsuProvincialExaminationsAuthority,sixtypesofreadingskillsneedtobetested.Usuallythesixskillsaresimplifiedasthefollowingfourskills:understandingthepurposeandgist,understandingexplicitlystatedinformation,makingjudgmentandinference,deducingthemeaningofunfamiliarwords.AccordingtoZou,agoodreadingtesttestsreadingskillsthosearerepresentative;testdesignersshouldavoidemphasizingoneorseveraltypesofreadingskills,suchasfocusingtomuchondeducingthemeaninganduseofunfamiliarlexicalitems.
Dataintheabovetableillustratethat2012mockpaperwellmeetsthestandardsandwellmockstheNMETintermsofthequestiontypes.Itcoversthreeofthefoursub-skillsofreadingcomprehension,withanappropriaterate.Questionsrequiringskillsofdeducingthemeaningofunfamiliarwordsarenotmentionedinthispaper.Yet,sincethecircumstancesoccurredinthe2011NMETaswellandthereisnorequirementinsyllabusoftestingallthesub-skillsinasingletest,itcouldberegardedasacceptable.Meanwhile,numbersoftheotherthreetypesofquestionsareverymuchclosetotheaveragedataof2009-2011NMET.Moreover,allthequestionsarebasedonthematerials,nodeviationfromthepassages.Thesefactsofquestiontypesaddtothevalidationevidenceofreadingcomprehensionpartofthismockpaper.
3.Theanalysisanddiscussionofstudents’performanceandscores
Inordertogathertheinformationofstudents’feelingsandperformances,aquestionnairetothestudentsandinterviewswithtwooftheirteacherswereconductedbyoneofmycolleague.Whenfinishingthequestionnaire,thestudentsrecalledthecoursetoobtainsolution.Allthesixsubjectsthoughtthatthereadingpassages,especiallythethirdone,areabitmoredifficultthanusual,inwhichtheycameacrossmanynewwordsandsomelongandcomplexsentences.Fourofthemspentover35minuteswhilethesuggestedtimingis30.Thetwoteachersbelievedthatthereadingsectionisonthewholeawell-designedone.Theythoughthighlyofthechoiceofmaterials,bothintopicsandingenres.Buttheyheldthesameviewonthedifficultylevelastheirstudents.Bythistoken,themajorprobleminthissectionisthedifficultylevel.
五、Conclusion
1.Summaryofthestudy
Throughthethoroughanalysisofmuchrelateddata,IthinkthatthereadingtestinthefirstmockNMETin2012,Yangzhou,succeedstosomeextentinthefollowingtwoaspects:First,itwellanswersthenationalEnglishcurriculumstandardandtheprovincialtestingsyllabus.Inotherwordsitreachesahighdegreeofvalidity.Second,itsuccessfullymockstheNMETpapers.Soitcanembodystudents’truelevelinreadingcomprehensionbyandlarge.Andwiththecorrectfeedbackfromthetest,bothteachersandstudentsmayappropriatelyadjusttheirstrategiesintherevisionofreading.
However,someflawsdoexistinthetestdesignthatcouldjeopardizeitsvalidity.Themostsignificantoneisthatitisalittledifficultfortheintendedtesttakersbasedonitshighreadingspeed,itsrelativelylowscoringaverageandstudents’feelings.Itisquitecommoninmycontextthatreadingtestsofwhichtherequirementsareabovestudents’proficiencylevelarepreferredbyteachers.Forinstance,theyoftenadoptthereadingtestsofCET-4(CollegeEnglishTestBand4)level,thinkingitcouldbebeneficialtopromotestudents’learning.YetIbelievethesetestscouldnothelptoincreasecomprehensibleinputandisnothelpfulforstudents’improvementofreadingproficiency.
2.Limitationsofthestudy
Dataofthetestpapersandquestionnairesarecollectedandanalyzedcarefullyinthisstudy.However,sincedifferentpeoplemayhavedifferentcognitionanddifferentdatacollectingprocedures,thedata,especiallythesubjectiveones,mightbeslightlydifferentfromthefacts.Besides,duetolackofofficialdataofstudents’NMETscoresinrecentyearsandnoaccesstothetestdesignersofthismockexam,thisstudydoesnotanalyzethoroughlyanddetailedlyaspectslikedifficultydegreeanditemdiscrimination.Also,thewashbackeffectsofmockNMET,whichisnotreferredtointhispaper,couldmeritfurtherenquiry.