A Task Allocation Method for Stream Processing with Recovery Latency Constraint

(整期优先)网络出版时间:2018-06-16
/ 1
Streamprocessingapplicationscontinuouslyprocesslargeamountsofonlinestreamingdatainrealtimeornearrealtime.Theyhavestrictlatencyconstraints.However,thecontinuousprocessingmakesthemvulnerabletoanyfailures,andtherecoveriesmayslowdowntheentireprocessingpipelineandbreaklatencyconstraints.Theupstreambackupschemeisoneofthemostwidelyappliedfault-tolerantschemesforstreamprocessingsystems.Itintroducescomplexbackupdependenciestotasks,whichincreasesthedifficultyofcontrollingrecoverylatencies.Moreover,whendependenttasksarelocatedonthesameprocessor,theyfailatthesametimeinprocessor-levelfailures,bringingextrarecoverylatenciesthatincreasetheimpactsoffailures.Thispaperstudiestherelationshipbetweenthetaskallocationandtherecoverylatencyofastreamprocessingapplication.Wepresentacorrelatedfailureeffectmodeltodescribetherecoverylatencyofastreamtopologyinprocessor-levelfailuresunderataskallocationplan.Weintroducearecovery-latencyawaretaskallocationproblem(RTAP)thatseekstaskallocationplansforstreamtopologiesthatwillachieveguaranteedrecoverylatencies.WediscussthedifferencebetweenRTAPandclassictaskallocationproblemsandpresentaheuristicalgorithmwithacomputationalcomplexityofO(nlog2n)tosolvetheproblem.Extensiveexperimentswereconductedtoverifythecorrectnessandeffectivenessofourapproach.Itimprovestheresourceusageby15%-20%onaverage.