Differential approximation and sprinting for multi-priority big data engines

Robert Birke, Isabelly Rocha, Juan Perez, Valerio Schiavoni, Pascal Felber, Lydia Y. Chen

Research output: Chapter in Book/InformConference contribution

1 Scopus citations

Abstract

Today’s big data clusters based on the MapReduce paradigm are capable of executing analysis jobs with multiple priorities, providing differential latency guarantees. Traces from production systems show that the latency advantage of high-priority jobs comes at the cost of severe latency degradation of low-priority jobs as well as daunting resource waste caused by repetitive eviction and re-execution of low-priority jobs. We advocate a new resource management design that exploits the idea of differential approximation and sprinting. The unique combination of approximation and sprinting avoids the eviction of low-priority jobs and its consequent latency degradation and resource waste. To this end, we designed, implemented and evaluated DiAS, an extension of the Spark processing engine to support deflate jobs by dropping tasks and to sprint jobs. Our experiments on scenarios with two and three priority classes indicate that DiAS achieves up to 90% and 60% latency reduction for low- and high-priority jobs, respectively. DiAS not only eliminates resource waste but also (surprisingly) lowers energy consumption up to 30% at only a marginal accuracy loss for low-priority jobs.

Original languageEnglish (US)
Title of host publicationMiddleware 2019 - Proceedings of the 2019 20th International Middleware Conference
PublisherAssociation for Computing Machinery
Pages202-214
Number of pages13
ISBN (Electronic)9781450370097
DOIs
StatePublished - Dec 9 2019
Event20th ACM/IFIP/USENIX Middleware Conference, Middleware 2019 - Davis, United States
Duration: Dec 9 2019Dec 13 2019

Publication series

NameMiddleware 2019 - Proceedings of the 2019 20th International Middleware Conference

Conference

Conference20th ACM/IFIP/USENIX Middleware Conference, Middleware 2019
Country/TerritoryUnited States
CityDavis
Period12/9/1912/13/19

All Science Journal Classification (ASJC) codes

  • Software

Fingerprint

Dive into the research topics of 'Differential approximation and sprinting for multi-priority big data engines'. Together they form a unique fingerprint.

Cite this