TY - GEN
T1 - Differential approximation and sprinting for multi-priority big data engines
AU - Birke, Robert
AU - Rocha, Isabelly
AU - Perez, Juan
AU - Schiavoni, Valerio
AU - Felber, Pascal
AU - Chen, Lydia Y.
N1 - Funding Information:
The research leading to these results has received funding from the European Union’s Horizon 2020 research and innovation programme under the LEGaTO Project (legato-project. eu), grant agreement No 780681. This work has been partly funded by the Swiss National Science Foundation NRP75 project 407540_167266.
Publisher Copyright:
© 2019 Association for Computing Machinery.
Copyright:
Copyright 2020 Elsevier B.V., All rights reserved.
PY - 2019/12/9
Y1 - 2019/12/9
N2 - Today’s big data clusters based on the MapReduce paradigm are capable of executing analysis jobs with multiple priorities, providing differential latency guarantees. Traces from production systems show that the latency advantage of high-priority jobs comes at the cost of severe latency degradation of low-priority jobs as well as daunting resource waste caused by repetitive eviction and re-execution of low-priority jobs. We advocate a new resource management design that exploits the idea of differential approximation and sprinting. The unique combination of approximation and sprinting avoids the eviction of low-priority jobs and its consequent latency degradation and resource waste. To this end, we designed, implemented and evaluated DiAS, an extension of the Spark processing engine to support deflate jobs by dropping tasks and to sprint jobs. Our experiments on scenarios with two and three priority classes indicate that DiAS achieves up to 90% and 60% latency reduction for low- and high-priority jobs, respectively. DiAS not only eliminates resource waste but also (surprisingly) lowers energy consumption up to 30% at only a marginal accuracy loss for low-priority jobs.
AB - Today’s big data clusters based on the MapReduce paradigm are capable of executing analysis jobs with multiple priorities, providing differential latency guarantees. Traces from production systems show that the latency advantage of high-priority jobs comes at the cost of severe latency degradation of low-priority jobs as well as daunting resource waste caused by repetitive eviction and re-execution of low-priority jobs. We advocate a new resource management design that exploits the idea of differential approximation and sprinting. The unique combination of approximation and sprinting avoids the eviction of low-priority jobs and its consequent latency degradation and resource waste. To this end, we designed, implemented and evaluated DiAS, an extension of the Spark processing engine to support deflate jobs by dropping tasks and to sprint jobs. Our experiments on scenarios with two and three priority classes indicate that DiAS achieves up to 90% and 60% latency reduction for low- and high-priority jobs, respectively. DiAS not only eliminates resource waste but also (surprisingly) lowers energy consumption up to 30% at only a marginal accuracy loss for low-priority jobs.
UR - http://www.scopus.com/inward/record.url?scp=85078012099&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85078012099&partnerID=8YFLogxK
U2 - 10.1145/3361525.3361547
DO - 10.1145/3361525.3361547
M3 - Conference contribution
AN - SCOPUS:85078012099
T3 - Middleware 2019 - Proceedings of the 2019 20th International Middleware Conference
SP - 202
EP - 214
BT - Middleware 2019 - Proceedings of the 2019 20th International Middleware Conference
PB - Association for Computing Machinery
T2 - 20th ACM/IFIP/USENIX Middleware Conference, Middleware 2019
Y2 - 9 December 2019 through 13 December 2019
ER -