Post-Doctoral Research Visit F/M Data management and job scheduling for Geo-distributed Workflows

1000scholars; Rennes; Hybrid; EUR 38,000 - 46,000; Full time

Job summary

Inria Rennes Center invites applications for a postdoc position within the MAGELLAN team, as part of the IPCEI-CIS DXP project in collaboration with Amadeus and partner teams. The position is for one year, with the possibility to extend to 24 months, hosted at Rennes University.

You will study geo-distributed data workflows, optimize scheduling and data movement to improve performance and cost. Activities include literature synthesis, design and implementation of new policies, large-scale

Qualifications

• Ph.D. in computer science or related field required.
• Strong background in distributed systems and data-heavy workflows.
• Experience with experimental systems research and benchmarking.
• Proven ability to build and evaluate large-scale systems and tools.
• Good command of English for collaboration and documentation.

Responsibilities

• Read and synthesize literature on geo-distributed workflows.
• Design scheduling policies and data management for geo-distributed workloads.
• Implement and validate solutions at scale.
• Participate in project meetings with partners.
• Prepare research papers and present results at meetings and conferences.

Skills

Distributed systems

Big Data

English communication

Experimentation & benchmarking

System development

Education

Ph.D. in computer science

Tools

Cloud computing tooling

Job description

Context

Financial and working environment.

This postdoc position will be in the context of IPCEI-CIS (Important Project of Common European Interest – Next Generation Cloud Infrastructure and Services) DXP (Data Exchange Platform) project involving Amadeus and three Inria research teams (COAST, CEDAR and MAGELLAN). This project aims to design and develop an open-source management solution for a federated and distributed data exchange platform (DXP), operating in an open, scalable, and massively distributed environment (cloud-edge continuum). The position will be recruited and hosted at the Inria Center at Rennes University; and the work will be carried out within the MAGELLAN team in collaboration with other partners.

The position is for one year, with the possibility of an extension to 24 months.

Assignment

Context: Data are usually hosted in multiple geo-distributed locations (private and public clouds, distributed caches and across the edge-to-cloud continuum). Collectively processing these data is a must (e.g. distributed data analysis and queries), but this presents several challenges to existing data-intensive distributed workflow frameworks (e.g. MapReduce [1], Spark [2], TensorFlow [3] and Dataflow [4]). This is due to the low capacity of wide area network (WAN) links, as well as the heterogeneity of networks, computation power and monetary cost in geo-distributed environments [5].

Much effort has devoted on optimizing the performance of geo-distributed workflows [5, 6, 7, 8, 9]. These efforts mainly focus on reducing cross-data center data transfer and optimal task placement according to performance heterogeneity of the data centers [5, 6, 7]. However, these efforts do not consider the storage services heterogeneity [10] (input and intermediate data are stored on different devices), monetary cost heterogeneity in terms of computing, storage and network, or multi-tenancy when multiple workflows run concurrently.

Objectives

We will study the interplay and correlation between the different factors that contribute to the performance of geo-distributed workflows (i.e., input data, intermediate data, number of iterations, capacity of site, etc). Accordingly, we will design scheduling policies and associated data movement to improve the overall performance, monetary cost, and resource utilization by considering the input location, the network cost and status, intermediate data, and the capacity of the different sites, when scheduling multiple workflows across massively distributed environments.

[1] J. Dean and S. Ghemawat, “MapReduce: Simplified Data Processing on Large Clusters,” Commun. ACM, pp. 107–113, Jan. 2008.

[2] M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica, “Spark: Cluster Computing with Working Sets,” in Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing (HotCloud’10). USENIX Association, 2010, pp. 10–10.

[3] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: a system for large-scale machine learning. In Proceedings of the 12th USENIX conference on Operating Systems Design and Implementation (OSDI'16).

[4] Tyler Akidau, Robert Bradshaw, Craig Chambers, Slava Chernyak, Rafael J. Fernández-Moctezuma, Reuven Lax, Sam McVeety, Daniel Mills, Frances Perry, Eric Schmidt, and Sam Whittle. 2015. The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing. Proc. VLDB Endow.

[5] Qifan Pu, Ganesh Ananthanarayanan, Peter Bodik, Srikanth Kandula, Aditya Akella, Paramvir Bahl, and Ion Stoica. Low latency geo-distributed data analytics. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication, SIGCOMM’15, pages 421–434, New York, NY, USA, 2015. ACM.

[6] Sharad Agarwal, John Dunagan, Navendu Jain, Stefan Saroiu, Alec Wol- man, and Harbinder Bhogan. Volley: Automated data placement for geo- distributed cloud services. In Proceedings of the7th USENIX Conference on Networked Systems Design and Implementation, NSDI’10, pages 2–2, Berkeley, CA, USA, 2010. USENIX Association.

[7] WANG, Qingyuan, GAO, Bin, ZHOU, Zhi, et al. Dag-aware optimization for geo-distributed data analytics. In: Proceedings of the 52nd International Conference on Parallel Processing. 2023. p. 472-481.

[8] Ashish Vulimiri, Carlo Curino, Philip Brighten Godfrey, Thomas Jungblut, Konstantinos Karanasos, Jitendra Padhye, and George Varghese. Wanalytics: Geo-distributed analytics for a data intensive world. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, SIGMOD’15, pages 1087–1092, New York, NY, USA, 2015. ACM.

[9] Chien-Chun Hung, Leana Golubchik, and Minlan Yu. Scheduling jobs across geo-distributed datacenters. In Proceedings of the Sixth ACM Symposium on Cloud Computing, SoCC’15, pages 111–124, New York, NY, USA, 2015. ACM.

[10] Hao Wu, Junxiao Deng, Hao Fan, Shadi Ibrahim, Song Wu, Hai Jin, “QoS-Aware and Cost-Efficient Dynamic Resource Allocation for Serverless ML Workflows,” 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS), St. Petersburg, FL, USA, 2023, pp. 886-896.

Main activities

• Read and synthesize literature work.
• Design new scheduling policies and data management for Geo-distributed Workflows
• Implementation and large-scale validation.
• Participate in project meetings and discussions with other partners.
• Write research papers and disseminate results through presentations at project meetings, conferences, and workshops.

Skills

• A Ph.D. in computer science
• A solid background in the area of distributed systems
• Ability to conduct experimental systems research
• Experience with building systems and tools
• Working experience in the areas of Big Data management, Cloud Computing, Data Analytics are advantageous
• Very good communication skills in oral and written English