With the development of big data, machine learning, and AI, existing software engineering techniques must be re-imagined to provide the productivity gains that developers desire. This talk will review emerging roles of data scientists and the tools they need to build scalable, correct, and efficient software for a data centric world.

Kim will present a large-scale study of about 800 data scientists in collaboration with Microsoft Research, which looked at data scientists’ educational background, problem topics that they work on, tools they use, and activities. From the gathered data, she has identified nine distinct clusters of data scientists and best practices and challenges faced by each cluster.

In the second half of this talk, she will discuss the needs of re-targeting SE research community’s directions to address new challenges in the era of data-centric software development. In particular, she will detail some examples of her group’s work that re-invents debugging and testing for big data distributed systems such as Apache Spark. She will conclude with open SE problems in ML and heterogeneous computing that support data-centric software development.

Miryung Kim

University of California, Los Angeles

United States