What is Wrong with Topic Modeling? (and How to Fix it Using Search-based Software Engineering)
Context Topic modeling finds human-readable structures in unstructured textual data. A widely used topic modeling technique is Latent Dirichlet allocation. When running on different datasets, LDA suffers from “order effects”, i.e., different topics are generated if the order of training data is shuffled. Such order effects introduce a systematic error for any study. This error can relate to misleading results; specifically, inaccurate topic descriptions and a reduction in the efficacy of text mining classification results.
Objective To provide a method in which distributions generated by LDA are more stable and can be used for further analysis.
Method We use LDADE, a search-based software engineering tool which uses Differential Evolution (DE) to tune the LDA’s parameters. LDADE is evaluated on data from a programmer information exchange site (Stackoverflow), title and abstract text of thousands of Software Engineering (SE) papers, and software defect reports from NASA. Results were collected across different implementations of LDA (Python+Scikit-Learn, Scala+Spark) across Linux platform and for different kinds of LDAs (VEM, Gibbs sampling). Results were scored via topic stability and text mining classification accuracy.
Results In all treatments: (i) standard LDA exhibits very large topic instability; (ii) LDADE’s tunings dramatically reduce cluster instability; (iii) LDADE also leads to improved performances for supervised as well as unsupervised learning.
Conclusion Due to topic instability, using standard LDA with its “off-the-shelf” settings should now be depreciated. Also, in future, we should require SE papers that use LDA to test and (if needed) mitigate LDA topic instability. Finally, LDADE is a candidate technology for effectively and efficiently reducing that instability.
Thu 14 Nov
16:00 - 16:20 Talk | What is Wrong with Topic Modeling? (and How to Fix it Using Search-based Software Engineering) Amritanshu AgrawalWayfair, Wei FuDepartment of Computer Science, North Carolina State University, Tim MenziesNorth Carolina State University Link to publication | |||||||||||||||||||||||||||||||||||||||||
16:20 - 16:40 Talk | Cautious Adaptation of Defiant Components Paulo MaiaState University of Ceará, Lucas VieiraState University of Ceará, Matheus ChagasState University of Ceará, Yijun YuThe Open University, UK, Andrea ZismanThe Open University, Bashar NuseibehThe Open University (UK) & Lero (Ireland) | |||||||||||||||||||||||||||||||||||||||||
16:40 - 17:00 Talk | Better Development of Safety Critical Systems:Chinese High Speed Railway System Development Experience Report Zhiwei WuEast China Normal University, Jing LiuEast China Normal University, Xiang ChenCASCO Signal Ltd. | |||||||||||||||||||||||||||||||||||||||||
17:00 - 17:20 Talk | Active Hotspot: An Issue-Oriented Model to Monitor Software Evolution and Degradation Qiong FengDrexel University, Yuanfang Cai Drexel University, Rick KazmanUniversity of Hawai‘i at Mānoa, Di CuiXi'an Jiaotong University, Ting LiuXi'an Jiaotong University, Hongzhou FangDrexel University | |||||||||||||||||||||||||||||||||||||||||
17:20 - 17:30 Talk | Automated Trainability Evaluation for Smart Software Functions Ilias GerostathopoulosTechnical University of Munich, Stefan KugeleTechnical University of Munich, Christoph SeglerBMW Group Research, New Technologies, Innovations, Tomas BuresCharles University, Czech Republic, Alois KnollTechnical University of Munich Pre-print | |||||||||||||||||||||||||||||||||||||||||
17:30 - 17:40 Demonstration | Lancer: Your Code Tell Me What You Need Shufan ZhouSchool of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Beijun ShenSchool of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Hao ZhongShanghai Jiao Tong University |