DeepMutation++: a Mutation Testing Framework for Deep Learning Systems
Thu 14 Nov 2019 12:00 - 12:10 at Cortez 2&3 - Deep Models Chair(s): Nazareno Aguirre
Deep neural networks~(DNNs) are increasingly expanding their real-world applications across domains, e.g., image processing, speech recognition and natural language processing. However, there is still limited tool support for DNN testing in terms of test data quality and model robustness. In this paper, we introduce a mutation testing-based tool for DNNs, DeepMutation++, which facilitates the DNN quality evaluation, supporting both feed-forward neural networks~(FNNs) and stateful recurrent neural networks~(RNNs). It not only enables to statically analyze the robustness of a DNN model against the input as a whole, but also allows to identify the vulnerable segments of a sequential input (e.g. audio input) by runtime analysis. It is worth noting that DeepMutation++ specially features the support of RNNs mutation testing. The tool demo video can be found on the project website https://sites.google.com/view/deepmutationpp.
Wed 13 Nov
10:00 - 10:40 Demonstration | TsmartGP: A Tool for Finding Memory Defects with Pointer Analysis Yuexing WangTsinghua University, Guang ChenTsinghua University, Min ZhouTsinghua University, Ming GuTsinghua University, Jiaguang SunTsinghua University | |||||||||||||||||||||||||||||||||||||||||
10:00 - 10:40 Demonstration | BuRRiTo: A Framework to Extract, Specify, Verify and Analyze Business Rules Pavan ChittimalliTCS Research, Kritika AnandTCS Research, Shrishti PradhanTCS Research, Sayandeep MitraTCS Research, Chandan PrakashTCS Research, Rohit ShereTCS Research, Ravindra NaikTCS Research, TRDDC, India | |||||||||||||||||||||||||||||||||||||||||
10:00 - 10:40 Demonstration | Lancer: Your Code Tell Me What You Need Shufan ZhouSchool of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Beijun ShenSchool of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Hao ZhongShanghai Jiao Tong University | |||||||||||||||||||||||||||||||||||||||||
10:00 - 10:40 Demonstration | TestCov: Robust Test-Suite Execution and Coverage Measurement Pre-print Media Attached File Attached | |||||||||||||||||||||||||||||||||||||||||
10:00 - 10:40 Demonstration | Prema: A Tool for Precise Requirements Editing, Modeling and Analysis Yihao HuangEast China Normal University, Jincao FengEast China Normal University, Hanyue ZhengEast China Normal University, Jiayi ZhuEast China Normal University, Shang WangEast China Normal University, Siyuan JiangEastern Michigan University, Weikai MiaoShanghai Key Lab for Trustworthy Computing, School of Computer Science and Software Engineering, East China Normal University, Geguang PuEast China Normal University&Shanghai Trusted Industrial Control Platform Co., Ltd | |||||||||||||||||||||||||||||||||||||||||
10:00 - 10:40 Demonstration | XRaSE: Towards Virtually Tangible Software using Augmented Reality Rohit MehraAccenture Labs, India, Vibhu Saujanya SharmaAccenture Labs, Vikrant KaulgudAccenture Labs, India, Sanjay PodderAccenture | |||||||||||||||||||||||||||||||||||||||||
10:00 - 10:40 Demonstration | MuSC: A Tool for Mutation Testing of Ethereum Smart Contract Zixin LiNanjing University, Haoran WuState Key Laboratory for Novel Software Technology, Nanjing University, Jiehui XuNanjing University, Xingya WangState Key Laboratory for Novel Software Technology, Nanjing University, Lingming ZhangThe University of Texas at Dallas, Zhenyu ChenNanjing University | |||||||||||||||||||||||||||||||||||||||||
10:00 - 10:40 Demonstration | VeriSmart 2.0: Swarm-Based Bug-Finding for Multi-Threaded Programs with Lazy-CSeq Bernd FischerStellenbosch University, Salvatore La TorreUniversità degli Studi di Salerno, Gennaro ParlatoUniversity of Molise | |||||||||||||||||||||||||||||||||||||||||
10:00 - 10:40 Demonstration | DeepMutation++: a Mutation Testing Framework for Deep Learning Systems Qiang HuKyushu University, Japan, Lei MaKyushu University, Xiaofei XieNanyang Technological University, Bing YuKyushu University, Japan, Yang LiuNanyang Technological University, Singapore, Jianjun ZhaoKyushu University | |||||||||||||||||||||||||||||||||||||||||
10:00 - 10:40 Demonstration | Manticore: A User-Friendly Symbolic Execution Framework for Binaries and Smart Contracts Mark MossbergTrail of Bits, Felipe ManzanoTrail of Bits, Eric HennenfentTrail of Bits, Alex GroceTrail of Bits, Gustavo GriecoTrail of Bits, Josselin FeistTrail of Bits, Trent BrunsonTrail of Bits, Artem DinaburgTrail of Bits Media Attached | |||||||||||||||||||||||||||||||||||||||||
10:00 - 10:40 Demonstration | ConVul: An Effective Tool for Detecting Concurrency Vulnerabilities Ruijie MengUniversity of Chinese Academy of Sciences, Biyun ZhuUniversity of Chinese Academy of Sciences, Hao YunUniversity of Chinese Academy of Sciences, Haicheng LiUniversity of Chinese Academy of Sciences, Yan CaiInstitute of Software, Chinese Academy of Sciences, Zijiang YangWestern Michigan University | |||||||||||||||||||||||||||||||||||||||||
10:00 - 10:40 Demonstration | mCUTE: A Model-level Concolic Unit Testing Engine for UML State Machines Reza AhmadiQueen's University, Karim JahedQueen's University, Juergen DingelQueen's University, Kingston, Ontario |
Thu 14 Nov
10:40 - 12:20: Papers - Deep Models at Cortez 2&3 Chair(s): Nazareno AguirreDept. of Computer Science FCEFQyN, University of Rio Cuarto | ||||||||||||||||||||||||||||||||||||||||||
10:40 - 11:00 Talk | Wuji: Automatic Online Combat Game Testing Using Evolutionary Deep Reinforcement LearningACM SIGSOFT Distinguished Paper Award Yan ZhengTianjin University, Xiaofei XieNanyang Technological University, Ting SuETH Zurich, Lei MaKyushu University, Jianye HaoTianjin University, Zhaopeng MengTianjin University, Yang LiuNanyang Technological University, Singapore, Ruimin ShenFuxi AI Lab in Netease, Yinfeng ChenFuxi AI Lab in Netease, Changjie FanFuxi AI Lab in Netease | |||||||||||||||||||||||||||||||||||||||||
11:00 - 11:20 Talk | A Study of Oracle Approximations in Testing Deep Learning Libraries | |||||||||||||||||||||||||||||||||||||||||
11:20 - 11:40 Talk | Property Inference for Deep Neural Networks Divya GopinathCarnegie Mellon University, Hayes ConverseThe University of Texas at Austin, Corina S. PasareanuCarnegie Mellon University Silicon Valley, NASA Ames Research Center, Ankur TalyGoogle | |||||||||||||||||||||||||||||||||||||||||
11:40 - 12:00 Talk | An Empirical Study towards Characterizing Deep Learning Development and Deployment across Different Frameworks and Platforms Qianyu GuoTianjin University, Sen ChenNanyang Technological University, Singapore, Xiaofei XieNanyang Technological University, Lei MaKyushu University, Qiang HuKyushu University, Japan, Hongtao LiuTianjin University, Yang LiuNanyang Technological University, Singapore, Jianjun ZhaoKyushu University, Li XiaohongTianJin University Pre-print | |||||||||||||||||||||||||||||||||||||||||
12:00 - 12:10 Demonstration | DeepMutation++: a Mutation Testing Framework for Deep Learning Systems Qiang HuKyushu University, Japan, Lei MaKyushu University, Xiaofei XieNanyang Technological University, Bing YuKyushu University, Japan, Yang LiuNanyang Technological University, Singapore, Jianjun ZhaoKyushu University | |||||||||||||||||||||||||||||||||||||||||
12:10 - 12:20 Demonstration | DeepHunter: A Coverage-Guided Fuzzer for Deep Neural Networks Xiaofei XieNanyang Technological University, Hongxu ChenNanyang Technological University, Yi LiNanyang Technological University, Lei MaKyushu University, Yang LiuNanyang Technological University, Singapore, Jianjun ZhaoKyushu University |