As Introductory Programming classes continue to grow, it becomes ever more important to ensure that automatic grading is both efficient yet still maintains high accuracy. Specifically, the use of automated testing for this type of grading has become standard, but it is not without fault. Instructor test suites can often misclassify correct student submissions as incorrect or vice versa. To address this discrepancy, we propose a novel approach which adapts the technique of Test Suite Augmentation to an educational setting in order to improve instructor test suite quality. Our approach first partitions student programs into clusters based on Behavioral Equivalence. We then use these clusters to better guide the Pex test-generator in generating a minimal set of tests which can augment an instructor’s suite and improve grading accuracy. We evaluated our approach on an extensive set of student submissions from a CS1 course and were able to find flaws in existing instructor test suites, and then augment these suites to substantially improve their quality.
Tue 12 Nov
13:40 - 15:20: Papers - Testing and Verification at Cortez 1 Chair(s): Weihang WangUniversity at Buffalo, SUNY | ||||||||||||||||||||||||||||||||||||||||||
13:40 - 14:00 Talk | Systematically Covering Input Structure Nikolas HavrikovCISPA Helmholtz Center for Information Security, Andreas ZellerCISPA Helmholtz Center for Information Security Pre-print | |||||||||||||||||||||||||||||||||||||||||
14:00 - 14:20 Talk | SEGATE: Unveiling Semantic Inconsistencies between Code and Specification of String Inputs Pre-print | |||||||||||||||||||||||||||||||||||||||||
14:20 - 14:40 Talk | Detecting Error-Handling Bugs without Error Specification Input Zhouyang JiaNational University of Defense Technology, Shanshan LiNational University of Defense Technology, Tingting YuUniversity of Kentucky, Xiangke LiaoNational University of Defense Technology, China, Ji WangNational University of Defense Technology, Xiaodong LiuNational University of Defense Technology, Yunhuai LiuPeking University | |||||||||||||||||||||||||||||||||||||||||
14:40 - 15:00 Talk | Test Automation and its Limitations Ahyoung SungSamsung Electronics, Yangsu KimSamsung Electronics, Sangjun KimSamsung Electronics, Jongin KimSamsung Electronics, Neo JangSamsung Electronics | |||||||||||||||||||||||||||||||||||||||||
15:00 - 15:10 Talk | Grading-Based Test Suite Augmentation Jonathan Osei-OwusuUniversity of Illinois at Urbana-Champaign, Angello AstorgaUniversity of Illinois at Urbana-Champaign, Liia ButlerUniversity of Illinois at Urbana-Champaign, Tao XiePeking University, Geoffrey ChallenUniversity of Illinois at Urbana-Champaign | |||||||||||||||||||||||||||||||||||||||||
15:10 - 15:20 Demonstration | MutAPK: Source-Codeless Mutant Generation for Android Apps Camilo Escobar-VelásquezUniversidad de los Andes, Michael Osorio-RiañoUniversidad de los Andes, Mario Linares-VásquezSystems and Computing Engineering Department , Universidad de los Andes , Bogotá, Colombia |