Automated Testing as Production Simulation: Research Opportunities and Challenges
Testing has always been about simulation, although often implicit. For example, when a software tester mocks out the behaviour of a code unit, they are effectively stimulating its behaviour. When a test tool creates a test user account in order to log in and run a test it is essentially simulating a potential user behaviour. Traditionally, software engineers, and the research that supports them, has tended to apply greater focus to problems and challenges associated with test mock techniques and applications. However, as software is increasingly deployed to user communities that interact through the software platform, effective techniques for simulating user behaviour are becoming increasingly important. In this talk we outline engineering trade-offs between test (simulation) realism, which we seek to maximise, and various testing costs we seek to minimise. We present an overview of and results from the deployment, at Meta, of two simulation-inspired testing technologies: Sapienz and Web Enabled Simulation, both of which aim to realistically simulate user behaviour, on client side and server side respectively. We show how a simulation-based testing approach opens new perspectives on new and existing automated software engineering challenges and opportunities.
