Network partitions are inevitable in large-scale cloud systems. Despite developers’ efforts in handling network partitions throughout designing, implementing and testing cloud systems, bugs caused by network partitions, i.e., partition bugs, still exist and cause severe failures in production clusters. It is challenging to expose these partition bugs because they often require network partitions to start and stop at specific timings.

In this paper, we propose Consistency-Guided Fault Injection (CoFI), a novel technique that smartly injects network partitions to effectively expose partition bugs. We observe that, network partitions can leave cloud systems at inconsistent states, where partition bugs are more likely to occur. Based on this observation, CoFI first infers invariants (i.e., consistent states) among different nodes in a cloud system. Once observing a violation to the inferred invariants (i.e., inconsistent states) while running the cloud system, CoFI injects network partitions to prevent the cloud system from recovering back to consistent states, and thoroughly tests whether the cloud system still proceeds correctly at inconsistent states.We have applied CoFI to three widely-deployed cloud systems, i.e., Cassandra, HDFS, and YARN. CoFI has detected 7 previously-unknown bugs, and three of them have been confirmed by developers.

Haicheng Chen

The Ohio State University, USA

United States

Wensheng Dou

Institute of Software, Chinese Academy of Sciences

China

Dong Wang

Institute of software, Chinese academy of sciences

China

Feng Qin

Ohio State University, USA

In recent years the use of non-traditional computing mechanisms has grown rapidly. One paradigm uses chemical reaction networks (CRNs) to compute via chemical interactions. CRNs are used to prototype molecular devices at the nanoscale such as intelligent drug therapeutics. In practice, these programs are first written and simulated in environments such has MatLab and later compiled into physical molecules such as DNA strands. However, techniques for testing the correctness of CRNs are lacking. Current methods of validating CRNs include model checking and theorem proving, but these are limited in scalability. In this paper we present an automated testing framework, ChemTest. In contrast to model checking, ChemTest evaluates test oracles on individual simulation traces and supports functional, metamorphic, internal and hyper test cases. It also allows for flakiness and programs that are probabilistic. We performed a large case study demonstrating that ChemTest can find seeded faults and scales beyond model checking. Of our tests, 21% are inherently flaky, suggesting that systematic support for this paradigm is needed. On average functional tests find 66.5% of the faults, while metamorphic tests find 80.4%, showing the benefit of using metamorphic relationships in our test framework. In addition, we show how time of evaluation impacts fault detection.

Michael C. Gerten

Iowa State University

United States

James I. Lathrop

Iowa State University

Myra Cohen

Iowa State University

United States

Titus H. Klinge

Drake University

United States

As a common IT infrastructure, APM (Application PerformanceManagement) systems have been widely adopted to monitor call request to an on-line service. Usually, each request may contain multi-dimensional attributes(e.g., City, ISP, Platform, etc. ), which may become the reason for a certain anomaly regarding DSR (Declining Success Rate) of service calls either solely or as a combination. Moreover, each attribute may also have multiple values (e.g., ISP could be T-Mobile, Vodafone, CMCC, etc.), rendering intricate root causes and huge challenges to identify the root causes. In this paper, we propose a prototype tool, Imp APTr(Imp act Analysis based on Pruning Tree), to identify the combination of dimensional attributes as the clues leading to the root causes of anomalies regarding DSR of a service call in a timely manner. Imp APTr have been evaluated in MeiTuan, one of the biggest on-line service providers. Performance regarding the accuracy outperforms several previous tools in the same field.

hao wang

Guoping Rong

Nanjing University

Yangchen Xu

Nanjing University

Yong You

Meituan-Dianping Group