Modern programs, and especially low-level libraries, often rely on concurrent memory accesses that do not enforce memory ordering, such as C++’s memory_order_relaxed accesses. Concurrent garbage collectors commonly rely on such accesses, as does a surprising amount of other performance-critical software. On some common architectures, it is both unnecessary and quite expensive to guarantee ordering for these accesses.

Unfortunately, we still have no agreement on an acceptable semantics for such accesses. Hence verification of algorithms like realistic concurrent garbage collectors appears infeasible. We review the so-called “out-of-thin-air” problem, the technical obstacles, and the ongoing controversy surrounding performance of a possible solution. We focus on some examples to illustrate that existing implementations may in fact produce extremely surprising results for code that includes unordered memory accesses.

Hans-J. Boehm

Google

Deep learning models are becoming larger and will not fit in the limited memory of accelerators such as GPUs for training. Though many methods have been proposed to solve this problem, they are rather ad-hoc in nature and difficult to extend and integrate with other techniques. In this paper, we tackle the problem in a formal way to provide a strong foundation for supporting large models. We propose a method of formally rewriting the computational graph of a model where swap-out and swap-in operations are inserted to temporarily store intermediate results on CPU memory. By introducing a categorized topological ordering for simulating graph execution, the memory consumption of a model can be easily analyzed by using operation distances in the ordering. As a result, the problem of fitting a large model into a memory-limited accelerator is reduced to the problem of reducing operation distances in a categorized topological ordering. We then show how to formally derive swap-out and swap-in operations from an existing graph and present rules to optimize the graph. Finally, we propose a simulation-based auto-tuning to automatically find suitable graph-rewriting parameters for the best performance. We developed a module in TensorFlow, called LMS, by which we successfully trained ResNet-50 with a 4.9x larger mini-batch size and 3D U-Net with a 5.6x larger image resolution.

Tung D. Le

IBM Research - Tokyo

Haruki Imai

IBM Research - Tokyo

Japan

Yasushi Negishi

IBM Research - Tokyo

Kiyokuni Kawachiya

IBM Research - Tokyo

Memory fragmentation is a widely studied problem of dynamic memory allocators. It is well known that fragmentation can lead to premature out-of-memory errors and poor cache performance.

With the recent emergence of dynamic memory allocators for SIMD accelerators, memory fragmentation is becoming an increasingly important problem on such architectures. Nevertheless, it has received little attention so far. Memory-bound applications on SIMD architectures such as GPUs can experience an additional slowdown due to less efficient vector load/store instructions.

We propose CompactGpu, an incremental, fully-parallel, in-place memory defragmentation system for GPUs. CompactGpu is an extension to the DynaSOAr dynamic memory allocator and defragments the heap in a fully parallel fashion by merging partly occupied memory blocks. We developed several implementation techniques for memory defragmentation that are efficient on SIMD/GPU architectures, such as finding defragmentation block candidates and fast pointer rewriting based on bitmaps.

Benchmarks indicate that our implementation is very fast with typically higher performance gains than compaction overheads. It can also decrease the overall memory usage.

Matthias Springer

Tokyo Institute of Technology

Germany

Hidehiko Masuhara

Tokyo Institute of Technology

Japan

Parallel copying garbage collection (GC) is widely used in the de facto Java virtual machines such as OpenJDK and OpenJ9. OpenJDK uses work-stealing for copying objects in the Parallel GC and Garbage-First (G1) GC policies to balance the copying task among GC threads. When a thread has no task in its own queue, it tries to steal a task from another thread's queue as a thief. When a thief succeeds in stealing a task, it processes the task and enqueues the children of the task into its queue, which is accessible from other thieves.Unfortunately, the overhead of the work-stealing framework becomes non-negligible when we aim to achieve a minimum GC pause time by increasing the number of GC threads. Since the number of tasks processed per thread decreases, thieves frequently try to steal tasks from others at a low success rate. When a thief fails in steals continuously, it needs to wait in a spin loop on the termination protocol of the work-stealing framework. Spinning in a loop frequently results in high CPU utilization, which is not acceptable in a large-scale data center where severe power management is required. This paper proposes two approaches named steal-best-of-many selection and spin-less termination to reduce the overhead in the work-stealing framework. Steal-best-of-many selection reduces steal failures by changing the number of queue selections to steal in accordance with the number of GC threads. Spin-less termination moves a part of the object copies into a spin loop by changing the procedure of copying GC. It reduces part of the GC pause time for the object copy as well as the CPU utilization for the spin loop. We developed a prototype on OpenJDK8 and evaluated it using SPECjbb2015 and SPECjvm2008 benchmarks. Critical-jOPS performance of SPECjbb2015 improved by 18% at maximum and scores of the SPECjvm2008 benchmarks improved by 1-5%.

Michihiro Horie

IBM Research - Tokyo

Kazunori Ogata

IBM Research, Japan

Mikio Takeuchi

IBM Research - Tokyo

Hiroshi Horii

IBM Research, Japan

Apache Spark is a popular cluster computing framework for iterative analytics
workloads due to its use of Resilient Distributed Datasets (RDDs) to cache data for in-memory processing. We have revealed that the performance of Spark RDD cache can be severely limited if its capacity falls short to the needs of the workloads. In this paper, we have explored different memory hybridization strategies to leverage emergent Non-Volatile Memory (NVM) devices for Spark's RDD cache. We have found that a simple layered hybridization approach does not offer an effective solution. Therefore, we have designed a flat hybridization scheme to leverage NVM for caching RDD blocks, along with several architectural optimizations such as dynamic memory allocation for block unrolling, asynchronous migration with preemption, and opportunistic eviction to disk. We have performed an extensive set of experiments to evaluate the performance of our proposed flat hybridization strategy and found it to be robust in handling different system and NVM characteristics. Our proposed approach uses DRAM for a fraction of the hybrid memory system and yet manages to keep the increase in execution time to be within 10% on average. Moreover, our opportunistic
eviction of blocks to disk improves performance by up to 7.5% when utilized alongside the current mechanism.

Md Muhib Khan

Florida State University

Muhammad Ahad Ul Alam

Florida State University, USA

Amit Kumar Nath

Florida State University, USA

Weikuan Yu

Florida State University, USA

Generational garbage collectors are one of the most common types of automatic memory management. We can minimize the costs they incur by carefully choosing the points in a program's execution at which they run. However, this decision is generally based on simple, crude heuristics. Instead, we train random forest classifiers to decide when to collect based on features gathered from a running program. This reduces the total cost of collection in both time and space. We demonstrate useful generalization of learned policies to unseen traces of the same program, showing this approach may be fruitful for further investigation.

Nicholas Jacek

UMass Amherst

Eliot Moss

University of Massachusetts Amherst

Memory allocation is increasingly important to parallel performance, yet it is challenging because a program has data of many sizes, and the demand differs from thread to thread. Modern allocators use highly tuned heuristics but do not provide uniformly good performance when the level of concurrency increases from a few threads to hundreds of threads.

This paper presents a new timescale theory to model the memory demand in real time. Using the new theory, an allocator can adjust its synchronization frequency using a single parameter called allocations per fetch (apf ). The paper presents the timescale theory, the design and implementation of APF tuning in an existing allocator, and evaluation of the effect on program speed and memory efficiency. APF tuning improves the throughput of MongoDB by 55%, reduces the tail latency of a Web server by over 60%, and increases the speed of a selection of synthetic benchmarks by up to 24× while using the same amount of memory.

Pengcheng Li

Google, Inc

China

Hao Luo

University of Rochester

Chen Ding

University of Rochester

Bugs have always been a big threat to the reliability and efficiency of software systems. In the era of big data and cloud computing, they continue to take effect, but often with new forms.

This talk will first discuss our recent study of Microsoft Azure incidents, understanding what led to production cloud service incidents. It will then go into details about two types of interesting bugs, (1) memory and persistent data races in the cloud and (2) inefficiencies in Object-Relational Mapping systems. Good understanding of these bugs has led to our tools discovering more than 1000 bugs in latest versions of popular open-source applications.

Shan Lu

University of Chicago

Shan Lu is an Associate Professor in the Department of Computer Science at the University of Chicago. She received her Ph.D. at University of Illinois, Urbana-Champaign, in 2008. She was the Clare Boothe Luce Assistant Professor of Computer Sciences at University of Wisconsin, Madison, from 2009 to 2014. Her research focuses on software reliability and efficiency, particularly detecting, diagnosing, and fixing functional and performance bugs in large software systems.

Shan won Alfred P. Sloan Research Fellow in 2014, Distinguished Alumni Educator Award from Department of Computer Science at University of Illinois in 2013, and NSF Career Award in 2010. Her co-authored papers won Google Scholar Classic Paper 2017, Best Paper Awards at USENIX OSDI 2016 and USENIX FAST 2013, ACM-SIGSOFT Distinguished Paper Awards at ICSE 2019, ICSE 2015, and FSE 2014, an ACM-SIGPLAN Research Highlight Award at PLDI 2011, and an IEEE Micro Top Picks in ASPLOS 2006.

Shan currently serves as the Vice Chair of ACM-SIGOPS (2015–) and the Associate Editor for IEEE Computer Architecture Letters. She also serves as the technical program co-chair for USENIX Symposium on Operating Systems Design and Implementation (OSDI) in 2020, USENIX Annual Technical Conference (ATC) in 2015, and ACM Asia-Pacific Systems Workshop (APSys) in 2018.

Efficient garbage collection is a key goal in engineering high-performance runtime systems.
To reduce pause times, many collector designs traverse the object graph concurrently with the application, an optimization known as concurrent marking.
Traditional concurrent marking imposes strict invariants on the object shapes: 1) static type layout of objects, 2) static object memory locations, 3) static object sizes.
High performance virtual machines for dynamic languages, for example, the V8 JavaScript virtual machine used in the Google Chrome web browser, generally violate these constraints in pursuit of high throughput for a single thread.
Taking V8 as an example, we show that some object shape changes are safe and can be handled by traditional concurrent marking algorithms.
For unsafe shape changes, we introduce novel wait-free object snapshotting and lock-based concurrent marking algorithms and prove that they preserve key invariants.
We implemented both algorithms in V8 and achieved performance improvements on various JavaScript benchmark suites and real-world web workloads.
Concurrent marking of shape-changing objects using the wait-free object snapshotting algorithm is enabled by default in Chrome since version 64.

Ulan Degenbaev

Google

Michael Lippautz

Google

Hannes Payer

Google

One common characteristic among current lock-free memory allocators
is that they rely on the operating system to manage memory since
they lack a lower-level mechanism capable of splitting and
coalescing blocks of memory. In this paper, we discuss this problem
and we propose a generic scheme for an efficient lock-free best-fit
coalescing-capable mechanism that is able of satisfying memory
allocation requests with desirable low fragmentation
characteristics.

Ricardo Leite

University of Porto

Portugal

Ricardo Rocha

University of Porto, Portugal

Portugal

snmalloc is an implementation of malloc aimed at workloads in which objects are typically deallocated by a different thread than the one that had allocated them. We use the term producer/consumer for such workloads. snmalloc uses a novel message passing scheme which
returns deallocated objects to the originating allocator in batches without taking any locks. It also uses a novel bump pointer-free list data structure with which just 64-bits of meta-data are sufficient for each 64 KiB slab. On such producer/consumer benchmarks our approach performs better than existing allocators.

Snmalloc is available at https://github.com/Microsoft/snmalloc.

Paul Lietar

Theodore Butler

Drexel University, USA

Sylvan Clebsch

Imperial College London

Sophia Drossopoulou

Imperial College London

United Kingdom

Juliana Franco

Microsoft Research, Cambridge

United Kingdom

Matthew Parkinson

Microsoft Research, UK

United Kingdom

Alex Shamis

Microsoft Research / Imperial College London

Christoph M. Wintersteiger

Microsoft Research, UK

David Chisnall

University of Cambridge

Write barriers are a fundamental mechanism that most production
garbage collection algorithms depend on. They inform the collector
of mutations to the object graph, enabling partial heap collections,
concurrent collection, and reference counting. While in principle,
write barriers remember only the pointers within the object graph
that were changed and do so just once, widely-used write barriers
are less precise, sacrificing temporal and/or spatial precision to
performance. The idea of precisely remembering just the pointers
that were changed is not new, however implementing performant
field-precise barriers has proved elusive.
We describe a technique for efficiently implementing field-logging
barriers. We implement a number of barriers and evaluate them on a
range of x86 hardware.
A generational field-logging barrier performs with 0.1% to 1%
mutator overhead compared to a highly tuned object-logging barrier,
while a preliminary implementation of a reference counting
field-logging barrier performs with around 1% to 2% overhead compared to
a highly tuned object-logging reference counting barrier.
These results suggest that garbage collection algorithms that
require the precision of exactly remembering field mutations without
sacrificing performance may now be possible, adding a new mechanism
to the design toolkit available to garbage collection researchers.

Steve Blackburn

Australian National University

Australia

My research interests are centered on the challenge of making software run faster and more power-efficiently on modern hardware. My primary interests include: microarchitectural support for managed languages, fast and efficient garbage collection, and the design and implementation of virtual machines. As a backdrop to this I have a longstanding interest in role of sound methodology and infrastructure in successful research innovation.

Ruby is a popular object-oriented programming language, and the performance of the Ruby garbage collector (GC) directly affects the execution time of Ruby programs.
Ruby 2.0 and earlier versions employed an inefficient non-generational conservative mark-and-sweep GC.
To improve this and make it a generational collector, it is necessary to introduce write barriers (WBs), but this requires huge modification to existing source code, including third-party C-extensions.
To avoid the need for adding WBs around legacy code, we invented a new concept called ``WB-unprotected objects'', which indicates to the GC to treat such objects more conservatively.
By leveraging this design, we were able to improve the performance of Ruby 2.1 with a generational GC and of Ruby 2.2 with an incremental GC while preserving compatibility with existing C-extensions.
Another significant advantage of this approach is that WBs can be added gradually, which reduces the difficulties associated with updating existing code.

Link to Publication: https://dl.acm.org/citation.cfm?id=3329986

Koichi Sasada

Cookpad, Japan

Japan

Koichi Sasada is a programmer, mainly developing Ruby interpreter (CRuby/MRI). He received Ph.D (Information Science and Technology) from the University of Tokyo, 2007. He became a faculty of University of Tokyo (Assistant associate 2006-2008, Assistant professor 2008-2012). After the 13 years life in university, he had joined a member of Matz’s team in Heroku, Inc. Now he is a member of Cookpad Inc (engineer). He is also a director of Ruby Association.