Abstract

Optimizations are an omnipresent topic when working on Virtual Machines (VMs). There is a plethora of different optimizations available ranging from simple tweaks and tricks to full evaluation concepts requiring a complex infrastructure. Depending on the complexity of an optimization and the performance increase it is important to choose the right kinds of optimizations. Based on a high-level language VM as a case study we argue in favor of transparent optimizations which do not require changes in the interpreted language's semantics. Furthermore it is necessary to write and properly evaluate benchmarks to be able to track the performance impact of a certain optimization. When building a high-level language VM the underlying system -- traditionally a C or C++ core -- does not share many concepts with the implemented language. Hence some optimizations emerging from the low-level VM core are orthogonal to the high-level concepts of the interpreted language. Focusing on such optimizations can strongly limit the dynamic capabilities of a high-level language. These non-transparent optimizations require the semantics of the interpreted language to be changed. Changes in the language's semantics can require extensive changes in the sources which is an undesired property. However transparent optimizations preserve semantics of the language. Using transparent optimizations helps to separate the low-level requirements of the VM form the high-level design decisions of the language. We argue that non-transparent optimizations should only be applied to a high-level language VM in an early development stage. Furthermore each non-transparent optimization should be paired with a compatible way to reintroduce the lost or altered semantics. To make valid statements about optimizations it is necessary to write adequate benchmarks. The benchmarks have to be reproducible and the evaluation has to be statistically sound, furthermore the benchmarks should focus on covering specific use cases to help locating performance issues in the VM code. Benchmarks are to be used in a similar way as unit tests. Optimizations can only be evaluated in a sound way when the corresponding benchmarks produce deterministic values. Furthermore we state that focusing on micro benchmarks helps to locate and track performance critical code segments of the VM.

Links and resources

Tags