MIcro KErneL virtualizAtioN for hiGh pErfOrmance cLOud and hpc systems

Project Information

MIKELANGELO

Grant agreement ID: 645402

DOI

10.3030/645402

Project closed

EC signature date 11 December 2014

Start date 1 January 2015

End date 31 December 2017

Funded under

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Total cost

€ 5 993 419,00

EU contribution

€ 5 993 419,00

Coordinated by

XLAB RAZVOJ PROGRAMSKE OPREME IN SVETOVANJE DOO
Slovenia

Periodic Reporting for period 3 - MIKELANGELO (MIcro KErneL virtualizAtioN for hiGh pErfOrmance cLOud and hpc systems)

Reporting period: 2017-01-01 to 2017-12-31

Summary of the context and overall objectives of the project

The current Cloud and HPC architectures are a trade-off between efficiency, stability and security. Legacy and compatibility requirements have through the years amassed a significant amount of layers, on top of which the application code runs. This complexity requires complex setup and management tools that have to follow the development of many pieces of code. Having so diverse and large infrastructure results in large security attack surface, reduced only with sophisticated networking security measures. Therein lie the reasons for inefficiency of the current Cloud and HPC architectures, which we try to reduce.
The goal of MIKELANGELO is to provide improved responsiveness, flexibility and security of virtual infrastructure through novel components, supporting different setups or variants of the HPC, Cloud and HPC-Cloud. MIKELANGELO relies on optimisation of guest, hypervisor, their joint collaboration and on the set of approaches, devised to simplify and increase performance in virtual IO.
Unikernel OSv is used to reduce the size of guest virtual machine to a bare minimum, to reduce its complexity while offering the best support for legacy applications among the unikernels. The KVM hypervisor was chosen due to its popularity among our target audiences and has been improved with IO-optimising approaches. A set of additional components was developed, each supporting a different infrastructure and/or deployment scenario.
To ensure longevity of the project, the developed components are open-source and most already upstreamed. The software, interfaces and methodology are developed in standards-compliant way. MIKELANGELO software can thus be used and integrated in larger projects (e.g. integrated in OpenStack, Kubernetes without any overhead).
Targets are applications in private Clouds and HPC centres, usually including simulations and modeling, but shifting towards (High Performance) Data Analytics. They are addressed through use-cases, included in the project.
MIKELANGELO consists of XLAB, University of Stuttgart, IBM, INTEL, GWDG, PIPISTREL, SCYLLA, Ben-Gurion University and Huawei.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

MIKELANGELO used all possible sources (e.g. use-cases, market research, new components, etc.) to position itself in the overlapping area of guest (OSv) and host (KVM) and the overarching management system (OpenStack, Kubernetes). We can use MIKELANGELO components quite independently and in many different environments (eg. AWS, GCE or under PBS Torque). This demonstrates the versatility of the MIKELANGELO approach. We divide the work done into four categories, described below.
In the first category are contributions to the OSv. OSv improvements have all been accepted upstream. We improved Seastar - it now has additional components, related to the IO improvement. Finally, the OSv management software has been significantly improved and is now under one brand Lightweight Execution Environment Toolbox - LEET. This includes complete OSv management utilities, which have been accepted into DellEMC’s UniK project (https://github.com/emc-advanced-dev/unik) and into MirantisIT’s Virtlet project (https://github.com/Mirantis/virtlet).
The second are improved OSv application compatibility and management. We improved the application compatibility in OSv and added support for languages Go and Node.js improved OSv's OpenFOAM and Open MPI compatibility and ported Apache Hadoop Distributed File System and Apache Spark. Under LEET, we enabled application composition and overall management of OSv, making OSv integration into different environments easier.
Thirdly, we extended the KVM IO management. We introduced IO core manager (IOcm) facilitating dynamic optimisation of CPU core allocation for IO intensive workloads. ZeCoRX (Zero Copy RX) has been developed to PoC, removing redundant data copying within the network stack. Both approaches are novel paravirtualized IO models for KVM. We developed SCAM, preventing side-channel attacks on KVM hypervisor. We designed the vRDMA approach, with guest and host implementations in the form of Virtio additions for KVM/OSv, as a patch. Finally, we designed and implemented UNCLOT - another approach for fast, shared memory-based data exchange between guests on the same host.
The fourth is compatibility with the Cloud and HPC management systems. We developed vTorque, which enables virtual machine (OSv or others) deployment in HPC batch system Torque. As telemetry and performance monitoring are of utmost importance, we developed management and monitoring approaches, optimising virtual environments for optimal elasticity. Here, we used Snap and developed appropriate plugins for MIKELANGELO. Finally, we have developed Scotty, a synthetic testbed for benchmarking of KVM and OSv in Openstack and actuator.py which provides an extensible framework for configuring software components.
MIKELANGELO ensured exploitation of the project’s results. The source code has been, in most cases, included in upstream projects. Our contributions have been accepted to relevant community projects. We presented MIKELANGELO in different settings on ISC-HPC 2016, 2017, SC 2016 and 2017, KubeCon and OpenStack Barcelona Summit.
At the end of this RIA project, we can sum up exploitation with the following data: number of follow-up/attributed projects is 7; commercial contracts stemming from the project is at 7 and the number of components already used in production environments is at 3.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

OSv has been significantly improved from the perspective of its performance, application compatibility (added Node.js and Go support) and application composition. The improvements cover additional system calls and C-lib functions, new build environments, a better NFS client, isolated threads, partial NUMA support, cloud-init support, improved DHCP, file system performance, Open MPI support etc.
Seastar has been improved from the management and performance perspectives. It is a state-of-the-art high-performance library for new software projects. Seastar improvements cover improved RPC, IO scheduling, IOtune, CPU scheduling, monitoring, ext4 support, a log-structured memory allocator, and DPDK support.
KVM was improved with two novel paravirtualized IO models - IOcm (fully developed) and ZeCoRX (Proof of Concept), vRDMA and UNCLOT. IOcm is a dynamic IO core manager. It samples IO statistics, estimates the IO pressure, and allocates the right amount of IO cores. vRDMA offers the use of a verbs-based API in the front-end driver, improving the overall performance. UNCLOT offers shared memory based communication between OSv guests on the same host.
Security in hypervisor has been improved with SCAM module, preventing side-channel attacks - SCAM offers monitoring, profiling, and initial mitigation of attacks.
Extensions for Torque enable deployment of virtual machines in HPC batch system (vTorque). Current release offers transparent provisioning of Linux and OSv instances for virtual workloads on HPC.
Major work has been done in the field of infrastructure management for scientific experiments. Three new tools have been developed - Scotty, the continuous experimentation framework, Actuator.py that provides a network for software components configuration and MIKELANGELO Cloud Manager, which provides a layer of abstraction above infrastructure management and telemetry systems.
Finally, newly developed collectors and approaches have been made for the telemetry and instrumentation system Snap.

The MIKELANGELO Component Diagram

The MIKELANGELO Project Objectives Overview

The MIKELANGELO Final Architecture