Periodic Reporting for period 2 - MICROCARD (Numerical modeling of cardiac electrophysiology at the cellular scale)
Periodo di rendicontazione: 2022-10-01 al 2024-09-30
Cardiovascular diseases are the most frequent cause of death worldwide and half of these deaths are due to cardiac arrhythmia, disorders of the heart's electrical synchronization system. This system, based on electrical connections between the cells and myriads of molecular-scale current generators in each cell membrane, is so complex that it can only be understood with the help of computer models.
These models are already very sophisticated and widely used, but currently they are not powerful enough to take the heart's (2 billion!) individual cells into account. They must therefore assume that hundreds of cells are doing approximately the same thing. Due to this limitation, current models cannot reproduce the events in aging and structurally diseased hearts, in which reduced electrical coupling leads to large differences in behaviour between neigbouring cells, with possibly fatal consequences. But if we want to model the heart cell by cell, we face a mathematical problem that is a million times larger, and also harder to solve. We will need larger supercomputers than those that exist today, and a lot of inventiveness to solve our problem efficiently on these future machines.
The main purpose of the MICROCARD project was to develop software that can solve this problem on exascale supercomputers. To this end the project has developed algorithms that are tailored to the specific mathematical problem, to the size of the computations, and to the particular design of these future computers, which will owe most of their compute power to ultra-parallel computing elements such as Graphics Processing Units (GPUs). Some of these algorithms are implemented in the open-source software library Ginkgo (https://github.com/ginkgo-project/ginkgo/(si apre in una nuova finestra)) and thus available to a much larger community. The openCARP electrophysiology simulation package was extended by numerous features to facilitate use on HPC systems. This includes asynchronous data output, automatic code generation, fault tolerance, installation through Spack and tailored documentation.
In addition, MICROCARD has developed geometrical models of cardiac tissue that it needs to perform simulations, as well as the software needed to create these enormously large and complex models. This was done by analysing microscopic imaging data, and also by making completely artificial models, which could be made much larger than the small volumes of biological data that could be imaged. In order to make this possible we have greatly improved the open-source software Mmg (http://www.mmgtools.org/(si apre in una nuova finestra)) making it more robust and able to treat larger geometries.
These models are already very sophisticated and widely used, but currently they are not powerful enough to take the heart's (2 billion!) individual cells into account. They must therefore assume that hundreds of cells are doing approximately the same thing. Due to this limitation, current models cannot reproduce the events in aging and structurally diseased hearts, in which reduced electrical coupling leads to large differences in behaviour between neigbouring cells, with possibly fatal consequences. But if we want to model the heart cell by cell, we face a mathematical problem that is a million times larger, and also harder to solve. We will need larger supercomputers than those that exist today, and a lot of inventiveness to solve our problem efficiently on these future machines.
The main purpose of the MICROCARD project was to develop software that can solve this problem on exascale supercomputers. To this end the project has developed algorithms that are tailored to the specific mathematical problem, to the size of the computations, and to the particular design of these future computers, which will owe most of their compute power to ultra-parallel computing elements such as Graphics Processing Units (GPUs). Some of these algorithms are implemented in the open-source software library Ginkgo (https://github.com/ginkgo-project/ginkgo/(si apre in una nuova finestra)) and thus available to a much larger community. The openCARP electrophysiology simulation package was extended by numerous features to facilitate use on HPC systems. This includes asynchronous data output, automatic code generation, fault tolerance, installation through Spack and tailored documentation.
In addition, MICROCARD has developed geometrical models of cardiac tissue that it needs to perform simulations, as well as the software needed to create these enormously large and complex models. This was done by analysing microscopic imaging data, and also by making completely artificial models, which could be made much larger than the small volumes of biological data that could be imaged. In order to make this possible we have greatly improved the open-source software Mmg (http://www.mmgtools.org/(si apre in una nuova finestra)) making it more robust and able to treat larger geometries.
In the first half of the project we concentrated on the construction of building blocks, a prototype implementation, and the design of an exascale-ready code. Functionality that we need was added to the existing software libraries on which the project builds, and is already available for others to use through recent releases of the ParMmg code (https://mmgtools.org(si apre in una nuova finestra)) the Ginkgo linear algebra library (https://github.com/ginkgo-project/ginkgo(si apre in una nuova finestra)) and the openCARP cardiac simulation software (https://opencarp.org(si apre in una nuova finestra)). In addition we worked on the interpretation of imaging data to construct tissue geometries, and we developed a code that can produce artificial geometries that are much larger than would be possible with microscope images.
In the second half of the project we have integrated the components. A cell-by-cell simulation module was developed for the openCARP software, using the newly added solvers and preconditioners via the Ginkgo back-end which we added earlier. Meanwhile components were being improved. We developed and implemented a problem-tailored BDDC preconditioner, and a mesh partitioning method that fit this particular scheme. The Mmg software was robustified such that it no longer had problems with the extremely challenging meshes that we produce. Thanks to these improvements, the synthetic tissue generation code can now reliably produce volumes of tissue as large as the computer memory permits.
The project has made several contributions to the openCARP code, usable also outside the cell-by-cell version: GPU support, accelerated membrane models, asynchronous output, automatic restart after system failure, automated experiment packaging, and Spack support.
The project has supported the publication of 20 peer-reviewed journal papers so far, and over 80 presentations at scientific conferences and seminars. We have also participated in events aimed at a larger audience, and we organized a summer school to teach students about cell-by-cell cardiac simulations on HPC machines.
In the second half of the project we have integrated the components. A cell-by-cell simulation module was developed for the openCARP software, using the newly added solvers and preconditioners via the Ginkgo back-end which we added earlier. Meanwhile components were being improved. We developed and implemented a problem-tailored BDDC preconditioner, and a mesh partitioning method that fit this particular scheme. The Mmg software was robustified such that it no longer had problems with the extremely challenging meshes that we produce. Thanks to these improvements, the synthetic tissue generation code can now reliably produce volumes of tissue as large as the computer memory permits.
The project has made several contributions to the openCARP code, usable also outside the cell-by-cell version: GPU support, accelerated membrane models, asynchronous output, automatic restart after system failure, automated experiment packaging, and Spack support.
The project has supported the publication of 20 peer-reviewed journal papers so far, and over 80 presentations at scientific conferences and seminars. We have also participated in events aimed at a larger audience, and we organized a summer school to teach students about cell-by-cell cardiac simulations on HPC machines.
The MICROCARD project has brought its main achievement, a code for macroscopic cell-by-cell simulation of cardiac electrophysiology, to near completion. It has thus helped to move the state of the art in cardiac simulation from homogenized models to models in which individual cells are represented.
The project has further produced building blocks for this code which in themselves represent advances in the state of the art in different domains. We designed and implemented a novel approach to run simulations with solution-adapted resolution, which reduces the computational workload by a factor up to eight. This is usually not achieved with traditional mesh refinement and coarsening. We enhanced the widely used openCARP simulation code with strongly accelerated membrane models for both CPU and GPU platforms, thanks to a dedicated model compiler based on the MLIR/LLVM framework. We further provided openCARP with asynchronous output, GPU support for linear algebra, and support for deployment and for the sharing of experiment designs. We implemented several new algorithms in the Ginkgo linear algebra library, bringing GPU support to our code but also to other software. We robustified the mmg3d software, so that it can reliably process our meshes with numerous and tighthly apposed internal boundaries. This progress benefits other applications as well, from biomedical to aeronautics. We also implemented new functionality in the ParMmg code, the parallel cousin of mmg3d. Finally, we developed a completely new code to produce artificial cardiac tissue meshes.
By the end of the project there was still no exascale supercomputer available for research in Europe, and our code was also not entirely ready to run on it. However, a working code was completed a month after the project, and will be further tested and developed by the Centre of Excellence (CoE) MICROCARD-2, which started in November 2024. The CoE will also continue to improve solver and preconditioner algorithms, and code to create artificial tissue meshes, to allow large-scale testing.
In summary, the MICROCARD project has helped one important application, simulation of the heart, to benefit from exascale machines. We expect that this will also help related fields, such as brain research, to make better use of future exascale supercomputers. Some of our work, e.g. in Ginkgo and in Mmg, will even benefit areas of research very far from ours. The tools that MICROCARD has developed will soon be usable for cardiology research and will allow a vast array of urgent questions to be answered, leading to a better understanding and diagnosis of cardiac arrhythmia.
The project has further produced building blocks for this code which in themselves represent advances in the state of the art in different domains. We designed and implemented a novel approach to run simulations with solution-adapted resolution, which reduces the computational workload by a factor up to eight. This is usually not achieved with traditional mesh refinement and coarsening. We enhanced the widely used openCARP simulation code with strongly accelerated membrane models for both CPU and GPU platforms, thanks to a dedicated model compiler based on the MLIR/LLVM framework. We further provided openCARP with asynchronous output, GPU support for linear algebra, and support for deployment and for the sharing of experiment designs. We implemented several new algorithms in the Ginkgo linear algebra library, bringing GPU support to our code but also to other software. We robustified the mmg3d software, so that it can reliably process our meshes with numerous and tighthly apposed internal boundaries. This progress benefits other applications as well, from biomedical to aeronautics. We also implemented new functionality in the ParMmg code, the parallel cousin of mmg3d. Finally, we developed a completely new code to produce artificial cardiac tissue meshes.
By the end of the project there was still no exascale supercomputer available for research in Europe, and our code was also not entirely ready to run on it. However, a working code was completed a month after the project, and will be further tested and developed by the Centre of Excellence (CoE) MICROCARD-2, which started in November 2024. The CoE will also continue to improve solver and preconditioner algorithms, and code to create artificial tissue meshes, to allow large-scale testing.
In summary, the MICROCARD project has helped one important application, simulation of the heart, to benefit from exascale machines. We expect that this will also help related fields, such as brain research, to make better use of future exascale supercomputers. Some of our work, e.g. in Ginkgo and in Mmg, will even benefit areas of research very far from ours. The tools that MICROCARD has developed will soon be usable for cardiology research and will allow a vast array of urgent questions to be answered, leading to a better understanding and diagnosis of cardiac arrhythmia.