Enabling dynamic and Intelligent workflows in the future EuroHPCecosystem

Informations projet

eFlows4HPC

N° de convention de subvention: 955558

DOI

10.3030/955558

Projet clôturé

Date de signature de la CE 17 Decembre 2020

Date de début 1 Janvier 2021

Date de fin 29 Février 2024

Financé au titre de

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Coût total

€ 7 663 216,25

Contribution de l’UE

€ 3 576 656,13

Coordonné par

BARCELONA SUPERCOMPUTING CENTER CENTRO NACIONAL DE SUPERCOMPUTACION
Spain

CORDIS fournit des liens vers les livrables publics et les publications des projets HORIZON.

Les liens vers les livrables et les publications des projets du 7e PC, ainsi que les liens vers certains types de résultats spécifiques tels que les jeux de données et les logiciels, sont récupérés dynamiquement sur OpenAIRE .

Livrables

Documents, reports (26)

Application bottlenecks and optimization opportunities on heterogeneous components

This report will include the identified bottlenecks as well as a preliminary analysis of such issues when considering the communication architecture, the storage architecture, and the use of heterogeneous components. This report will be the baseline for the work of the remaining tasks in this WP. This deliverable will collect theoutcome from task T3.1.

Design of the Pillar I use cases

Report on the choice of industrially relevant problems to be addressed in the WP. Report will identify success criteria as well as potential breakthroughs

Revision of Requirements and architecture design

Second version of the Pillars’ requirements and architecture design (output from tasks 1.1, 1.2, 4.1, 5.1, 6.1)

Requirements on the eFlows4HPC software stack from Pillar II and evaluation metrics.

This report will include a prioritized list of requirements that will guide the design and implementation of the different Pillar II use cases as well as of the core eFlows4HPC components. The deliverable will also include a set of metrics to be used in the evaluation of the developed workflows

Report on Implementing Containerization and Optimization Strategy

This deliverable will describe the implementation of the Optimization Strategy (D2.2) and possibly reevaluate the approach taken. Tasks 2.1, 2.2, 2.3 contribute to this deliverable.

Optimized kernels for heterogeneous components

This document will reflect the final version of the optimized kernels developed in the project for the use of heterogeneous components. It will compound application kernels and neural network kernels. This deliverable will collect the work from tasks T3.2and T3.4, including associated evaluation work from T3.6.

Second Dissemination and Communication Report

This deliverable will report on the dissemination and communication activities of the project done during the second year.

Validation of the Pillar I use cases

Report on the validation of the ROM models. Evaluation of success criteria.

Requirements on the eFlows4HPC software stack from Pillar I and evaluation metrics

Summary of conclusions on the pillar I requirements.

Optimized data management with new storage technologies

This document will reflect the optimizations performed with new storage technologies. This deliverable will collect the work from task T3.5 and associated evaluation work from T3.6.

Optimized kernels for EPI

In this report the optimized kernels for the EPI will be described. Their implementation on the emulated hardware platform will be provided and its projected performance and energy saving. This deliverable will collect the work from task T3.3, including associatedevaluation work from T3.6.

Initial draft of optimized kernels for EPI

In this report the draft version of the optimized kernels for the EPI will be described. Their implementation on the emulated hardware platform will be provided and its projected performance and energy saving. This deliverable will collect the work from task T3.3, including associated evaluation work from T3.6.

Final Report on Data Logistics Implementation

This report will comprise the description of the final version of Data Logistics Service, implementation of data pipelines motivated by Pillars. Task 2.3 and 2.4 contribute to this deliverable.

Training Plan

This deliverable will define the objectives of the project's training activities, the initial plans for organization of the training activities, as well as the materials that will be provided.

Technology Evaluation, Containerization and Optimization Strategy

Based on the available requirements, this deliverable will derive a strategy for optimizing deployment in all envisioned dimensions: by optimizing libraries and runtimes, using containers, and application of emerging storage solutions. Tasks 2.1, 2.2, 2.3 contribute to this deliverable.

Validation of requirements

This report will include the process and outcome of the validation of the requirements (internal and external evaluations).

Initial draft of optimized kernels for heterogeneous components

This document will reflect a first draft version of the optimized kernels developed in the project for the use of heterogeneous components. It will compound application kernels and neural network kernels. This deliverable will collect the work from tasks T3.2 and T3.4, including associated evaluation work from T3.6.

Requirements on the eFlows4HPC software stack from Pillar III and evaluation metrics

Summary of conclusions on the pillar III requirements.

Report of the organization of community workshops

This deliverable will report on the community workshops organized during the lifetime of the project.

Protocol for urgent HPC

Protocol definition and governance recommendations regarding urgent HPC.

Dissemination and Communication Plan

This deliverable will set out the dissemination and communication strategy and the activities to be undertaken to achieve this. Results of the dissemination work will be reported in the periodic and final reports.

Description of the use cases for Pillar III

Compendium of datasets and models employed for development and validation of Pillar III workflows

First Dissemination and Communication Report

This deliverable will report on the dissemination and communication activities of the project done during the first year.

Design of the Pillar II use cases

This report provides a complete design and comprehensive documentation of the software architecture of the Pillar II use cases

Final Dissemination and Communication Report

This deliverable will report on the final dissemination and communication activities of the complete project.

Requirements, metrics and architecture design

First version of the pillar’srequirements, evaluation metrics and architecture design (output from tasks 1.1, 1.2, 4.1, 5.1, 6.1)

Other (13)

eFlows4HPC interfaces and Iteration 1 software stack release

First version of the design and implementation of the eFlows4HPC software stack interfaces. Tasks 1.3, 1.4, 1.5 contribute to this deliverable.

Database of Earth models

Release of Earth models obtained for the usecase regions, together with associated metadata.

Pillar II - Iteration 2 Software Release

This deliverable relates to the software and documentation released at the end of Iteration 2 for the implementation of the Pillar II use cases.

First version of Data Logistics

This deliverable will be the first production version of Data Logistics service integration with selected storage technology and demonstration of a data pipeline motivated by the Pillars’ use cases. Task 2.3 and 2.4 contribute to this deliverable.

ROM Tools Release

Code Release of essential tools for ROM preparation

Data Catalogue

Based on the requirements report D1.1 this deliverable will analyse and describe the data sources used by the Pillars. This information will be made available in the form of an electronic document or service.

eFlows4HPC interfaces and final software stack release

Second version of the design and implementation of the eFlows4HPC software stack interfaces. Tasks 1.3, 1.4, 1.5, 1.6 and 1.7 contribute to this deliverable.

Iteration 1 workflows for urgent computing of natural hazards

Taskbased version of UCIS4EQ and PTF workflows.

Demo ROM

Release of demonstrator ROM model.

eFlows4HPC interfaces and Iteration 2 software stack release

Second version of the design and implementation of the eFlows4HPC software stack interfaces. Tasks 1.3, 1.4, 1.5, and 1.6 contribute to this deliverable.

Pillar II - Iteration 1 Software Release

This deliverable relates to the software and documentation released at the end of Phase 1 for the implementation of the Pillar II use cases

Iteration 2 workflows for urgent computing of natural hazards

Final releases of the UCIS4EQ and PTF workflows.

Release of HPCWaaS integrated solver stack

Release of software stack integrated in the HPCWaaS interface

Open Research Data Pilot (1)

Data Management Plan

Document outlining how data will be managed during the project from internal and external point of view. The DMP will include a table specifying how the datawill be exploited, shared for verification and reuse. Updates to this report will be provided in M12, M24 and M36.

Publications

Peer reviewed articles (31)

Use of Neural Networks for Tsunami Maximum Height and Arrival Time Predictions

Auteurs: Rodríguez, J.F.; Macías, J.; Castro, M.J.; de la Asunción, M.; Sánchez-Linares, C. Use of Neural Networks for Tsunami Maximum Height and Arrival Time Predictions. GeoHazards 2022, 3, 323-344. 
Publié dans: GeoHazards, Numéro 3(2), 2022, ISSN 2624-795X
Éditeur: MDPI
DOI: 10.3390/geohazards3020017

A Shape Optimization Pipeline for Marine Propellers by means of Reduced Order Modeling Techniques

Auteurs: Ivagnes, Anna; Demo, Nicola; Rozza, Gianluigi
Publié dans: The International Journal for Numerical Methods in Engineering, 2024, ISSN 0029-5981
Éditeur: John Wiley & Sons Inc.
DOI: 10.48550/arxiv.2305.07515

Fast truncated SVD of sparse and dense matrices on graphics processors

Auteurs: Andrés E. Tomás; Enrique S. Quintana-Orti; Hartwig Anzt
Publié dans: The International Journal of High Performance Computing Applications, 2023, ISSN 1094-3420
Éditeur: SAGE Publications
DOI: 10.1177/10943420231179699

Reformulating the direct convolution for high-performance deep learning inference on ARM processors

Auteurs: Sergio Barrachina, Adrián Castelló, Manuel F. Dolz, Tze Meng Low, Héctor Martínez, Enrique S. Quintana-Ortí, Upasana Sridhar, Andrés E. Tomás,
Publié dans: Journal of Systems Architecture, 2023, ISSN 1383-7621
Éditeur: Elsevier BV
DOI: 10.1016/j.sysarc.2022.102806

Geometrically Parametrised Reduced Order Models for Studying the Hysteresis of the Coanda Effect in Finite-elements-based Incompressible Fluid Dynamics

Auteurs: Bravo, J. & Stabile, Giovanni & Hess, M. & Hernández, Joaquin & Rossi, R. & Rozza, Gianluigi.
Publié dans: Journal of Computational Physics, 2023, ISSN 0021-9991
Éditeur: Academic Press
DOI: 10.48550/arxiv.2307.05227

Enhancing iteration performance on distributed task-based workflows

Auteurs: Alex Barcelo; Anna Queralt; Toni Cortes
Publié dans: Distributed, Parallel, and Cluster Computing, Numéro Volume 149, 2023, Page(s) 359-375, ISSN 0167-739X
Éditeur: Elsevier BV
DOI: 10.1016/j.future.2023.07.032

Empirical Interscale Finite Element Method (EIFEM) for modeling heterogeneous structures via localized hyperreduction

Auteurs: J.A. Hernández, A. Giuliodori, E. Soudah,
Publié dans: Computer Methods in Applied Mechanics and Engineering, Numéro Volume 418, Part A,, 2024, ISSN 0045-7825
Éditeur: Elsevier BV
DOI: 10.1016/j.cma.2023.116492

Block size estimation for data partitioning in HPC applications using machine learning techniques

Auteurs: Riccardo Cantini, Fabrizio Marozzo, Alessio Orsino, Domenico Talia, Paolo Trunfio, Rosa M. Badia, Jorge Ejarque & Fernando Vázquez-Novoa
Publié dans: JournalofBigData, 2024, ISSN 2196-1115
Éditeur: JournalofBigData
DOI: 10.1186/s40537-023-00862-w

Dynamic resource allocation for efficient parallel CFD simulations

Auteurs: G. Houzeaux; R.M. Badia; R. Borrell; D. Dosimont; J. Ejarque; M. Garcia-Gasulla; V. López
Publié dans: Distributed, Parallel, and Cluster Computing (cs.DC), 2021, ISSN 0045-7930
Éditeur: Pergamon Press Ltd.
DOI: 10.1016/j.compfluid.2022.105577

Urgent Computing for Protecting People From Natural Disasters

Auteurs: Domenico Talia, Paolo Trunfio
Publié dans: Computer, Numéro Volume: 56, Numéro: 4,, 2023, ISSN 0018-9162
Éditeur: Institute of Electrical and Electronics Engineers
DOI: 10.1109/mc.2023.3241733

Pyophidia: a python library for high performance data analytics at scale.

Auteurs: Donatello Elia, Cosimo Palazzo, Sandro Fiore, Alessandro D’Anca, Andrea Mariello, Giovanni Aloisio
Publié dans: SoftwareX, Numéro Volume 24, 2023, ISSN 2352-7110
Éditeur: SoftwareX
DOI: 10.1016/j.softx.2023.101538

A comparison of data-driven reduced order models for the simulation of mesoscale atmospheric flow

Auteurs: Arash Hajisharifi; Michele Girfoglio; Annalisa Quaini; Gianluigi Rozza
Publié dans: Finite Elements in Analysis and Design, Numéro 228, 2024, ISSN 0168-874X
Éditeur: Elsevier BV
DOI: 10.48550/arxiv.2307.08790

Programming parallel dense matrix factorizations and inversion for new-generation NUMA architectures

Auteurs: Sandra Catalán, Francisco D. Igual, José R. Herrero, Rafael Rodríguez-Sánchez, Enrique S. Quintana-Ortí,
Publié dans: Journal of Parallel and Distributed Computing, 2023, Page(s) 51-65, ISSN 0743-7315
Éditeur: Academic Press
DOI: 10.1016/j.jpdc.2023.01.004

An Ensemble Machine Learning Approach for Tropical Cyclone Localization and Tracking From ERA5 Reanalysis Data

Auteurs: Accarino, Gabriele; Donno, Davide; Immorlano, Francesco; Elia, Donatello; Aloisio, Giovanni
Publié dans: Earth and Space Science, Numéro 23, 2023, Page(s) Volume 10, Numéro 11, ISSN 2333-5084
Éditeur: American Geophysical Union (AGU)
DOI: 10.1029/2023ea003106

Multiscale modeling of prismatic heterogeneous structures based on a localized hyperreduced-order method

Auteurs: A. Giuliodori, J.A. Hernández, E. Soudah,
Publié dans: Computer Methods in Applied Mechanics and Engineering, Numéro Volume 407, 2023, ISSN 0045-7825
Éditeur: Elsevier BV
DOI: 10.1016/j.cma.2023.115913

Toward Matrix Multiplication for Deep Learning Inference on the Xilinx Versal

Auteurs: Lei, Jie; Flich, José; Quintana-Ort, Enrique S.
Publié dans: Euromicro Conference on Parallel, Distributed and Network-Based Processing 2023, 2023, ISSN 2377-5750
Éditeur: IEEE
DOI: 10.1109/pdp59025.2023.00043

A continuous convolutional trainable filter for modelling unstructured data

Auteurs: Coscia, D; Meneghetti, L; Demo, N; Stabile, G; Rozza, G
Publié dans: Computational Mechanics, Numéro 72, 2023, Page(s) 253–265, ISSN 0178-7675
Éditeur: Springer Verlag
DOI: 10.1007/s00466-023-02291-1

A BLIS-like matrix multiplication for machine learning in the RISC-V ISA-based GAP8 processor

Auteurs: C. Ramirez, Adrián Castelló, Enrique S Quintana-Orti
Publié dans: The Journal of Supercomputing, 2022, ISSN 0920-8542
Éditeur: Kluwer Academic Publishers
DOI: 10.1007/s11227-022-04581-6

CECM: A continuous empirical cubature method with application to the dimensional hyperreduction of parameterized finite element models

Auteurs: J.A. Hernández, J.R. Bravo, S. Ares de Parga
Publié dans: Computer Methods in Applied Mechanics and Engineering, Numéro Volume 418, Part B,, 2024, ISSN 0045-7825
Éditeur: Elsevier BV
DOI: 10.1016/j.cma.2023.116552

An enriched finite element/level-set model for two-phase electrohydrodynamic simulations

Auteurs: Christian Narváez-Muñoz; Mohammad R. Hashemi; Pavel B. Ryzhakov; Jordi Pons-Prats
Publié dans: Physics of Fluids, Numéro 6, 2023, Page(s) 35, ISSN 1089-7666
Éditeur: AIP Publishing
DOI: 10.1063/5.0127274

An Ensemble Machine Learning Approach for Tropical Cyclone Detection Using ERA5 Reanalysis Data

Auteurs: Accarino, Gabriele; Donno, Davide; Immorlano, Francesco; Elia, Donatello; Aloisio, Giovanni
Publié dans: Earth and Space Science, Numéro 3, 2023, ISSN 2333-5084
Éditeur: AGU Journals
DOI: 10.48550/arxiv.2306.07291

A kinematically stabilized linear tetrahedral finite element for compressible and nearly incompressible finite elasticity

Auteurs: Guglielmo Scovazzi; Rubén Zorrilla; Riccardo Rossi
Publié dans: Computer Methods in Applied Mechanics and Engineering, Numéro Volume 412, 2023, ISSN 0045-7825
Éditeur: Elsevier BV
DOI: 10.1016/j.cma.2023.116076

Enabling dynamic and intelligent workflows for HPC, data analytics, and AI convergence

Auteurs: Jorge Ejarque; Rosa M. Badia; Loïc Albertin; Giovanni Aloisio; Enrico Baglione; Yolanda Becerra; Stefan Boschert; Julian R. Berlin; Alessandro D’Anca; Donatello Elia; François Exertier; Sandro Fiore; José Flich; Arnau Folch; Steven J. Gibbons; Nikolay Koldunov; Francesc Lordan; Stefano Lorito; Finn Løvholt; Jorge Macías; Fabrizio Marozzo; Alberto Michelini; Marisol Monterrubio-Velasco; Mart
Publié dans: EPIC3Future Generation Computer Systems, Numéro 134, 2022, Page(s) 414-429, ISSN 0167-739X
Éditeur: Elsevier BV
DOI: 10.1016/j.future.2022.04.014

Programming Big Data Analysis: Principles and Solutions

Auteurs: Loris Belcastro, Riccardo Cantini, Fabrizio Marozzo, Alessio Orsino, Domenico Talia & Paolo Trunfio
Publié dans: Journal of Big Data, Numéro 9:4, 2022, ISSN 2196-1115
Éditeur: SpringerOpen
DOI: 10.1186/s40537-021-00555-2

PyCOMPSs as an instrument for Translational Computer Science

Auteurs: Rosa M. Badia; Javier Conejero; Jorge Ejarque; Daniele Lezzi; Francesc Lordan
Publié dans: Computing in Science & Engineering, Numéro 24(2), 2022, ISSN 1558-366X
Éditeur: IEEE
DOI: 10.22541/au.164557536.67201934/v1

Automatizing the creation of specialized high-performance computing containers

Auteurs: Jorge Ejarque; Rosa M Badia
Publié dans: The International Journal of High Performance Computing Applications, 2023, ISSN 1094-3420
Éditeur: SAGE Publications
DOI: 10.1177/10943420231165729

Revisiting active object stores: Bringing data locality to the limit with NVM

Auteurs: Alex Barceló; Anna Queralt; Anna Queralt; Toni Cortes; Toni Cortes
Publié dans: Future Generation Computer Systems, Numéro Volume 129, 2021, Page(s) 425-439, ISSN 0167-739X
Éditeur: Elsevier BV
DOI: 10.1016/j.future.2021.10.025

Sparse matrix‐vector and matrix‐multivector products for the truncated SVD on graphics processors

Auteurs: José I. Aliaga; Hartwig Anzt; Enrique S. Quintana‐Ortí; Andrés E. Tomás
Publié dans: Concurrency and Computation: Practice and Experience, 2023, ISSN 1532-0626
Éditeur: John Wiley & Sons Inc.
DOI: 10.1002/cpe.7871

Boosting HPC data analysis performance with the ParSoDA-Py library

Auteurs: Belcastro, L., Giampà, S., Marozzo, F,Rosa M. Badia, Jorge Ejarque & Nihad Mammadli,Loris Belcastro, Salvatore Giampà, Fabrizio Marozzo, Domenico Talia & Paolo Trunfio
Publié dans: The Journal of Supercomputing, 2024, ISSN 0920-8542
Éditeur: Kluwer Academic Publishers
DOI: 10.1007/s11227-023-05883-z

Generative Adversarial Reduced Order Modelling

Auteurs: Coscia, Dario; Demo, Nicola; Rozza, Gianluigi
Publié dans: Sci Rep, Numéro 14, 2024, ISSN 2045-2322
Éditeur: Nature Publishing Group
DOI: 10.48550/arxiv.2305.15881

A memory-efficient MultiVector Quasi-Newton method for black-box Fluid-Structure Interaction coupling

Auteurs: Zorrilla Martínez, Rubén; Rossi, Riccardo
Publié dans: Computers & Structures, Numéro 275, 2022, ISSN 0045-7949
Éditeur: Pergamon Press Ltd.
DOI: 10.1016/j.compstruc.2022.106934

Conference proceedings (5)

A Community Roadmap for Scientific Workflows Research and Development

Auteurs: Rafael Ferreira da Silva, Henri Casanova, Kyle Chard, Ilkay Altintas, Rosa M Badia, Bartosz Balis, Tainã Coleman, Frederik Coppens, Frank Di Natale, Bjoern Enders, Thomas Fahringer, Rosa Filgueira, Grigori Fursin, Daniel Garijo, Carole Goble, Dorran Howell, Shantenu Jha, Daniel S. Katz, Daniel Laney, Ulf Leser, Maciej Malawski, Kshitij Mehta, Loïc Pottier, Jonathan Ozik, J. Luc Peterson, Lavanya
Publié dans: Rafael Ferreira da Silva, Henri Casanova, Kyle Chard, Ilkay Altintas, Rosa M Badia, Bartosz Balis, Tainã Coleman, Frederik Coppens, Frank Di Natale, Bjoern Enders, Thomas Fahringer, Rosa Filgueira, Grigori Fursin, Daniel Garijo, Carole Goble, Dorran Howell, Shantenu Jha, Daniel S. Katz, Daniel Laney, Ulf Leser, Maciej Malawski, Kshitij Mehta, Loïc Pottier, Jonathan Ozik, J. Luc Peterson, Lavanya, 2021, ISSN 978-1-6654-1136
Éditeur: IEEE
DOI: 10.1109/works54523.2021.00016

Towards Efficient Neural Network Model Parallelism on Multi-FPGA Platforms

Auteurs: Rodríguez-Agut, David; Tornero-Gavilá, Rafael; Flich Cardo, José
Publié dans: 2023 Design, Automation & Test in Europe Conference & Exhibition (DATE), Numéro 1, 2023
Éditeur: IEEE
DOI: 10.23919/date56975.2023.10137117

End-to-End Workflows for Climate Science: Integrating HPC Simulations, Big Data Processing, and Machine Learning

Auteurs: Donatello Elia; Sonia Scardigno; Jorge Ejarque; Alessandro D’Anca; Gabriele Accarino; Enrico Scoccimarro; Davide Donno; Daniele Peano; Francesco Immorlano; Giovanni Aloisio
Publié dans: SC-W '23: Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, Numéro 1, 2023
Éditeur: Association for Computing Machinery (ACM)
DOI: 10.1145/3624062.3624283

Convolution Operators for Deep Learning Inference on the Fujitsu A64FX Processor

Auteurs: M. F. Dolz H. Martínez P. Alonso E. S. Quintana-Ortí
Publié dans: 2022, ISBN 978-1-6654-5155-0
Éditeur: IEEE
DOI: 10.1109/sbac-pad55451.2022.00027

A Community Roadmap for Scientific Workflows Research and Development

Auteurs: Rafael Ferreira da Silva, Henri Casanova, Kyle Chard, Ilkay Altintas, Rosa M Badia, Bartosz Balis, Tainã Coleman, Frederik Coppens, Frank Di Natale, Bjoern Enders, Thomas Fahringer, Rosa Filgueira, Grigori Fursin, Daniel Garijo, Carole Goble, Dorran Howell, Shantenu Jha, Daniel S. Katz, Daniel Laney, Ulf Leser, Maciej Malawski, Kshitij Mehta, Loïc Pottier, Jonathan Ozik, J. Luc Peterson, Lavanya
Publié dans: 2021 IEEE Workshop on Workflows in Support of Large-Scale Science, 2021, ISBN 978-1-6654-1137-0
Éditeur: IEEE
DOI: 10.48550/arxiv.2110.02168

Other (1)

11th EGU Galileo Conference: Solid Earth and Geohazards in the Exascale Era Consensual Document

Auteurs: Folch, Arnau; Bhihe, Cedric; Caviedes-Vouillième, Daniel; de la Puente, Josep; Esposti Ongaro, Tomaso; Garg, Deepak; Gibbons, Steven J.; Kaus, Boris; Monterrubio, Marisol; Räss, Ludovic; Reis, Claudia; Scaini, Chiara; Srivastava, Nishtha; Vilarrasa, Víctor; Zwinger, Thomas
Publié dans: 11th EGU Galileo Conference: Solid Earth and Geohazards in the Exascale Era Consensual document, 2023
Éditeur: CSIC
DOI: 10.20350/digitalcsic/15439