5G Applications and Devices Benchmarking

Project Information

TRIANGLE

Grant agreement ID: 688712

DOI

10.3030/688712

Project closed

EC signature date 10 November 2015

Start date 1 January 2016

End date 31 December 2018

Funded under

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Total cost

€ 3 541 981,25

EU contribution

€ 2 903 690,00

Coordinated by

KEYSIGHT TECHNOLOGIES BELGIUM
Belgium

Periodic Reporting for period 3 - TRIANGLE (5G Applications and Devices Benchmarking)

Reporting period: 2018-03-01 to 2018-12-31

Summary of the context and overall objectives of the project

The future 5G system is getting day by day more defined with 3GPP releasing the specification of the system and operator starting to move towards trials and deployment. This very sharp cut has then been stressing how the initial introduction of new services will be happening in the so-called 4.5/4.9G system (LTE-A Pro Evolution, LTE-M, LTE NB-IoT).
On the other side of the coin, the mobile App and services market is experiencing a constant growth in both size and revenue, even though a common ground for verifying the App performances in a standardized fashion is still lacking. Only traditional software testing, which considers the network a reliable data pipe, is performed on the code itself. While it has been an acceptable void so far, the requirements foreseen for the 5G Apps and services will make it impossible to develop software without considering its performance in varying network conditions.
This gap brings along the lack of a homogeneous testing and certification framework that encompasses a wide range of use cases, network scenarios, and Key Performance Indicators (KPIs) that are able to cover not only the different traditional telco needs, but also the End-to-End (E2E) side, given the incredibly large variety of available and foreseen Apps and services. A new and more holistic set of tools, covering also the mobile device behavior at Operating System level and the Evolved Packet Core (EPC) network would be need to support this work.
The TRIANGLE project is funded under the FIRE/FIRE+ objective, it started in January 2016, and had a duration of three years. The primary objective of the project is to promote the testing and benchmarking of mobile applications and devices in Europe as the industry moves towards 5G and to provide a pathway towards certification in order to support qualified apps and mobile developments using FIRE testbeds as testing framework available at the University of Màlaga.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

One of the key achievements of the project was to develop a strategy around the creation of an automated measured QoE, which is crucial in the acceptance of the application by the end user. In TRIANGLE, the QoE is reported in function of the application or device performance against a set of pre-defined domains (power consumption, reliability…). These domains are then aggregated in a synthetic QoE number which is the TRIANGLE mark. To validate this approach, multiple measurement campaigns have been run including a calibration of the synthetic QoE value against a real set of users. Results obtained in TRIANGLE demonstrate that the direction taken is a good prediction of the effective QoE.

The concept around QoE evaluated with an End-to-End testing approach also gained traction within the industry. The approach was presented at the Mobile World Congress 2017 and Mobile World Congress 2018. In January 2018, NGMN published a position paper “DEFINITION OF THE TESTING FRAMEWORK FOR THE NGMN 5G PRE-COMMERCIAL NETWORKS TRIALS”, which endorses the TRIANGLE approach, the TRIANGLE service definitions, and KPI for E2E QoE measurements. The concept and approach were also presented different occasions in leading industrial bodies such as the Global Certification forum (GCF) and the International Wireless Industry Consortium (IWPC).

Over the project duration, 4 testbed releases where created expanding capabilities and tests libraries, 42 companies and universities applied to use the testbed to run experiments and 13 applied for extensions. The test approach has been so interesting and valuable that some of the applicants not selected in the Open Call have requested access to the testbed services without funding. Open Call 5 reused this non-funded approach to ensure applicants were truly interested in the testbed and not just receiving funding. Towards the end of the project, an extra open call 6 (OC6) was launched to attract further users with low amount of funding but with a high 5G content.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The creation of a clearly identifiable “label” can help not only the business actors in the 5G ecosystem in creating a more aligned and stable E2E technology, but could impact the end users of the system providing them with a tool for simply and effectively helping in the selection of the device, app, or service to purchase, exactly as the EU energy label helps purchasing everyday objects, from cars to fridges. Finally, the creation of the certification scheme would open market possibilities for the entire testing equipment and services value chain. The consortium is working with the respective standardization bodies to gain traction on the approach.

Usage flow illustrated