Converging broadcast and user generated content for interactive ultra-high definition services

Project Information

COGNITUS

Grant agreement ID: 687605

DOI

10.3030/687605

Project closed

EC signature date 12 November 2015

Start date 1 January 2016

End date 31 December 2018

Funded under

• INDUSTRIAL LEADERSHIP - Leadership in enabling and industrial technologies - Information and Communication Technologies (ICT)

Total cost

€ 4 068 691,25

EU contribution

€ 3 643 018,26

Coordinated by

BRITISH BROADCASTING CORPORATION
United Kingdom

Periodic Reporting for period 2 - COGNITUS (Converging broadcast and user generated content for interactive ultra-high definition services)

Reporting period: 2017-01-01 to 2018-12-31

Summary of the context and overall objectives of the project

COGNITUS offers an end-to-end platform to aggregate User Generated Content (UGC) during events, enhance to Ultra-High Definition (UHD) quality and provide semantic enrichment to facilitate the smooth integration in broadcast productions. COGNITUS includes innovative tools for the producers to manage calls for UGC contributions and access the enhanced content using a number of smart search methods, including via text, visual similarities and audio correlation. A suite of applications integrating innovative technological modules provide the overall functionality: a mobile app for UGC contributors; plot authoring for producers; a set top box and companion app for end-users; and a system administration application.

The COGNITUS platform was developed in response to an era of multimedia capture and consumption trends: (i) the general public progressively capturing more multimedia content via mobile phones, and sharing via social media platforms; (ii) changing media consumption by audiences, with increasing demand for superior picture quality evidenced by the increasingly wide availability of UHD TVs, High Dynamic Range (HDR) TVs and new display technologies. While these trends presented opportunities to provide new engaging experiences to audiences, COGNITUS identified a number of significant gaps in bridging the necessary technologies. Notably, a lack in variety of good quality UHD content existed at the outset of COGNITUS (partly due to very high production costs). Furthermore, available UGC media was typically poor technical and editorial quality. Any existing market solutions for exploitation of crowd-sourced multimedia content (e.g. from social media platforms) lacked reliable quality control and enhancement mechanisms, which COGNITUS technical innovations stepped in to address.

COGNITUS offers solutions to facilitate the convergence of broadcast and UGC for future interactive UHD services. The project outcomes provide proof of concept to cement the viability of interactive UHD content production and UGC exploitation, through use case demonstrators at large events. The demonstrators were based on two use cases drawn from real-life events, providing examples of the fascinating and potentially unlimited new services that could be unleashed by the confluence of UHD broadcasting technology and smart social mobile UGC brought by COGNITUS.

The COGNITUS project was realized into a robust platform and suite of applications, able to address the vision: to create a vibrant community for contributing UGC and provide high fidelity professional productions based on UGC content. COGNITUS facilitates reductions in the cost of producing high quality UHD content, by leveraging enhanced crowd-sourced content. Furthermore, the COGNITUS platform offers a means of engagement for the general public to get involved in broadcast content production, further enhancing their creativity, providing an immense sense of achievement through active participation rather than passive viewing.

Work performed from the beginning of the project to the end of the period covered by the report and main results achieved so far

The COGNITUS technical innovations culminated in a novel platform and suite of applications. Two installations were set up at the BBC in the UK and at FORTHNET in Greece, and large-scale deployments were launched at high profile cultural entertainment and sports events across both countries. UGC video content was captured, enhanced and enriched by the COGNITUS technology components, and validation was provided by UGC contributors and producers. The ability of the platform to support the COGNITUS mission was demonstrated successfully. Very positive results were received on the platform’s performance under real conditions, and from the contributors and producers interacting with the platform.

The various content processing algorithms at the heart of the technical innovations and associated results are available to the scientific community as research publications, and open access datasets. New technology introduced is now available to the consortium partners, and beyond with some software results also available as open source. COGNITUS UGC tools can be used for research purposes by various institutions in the future.

COGNITUS gained wide interest at national and international workshops and trade shows, including IBC 2018, broadcasting events such as EBU conferences, and two TV appearances. COGNITUS elicited interest from broadcasters, content producers, technology providers and system integrators. Exploitation opportunities centre around the COGNITUS modular product (as a core product with add-ons to provide the functionality according to the needs of the productions). COGNITUS CORE consists of ten modules responsible for media asset management operations, module coordination, and UGC applications for the end users and the professionals. The add-ons provide extra functionality for quality assessment and video and audio enhancement, narrative event plotting, metadata enrichment, quality assessment and set top box adaptive streaming.

Progress beyond the state of the art and expected potential impact (including the socio-economic impact and the wider societal implications of the project so far)

The COGNITUS project produced technological results beyond the state-of-the-art through the final unified solution, and the individual technological components which leverage artificial intelligence based algorithms for media enhancements. Innovations centred around: UGC and production convergence; quality of experience, UHD broadcasting and HDR, content enhancement, reward schemes for media content crowdsourcing in the creative industries, data compression and data coding, and creation of an immersive sound experience from the audio content of UGCs.

Key innovative features of the COGNITUS platform include quality assessment for quality control and enhancement of the UGC for creating high quality UHD experiences. Features include spatial upscaling, framerate up-conversion, and dynamic range adaptation as visual enhancement mechanisms. Refinement modules, such as video stabilisation and portrait-to-landscape modules are also offered. Semantic enrichment of the metadata is achieved deep-learning algorithms which associate media annotations with respective visual and text data. Various audio enhancement tools handle a number of distortions that can occur in UGC such as wind noise or clipping, as well as audio synchronisation for alignment of overlapping audio and video streams.

COGNITUS offers an advanced, fast and efficient video delivery system, capable of supporting new video formats (such as UHD, HDR and High Frame Rates (HFR)). An HTTP adaptive streaming paradigm, and capable of efficiently compressing using an open source HEVC encoder is supported.

The innovation COGNITUS results are published across reputed and peer-reviewed journals and conferences. The published knowledge will lead to improved research environments and influence in academia. COGNITUS may further influence education on media in areas such as UGC-based production techniques as well as reward schemes for crowdsourcing services.

COGNITUS contributes to social innovation by providing tools that aim to increase productivity in specific aspects of media industry. The media industry is often considered as one of most rapidly growing, leading to increased employment of younger generations, which is not typical for all sectors. Most importantly, the COGNITUS concept in which all people can become content creators (and producers have access to tools to create new programmes) has potential to enable new creative services, processes and experiences.

COGNITUS logo