The ToHyVE project is organized in 10 work packages (WP). The actual research and development work is carried out in WP3 to WP9. In addition, there are work packages for project coordination (WP1), for specification of requirements (WP2) and for the transfer of the results into economy and society (WP10).

Hybrid event platform

The aim of this work package is the conception and design of an open ecosystem of hybrid event platforms that are linked to one another as seamlessly as possible and of technologies and tools for hybrid events that can be used across these platforms. To this end, innovative, hybrid, immersive representation, interaction and presentation formats are to be developed on the basis of these platforms and tools. A scalable approach is intended to achieve a high reach and easy access on different end devices. The usability, the user experience and the guarantee of data security and data protection are important factors.

Capture and representation of real scenes and of participants

The aim of the work package is to capture real scenes of face-to-face events as well as the present and remote participants, including their location. The audiovisual presentation should be as natural as possible, with the focus on conveying elements of non-verbal communication (gestures, eye contact, posture). For this purpose, different sensor and camera systems are being set up and tested, and AI methods for capturing the situational context are being developed.

Presentation of virtual scenes and of participants

The aim of WP5 is the development of methods and systems for the most realistic possible representation of the situation and the people at an event location in virtual environments, as well as the representation of virtual participants in a face-to-face event using the results of WP4. This includes the development of a web-based platform for low-threshold participation with low-complexity equipment and low bandwidth connections (e.g. mobile) as well as 3D platforms with higher complexity for the highly immersive participation. In addition, methods for display and loudspeaker arrangements, for realistic fusion of real and virtual content and for the coherent presentation of the auditory and visual components are developed and innovative exhibit formats are designed.

Multimodal Interaction

The goal of WP6 is to track the reactions of listeners and viewers during an event by detecting cross-language concept and relation types through real-time language processing and through information from social networks. By recognizing relevant entities and relationships, unstructured information can be structured and persisted for further use. In addition, an aspect-oriented sentiment analysis will be used to capture the mood in relation to certain topics during the event. With methods for automatic language recognition and machine translation, we meet the requirements for multimodality and multilingualism. As a result, we enable the audience to act during the event and the organizers to learn and react from its input. The structured textual information also serves as a knowledge base for a Multimodal Conversational Concierge, an event chatbot that serves as a user interface for real-time information and social media resources. The Concierge was developed with a user-centric design approach and can be used for various scenarios such as ticket purchase, registration and feedback.

Streaming Tools

The objective of this work package is to provide tools, protocols, APIs and modules for low latency live streaming, which can be used in interactive hybrid environments. To this end, audio and 2D video streaming processes are to be further developed, 3D audio and video streaming processes are defined and these processes are then implemented. There is also a focus on deploying these tools in cloud environments for global use. Therefore suitable technologies such as edge cloud processing, which meet real-time requirements are investigated in order to offload computing capacity, thus enabling the use of end devices with limited computing power.

Security und Privacy​

The objective of this work package is to identify and implement suitable measures that ensure security as well as privacy and data protection, taking into account existing legal frameworks. Experts in data security and privacy ​accompany the development of hybrid event platforms from the specification to the implementation phase in order to technically anchor the requirements for data protection, data security and protection of personal rights in the system. GDPR requirements and risks to privacy, such as identifiability, are particularly relevant. ​Specific innovative technical measures for end-to-end security are implemented, by which we mean consistent state-of-the-art encryption with certificates from an accredited trust center and hard cryptographic protection of data processing in the cloud.

Demonstrators and Evaluation

Workpackage WP9 aims to integrate the developed technologies into demonstrators for the application scenarios of the industrial partners as well as of the Forum Digital Technologies. It also includes the evaluation of sub-components and a summative evaluation of the system for different event formats. This includes the system integration, the implementation of the demonstrators, the technical evaluation and the system optimization. The results are running demonstrators, user feedback from using these demonstrators and the transfer of the results into product development.