Description
Multi-modal Data is defined in 3GPP to describe the input data from different kinds of devices or sensors, or the output data to different kinds of destinations (for example, one or more UEs), required for the same task or application. It consists of more than one Single-modal Data, and there is strong dependency among each Single-modal Data. Single-modal Data can be seen as one type of data — for example, video from a camera, audio from a microphone, temperature from a sensor, or text from an application. The defining characteristic of multi-modal data is not merely the presence of multiple data types, but the strong dependency among them: the data types are jointly required to accomplish a single task, and their meaning or usefulness depends on their combination.
The concept emerged in the context of AI/ML-enabled 5G services studied in 22.847 and 23.700, and in the radio access network context addressed in 38.835. In these studies, multi-modal data is the object that flows through the network to support applications such as multi-modal sensing, collaborative perception, and AI/ML inference that fuses inputs from multiple sources. For example, an autonomous driving or industrial robot application may require simultaneous video, LiDAR, radar, and sensor telemetry from multiple devices, all correlated in time and space, to make a control decision. Each stream is a Single-modal Data; together they form Multi-modal Data.
Architecturally, handling multi-modal data raises requirements that differ from single-modal traffic. The data types may originate from different devices or sensors, traverse different network paths, and arrive at different destinations, yet they must be correlated and delivered with consistent timing and quality. The strong dependency among Single-modal Data means that losing or delaying one modality can degrade or invalidate the entire task. Consequently, the network must coordinate QoS across modalities, synchronize delivery, and potentially perform in-network processing such as fusion or caching. The 3GPP studies examine how 5G systems can support these requirements through mechanisms such as QoS flows, time-sensitive communication, edge computing, and AI/ML data transport.
In 38.835, the RAN-level study addresses AI/ML data collection and transfer, where multi-modal data from different sources may need to be collected and delivered to an AI/ML entity for training or inference. The study considers how the RAN can support the transport of multi-modal data with appropriate reliability, latency, and synchronization. In 22.847 and 23.700, the service and system aspects are studied, including how multi-modal data relates to application requirements, network exposure, and edge computing. Together these specifications define the concept and explore the architectural implications without yet prescribing a single normative solution.
The term was introduced in Release 18 and is defined only in that release. Its introduction reflects the growing importance of AI/ML-driven services and the need for a precise vocabulary to describe data that spans multiple modalities and sources. As 5G-Advanced and 6G evolve, multi-modal data is expected to remain a foundational concept for services that fuse sensing, communication, and computation.
Purpose & Motivation
Multi-modal Data exists to provide a precise term for data that spans multiple types and sources but is jointly required for a single task or application. Previous vocabulary treated data streams independently, which was adequate for single-modal services such as voice or video but insufficient for emerging AI/ML services that fuse inputs from cameras, sensors, and other devices. The definition addresses this gap by emphasizing the strong dependency among Single-modal Data, making clear that the modalities cannot be handled in isolation.
The concept was motivated by the rise of AI/ML-enabled applications in 5G-Advanced, including collaborative sensing, autonomous systems, and edge inference, where the network must transport and coordinate correlated data from multiple devices. By defining Multi-modal Data in Release 18 studies, 3GPP created a common basis for analyzing QoS, synchronization, and architectural requirements for these services, laying groundwork for future normative work in 5G-Advanced and 6G.
Evolution Across Releases
An overview written from the defining specifications, not a verified change history. For the exact text, open the specifications listed below.
Defining Specifications
3GPP specifications that define or reference Multi-modal Data, with the latest known release. Sourced from the 3GPP document catalog — see methodology.
| Specification | Title | Release |
|---|---|---|
| TR 22.847 vi20 | Technical Report | Rel-18 read |
| TS 23.700 vk10 | AI/ML Application Layer Support Phase 2 | Rel-20 read |
| TR 38.835 vi01 | Technical Report on XR Enhancements for NR | Rel-18 read |