Live streaming has become one of the most popular formats on various content platforms in recent years. To distinguish their streams and appeal to audiences, many streamers seek to leverage cutting-edge technologies when delivering content. One of the latest technologies that are currently being explored is 3D live streaming. Stereoscopic video has existed for over a century, and it can create illusions of depth and immersion. 3D live streaming incorporates 3D technologies such as multi-view video, 3D audio, and free-viewpoint video within a production pipeline. Stereoscopic video channels are encoded in a video, and synchronized 3D audio signals are simultaneously live streamed to provide a more interactive user experience. Additionally, multi-view-plus-depth datasets are transformed into inversely rendered novel views in free-viewpoint video, which appear to be compelling for natural phenomena. While there is more to the technology, this paper will dissect the aspects of its techniques, applications, and guidelines in intricate detail.

The applications of 3D live streaming are widespread in various industries, such as broadcasting TV shows, sports, cultural performances, education, online conferences, and even company meetings. While they present new real-time possibilities, the applications appeal to crucial industries. There are different technologies such as multi-views or point clouds and encoding/decoding processes that allow for the 3D experience and realize the interactive immersive viewing experience. We envision that the future of 3D live content will have a substantial positive impact on different applications and industries. The main goal of this essay is to provide an in-depth exploration of the different components of 3D live streaming technology, such as the techniques, use cases, and guidelines. Roughly speaking, these three components correspond to the variety of technologies, the broad applications, and the recommended ways of creating 3D content.

Evolution of Live Streaming Technologies

The emergence of 3D live streaming has caught the attention of many professionals and enthusiasts alike. With the processing power and the most cutting-edge research, it has now been made even more robust and engaging than ever. It is such that audiences feel like they are sitting inside a gymnasium, enjoying the game, as though spectators at a live event. Taking note of the evolution of how we have reached this stage in entertainment provides invaluable context. The success of the aforementioned stages has led us right to where we stand today with 3D live streaming. With the transition from traditional broadcasting to the internet, live streaming saw true growth. Along with it, various new streaming protocols came into existence. Large-scale broadcasting companies, along with social media companies, joined the live streaming fold. With this increased competition, new formats emerged, including music concerts, live programming, movie broadcasts on television, sports events, news reports, political debates, public committee meetings, streaming social events, product releases, environmental or animal news, pet cams, streaming in production, and serious streaming. Higher internet access embedded faster speeds, improved real-time communication systems, more CPU speed, as well as better video compression and delivery. Protocols such as Real-Time Protocol and Real-Time Streaming Protocol facilitated real-time content delivery. Social media turned to live video. Sports leagues started live streaming. Animated GIFs and short videos made for an engaging streaming experience. Many new live streaming startups appeared at this time. Many of these are not as successful as the commercial juggernauts, but several brought new innovations. User demand drives technology. It is the audience demand that raises the bar, so it is the viewer’s engagement that drives innovation. In the future, as technology grows, it will keep us in a cycle of looking for more events to broadcast live. On the technological side, more broadcasts will improve the viewpoint of the viewer. Technological resources and the internet combine to train new content creators. This has again fueled new entertainment and arts. The parallel with the 3D live streaming project comes from here, although not directly linked to the viewer-driven nature of the success of 4K, 8K, and HDR.

Understanding 3D Technology

3D technology, often known simply as 3D, is what makes it possible to view images and videos as three-dimensional rather than as flat screens with two-dimensional information. It does so by employing various techniques and technologies to offer additional perspectives beyond the two that are typically used. Within the field of 3D, there are several different approaches to the development and implementation of 3D effects, including such prominent types as stereoscopy and volumetrics. Each of these technologies functions in unique ways, has its own benefits and drawbacks, and has different potential applications. Regardless of which approach is used, however, one of the primaries aims of 3D technology is to create an enhanced and more immersive viewing experience that is able to adjust itself to the way viewers see the wider world in three dimensions, making it more natural and real.

One of the magical possibilities that 3D technology can achieve is its creation of depth in the images and scenes it creates. There are certain base elements in 3D rendering technologies, like stereopsis and binocular vision, that contribute to this. Modern 3D technologies also help to rectify issues with 3D’s visualization abilities, including the field of view and the types of devices that are able to visualize such content. 3D content also has a number of hardware requirements, such as 3D cameras for the recording process and 3D displays for a viewer’s use. Thanks to these physical requirements, 3D live streaming necessitates the appropriate hardware infrastructure. This technology is also shaped and developed by a variety of other hardware and software tools and applications, which often deal with the creation, editing, and distribution of 3D content. In the context of 3D live streaming, the primary types of hardware are 3D cameras for capturing the moving 3D images, as well as 3D displays that cope with the actual playback of the content for a viewer’s consumption. Software tools used for 3D broadcasting can include anything used for the camera’s operation and footage capture, as well as post-production software for 3D video editing, and eventually streaming software that allows the content to be shared with viewers.

A 3D movie captures scenes such as action on a field and session presentations in the canteen. 3D images give more depth and texture to the view, making it feel more real than was previously agreed upon. Initially, the 3D content on the big screens, like cinemas and IMAX, were significant captivations of the audience, but now 3D videos have shifted to our palms with the advent of 360-degree 3D videos. With more visualized people, 3D live streaming, a technique used to demonstrate captivating scenes, is gaining prominent interest. However, it has a few challenges since it involves the simultaneous capturing of 3D data – FHD videos and audio – from two or more views, editing in real time, converting the data to a compressed format, and transmitting it within an acceptable time. Although virtual reality is also a novel concept that can elevate visual interaction and expression, it has other challenges such as the need for special viewing equipment, unlike 360 degrees that can be accessed without constraints.

Applications of 3D Live Streaming

3D live streaming has strong cross-sector applicability and can embrace multiple profit-driven business models. In the entertainment industry, musicians staged unique and immersive virtual concerts for millions of fans worldwide during the pandemic via a platform. A gaming giant broadcasted a listening party of a famous singer’s new album, aiming to recreate and share the vibes and urban lifestyle through this interactive and digital 3D live event. Earlier, another human-machine interaction simulation game attracted millions of fans to participate in a live online mash-up event between the players and a rap superstar, whose digital 3D figure created a magical virtual experience across time and space.

3D live streaming also finds its place in the education and training industry, such as 3D live streaming of medical operations for practical surgical training of medical students and doctors. For real estate and tourism applications, 3D live streaming enables a 360-degree panoramic aerial view from multiple angles and broadcasts clear video content of surrounding residential buildings. In an emergency, 3D live streaming can broadcast a mobile phone screen 2D image into 3D directly in real time for emergency operations. It can help professional emergency personnel take correct actions for the injured. Moreover, 3D telemedicine services can also convert physical medicine health instructions into digital ones, which are then transformed into 3D animations after the one-click transmission for sharing among patients from different regions.

In advertising or marketing fields, it can create a unique and irreplaceable experience for potential customers. As a successful case, a global lifestyle consumer electronics brand showcased sophisticated and innovative spatial 3D effects at a global online platform and technology product launch event, attracting more than 40,000 online audiences. At the booth of a company working on 3D live streaming technology at an expo, a highly impressed gamer clearly mentioned that she was impressed by the 3D technology that allows her opponents to physically feel like they’re right in front of you in the live streaming of the video game of VR arcade.

Challenges and Future Directions

3D live streaming has many challenges. First, higher bandwidth requirements are a barrier to widespread deployment. We find that people of different ages, education levels, and skills have relatively lower bandwidth, and people who live in rural areas have the worst conditions. Second, latency is a challenge for different kinds of large-scale events. Third, 3D live streaming platforms often require participants to use specific kinds of VR/AR devices, which are sometimes too expensive and not easily available. The 3D live streaming head display can easily cause discomfort to the wearer; even some people may experience motion sickness or dizziness. Moreover, the hardware update cycle of 3D live streaming devices is relatively short, which requires a relatively higher cost for daily maintenance. Fourth, 3D live streaming needs specific technical support and services during content creation.

3D un-authored VR/AR live streaming will become one of the main forms of VR/AR live streaming, bringing a sense of presence that makes you feel like you are really in the scene. With the continuous development of artificial intelligence, a lot of research has used GANs to achieve 2D to 3D image conversion, showing that we can generate real-world 3D models using AI from 2D real-world video. There are already some breakthroughs in technology that directly generate 3D world views from 3D sensors or videos, but the technology is still immature due to neural network memory issues. Researchers will investigate the encoding and decoding of neural networks in combination with the quantization of GANs, and the 3D AI model will be uploaded to the cloud with performance analysis feedback. The 3D AI model will be downloaded after paired feedback, edited, and then used in real-time mode. However, with the development of innovative technology, the user interface of VR/AR can rely entirely on software and not be dependent on hardware. When conducting interviews, professionals in the VR/AR field suggest that 3D AI using GAN technology will be able to be used in 3D live streaming in the future, which has potential in AI. Furthermore, some leaders in the 360-degree live streaming field indicate that applications of GAN networks are developing alternative hardware solutions, showing the strategic positioning of 3D AI live streaming. This cloud streaming approach solves issues of battery life, comfort on the head, and balance; it allows for a ‘virtual’ and ‘real’ appearance, merging with real-world cameras. This trend signals the possible future of 3D AI live streaming. However, laypeople do not understand this technology and still require professional engineers to handle it; this hardware solution marks a further step closer to personalized solutions.