Thought Leadership: MediaKind on taking VR in sport forward with live 360-degree video streams and beyond

Basketball goes virtual with the Telekom Baskets Bonn

On 9 December 2018 a live 360-degree video stream of a basketball match between Telekom Baskets Bonn and EWE Baskets Oldenburg was delivered from the Telekom Dome in Bonn, Germany, to Deutsche Telekom’s Magenta Virtual Reality (VR) consumer app at 6K resolution.

SVG Europe spoke to Stuart Boorn, head of portfolio management at MediaKind, about how this industry world first was achieved.

What was the aim of this project in terms of benefit to Deutsche Telekom and its consumers?
Consumer demand for more personalised, immersive and interactive visual experiences is at an all-time high, and we can see this in the way the media and entertainment industry is responding.

The growing transition to 4K UHD, HDR and Wide Colour Gamut (WCG) all reflect a collective desire to enrich content and ultimately create viewing experiences that match ‘being there’ for live events. Deutsche Telekom wanted to test potential methods to enhance the quality of its existing 360-degree service.

This project [commissioned by Deutsche Telekom] was designed to explore the possibility of efficiently delivering higher quality, immersive 360-degree video streams using existing technology within the critical 15MBps bandwidth parameter. The project was also designed to test its wider commercial viability by making the streams available to most users of Deutsche Telekom’s Magenta VR service.

What were the challenges that this cutting edge project presented for you?
For a truly immersive experience, the resolution of the 360-degree image needs to be very large. Delivering these large images to consumer devices such as mobile phones and tablets presents two issues.
First was the bandwidth, an issue compounded by the fact that compression artefacts, which might be imperceptible on a TV screen, can often be clearly seen when viewed close-up on a tablet or head-mounted device (HMD).

The second issue concerned the devices we were aiming to target. To achieve widespread adoption, we needed to target consumer devices, phones and tablets, not the VR hardware most commonly associated with gaming. With the exception of the most recent arrivals, these devices don’t support the decoding of 8K images.

Both of these issues are solved by using so-called viewport adaptive streaming. For our project with Deutsche Telekom, we used the viewport adaptive technology from TiledMedia which exploits the fact that whilst the source image needs to be very large, at any one time the viewer is looking at only 15% of the full 360-degree image.

“The main challenge for delivering 360-degree video for sports is in meeting the quality requirements in a live workflow. From a technical point of view, the main limiting factors at this time are the availability of high-quality 360-degree camera rigs and the capacity of the stitching servers.”

This means the remaining 85% is effectively redundant. It works by splitting the source image up into small tiles and encoding and packaging them separately. The client then chooses to download the tiles needed to fill the current view. This way, both the bandwidth and the decode capabilities of the device are maximised.

Of course, encoding such large images brings its own challenges. The tiled method helps here too though. By leveraging the elasticity of cloud platforms we can deploy many lightweight encoders to handle the individual tile encodes. This way we are scalable to any resolution which will surely be necessary as device resolutions increase.

Performing the encode in the cloud means that data has to get from the event site into the datacentre. Fortunately, reliable 1Gbps links from sports venues are now pretty common and this was less of an issue than anticipated.

What is the benefit of using 6K for VR viewing? Why not 4K? Why push this boundary?
The resolution of the source image is, to a large degree, determined by the end device resolution. At any one time, the user is looking at around 15% of the full 360 view. If the user device has a screen resolution of HD then in order for the pixels of the source image to not be visible, the source image should be 8K x 4K or the equivalent of 16 HD broadcast videos.

With a 4K source image, the user’s view is pixelated or blurry, which significantly detracts from the immersion. Currently, the limitation on the resolution lies in the camera quality and the power of the stitching servers. So for this event, we compromised slightly and went for 6K x 3K.

What are the challenges within 360-degree video delivery specifically for sports broadcast, and what is holding it back right now?
The main challenge for delivering 360-degree video for sports is in meeting the quality requirements in a live workflow. From a technical point of view, the main limiting factors at this time are the availability of high-quality 360-degree camera rigs and the capacity of the stitching servers to meet the resolution requirements in real-time. That said, delivering a 6K or 8K 360-degree video service, either for a VR headset or mobile device is entirely feasible.

One limitation for 360-degree video for sports broadcasting is in the perception that it must be viewed on a head-mounted display (HMD). Given live sports and music events are primarily social occasion watched alongside other people, you can understand the reluctance to don a headset.

For that reason, we see the potential for 360-degree video delivered to tablets or phone screens on what we call a ‘flat’ app, which allows the user to choose the direction of view by swiping. It provides viewers with an opportunity to personalise their experience on their device whilst sharing in the main broadcast. For instance, this might involve selecting a certain camera angle or focusing on a particular area of the pitch or stage. At the same time, the viewer will still have access to follow the director’s curated action on the main screen.

This is a technology space that is evolving extremely quickly as higher density display technologies, compute, battery and wireless elements advance towards a goal of lightweight, affordable, untethered and consumer-friendly models.

How can you see this technology moving forward, and how will it be used for sports broadcasts in the future?
It is likely that that the first generation of 360-degree video deployments will act as a complementary experience via a second screen device that will augment the traditional broadcast. It will provide sports fans with more freedom to “direct the match” from a greater variety of camera views, angles, levels of zoom plus value added features that complement rather than replace the traditional at-home viewing experience.
Whatever its use case – such as a stadium seating view, player or vehicle cam, or an instant replay offering a specific viewpoint of the action – the ability to offer a unique viewpoint could be a compelling differentiator between broadcasters in an extremely competitive market. For instance, by expanding the consumer’s field of vision, the additional 360-degree functionality will deliver new creative opportunities for advertisers to monetise across a range of different formats and screens.

How did you manage the workflow within this project?
The project was a collaborative effort led by MediaKind and Deutsche Telekom with expert participants including Tiledmedia, Magnum Film, INVR.SPACE and Iconic Engine. Magnum Film acted as a general contractor, responsible for the production on site and for managing the other partners together with Deutsche Telekom.

INVR.SPACE was responsible for recording the 360-degree streams in 6K resolution. MediaKind transferred the streams to the cloud and processed, encoded and distributed them via a CDN for Magenta VR users. Tiledmedia’s innovative tile-based streaming technology was integrated in the MediaKind platform as well as the Magenta VR app, allowing streaming in high quality via regular internet connections and even over 4G. Iconic Engine implemented support for the tile-based live stream in the Magenta VR App for an intuitive, immersive and interactive experience.

In the cloud, MediaKind’s stateless microservices environment transformed the streams into formats suitable for 360-degree viewing including conversion to a cubemap projection, splitting the images into tile-segments according to a defined tiling scheme and handling these frames into a CDN platform with specific optimisations for tiled delivery. This workflow allowed for real time delivery and to test the ability to scale the platform up to deal with multiple, simultaneous streams.

The result was a world first but more importantly, it proved that 360-degree video could be delivered at commercial scale to a real mixed device audience of fans with Oculus VR headsets as well as smart apps on tablets.