BT Sport on its vision for augmented live sports broadcasting becoming reality with 5G
Boxing with a 3D AR version of the match on your coffee table could become a reality with BT Sport
BT Sport recently became one of the winners of the UK government’s Department for Digital, Culture, Media & Sport (DCMS) funding boost for 5G, under the 5G Create competition.
Named 5G Edge-XR, the consortium led by BT’s Media and Research teams alongside TheGridFactory, Condense Reality, Bristol University, Dance East, and Salsa Sound, will develop virtual and augmented reality experiences for live sport, working closely with BT Sport.
5G Edge-XR will enable people to view immersive sporting events from all angles, across a broader range of devices including smartphones, tablets, AR and VR headsets and TVs.
The project is worth a total of £2,558,494, with £1,486,004 of that figure coming from the DCMS out of the 5G Create competition fund. The project will run over 20 months, using 15 full time employees.
Immersive sporting experiences
Speaking to SVG Europe, Matt Stagg, BT Sport’s director of mobile strategy, explained the idea behind 5G Edge-XR’s goal: “We’ve been looking for a long time at how we can provide more immersive experiences as technology evolves. We had a vision for how we can use 5G for doing exactly that, around how we can take the value we have as a broadcaster for people within stadiums and then using the technology and our capabilities [to work out] how we can bring that stadium experience into the home for people who can’t go.
“Already, we’ve got a lot of interactive and immersive things on the BT Sport app, but it’s [about] how do we take that to the next level, and that’s all around volumetric capture, augmented reality, and virtual reality,” he notes.
“This is the most exiting area of sport technology I’ve ever been involved in, the vision for 5G and augmented experiences for sport is coming to life”
BT Sport is pushing everything within this vision under the mixed or cross reality (XR) banner, “because XR really changes your actual reality to increase the experience either through immersive means or access to content and experiences you wouldn’t have had otherwise, with the ability to be able to turn that on and off, to be the curator of your own experience in the home and outside of the home,” notes Stagg.
Stagg wrote the vision for this project at the beginning of 2019.
View the video of this vision
He says the question now is, “how do we bring this to life? ” He goes on: “That’s why I’m so excited about it is because we put a visionary video out there and everybody said that looks amazing, but people were sort of sceptical that that could ever happen or any of that could actually be done, but we were already working on it so we knew it could be done, but we didn’t know how to do it at scale or what the correct architecture was, and how that worked out from a commercial viewpoint, from a technical viewpoint, you know, can we do these things?
“Realistically we have to do a proof of concept to make sure this work, but the learnings from that will come out [as we work out how to] do something commercially; is there a model for this? Is it too processor intensive or is it unrealistic now that we know how much [processing power] it takes to do it?”
MotoGP is working with the 5G Edge-XR consortium to bring the millions of data points captured in each race to your living room
Creating and testing
Making this work in a live sports environment for distribution to wearable devices involves volumetric capture. That may include a set up using 16 x stereo pairs in four vertical clusters, capture with 32 x 4K cameras at 120fps, eight capture PCs, 1 Fusion PC to encode and contribute, cloud processing and encoding, cloud-based editorial tools, and delivery to apps at 30Mbps.
Ingest will go over fibre or 5G, depending on what is available, while distribution will go out over the 5G network. Location and tracking from the wearable devices will need to feed back into the system with 10ms to 30ms latency to create a realistic, enhanced XR experience for the end user.
A number of use cases are going to be the object of the project, including one with the data-rich sport of MotoGP to create a sports fan experience delivered over 5G and powered by edge cloud compute GPU facilities, enabling a photoreal, real time MotoGP Viewing Suite to be rendered and streamed to viewers’ smartphones, tablets and headset devices.
“I’m really, really passionate about this because it’s limitless. Some of these experiences will change the future way of how some things are done”
The MotoGP demonstrator will focus on a 5G delivered cloud XR rendered AR experiences, using high polygon (5M) 3D models, multiple video streams, immersive audio and data integration. Among other functionalities, it will hopefully deliver a video wall of multiple live race video streams with corresponding immersive audio, a 3D track view of riders, immersive halos for 8K 360 video, and rider and team profiles, stats and a replay area.
Comments Stagg: “We’ve been working on a demonstration MotoGP. MotoGP is a very good way of showing what augmented experiences can bring because you have so many data points and so much content coming from it, and then you look at how you create your own experiences from that data.”
Another demonstrator will be with a boxing event, allowing a real time virtual volumetric hologram to be interacted with on the viewer’s coffee table at home, allowing them to watch live action and replays from all angles, simultaneously alongside BT Sport’s live feed, creating an editorially-driven experience including commentary.
Yet another demonstrator will look into the fan experience within stadiums such as Wembley or race tracks including Silverstone, and even on the street at marathons and the like. This will look into using 5G to provide real time, mixed reality overlays on smartphones, tablets and AR headsets for fans at events.
5G Edge-XR topography for bringing XR to wearable devices, smartphones and tablets
Meaningful applications
Stagg says that other than all of the data that BT Sport has as a broadcaster to be able to enhance the experience of a fan, this project could have many meaningful applications, from showing fans in stadiums where the nearest toilet is, to showing them where the nearest exit is if there is an issue, to showing season ticket holders who have only ever sat in one seat in the stadium the view of the match from the opposite side.
Stagg adds: “As a season ticket holder, I’d like to see the penalty from the other side of the stadium. This technology enables you to do that and see how many miles [a footballer has] run, [whether Messi] is walking everywhere, what’s the sprint speed [of a player], the acceleration, all of that.”
This project is going to provide the viewers who want more than a broadcaster can currently offer in terms of information, statistics and viewpoints, provided live during a sporting event, with what they want, explains Stagg: “As a broadcaster, we have to broadcast to the middle ground [of viewers]; the majority of people who just watch the curated experience of post production, which is fantastic. That’s what we specialise in [as a broadcaster] and that’s why people want our content. But you take MotoGP, and you have some people who want to know absolutely everything; they want to be able to see every camera, they want to know the tire pressure. And you have the other end of people who don’t want any of that; they just want to watch the screen, the bike, the track.
“It’s that whole thing, it’s your choice [as to] how you build your experience,” he adds.
Massive processing power
Now, BT Sport and the consortium have to work out the topical infrastructure of a 5G network that would enable enough processing power to make these experiences work. Says Stagg: “We want to understand exactly how in a 5G world with 5G technology, GPU technology, how we make this come together in the most technically viable and commercially viable way. You have low latency and a huge amount of bandwidth so you can get the feed; all we need now is to bring in the processing power and we have everything we need to do these really immersive experiences.”
One key point the consortium is acutely aware of at the onset of the project is the sheer amount of processing power needed to translate the volumetric capture of live sporting events into an enhanced reality for the viewer.
Comments Stagg: “One of the things we’ve learned is that the processing power that is needed for some of these experiences is huge. When you look at your full volume match with 32 cameras and full volumetric compute to be able to render that, and then to be able to interact with that content, it’s massive.”
Part of the project is finding out how that processing requirement will affect the actual validity of idea. Stagg explains: “If we were going to do this, how many [interactive viewers] could we support? Could we serve this to 10 people or how many, and how big is this, are these experiences? It’s so processor intensive do we need to think of another way of doing it or dialling it down a bit. So it’s really understanding all of the vision that we have at BT Sport; let’s really take that vision we wrote and work out how we do it. We know what we need; we know we need massive processing power, over and above all of the functionality and capabilities of 5G. We know that, and now we need to know how we do all of the rest of it.”
Augmented stadiums for fans on-site could become a reality with 5G Edge-XR
Wearables for one-to-one
BT Sport is looking at wearables, which are now looking much more like regular accessories, to enable those that want to build on their experience of watching sport to use these devices as an access route to an enhanced reality.
Says Stagg: “With wearables, this could mean a viewer could watch the race on a big screen that takes up an entire wall, as broadcast by BT Sport, then also see a holographic representation on the track and bikes on the floor in front of them, with data points they can enhance or select to tailor the content for themselves.”
Adds Stagg: “Your whole room becomes your canvas for how you generate your own experience. That’s why we’ve come up with this [consortium] and are working with it, in order to make some of that stuff actually come to life.”
On whether this experiment will result in something tangible that works along the lines of Stagg’s original vision, he says: “I’m excited. I think it will happen. The glasses getting lighter, the wearables are looking more like the sort of thing that you would actually wear. I believe that this is going to revolutionise the way that people consume sport media and entertainment. As a broadcaster ourselves we’ve gone from what’s always been a one-to-many experience, and now what we need to be able to do with the technology that we’ve got is go to a one-to-one experience.
“We always have a fantastic post production output, but what we need to do is to be able to offer our fans more choice and more immersive [experiences] so that we can start to address our fans [needs on another level]. When we have a one-to-one relationship with our fans we can provide them with experiences that make the game more enjoyable,” adds Stagg.
Concludes Stagg: “I’m really, really passionate about this because it’s limitless. Some of these experiences will change the future way of how some things are done.
“This is the most exiting area of sport technology I’ve ever been involved in, the vision for 5G and augmented experiences for sport is coming to life.”