Live from the UEFA Youth League Semi-Final cloud production showcase

Director Rob Levi, foreground, with trainee director Russell Yates

UEFA and TEAM Marketing invited a group of European broadcasters – and SVG Europe – to the BT Sport studios in London for a live demonstration of a cloud workflow of the 2023 UEFA Youth League Semi-Finals on 21 April in Geneva.

The UEFA Youth League was introduced in 2013 with the aim of developing youth football, providing players at the Under-19 level with the opportunity and experience of competing in an international tournament. It has also served as an opportunity for UEFA Youth League broadcaster BT Sport and its technology partners to try some innovative – and more sustainable – approaches to live production.

Organised by TEAM – UEFA’s global marketing partner – the showcase was attended by a host of rights-holding European broadcasters who gathered at BT Sport’s studios in Stratford, east London to watch a live cloud production (that ran in parallel to the main, traditional UEFA host broadcast) of the UEFA Youth League semi-final tie between Portugal’s Sporting CP and Dutch club AZ Alkmaar at the Stade de Genève, Geneva, Switzerland.

Hardware is so hard to get hold of, and switches are in short supply, so if you want to build something quickly it is almost impossible – the future is clearly software-based.

BT Sport chief engineer Andy Beale explains that historically, a production of a UEFA Youth League tie with eight cameras would still require some “heavy lifting” in terms of infrastructure and kit.

He says: “Typically, we would send a large articulated OB truck, scanner, generators, satellite truck and around 25 people to every game, just to create the multilateral world feed. And then back here at BT Sport in Stratford we would have another team in the gallery taking that satellite feed and adding BT Sport unilateral personalisation, such as commentary, GFX, replay, and so on.

“So that is 30-something people and a significant amount of carbon by the time you have driven all those vehicles to the venue. That is how we did it for years, and of course it was reliable and successful, but we wanted to consider if there was a better way.”

That led to BT Sport staging a multicamera cloud-based production of the UEFA Youth League tie between Tottenham Hotspur and Sporting Lisbon in Enfield, North London in October last year – a setup that provided the basis for the semi-final showcase between Sporting CP and AZ Alkmaar.

“In Enfield last year, there was one vehicle and that was a small, two-ton van to take tripods and scaffold clamps, cameras, lenses and LiveU backpacks and pre-charged batteries for the day. So we had taken off the road all of those vehicles and backhauled those signals into the cloud via the LiveU cloud service.

“Our friends at Limitless Broadcast who were doing the OB and shading the cameras weren’t on site either; shading was done by engineers at the Limitless HQ in Woking outside the M25, but with exactly the same camera spec – we didn’t change anything in terms of what we provided. And our production team was based here at BT Sport in east London. That is the beauty of the cloud; collaboration is relatively simple and there is no need for everyone to be co-located.”

NDI was the main cloud production codec, with the NDI Tools suite of applications a “critical” part of BT Sport’s cloud infrastructure. The setup also used LiveU packs on site, with a LiveU cloud-based decoder used to break out the LiveU camera signals as NDI streams.

Viz Vectar was the vision mixer and audio mixer of choice, with Viz (NewTek) 3Play engines running in the cloud, Chyron Prime for GFX and Kiloview multiviewers used for a couple of the ancillary multiviewers with Unity talkback systems for comms. Comprimato encoders were used for handling signals for transmission to Red Bee Media and on pass for the world feed.

Eric Orengo, UEFA, Senior Broadcast Engineering Expert, Andy Beale, BT Sport chief engineer and Claire Wilkie, managing director, Limitless Broadcast in the gallery at BT Sport in London.

“We went from 24 to eight people on site: six camera ops who also rigged the cameras, an audio guarantee providing audio and general video support, plus a floor manager,” says Beale.

“We also ran a variety of microservices in the cloud: LiveU decode, vision mix, replay, Chyron GFX, audio mix – those were then brought back to ground using the NDI Bridge tool, which we ran in the cloud and on on-prem. Bridge allows us to pull any of those NDI streams – whether camera sources or raw cameras, multiviewers and so on – which can then be routed over NDI Bridge into a cloud gallery for the production of the programme.”

BT Sport produced a few UEFA Youth League games using that model, which Beale describes as “super reliable”. He says: “The only thing we had to check is the internet capacity at each venue and whether there was adequate 4G coverage. With relatively small crowds at Youth League matches we don’t have huge contention to deal with, so we are usually able to get good performance off those mobile networks.”

For the cloud production of the semi-final, one of the biggest changes to the setup described above is an increase in the number of cameras, from 8 to 12.

“The host broadcaster is providing 10 cameras, and we have added one for ourselves because we will be doing a shadow cut and our director wanted the confidence they wouldn’t be beholden to the camera under the control of main host director, so we have a clone of camera two – which is our camera 11 – with a big lens on the gantry.

“And we have our own beauty cameras in the corner for somewhere to go when the host is doing their own thing we need somewhere to go safely, so we have two ‘get out of jail’ cameras to make sure we do a good job of the coverage today,” says Beale. The 10 host camera sources plus two additional cameras are taken as feeds from the NEP OB truck on site, and then fed into three LiveU LU800 packs via wired ethernet connections.

UEFA Nyon remote VAR set-up

Another difference to previous Youth League matches is the addition of a VAR setup at UEFA’s headquarters in Nyon, Switzerland.. “We want to prove that this is completely interoperable and the opportunities are there for wider sharing,” says Beale, who explains that the VAR setup is based on SimplyLive RefBox running in the cloud. Usually, camera feeds would be handed off to Hawkeye via the tailboard of an OB truck at the venue, but for today’s demonstration feeds are instead sent as NDI streams to Nyon where the SimplyLive operator and a match official are based.

Lessons learnt

Beale says that when BT Sport first started producing match coverage in the cloud a fair amount of time and energy was spent learning and discovering some pitfalls.

“There are so many things that you need to be aware of, some of them quite small, but all important for the production,” he says. “Our first test last year went really well, but some of the GPU instances in AWS were slightly on the edge, so one of our engineers thought it best to go up a stack, but they didn’t notice that tiering isn’t linear so even though we went to the next one up, the GPU association with it was a quarter of the previous one; the CPU was greater, but the GPU dropped off and we started to drop some packets.

UEFA Youth League Semi Final: BT Sport’s additional beauty shot

“So you have to be careful about sizing and shaping your instance. But also, it’s important not to over spec because you don’t want to burn more carbon than is necessary. So getting the instances right is a key thing.”

The design criteria was also to create a “normal” set of production tools for users, and for operators to be unaware that they are working in a cloud-based gallery.

BT Sport broadcast cloud architect Pete Smith was tasked with building the AWS backend. “With cloud it is very easy to have someone who becomes your cloud engineer, developer as well as being audio engineer and vision engineer, because they can do it all from one laptop. So one of the difficulties is abstracting the bits people need,” says Smith.

Smith created a web page that lists all the instances and provides a one-click log in to a machine so that engineers don’t need to be an AWS developer – they can just start and stop applications via a GUI.

For the Youth League, a script was developed so that the day prior to a production, the gallery can be remotely set up, with the script prompting things like the vision mixer software to boot up, auto log in, and open Viz Vectar.

“The day before would be a normal building day, so we just need to make sure everything boots and check the gallery works, just as we would for any show. The Limitless engineer can then join on the day. And then when it’s over, we can shelve the production setup until we need it again.”

Adding super slow-mo and HDR

Two of the limitations highlighted by Beale were the use of super slow mo and HDR. “There isn’t currently a mechanism for doing phases into the 3Play system, but SimplyLive is now supporting phases through the UI gateway, so I am sure it’s coming to other vendors at some point. For our next test we will definitely have a super slow mo element.

“And in terms of HDR, the limitation is purely to do with the implementation of the 10-bit ecosystem. On paper, NDI has support for 10-bit, but no one has implemented it. Viz announced with AWS they will deploy it in Q4 this year, but we will need other vendors to play their part for it to work properly, but there is progress.”

When considering what’s next for BT Sport’s use of cloud production, Beale says ‘pass through’ productions of a single feed that require elements such as commentary and basic graphics are relatively easy and being produced in the cloud now. But the availability of other events to work on is limited, because BT Sport has committed to 10-bit HDR, so the lack of 10-bit support creates an issue. “We can’t reverse that shift to 10-bit,” he says.

Looking further ahead, the remodelling of the OB setup and the shift to software and cloud-based production is being explored via one of this year’s IBC Accelerators, says Beale. “Hardware is so hard to get hold of, and switches are in short supply, so if you want to build something quickly it is almost impossible – the future is clearly software-based.

“That said, we still see a need on big events for truck – we won’t be taking them off the road for the Champions League any time soon – but the IBC Accelerator project ‘Connect & Produce Anywhere’, which will focus on the build of a distributed edge and cloud computing system to remotely produce a live sports event, is looking at what the OB truck of future will look like and if not it’s not hardware based, and with the move to the cloud, what can stay in the truck and what can remain on premise. These are all questions that through the Accelerator – and today’s showcase – we’re trying to answer.”

Read more IBC announces eight Accelerator Media Innovation projects with three focused on sport